# MVGrasp: Real-Time Multi-View 3D Object Grasping in Highly Cluttered
Environments

Hamidreza Kasaei1, Mohammadreza Kasaei2 1Department of Artificial
Intelligence, Bernoulli Institute, University of Groningen, 9747 AG, The
Netherlands. Email<EMAIL_ADDRESS>University of Aveiro
3810-193 Aveiro, Portugal. Email<EMAIL_ADDRESS>thank NVIDIA
Corporation for their generous donation of GPUs which was partially used in
this research.

###### Abstract

Nowadays service robots are entering more and more in our daily life. In such
a dynamic environment, a robot frequently faces pile, packed, or isolated
objects. Therefore, it is necessary for the robot to know how to grasp and
manipulate various objects in different situations to help humans in everyday
tasks. Most state-of-the-art grasping approaches addressed four degrees-of-
freedom (DoF) object grasping, where the robot is forced to grasp objects from
above based on grasp synthesis of a given top-down scene. Although such
approaches showed a very good performance in predefined industrial settings,
they are not suitable for human-centric environments as the robot will not
able to grasp a range of household objects robustly, for example, grasping a
bottle from above is not stable. In this work, we propose a multi-view deep
learning approach to handle robust object grasping in human-centric domains.
In particular, our approach takes a point cloud of an arbitrary object as an
input, and then, generates orthographic views of the given object. The
obtained views are used to estimate pixel-wise grasp synthesis for each
object. To evaluate the performance of the proposed approach, we performed
extensive experiments in both simulation and real-world environments within
the pile, packed, and isolated objects scenarios. Experimental results showed
that our approach can estimate appropriate grasp configurations in only $22$ms
without the need for explicit collision checking. Therefore, the proposed
approach can be used in real-time robotic applications that need closed-loop
grasp planning.

Figure 1: Overview of the proposed approach: we designed a mixed autoencoder
for multi-view object grasping to be used in both isolated and highly crowded
scenarios. First, multiple views of a given object are generated. The view
selection module selects which view is the best one in terms of grasping and
then fed it to the grasp network to obtain a pixel-wise grasp synthesis. As
shown in the most right part of the figure, the best grasp configurations are
finally ranked and transformed from 2D to 3D using a set of known
transformations.

##  I Introduction

Industrial robots are mainly designed to perform repetitive tasks in
controlled environments. Recent years have seen increasing interest in
deploying service robots to human-centric environments. In such unstructured
settings, object grasping is a challenging task due to the high demand for
real-time and accurate responses for a vast number of objects with a wide
variety of shapes and sizes under various clutter and occlusion conditions.
Although several grasping approaches have been developed successfully for 4DoF
$(x,y,z,\theta)$, many challenges still remain towards robust grasping. One
major obstacle to achieving a robust object grasping is the cost of collecting
efficient training data by real robots through self-supervised trial and error
experiments. To address this challenge, recent works in grasp synthesis have
mainly focused on developing end-to-end deep convolutional learning-based
approaches to plan grasp configuration directly from sensor data. Although it
has been proven that these approaches can outperform hand-crafted grasping
methods, the grasp planning is mainly constrained to top-down grasps from a
single depth sensor [1, 2]. These approaches assume a favorable global camera
placement and force the robot to grasp objects from a single direction, which
is perpendicular to the image plane. Such constraints bound the flexibility of
the robot, and the robot will not be able to grasp a range of household
objects robustly, e.g., bottles, boxes, etc. Furthermore, a certain set of
objects has convenient parts to be grasped, e.g., the ear of a mug, or in some
situation, it is easier to approach a household object from different
directions.

In this work, we propose a multi-view deep learning approach to handle real-
time object grasping. Figure 1 depicts an overview of our work. In particular,
our approach takes a partial point cloud of a scene as an input. We then
generate multi-view depth images of the objects and import them into a view
selection function. The best view is then fed to a deep network to estimate a
pixel-wise grasp synthesis (grasp quality, orientation, and width of a grasp).
It is worth mentioning that we trained the network end-to-end using a very
small isolated object grasping dataset. We based our approach on the
hypothesis that considering multi-view of the objects forces the network to
capture collisions among the gripper, objects, and the environment, which is
necessary for object manipulation in human-centric environments. We validate
this hypothesis by performing several experiments in both simulated and real-
robot settings. More specifically, we evaluate the performance of the proposed
method on the isolated, pile, and packed objects scenarios. Extensive
quantitative and qualitative evaluations showed that our approach
significantly outperformed the state-of-the-art methods. Our approach, on
average, could estimate stable grasp configurations in less than $22$ms
without the need for explicit collision checking. Therefore, the proposed
approach is suitable to be used in real-time robotic applications that need
closed-loop grasp planning.

##  II Related work

Traditional object grasping approaches explicitly model how to grasp different
objects mainly by considering prior knowledge about object shape, and pose
[3]. It has been proven that it is hard to obtain such prior information for
never-seen-before objects in human-centric environments [4]. More recent
approaches try to tackle this limitation by formulating object grasping as an
object-agnostic problem, in which grasp synthesis is detected based on learned
visual features without considering prior object-specific information.
Therefore, these approaches are able to generalize the learned grasping
policies to new objects without the need for having the full model of the
object in advance. In this vein, much attention has been given to object
grasping approaches based on Convolutional Neural Network (CNN) [5, 1, 6, 7].

Deep-learning approaches for object grasping falling into three main
categories depending on the input to the network. Volume-based approaches
represent objects as 3D voxel grid and then fed them to a CNN with 3D filter
banks [8, 7, 9, 10]. In particular, these approaches first estimate the
complete shape of the target object using a variational autoencoder network,
and then sample depth images of the object from different directions. Finally,
all the obtained views are used for generating grasp synthesis for the given
object. These approaches are computationally very expensive as they
incorporate a shape complete network. Furthermore, training such network
required enormous amount of data.

Pointset-based methods directly use point cloud data as input [11, 12, 13].
One of the biggest bottlenecks with these approaches is the execution time and
sensitivity to calibration errors. Unlike these methods, our approach
generates three virtual depth images of the object, and then generates grasp
synthesis for the obtained object’s views.

View-based approaches receive a depth image as the input of network. For
instance, DexNet [1] and QT-Opt [4] learn only top-down grasping based on
images from a fixed static camera. GG-CNN [6], DexNet [1], and QT-Opt [4],
generate a grasp map per scene. Unlike these approaches, our approach
generates a grasp synthesis map per object. Furthermore, these approaches only
work for top-down camera-settings and have mainly focused on solving 4DoF
grasping, where the gripper is forced to approach objects from above. The
major drawbacks of these approaches are inevitably restricted ways to interact
with objects. Moreover, the robot is not able to immediately generalize to
different task configurations without extensive retraining. We tackle these
problems by proposing a multi-view approach for object grasping in highly
crowded scenarios.

More recent approaches receive 3D representation of an object, e.g., in the
form of point cloud, and generate multiple views of the object. Our work
belongs to this category. These approaches appear to be most effective in 3D
object recognition and grasping, as shown by [14, 15, 16, 1, 6, 17, 18, 19].
In these approaches, 2D images are extracted from the point cloud of the
object by projecting the object’s points onto 2D planes. The obtained view is
then fed to a CNN to generate grasp syntheses. Finally, the robot executes the
highest-ranked grasp [1]. Most of the approaches take a very long time to
sample and rank grasp candidates individually [5, 1]. Unlike our approach,
these approaches are used in open-loop control scenarios and are not suitable
for real-time applications.

Recent researches on multi-view grasping [20, 21], have been done using a real
single camera moving over a predefined trajectory, where each point of the
trajectory is considered as a view. In [20], the most informative view is then
chosen by means of entropy measures. In our work, we instead generate multiple
views for each object and select the most informative view, based on a
viewpoint entropy measure, for grasping.

There are some approaches that addressed closed-loop object grasping. These
methods frequently receive visual inputs and update grasp configuration during
execution, mainly based on visual serving. Most of these methods considered
several constraints (e.g., top grasp only) to reduce the amount of required
training data. For example, Morrison et al., [6] proposed Generative Grasping
CNN (GG-CNN), a solution where grasp poses are generated for every pixel using
a small CNN. Similar to our approach, GG-CNN is designed to use visual
feedback in real-time applications. Unlike GG-CNN that uses an eye-in-hand
visual feedback, our approach works in an eye-to-hand system and use a wider
view of the environment rather than a narrow top-down view. In service robots,
a wider view of the environment is better suited for household tasks as the
robot has to consider an entire scene for motion planning.

##  III Problem Formulation

In this work, the robot uses a single Kinect camera to perceive the world. We
formulate grasp synthesis as a learning problem of planning parallel-jaw
grasps for objects in clutter. In particular, we intend to learn a function
that receives a collection of rendered depth images of a 3D object as input,
and returns (i) the best approaching direction, and (ii) a grasp map
representing per-pixel grasp configuration for a selected view. The grasp map
is then used to find the best grasp configuration for removing a target object
from the workspace. In this work, we assume that the object has been segmented
from a scene, and refer the reader to our previous works for this purpose [22,
23].

###  III-A Generating multiple views of objects

A three-dimensional (3D) object is usually represented as point cloud,
$p_{i}:i\in\\{1,\dots,n\\}$, where each point is described by their 3D
coordinates $[x,y,z]$. To capture depth images from a 3D object, we need to
set up a set of virtual cameras around the object, where the Z axes of cameras
are pointing towards the centroid of the object. We first calculate the
geometric center of the object as the average of all points of the object.
Then, a local reference frame is created by applying principal component
analysis on the normalized covariance matrix, $\Sigma$, of the object, i.e.,
$\Sigma\textbf{V}=\textbf{EV}$, where $\textbf{E}=[e_{1},e_{2},e_{3}]$
contains the descending sorted eigenvalues, and
$\textbf{V}=[\vec{v}_{1},\vec{v}_{2},\vec{v}_{3}]$ shows the eigenvectors.
Therefore, $\vec{v}_{1}$, shows the largest variance of the points of the
object. In this work, $\vec{v}_{1}$ and the negative of the gravity vector are
considered as X and Z axes, respectively. We define the Y axis as the cross
product of $\vec{v}_{1}\times\textbf{Z}$. The object is then transformed to be
placed in the reference frame.

Figure 2: Two examples of generating bounding box, local reference frame, and
three projected views for: (left) a glass-cleaner; (right) a juice-box.

From each camera pose, we map the point cloud of the object into a virtual
depth image based on z-buffering. In particular, we first project the object
to a square plane, $M$, centered on the camera’s center. The projection area
is then divided into $l\times l$ square grid, where each bin is considered as
a pixel. Finally, the minimum $z$ of all points falling into a bin is
considered as the pixel value. The size of the projection square area,
$l\times l$, is an important factor for object grasping tasks. In the case of
object-agnostic grasping, since the grasp configurations depend on the pose
and size of the target object, a view of the object should not be scale-
invariant, we consider a fixed size projection plane ($l\times l$). In our
setup, the size of each bin is defined as $f_{g}\times f_{g}$, where $f_{g}$
represents the size of the head of the robot’s finger, and the length of each
side of the projection plane, $M$, is defined as $l\times f_{g}$.

The number of object views is an important parameter for object grasping. In
this work, we consider orthographic views including XoY, XoZ, and YoZ
projections. Figure 2 shows two examples of this procedure for a glass-cleaner
and a juice-box111Video of this procedure is available:
https://youtu.be/0UWXIlg2WC8. The intuition behind this selection is that most
of the household objects are stably graspable from top, side, or front views
[24]. The obtained projections are then normalized (i.e., mean-subtracted and
divided by standard deviation) and fed to the network.

We develop a function to define which of these views is the most-informative
view for approaching and grasping the target object. Therefore, we use
viewpoint entropy that nicely takes into account both the number of occupied
pixels and their values. We calculate the entropy of a view, $v$, as
$H(M)=-\sum_{i=1}^{l^{2}}p_{i}\log_{2}(p_{i})$, where $p_{i}$ represents the
value of $i$th pixel, and then sort the views based on their entropy value.
The view with maximum entropy is considered as the most-informative view. We
also consider the kinematic feasibility and distance that the robot would
require to travel in the configuration space. In the case of large objects, or
a pile of objects, there is a clear advantage (e.g., collision-free) to grasp
from above, while for an isolated (cans, bottles, boxes, toys),

Figure 3: Example of grasping a Colgate box in two different situations.

it completely depends on the pose of the object relative to the camera (see
Fig. 3). The gripper finally approaches the object in an orthogonal direction
to the most-informative view. It should be noted that the view selection
function can be easily adapted to any other tasks’ criteria.

###  III-B Network architecture

We aim to learn a function that maps an input object’s view to multiple
outputs representing pixel-wise antipodal grasp configurations,
$f_{\theta}:\mathcal{X}\rightarrow\mathbf{\mathcal{Y}}$. Towards this goal, we
have designed a convolutional autoencoder that receives a depth image with
height $h$ and width $w$ pixels as input, $x_{(i,j)}\in\mathbb{R}^{h\times
w}$, and returns a pixel-wise grasp configuration map, G, i.e.,
$y_{(i,j)}=[\textbf{G}_{(i,j)}]$. Since we want to use the network in a
closed-loop control scenario, we add a constraint on number of trainable
parameters to be less than $120$K.

The network is parameterized by its weights $\theta$. Our model is a single-
input multiple-outputs network and is constructed using three types of layers,
including convolution, deconvolution, and batch normalization.

The encoder part is composed of six convolutional layers (C$1$ to C$6$). We
use a Rectified Linear Unit (ReLU) as the activation function of all layers of
the encoder part to force the negative values to zero and eliminating the
vanishing gradient problem which is observed in other types of activation
functions. We added a batch normalization layer after each convolutional layer
to stabilize the learning process and reducing the number of training epochs
by keeping the mean output and standard deviation close to $0$ and $1$,
respectively. The decoder part is composed of six transposed convolutional
layers (T$1$ to T$6$), followed by three separate output layers for predicting
grasp quality, width, and rotation. Similar to the encoder part, we use the
ReLU activation function for all layers and add a batch normalization after
each layer. We use the same padding in all convolution and transposed
convolution layers to make the input and output be of the same size (see
Section IV).

In this work, an antipodal grasp point is represented as a tuple,
$g_{i}=\langle(u,v),\phi_{i},w_{i},q_{i}\rangle$, where $(u,v)$ stands the
center of grasp in image frame, $\phi_{i}$ represents the rotation of the
gripper around the Z axis in the range of $[\frac{-\pi}{2},\frac{\pi}{2}]$,
$w_{i}$ shows the width of the gripper where $w_{i}\in[0,w_{max}]$, and the
success probability of the grasp is represented by $q_{i}\in[0,1]$. Given an
input view, the network generates multiple outputs, including:
$(\boldmath{\phi},\textbf{W},\textbf{Q})\in\mathbb{R}^{h\times w}$, where
pixel value of images indicate the measure of $\phi_{i},w_{i},q_{i}$
respectively. Therefore, from $f_{\theta}(I_{i})=\mathbf{G}_{i}$, the best
grasp configuration, $\operatorname{g^{*}}$, is the one with maximum quality,
and its coordinate indicate the center of grasp, i.e.,
$(u,v)\leftarrow\operatorname{g^{*}}=\operatorname*{argmax}_{\mathbf{Q}}~{}\mathbf{G}_{i}$.
Given a grasp object dataset, $\mathcal{D}$, containing $n_{d}$ images,
$D=\\{(x_{i},y_{i})|1\leq i\leq n_{d}\\}$, our model can be trained end-to-end
to learn $f_{\theta}(.)$.

After obtaining the grasp map of an input view, the Cartesian position of the
selected grasp point, $(u,v)$, can be transformed to object’s reference frame
since the transformation of the orthographic view relative to the object is
known. The depth value of the grasp point is estimated based on the minimum
depth value of the surrounding neighbors of $(u,v)$ that are within a radius
of $\Delta$, where $\Delta=5$mm.

##  IV Experimental Results

We performed several rounds of simulation and real-robot experiments to
evaluate the performance of the proposed approach. We were pursuing three
goals in these experiments: (i) evaluating the performance of object grasping
in three scenarios; (ii) investigating the usefulness of formulating object
grasping as an object-agnostic problem for general purpose tasks; (iii)
determine whether the same network can be used in both simulation and real-
robot systems without additional fine-tuning. Towards this goal, we employed
the same code and network (trained on the Cornell dataset) in both real and
simulation experiments.

###  IV-A Ablation study

We trained several networks with the proposed architecture but different
parameters including filter size, dropout rate, number of units in fully
connected layers, loss functions, optimizer, and various learning rates for
$100$ epochs each. We used the extended version of the Cornell dataset [25]
comprising $1035$ RGB-D images of $240$ household objects. In this work, we
considered the $5110$ positive grasp configurations and discard all the
negative labels. Furthermore, since the Cornell dataset is a small dataset, we
augmented the data by zooming, random cropping, and rotating functions to
generate $51100$ images. The $80\%$ of the augmented data is used for training
and the remaining $20\%$ is used as the evaluation set.

We report the obtained results based on the Intersection over Union (IoU)
metric, and speed. A grasp pose is considered as a valid grasp if the
intersection of the predicted grasp rectangle and the ground truth rectangle
is more than $25\%$, and the orientation difference between predicted and
ground truth grasp rectangles is less than $30$ degrees. The final
architecture is shaped as: C${}_{(9\times 9\times 8,~{}S_{3})}$,
C${}_{(5\times 5\times 16,~{}S_{2})}$, C${}_{(5\times 5\times 16,~{}S_{2})}$,
C(3×3×32), C(3×3×32), C(3×3×32), T(3×3×32), T(3×3×32), T(3×3×32),
T${}_{(5\times 5\times 16,~{}S_{2})}$, T${}_{(5\times 5\times 32,~{}S_{2})}$,
T${}_{(9\times 9\times 32,~{}S_{3})}$, where $S$ stands for strides. We used
Adam optimizer with a learning rate of $0.001$, and Mean Squared Error as a
loss function. The number of trainable parameters of our network is $116$K,
which makes it suitable for closed-loop control at the rate of up to $45$ Hz.

We compared our approach with six visual grasp detection baselines, including
Lenz et al. [5], Redmon et al. [26], Wang et al. [27], DexNet [1], GG-CNN [6],
and GG-CNN2 [2]. Results are reported in Table I. Our approach achieved state-
of-the-art performance and outperformed the selected approaches by a large
margin. Concerning IoU metric, our approach achieved $89.51\%$ which was
$15.61$, $1.51$, $4.21$, $0.51$, $16.51$, and $14.31$ percentage point (p.p)
better than Lenz et al. [5], Redmon et al. [26], Wang et al. [27], DexNet [1],
GG-CNN [6], and GG-CNN2 [2] respectively. In addition to the IoU metric, we
also computed the average grasp prediction time for each of the mentioned
approaches. The obtained results indicated that DexNet [1], Lenz et al. [5],
Wang et al. [27], and Redmon et al. [26] methods need a very long time to
predict grasp poses for a given object. Hence, these approaches are
computationally expensive, and not useful for real-time robotic applications,
where a closed-loop controller is usually needed. In contrast, the inference
times for the GG-CNN [6], GG-CNN2 [2], and the proposed method was less than
$25$ ms, which represents that these approaches are suitable for real-time
object grasping (i.e., a closed-loop control can work steadily in $\geq
45$Hz). The underlying reason for fast inference is that these approaches use
a relatively small CNN, while other selected approaches are based on very deep
neural networks with millions of parameters (e.g., DexNet has 18 million
parameters). Although GG-CNN [6] and GG-CNN2 [2] achieved slightly better
execution time than our approach (i.e., $3$ ms, and $1$ ms faster
respectively), the grasp performance of our approach is significantly better.
Note that, GG-CNN [6] and GG-CNN2 [2] have $62$k and $66$k network parameters,
respectively, whereas the proposed network has $116$k parameters, which is
almost twice as many.

TABLE I: Result of object grasping on the Cornell dataset [25].

approach | input data |  IoU (%) |  speed (ms)
---|---|---|---
Lenz et al. [5] | RGB-D | 73.9 | 1350
Redmon et al. [26] | RGB-D | 88.0 | 76
Wang et al. [27] | RGB-D | 85.3 | 140
DexNet [1]∗ | depth image | 89.0 | 2500
GG-CNN [6] | depth image | 73.0 | 19
GG-CNN2 [2]∗ | depth image | 75.2 | 21
Our approach | depth image | 89.51 | 22

$*$ retrained

Figure 4: Our experimental setups in: (left) simulation environment; and
(right) the real-world setting. It should be noted, both simulated and real
sets of objects used for evaluations are shown in these figures.

###  IV-B Grasp evaluation

Figure 4 shows our experimental setup in simulation (left) and real-robot
(right). In this work, we have developed a simulation environment in Gazebo
similar to our real-robot setup to extensively evaluate our approach. The
robot and the camera in the simulated environment were placed according to the
real-robot setup to obtain consistent performance. Our setup comprises a
Universal Robot (UR5e) with a two-fingered Robotiq 2F-140 gripper, a Kinect V1
camera mounted on a tripod, and a user interface to start and stop the
experiments.

To evaluate the performance of the proposed approach, we designed (i)
isolated, (ii) packed, and (iii) pile removal tasks (see Fig. 5). In each
round, we randomly generated a new scene consisting of four to six objects in
the case of packed and pile scenarios, while for the isolated object scenario,
we placed a randomly selected object in an arbitrary pose inside the robot’s
work-space. In all experiments, the robot knows in advance about the pose of
the bin object as the placing pose, while it has to detect and track the pose
of target objects to be removed from the table. Afterward, the robot needs to
predict grasp synthesis and select the best graspable pose of the target
object, picks it up, and puts it in the bin. This procedure is repeated until
all objects get removed from the table. Note that, at the beginning of each
experiment, we set the robot to a pre-defined setting, and randomly place
objects on the table. We used a set of $20$ simulated objects, imported from
different sources (e.g., the YCB dataset [28], Gazebo repository, and etc),
and $20$ real daily-life objects with different materials, shapes, sizes, and
weight (see Fig. 4). All objects were inspected to be sure that at least one
side fits within the gripper. We assessed the performance of our approach by
measuring success rate, i.e.,
$\frac{number~{}of~{}successful~{}grasps}{number~{}of~{}attempts}$. We
benchmark our approach against Grasp Pose Detection (GPD) [13] (an analytical
approach) and GGCNN (learning based approach) baselines.

Figure 5: Illustrative examples of three evaluation scenarios in Gazebo:
(left) isolated object; (center) packed objects; (right) pile of objects.

####  IV-B1 Isolated object scenario

In this round of experiments, each object was tested in both real-world and
simulation environments for $10$ and $50$ times, respectively. A particular
grasp was recorded as a success if the object is inside the bin at the end of
the experiment. The best grasp point is defined as the collision-free and
kinematically feasible one with the highest grasp quality. Note that, to speed
up the real-robot experiments, we randomly placed five objects on the table.
In each execution cycle, the robot selected the nearest object to the base and
tried to remove it from the table. Results are reported in Table II. By
comparing both real and simulation experiments, it is visible that our
approach outperformed GPD by a large margin. Particularly, in the case of
simulation experiments, we achieved a grasp success rate of $89.7\%$ (i.e.,
$897$ success out of $1000$ trials), while GPD and GGCNN obtained $78.7\%$,
and $72.6\%$, respectively. We visualized the best grasp configuration for
$10$ simulated objects in Fig. 6.

In the case of real-robot experiments, the success rate of our approach was
$90.5\%$ ($181$ success out of $200$ attempts), which was $9.5\%$ and $12.0$
better than GPD and GGCNN, respectively. The underlying reason was that the
proposed approach generated pixel-wise grasp configurations for the most-
informative view, resulting in a diverse set of grasps for the target object,
which we then selected the best kinematically feasible one. This is not the
case for the GPD approach, and GGCNN. More specifically, GPD generated a few
grasps for a target object, and sometimes,

TABLE II: Isolated Scenario

Method | Type | Success rate ($\%$)
---|---|---
GPD | sim | 78.7 (787/1000)
GGCNN | sim | 72.6 (726/1000)
Our | sim |  89.7 (897/1000)
Our (top-down) | sim | 73.2 (732/1000)
Our (random) | sim | 51.3 (513/1000)
GPD | real | 81.0 (162/200)
GGCNN | real | 78.5 (157/200)
Our | real |  90.5 (181/200)
Our (top-down) | real | 67.5 (135/200)
Our (random) | real | 49.0 (98/200)

none of the configurations led to a successful attempt. In the case of GGCNN
and our approach with top-only view selection, failures mainly happened in
grasping soda-can, bottle, human toy, and mustard object since the supporting
area around the selected grasp point was too small and therefore, the object
slipped and fall during manipulation. In the case of our approach with random
view selection, the main failures were due to collision with the table, e.g.,
grasping a toppled soda-can from side. Some failures also occurred when one of
the fingers of the gripper was tangent to the surface of the target object,
which led to pushing the object away. In the case of our approach, the failed
attempts were mainly due to inaccurate bounding box estimation, some objects
in specific pose had a very low grasp quality score, and collision between the
object and the bin (mainly happened for large objects e.g., Pringles and Juice
box).

Figure 6: Qualitative results for 10 never-seen-before household objects in
the isolated scenario: visualizing objects in Gazebo and their best grasp
configurations. These results showed that our approach learned very-well the
intended object-agnostic grasp function.

####  IV-B2 Pile and packed scenarios

To generate a simulated scene containing a pile of objects, we randomly spawn
objects into a box placed on top of the table. We wait for a couple of seconds
until all objects become stable, and then remove the box, resulting in a
cluttered pile of objects. To generate a packed scenario, we iteratively
placed a set of objects next together in the workspace. An example for each
scenario is shown in Fig. 5 (center and right). In the case of real-robot
experiments, we randomly put four to six objects in a box, then shake the box
to remove bias, and finally pour the box in front of the robot to make a pile
of objects. In the case of packed experiments, we manually generate scenes by
putting several objects together. In this round of evaluation, in addition to
the success rate, we report the average percentage of objects removed from the
workspace. An experiment is continued until either all objects get removed
from the workspace, or three failures occurred consecutively, or the quality
of the grasp candidate is lower than a pre-defined threshed, $\tau$. This
parameter plays an important role. We performed $100$ packed removal tests to
tune the grasp quality threshold by setting $\tau\in\\{0.8,0.9\\}$, i.e., $50$
simulation experiments for each $\tau$. In each iteration, we randomly
generated a scene including four objects. In each execution cycle, the robot
is instructed to execute a feasible grasp synthesis that has the maximum grasp
quality. We also performed experiments with GPD and GGCNN.

Figure 7: Qualitative results on packed scenarios: visualizing the top-three
grasp configurations on four different densely-packed objects.

Packed experiments: Results are reported in table III. It was observed that by
setting $\tau=0.9$, the robot could remove fewer objects from the workspace
while achieving a higher success rate, while setting $\tau$ to $0.8$ leads to
a good balance between success rate and percentage of object removal.

TABLE III: Performance on Packed Scenario

Method | Type | Success rate | Percent cleared
---|---|---|---
GPD | sim | 0.54 (139/256) | 0.70 (139/200)
GGCNN | sim |  0.55 (144/262) | 0.72 (144/200)
$\tau=0.8$ | sim | 0.84 (176/210) |  0.88 (175/200)
$\tau=0.9$ | sim |  0.94 (141/150) | 0.71 (141/200)
GPD | real | 0.64 (31/48) | 0.76 (31/40)
GGCNN | real | 0.46 (25/54) | 0.63 (25/40)
$\tau=0.8$ | real |  0.91 (38/42) |  0.88 (38/40)

Our approach outperformed both GPD and GGCNN with a large margin in both
simulated and real-robot experiments ($>25\%$). On closer inspection of real-
robot experiments, the proposed method could successfully grasp $38$ objects
out of $42$ attempts, resulting in $91\%$ success rate and $88\%$ percent
cleared, while GPD resulted in $64\%$ success rate and $76\%$ percent cleared.
In the case of GGCNN, the success rate and percent cleared degraded to $46\%$
and $63\%$ respectively by executing $29$ unsuccessful grasp attempts.

We found that mug-like objects and bottle-like objects are difficult to grasp
for GPD and GGCNN respectively, as the target object mostly slipped out of the
gripper during the manipulation phase. We observed that the proposed approach
is able to predict robust grasp quality scores for a given object. Figure 7
illustrates four examples of packed removal experiments.

Pile experiments: The obtained results are summarized in Table IV. We use an
example to explain the results. Figure 8 depicts a successful sequence of
removing a pile of four objects using the proposed approach. It was observed
that after removing Mustard and Colgate objects from the workspace (Fig. 8 a,
b), the complexity of the scene reduced significantly. Therefore, the robot
could find more grasp configurations whose grasp quality exceeds the threshold
(Fig. 8 c, d). As shown in this example, while the robot was interacting with
an object, the pose of the other objects changed completely, resulting in a
situation that the target object is not graspable (e.g., toppled Oreo box).
Such a situation was one of the main reasons for unsuccessful attempts.

TABLE IV: Performance on Pile Scenario

Method | Type | Success rate | Percent cleared
---|---|---|---
GPD | sim | 0.61 (131/214) | 0.66 (131/200)
GGCNN | sim | 0.64 (143/223) | 0.72 (143/200)
$\tau=0.8$ | sim |  0.80 (153/192) |  0.77 (153/200)
GPD | real | 0.72 (31/43) | 0.78 (31/40)
GGCNN | real | 0.78 (32/41) | 0.80 (32/40)
$\tau=0.8$ | real |  0.92 (35/38) |  0.88 (35/40)

Some other failures occurred due to the lack of friction, applying limited
force to the object, collision with other objects, and unstable grasps
predictions. A video summarizing these experiments is available online:
https://youtu.be/rBDEy5f7z0E

Figure 8: An example of successful sequence of removing a pile of objects: in
this experiment, the robot successfully removed Mustard, Colgate, Juice box,
and Coke can objects one by one. In each execution cycle, grasp configurations
that are collision-free and kinematically feasible are shown by green color.

##  V Conclusion

In this paper, we proposed a deep learning approach for real-time multi-view
3D object grasping in highly cluttered environments. We trained the approach
in an end-to-end manner. The proposed approach allows robots to robustly
interact with the environments in both isolated and highly crowded scenarios.
In particular, for a given scene, our approach first generates three
orthographic views, The best view is then selected and fed to the network to
predict a pixel-wise grasp configuration for the given object. The robot is
finally commanded to execute the highest-ranked grasp synthesis. To validate
the performance of the proposed method, we performed extensive sets of real
and simulation experiments in three scenarios: isolated, packed, and pile of
objects. Experimental results showed that the proposed method worked very well
in all three scenarios, and outperformed the selected state-of-the-art
approaches. In the continuation of this work, we would like to investigate the
possibility of improving grasp performance by learning a shape completion
function that receives a partial point cloud of a target object and generates
a complete model. We then use the full model of the object to estimate grasp
synthesis map. The other direction would be extending the proposed approach by
adding an eye-in-hand system to have more flexibility to reconstruct parts of
the environment that we are interested in. In particular, by moving the sensor
to the desired areas, we can capture significantly more details that would
otherwise not be visible. This information can be very helpful and lead to
better grasp planning.

## References

  * [1] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” _arXiv preprint arXiv:1703.09312_ , 2017.
  * [2] D. Morrison, P. Corke, and J. Leitner, “Learning robust, real-time, reactive robotic grasping,” _The International Journal of Robotics Research_ , vol. 39, no. 2-3, pp. 183–201, 2020.
  * [3] J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp synthesis—a survey,” _IEEE Transactions on Robotics_ , vol. 30, no. 2, pp. 289–309, 2013.
  * [4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke _et al._ , “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” in _Conference on Robot Learning_ , 2018.
  * [5] I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” _The International Journal of Robotics Research_ , vol. 34, no. 4-5, pp. 705–724, 2015.
  * [6] D. Morrison, P. Corke, and J. Leitner, “Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach,” in _Proc. of Robotics: Science and Systems (RSS)_ , 2018.
  * [7] M. Breyer, J. J. Chung, L. Ott, S. Roland, and N. Juan, “Volumetric grasping network: Real-time 6 dof grasp detection in clutter,” in _Conference on Robot Learning_ , 2020.
  * [8] J. Lundell, F. Verdoja, and V. Kyrki, “Beyond top-grasps through scene completion,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 545–551.
  * [9] Y. Li, L. Schomaker, and S. H. Kasaei, “Learning to grasp 3D objects using deep residual u-nets,” in _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2020, pp. 781–787.
  * [10] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen, “Shape completion enabled robotic grasping,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 2442–2447.
  * [11] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang, F. Sun, and J. Zhang, “Pointnetgpd: Detecting grasp configurations from point sets,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3629–3635.
  * [12] A. Mousavian, C. Eppner, and D. Fox, “6-dof graspnet: Variational grasp generation for object manipulation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2901–2910.
  * [13] M. Gualtieri, A. Ten Pas, K. Saenko, and R. Platt, “High precision grasp pose detection in dense clutter,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 598–605.
  * [14] A. Kanezaki, Y. Matsushita, and Y. Nishida, “RotationNet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5010–5019.
  * [15] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J. Guibas, “Volumetric and multi-view CNNs for object classification on 3D data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5648–5656.
  * [16] B. Shi, S. Bai, Z. Zhou, and X. Bai, “Deeppano: Deep panoramic representation for 3-d shape recognition,” _IEEE Signal Processing Letters_ , vol. 22, no. 12, pp. 2339–2343, 2015.
  * [17] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao, “Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 1386–1383.
  * [18] X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee, “Learning 6-DoF grasping interaction via deep geometry-aware 3D representations,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 3766–3773.
  * [19] D. De Gregorio, F. Tombari, and L. Di Stefano, “Robotfusion: Grasping with a robotic manipulator via multi-view reconstruction,” in _European Conference on Computer Vision_. Springer, 2016, pp. 634–647.
  * [20] D. Morrison, P. Corke, and J. Leitner, “Multi-view picking: Next-best-view reaching for improved grasping in clutter,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 8762–8768.
  * [21] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,” _The International Journal of Robotics Research_ , vol. 36, no. 13-14, pp. 1455–1473, 2017.
  * [22] S. H. Kasaei, M. Oliveira, G. H. Lim, L. S. Lopes, and A. M. Tomé, “Towards lifelong assistive robotics: A tight coupling between object perception and manipulation,” _Neurocomputing_ , vol. 291, pp. 151–166, 2018\.
  * [23] S. H. Kasaei, L. Seabra Lopes, and A. Tomé, “Local-LDA: Open-ended learning of latent topics for 3D object recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * [24] S. H. Kasaei, N. Shafii, L. Seabra Lopes, and A. M. Tomé, “Object learning and grasping capabilities for robotic home assistants,” in _RoboCup 2016: Robot World Cup XX_. Springer International Publishing, 2017, pp. 279–293.
  * [25] Y. Jiang, S. Moseson, and A. Saxena, “Efficient grasping from RGBD images: Learning using a new rectangle representation,” in _IEEE International conference on robotics and automation_. IEEE, 2011, pp. 3304–3311.
  * [26] J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 1316–1322.
  * [27] Z. Wang, Z. Li, B. Wang, and H. Liu, “Robot grasp detection using multimodal deep convolutional neural networks,” _Advances in Mechanical Engineering_ , vol. 8, no. 9, 2016.
  * [28] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Yale-cmu-berkeley dataset for robotic manipulation research,” _The International Journal of Robotics Research_ , vol. 36, no. 3, pp. 261–268, 2017.


respect to $u$, we can construct g-characteristic isotone embeddings
$l^{\prime},m\colon\triangledown(\kappa)\to G$ satisfying the condition (a-1).
We define $l\colon\triangledown(\kappa)\to G$ by
$l(\alpha)=\min\\{l^{\prime}(\alpha),m(\alpha+1)\\}$ if $\alpha<\kappa$, and
by $l(\kappa)=0_{G}$. Then $m$ is a g-characteristic isotone embedding, and
the pair $l,m\colon\triangledown(\kappa)\to G$ satisfies the conditions (a-1)
and (a-2).

For the sake of contradiction, we suppose that there exists a Cauchy filter
$\mathcal{F}$ on $(X,\Psi)$, which is not Cauchy on $(X,u)$. Then, we obtain
the following fact:

  1. (Fact 1)

there exits $\epsilon\in G_{>0}$ such that for all $F\in\mathcal{F}$, and for
all $x,y\in F$, we have $\epsilon\leq u(x,y)$.

Since $l\colon\triangledown(\kappa)\to G$ is a g-characteristic isotone
embedding and $\mathcal{F}$ is Cauchy on $(X,\Psi)$, we can take a map
$\mathfrak{B}\colon\kappa\to\mathcal{F}$ stated in Lemma 2.37 associated with
the map $l$.

According to the condition (2) in Lemma 2.37, the following statement holds
true:

  1. (Fact 2)

we have $\Psi(x,y)\leq l(\alpha)$ for all $\alpha<\kappa$, and for all
$x,y\in\mathfrak{B}(\alpha)$,

Thus, we obtain $\varrho_{u,A}(x)\lor\varrho_{u,A}(y)\leq l(\alpha)$. Then, by
(a-2), for each $x\in\mathfrak{B}(\alpha)$, there exists $a_{x}\in A$ with
$u(x,a_{x})<m(\alpha)$. By (a-1), we have
$u(a_{x},r(x))=u(r(x),r(a_{x}))<m(\alpha)$. Then, for all $\alpha<\kappa$, and
for all $x\in\mathfrak{B}(\alpha)$, we have $u(r(x),x)\leq u(r(x),a_{x})\lor
u(a_{x},x)\leq m(\alpha)\lor m(\alpha)=m(\alpha)$. By (a-2), for all
$\alpha<\kappa$, and for all $x\in\mathfrak{B}(\alpha)$, we obtain

(4.8) $\displaystyle u(r(x),x)\leq m(\alpha).$

By $r^{*}d(x,y)\leq\Psi(x,y)$, and by (Fact 2), we have $d(r(x),r(y))\leq
l(\alpha)$ for all $x,y\in\mathfrak{B}(\alpha)$ and for all $\alpha<\kappa$.
Thus, the pushout filter $r_{\sharp}\mathcal{F}$ of $\mathcal{F}$ by $r$ is a
Cauchy filter on $(A,d)$.

Since $(A,d)$ is complete, there exits a limit $p\in A$ of
$r_{\sharp}\mathcal{F}$. Since $d$ and $u$ generate the same topology on $A$,
the point $p$ is also a limit of $r_{\sharp}\mathcal{F}$ with respect to $u$.
Then, there exists $\theta<\kappa$ satisfying that

(4.9) $\displaystyle m(\theta)<\epsilon;$ (4.10) $\displaystyle
r(\mathfrak{B}(\theta))\subseteq U(p,\epsilon;u).$

From (4.8), (4.9), and (4.10), it follows that for all
$x\in\mathfrak{B}(\theta)$, we have

$u(p,x)\leq u(p,r(x))\lor u(r(x),x)<\epsilon\lor m(\alpha)\leq\epsilon.$

Namely, we obtain $\mathfrak{B}(\theta)\subseteq U(p,\epsilon;u)$. This
inclusion and the strong triangle inequality imply that $u(x,y)<\epsilon$ for
all $x,y\in\mathfrak{B}(\theta)$. This contradicts (Fact 1). Thus, the filter
$\mathcal{F}$ is Cauchy on $(X,u)$. This leads to the lemma. ∎

######  Remark 4.1.

The author does not know whether the metrics $d$ and
$\Psi[S,\Sigma[S,h,r,k],r,A,d]$ in Lemma 4.4 are uniformly equivalent to each
other or not.

###### Proof of Theorem 1.2.

We assume that $X$ is a topological space possessing an infinite metrizable
gauge. Let $\kappa$ be a regular cardinal with with
$\kappa\in\operatorname{\mathscr{MG}}(X)$. Let $G\in\mathrm{GRP}(\kappa)$. Let
$S$ be a g-characteristic subset of $G$. Let $A$ be a closed subset of $X$.

Remark that since $S$ is g-characteristic in $G$, the set $\lambda_{G}(S)$ is
characteristic in the set $\Xi(\mathbf{A}(G))$. Fix a characteristic isotone
embedding $m\colon\triangledown(\kappa)\to\Xi(\mathbf{A}(G))$ such that
$m(\alpha)\in\lambda_{G}(S)$ for all $\alpha\in\triangledown(\kappa)$. For
each $\alpha\in\triangledown(\kappa)$, we take $l_{\alpha}\in S$ such that
$\lambda_{G}(l_{\alpha})=m(\alpha)$. We define a map
$l\colon\triangledown(\kappa)\to S$ by $l(\alpha)=l_{\alpha}$. Then the map
$l$ is a characteristic isotone embedding, and the relation $\lambda_{G}\circ
l=m$ holds true. Let
$\zeta_{\Xi(\mathbf{A}(G)),m}\colon\Xi(\mathbf{A}(G))\to\triangledown(\kappa)$
be a map in Definition 2.12 associated with $m$. Put
$\xi=\zeta_{\Xi(\mathbf{A}(G)),m}$. We put $T=l(\triangledown(\kappa))$. Note
that $T$ is a Dedekind complete g-characteristic subset of $G$ and $T\subseteq
S$.

For all $\eta\in G_{>0}$, there exists $\alpha\in\triangledown(\kappa)$ such
that $m(\alpha+1)<\lambda_{G}(\eta)\leq m(\alpha)$. In this case, we have
$\xi\circ\lambda_{G}(\eta)=\alpha+1$, and hence
$l\circ\xi\circ\lambda_{G}(\eta)=l(\alpha+1)$. By the definition of $l$, by
$\lambda_{G}\circ l=m$, and by $m(\alpha+1)<\lambda_{G}(\eta)$, we have
$l\circ\xi\circ\lambda_{G}(\eta)<\eta$. Namely, the following holds true:

  1. (Fact 1)

for all $\eta\in G_{>0}$, we have $l\circ\xi\circ\lambda_{G}(\eta)<\eta$.

Fix $h\in\operatorname{UMet}(X;T)$ (see Proposition 2.24). Using Theorem 1.1,
we take a uniformly continuous retraction $r\colon X\to A$ with respect to
$h$.

For each $d\in\operatorname{Met}(A;G)$, we put $D[d]=\lambda_{G}\circ
d\in\operatorname{UMet}(A;\Xi(\mathbf{A}(G)))$ (see Lemma 2.20). Let
$k[d]=l\circ\xi\circ D[d](=l\circ\xi\circ\lambda_{G}\circ d)$. According to
Lemmas 2.21 and 2.22, we observe that $k[d]\in\operatorname{UMet}(X;T)$.

Put $u[d]=\Sigma[T,h,r,k[d]](=r^{*}k[e]\lor h)$. Then we have
$u[d]\in\operatorname{UMet}(X;T)$. According to Lemma 2.35, the metrics $d$
and $D[d]$ are uniformly equivalent to each other. Thus, $D[d]$ satisfies the
assumption in Lemma 4.4. By Lemma 4.4, the map $r$ is uniformly continuous
with respect to $u[d]$.

We define a map $\Phi\colon\operatorname{Met}(A;G)\to\operatorname{Met}(X;G)$
by

$\Phi(d)=\Psi[T,u[d],r,A,d].$

We now verify that the map $\Phi$ is a desired one. By Lemma 4.3, we observe
that $\Phi$ is certainly a map from $\operatorname{Met}(A;G)$ into
$\operatorname{Met}(X;G)$.

We now show that $\Phi$ satisfies (A1). Fix $d\in\operatorname{Met}(A;G)$ and
$\epsilon\in G_{>0}$. Take $\eta\in G_{>0}$ with $\eta<\epsilon$. Take
arbitrary $e\in\mathscr{V}(d;\eta)$, which means that for all $x,y\in A$ we
have

(4.11) $\displaystyle d(x,y)<e(x,y)\lor\eta;$ (4.12) $\displaystyle
e(x,y)<d(x,y)\lor\eta.$

Since $\lambda_{G}$ is isotone, by (4.11) and (4.12), for all $x,y\in A$, we
have

(4.13) $\displaystyle D[d](x,y)\leq D[e](x,y)\lor\lambda_{G}(\eta);$ (4.14)
$\displaystyle D[e](x,y)\leq D[d](x,y)\lor\lambda_{G}(\eta).$

Since $l$ and $\xi$ are isotone, by (Fact 1), and by (4.13), and (4.14), for
all $x,y\in A$, we obtain

(4.15) $\displaystyle k[d](x,y)\leq k[e](x,y)\lor\eta;$ (4.16) $\displaystyle
k[e](x,y)\leq k[d](x,y)\lor\eta.$

By (4.15) and (4.16), for all $x,y\in X$, we have

(4.17) $\displaystyle r^{*}k[d](x,y)\leq r^{*}k[e](x,y)\lor\eta;$ (4.18)
$\displaystyle r^{*}k[e](x,y)\leq r^{*}k[d](x,y)\lor\eta.$

These inequalities imply that for all $x,y\in X$, we have

(4.19) $\displaystyle u[d](x,y)\leq u[e](x,y)\lor\eta;$ (4.20) $\displaystyle
u[e](x,y)\leq u[d](x,y)\lor\eta.$

By (4.19) and (4.20), for all $x\in X$, we obtain

(4.21) $\displaystyle\varrho_{u[d],A}(x)\leq\varrho_{u[e],A}(x)\lor\eta;$
(4.22) $\displaystyle\varrho_{u[e],A}(x)\leq\varrho_{u[d],A}(x)\lor\eta.$

Thus, for all $x,y\in X$, the following hold true:

(4.23) $\displaystyle\Theta[u[d],A](x,y)\leq\Theta[u[e],A](x,y)\lor\eta;$
(4.24) $\displaystyle\Theta[u[e],A](x,y)\leq\Theta[u[d],A](x,y)\lor\eta.$

Therefore, by $\eta<\epsilon$, we conclude that
$\Phi(e)\in\mathscr{V}(\Phi(d);\epsilon)$, and hence the map $\Phi$ is
continuous. This implies the condition (A1).

By the definition of $\Phi$, we see that $\Phi$ satisfies (A2).

The latter part of Lemma 4.2 implies the conditions (A4) and (A5) (note that
$T\subseteq S$).

Note that $D[d\lor e]=D[d]\lor D[e]$ and $k[d\lor e]=k[d]\lor k[e]$. By
$k[d\lor e]=k[d]\lor k[e]$, we have
$\varrho_{u[d]}\lor\varrho_{u[e]}=\varrho_{u[d\lor e]}$. Then $\Theta_{k[d\lor
e]}=\Theta_{k[d]}\lor\Theta_{k[e]}$, and hence $\Phi(d\lor
e)=\Phi(d)\lor\Phi(e)$. Thus, the condition (A6) is satisfied. The condition
(A6) implies the condition (A3).

To show the condition (A7), we assume that there exists a complete $G$-metric
$D$ in $\operatorname{Met}(X;G)$. By Lemma 2.36, there exists a complete
$\triangledown(\kappa)$-ultrametric
$H\in\operatorname{UMet}(X;\triangledown(\kappa))$. Put $h^{\prime}=l\circ H$.
By Lemmas 2.21 and 2.35, we have $h^{\prime}\in\operatorname{UMet}(X;T)$ and
$h^{\prime}$ is complete. Using Theorem 1.1, we can take a uniformly
continuous retraction $r^{\prime}\colon X\to A$ with respect to $h^{\prime}$.
According to Lemmas 2.20 and 2.35, if $d\in\operatorname{Met}(A;G)$ is
complte, then so is $D[d]$. Note that $\Sigma[S,h^{\prime},r^{\prime},k[d]]$
is complete.

Using $h^{\prime}$ and $r^{\prime}$ instead of $h$ and $r$ in the argument
discussed above, by Lemma 4.4 ($D[d]$ and $d$ are uniformly equivalent to each
other), we obtain a map
$\Phi\colon\operatorname{Met}(A;G)\to\operatorname{Met}(X;G)$ satisfying that
if $d\in\operatorname{Met}(X;G)$ is complete, then so is the $G$-metrics
$\Phi(d)$. Namely, the map $\Phi$ satisfies the condition (A7).

This finishes the proof of Theorem 1.2. ∎

###### Proof of Theorem 1.3.

Let $S$ be a characteristic subset of $\mathbb{R}_{\geq 0}$. Let $X$ be an
$\mathbb{R}_{\geq 0}$-ultrametrizable space.

Put $G=\mathbb{P}(S)$. Then the set $S$ can considered as a g-characteristic
subset of $G$ (see Proposition 2.12). Let
$\Phi:\operatorname{Met}(A;G)\to\operatorname{Met}(X;G)$ be a map stated in
Theorem 1.2. Note that
$\operatorname{UMet}(X;S)\subseteq\operatorname{Met}(X;G)$ (see Lemma 2.23).
Put $\Upsilon=\Phi|_{\operatorname{UMet}(X;S)}$. By the condition (A5) in
Theorem 1.2, we confirm that $\Upsilon$ is a map from
$\operatorname{UMet}(A;S)$ into $\operatorname{UMet}(X;S)$. Similarly to the
proof of Theorem 1.2, we observe that $\Phi$ satisfies (B1)–(B5). In
particular, from the argument verifying that $e\in\mathscr{V}(d;\eta)$ implies
$\Phi(e)\in\mathscr{V}(\Phi(d);\epsilon)$, it follows that $\Upsilon$ is an
isometric embedding with respect to $\operatorname{\mathcal{UD}}_{A}^{S}$ and
$\operatorname{\mathcal{UD}}_{X}^{S}$.

Note that, to prove the condition (B5), we need to confirm that there exists a
complete $S$-ultrametric $D\in\operatorname{Met}(X;G)$. From [26, Proposition
2.17], there exists complete $D\in\operatorname{UMet}(X;S)$, and hence, we
have $D\in\operatorname{Met}(X;G)$. Thus, the space $X$ satisfies the
assumption in Theorem 1.2 for the condition (A7). Hence the condition (B5)
holds true.

This completes the proof of Theorem 1.3. ∎

###  4.2. Proof of Theorem 1.4

We next show Theorem 1.4.

The following lemma is stated in [48, Theorem 1.3] (see also [4, Proposition
2.2]. The proof is similary to the case of ordinary metric spaces.

######  Lemma 4.5.

We assume that $X$ is a topological space possessing an infinite metrizable
gauge. Let $\kappa$ be a regular cardinal with
$\kappa\in\operatorname{\mathscr{MG}}(X)$. Then the following are equivalent
to each other:

  1. (1)

Every $\kappa$-sequence in $X$ has a convergent $\kappa$-subsequence.

  2. (2)

Every open cover of $X$ whose cardinal is $\kappa$ has a subcover with
cardinal $<\kappa$.

  3. (3)

The space $X$ is finally $\kappa$-compact.

######  Lemma 4.6.

We assume that $X$ is a topological space possessing an infinite metrizable
gauge. Let $\kappa$ be a regular cardinal with
$\kappa\in\operatorname{\mathscr{MG}}(X)$. If $X$ is not finally
$\kappa$-compact, then there exists a closed subset $A$ of $X$ such that
$\operatorname{Card}(A)=\kappa$ and $A$ is discrete.

###### Proof.

Since $X$ is not finally $\kappa$-compact, by Lemma 4.5, there exists a
$\kappa$-sequence $\\{p_{\alpha}\\}_{\alpha<\kappa}$ whose all
$\kappa$-subsequence have no limits. Put
$A=\\{\,p_{\alpha}\mid\alpha<\kappa\,\\}$. Then, for all $p\in A$, there
exists an open set $U_{p}$ such that $p\in U_{p}$ and $U_{p}\cap A=\\{p\\}$.
Thus, the set $A$ is discrete. Since all $\kappa$-subsequences of
$\\{p_{\alpha}\\}_{\alpha<\kappa}$ have no limits, we conclude that for each
$p\in A$, we have $\operatorname{Card}(\\{\,\alpha<\kappa\mid
p_{\alpha}=p\,\\})<\kappa$. Thus, by the regularity of $\kappa$, we see that
$\operatorname{Card}(A)=\kappa$. ∎

###### Proof of Theorem 1.4.

We assume that $X$ is a topological space possessing an infinite metrizable
gauge. Let $\kappa$ be a regular cardinal with
$\kappa\in\operatorname{\mathscr{MG}}(X)$.

Since $\mathrm{GRP}(\kappa)\neq\emptyset$ and
$\mathrm{ORD}(\kappa)\neq\emptyset$, we obtain the implications
$(\ref{item:com3})\Longrightarrow(\ref{item:com2})$ and
$(\ref{item:com5})\Longrightarrow(\ref{item:com4})$. According to Lemma 2.40,
we have the implications $(\ref{item:com1})\Longrightarrow(\ref{item:com3})$
and $(\ref{item:com1})\Longrightarrow(\ref{item:com5})$.

To show $(\ref{item:com2})\Longrightarrow(\ref{item:com4})$, we assume the
statement (2). Then all $d\in\operatorname{UMet}(X;G_{\geq 0})$ are complete.
Put $S=G_{\geq 0}$. According to Proposition 2.15, we have
$S\in\mathrm{ORD}(\kappa)$. Hence the statement (4) holds true. Thus, we
obtain the implication $(\ref{item:com2})\Longrightarrow(\ref{item:com4})$.

To prove Theorem 1.4, we only need to show the implication
$(\ref{item:com4})\Longrightarrow(\ref{item:com1})$. We assume that $X$ is not
finally $\kappa$-compact. Then, by Lemma 4.6, there exists a closed subset $A$
of $X$ such that $\operatorname{Card}(A)=\kappa$ and $A$ is discrete. Take a
bijection $f\colon A\to\kappa$, and put $g=f^{-1}$. Let
$e=f^{*}M_{\triangledown(\kappa)}$, where $M_{\triangledown(\kappa)}$ is the
ultrametric on $\triangledown(\kappa)$ in Definition 2.14. Then
$e\in\operatorname{UMet}(A;\triangledown(\kappa))$.

Note that $\kappa\subseteq\triangledown(\kappa)$. Put
$G=\mathbb{P}(\triangledown(\kappa))$. The set $\triangledown(\kappa)$ can be
considered as a g-characteristic subset of $G$ (see Proposition 2.12), and
hence $e\in\operatorname{UMet}(X;G_{\geq 0})$. We take a map
$\Phi\colon\operatorname{Met}(A;G)\to\operatorname{Met}(X;G)$ satisfying the
conditions in Theorem 1.2. Put $d=\Phi(e)$. Then, by condition (A4) in Theorem
1.2, we obtain $d\in\operatorname{UMet}(X;G_{\geq 0})$ with $d|_{A^{2}}=e$.
Since $\\{\alpha\\}_{\alpha<\kappa}$ is Cauchy in
$(\triangledown(\kappa),M_{\triangledown(\kappa)})$ (see (3) in Corollary
2.43), the sequence $\\{g(\alpha)\\}_{\alpha<\kappa}$ is Cauchy in $(X,d)$.
Since $A$ is closed, the sequence $\\{g(\alpha)\\}_{\alpha<\kappa}$ has no
limits. Hence $(X,d)$ is not complete. Therefore we obtain the implication
$(\ref{item:com4})\Longrightarrow(\ref{item:com1})$. This finishes the proof
of Theorem 1.4. ∎

##  5\. Table of symbols

Symbol  | Description
---|---
$\operatorname{Met}(X;G)$ |  The set of all $G$-metrics that generate the same topology on a topological space $X$.
${\ooalign{$\bigcirc$\crcr\hss$\bullet$\hss}}_{S}$ |  The minimal element of a bottomed inearly ordered set $S$.
$\Xi(L)$ |  The one-point order extension of a linearly ordered set $L$. This is a ordered set by putting the minimal element ${\ooalign{$\bigcirc$\crcr\hss$\bullet$\hss}}_{\Xi(G)}$ to $L$ (Definition 1.4).
$\operatorname{\chi}(S)$ |  The character of $S$ (Definition 1.3).
$x\ll y$ |  For all $n\in\mathbb{Z}_{\geq 1}$, we have $n\cdot x<y$.
$x\asymp y$ |  $x$ and $y$ are Archimedean equivalent to each other. Namely, there exist $n,n\in\mathbb{Z}_{\geq 1}$ such that $y\leq n\cdot x$ and $x\leq m\cdot y$.
$[x]_{\asymp}$ |  The Archimedean equivalence class of $x$.
$\mathbf{A}(G)$ |  The set of all Archimedean classes of a linearly ordered Abelian group $G$.
$\alpha\mathrel{\mathchoice{\vbox to7.71942pt{\offinterlineskip\hbox{$\displaystyle\ll$}\vss\vbox to0.0pt{ \vss\hbox to12.77773pt{$\mspace{-0.5mu}\displaystyle{-}\hss{-}\mspace{-0.5mu}$}\kern 0.0pt{\displaystyle} 3 \kern 0.0pt{\displaystyle} 2 } }}{\vbox to7.71942pt{\offinterlineskip\hbox{$\textstyle\ll$}\vss\vbox to0.0pt{ \vss\hbox to12.77773pt{$\mspace{-0.5mu}\textstyle{-}\hss{-}\mspace{-0.5mu}$}\kern 0.0pt{\textstyle} 3 \kern 0.0pt{\textstyle} 2 } }}{\vbox to5.4036pt{\offinterlineskip\hbox{$\scriptstyle\ll$}\vss\vbox to0.0pt{ \vss\hbox to9.77773pt{$\mspace{-0.5mu}\scriptstyle{-}\hss{-}\mspace{-0.5mu}$}\kern 0.0pt{\scriptstyle} 3 \kern 0.0pt{\scriptstyle} 2 } }}{\vbox to3.85971pt{\offinterlineskip\hbox{$\scriptscriptstyle\ll$}\vss\vbox to0.0pt{ \vss\hbox to7.77771pt{$\mspace{-0.5mu}\scriptscriptstyle{-}\hss{-}\mspace{-0.5mu}$}\kern 0.0pt{\scriptscriptstyle} 3 \kern 0.0pt{\scriptscriptstyle} 2 } }}}\beta$ |  $x\ll y$ or $x\asymp y$, where $\alpha=[x]_{\asymp}$ and $\beta=[y]_{\asymp}$.
$\alpha\ll\beta$ |  $x\ll y$, where $\alpha=[x]_{\asymp}$ and $\beta=[y]_{\asymp}$. We use the same symbol to the order $\ll$ on $G$.
$\operatorname{\mathscr{MG}}(X)$ |  The set (class) of all metrizable gauges of a topological space $X$.
$\omega_{0}$ |  The least infinite cardinal. $\omega_{0}=\\{0,1,2,\dots,\\}$
$\mathrm{GRP}(\kappa)$ |  The class of all linearly ordered Abelian group $G$ with $\operatorname{\chi}(\mathbf{A}(G))=\kappa$.
$\mathrm{ORD}(\kappa)$ |  The class of all bottomed linearly ordered set $S$ with $\operatorname{\chi}(S)=\kappa$.
$\mathscr{V}(d;\epsilon)$ |  The set of all $e\in\operatorname{Met}(X;d)$ such that for all $x,y\in X$ we have $e(x,y)<d(x,y)\lor\epsilon$ and $d(x,y)<e(x,y)\lor\epsilon$.
$\operatorname{\mathcal{UD}}_{X}^{S}$ |  The infimum of $\epsilon\in S\sqcup\\{\infty\\}$ such that for all $x,y\in X$ we have $d(x,y)\leq e(x,y)\lor\epsilon$, and $e(x,y)\leq d(x,y)\lor\epsilon$.
$\mathrm{op}(S)$ |  The ordered set with the dual order to $(S,\leq_{S})$.
$\triangledown(\kappa)$ |  $\mathrm{op}(\kappa+1)$ (Definition 2.1).
$S^{\\#}$ |  The Dedekind completion of $(S,\leq)$.
$\mathrm{abs}(x)$ |  The absolute value of $x$ of $G$ (Definition 2.5).
$\lambda_{G}(x)$ |  The natural valuation (or covaluation) of $G$ (Definition 2.5).
$\mathbb{H}(L)$ |  The Hahn group induced from a linearly ordered set $L$.
$\mathbb{F}(G)$ |  The Hahn field induced from a linearly ordered Abelian group $G$.
$S^{\bigstar}$ |  $S^{\bigstar}=S\setminus\\{{\ooalign{$\bigcirc$\crcr\hss$\bullet$\hss}}_{S}\\}$ (Definition 2.8).
$\mathbb{P}(S)$ |  $\mathbb{P}(S)=\mathbb{F}(\mathbb{H}(\mathrm{op}(S^{\bigstar})))$ (Definition 2.10).
$\zeta_{S,l}$ |  The map defined in Definition 2.12 induced from $S\in\mathrm{ORD}(\kappa)$ and a characteristic isotone embedding $l\colon\triangledown(\kappa)\to S$.
$U(x,\epsilon;d)$ |  The open ball centered at $x$ with radius $\epsilon$ by a $G$-metric or $S$-ultrametric $d$.
$B(x,\epsilon;d)$ |  The closed ball centered at $x$ with radius $\epsilon$ by a $G$-metric or $S$-ultrametric $d$.
$M_{S}$ |  The $S$-ultrametric on $S$ defined in Definition 2.14.
$\varrho_{d,A}(x)$ |  The distance between the set $A$ and the point $x$ (Definition 3.1).
$\Theta[S,h,A]$ |  The $S$-pseudo-ultrametric defined in Definition 4.1.
$\Psi[S,h,r,A,d]$ |  The metric defined in Definition 4.3.
$f^{*}d$ |  The pullback induced from $d$ and a map $f$ (Definition 4.2).
$\Sigma[S,h,r,k]$ |  The metric defined in Definition 4.4.
$f_{\sharp}\mathcal{F}$ |  The pushout filter of $\mathcal{F}$ by a map $f$.

## References

  * [1] H. Abels and A. Manoussos, _Topological generators of abelian lie groups and hypercyclic finitely generated abelian semigroups of matrices_ , Adv. Math. 229 (2012), no. 3, 1862–1872.
  * [2] G. Artico, U. Marconi, and J. Pelant, _On supercomplete $\omega_{\mu}$-metric spaces_, Bull. Polish Acad. Sci. Math. 44 (1996), no. 3, 299–310.
  * [3] G. Artico and R. Moresco, _Some results on $\omega_{\mu}$-metric spaces_, Annali di Matematica Pura ed Applicata 128 (1981), no. 1, 241–252.
  * [4] by same author, _$\omega_{\mu}$ -additive topological spaces_, Rend. Sem. Mat. Univ. Padova 67 (1982), 131–141.
  * [5] N. Brodskiy, J. Dydak, J. Higes, and A. Mitra, _Dimension zero at all scales_ , Topology Appl. 154 (2007), no. 14, 2729–2740.
  * [6] K. A. Broughan, _A metric characterizing Čech dimension zero_ , Proc. Amer. Math. Soc. 39 (1973), 437–440.
  * [7] A. H. Clifford, _Note on Hahn’s theorem on ordered abelian groups_ , Proc. Amer. Math. Soc. 5 (1954), 860–863.
  * [8] A. B. Comicheo, _Generalized open mapping theorem for $X$-normed spaces_, $p$-Adic Numbers Ultrametric Anal. Appl. 11 (2019), no. 2, 135–150.
  * [9] J. Dancis, _Each closed subset of metric space $X$ with $\mathrm{Ind}\ X=0$ is a retract_, Houston J. Math 19 (1993), no. 4, 541–550.
  * [10] C. Delhommé, C. Laflamme, M. Pouzet, and N. Sauer, _Indivisible ultrametric spaces_ , Topology Appl. 155 (2008), no. 14, 1462–1478.
  * [11] A. Di Concilio and C. Guadagni, _Uniform continuity in $\omega_{\mu}$-metric spaces and uc $\omega_{\mu}$-metric extendability_, Acta Math. Hungar. 150 (2016), no. 1, 153–166.
  * [12] by same author, _Hypertopologies on $\omega_{\mu}$-metric spaces_, Filomat 31 (2017), no. 13, 4063–4068.
  * [13] D. Dordovskyi, O. Dovgoshey, and E. Petrov, _Diameter and diametrical pairs of points in ultrametric spaces_ , $p$-Adic Numbers Ultrametric Anal. Appl. 3 (2011), no. 4, 253–262.
  * [14] A. A. Dovgosheĭ and E. A. Petrov, _A subdominant pseudoultrametric on graphs_ , Mat. Sb. 204 (2013), no. 8, 51–72.
  * [15] O. Dovgoshey, _On ultrametric-preserving functions_ , Math. Slovaca 70 (2020), no. 1, 173–182.
  * [16] O. Dovgoshey, O. Martio, and M. Vuorinen, _Metrization of weighted graphs_ , Ann. Comb. 17 (2013), no. 3, 455–476.
  * [17] O. Dovgoshey and V. Shcherbak, _The range of ultrametrics, compactness, and separability_ , Topology Appl. 305 (2022), Paper No. 107899, 19 pages.
  * [18] J. Dugundji, _Absolute neighborhood retracts and local connectedness in arbitrary metric spaces_ , Compos. Math. 13 (1958), 229–246 (1958).
  * [19] R. Engelking, _On closed images of the space of irrationals_ , Proc. Amer. Math. Soc. 21 (1969), 583–586.
  * [20] A. J. Engler and A. Prestel, _Valued fields_ , Springer Monographs in Mathematics, Springer-Verlag, Berlin, 2005.
  * [21] H. Hahn, _Über die nichtarchimedischen Größensysteme_ , Sitz. ber. K. Akad. der Math. Nat. Kl. IIa 116 (1907), 601–655, (Hans Hahn Gresammelte Abhandlungen Band 1, Hans Hahn Collected Works Vol. 1, pp. 445–499, Springer, 1995).
  * [22] F. Hausdorff, _Erweiterung einer homöomorphie_ , Fund. Math. 16 (1930), 353–360.
  * [23] M. Hausner and J. G. Wendel, _Ordered vector spaces_ , Proc. Amer. Math. Soc. 3 (1952), 977–982.
  * [24] A. Hayes, _Uniformities with totally ordered bases have paracompact topologies_ , Proc. Cambridge Philos. Soc. 74 (1973), 67–68.
  * [25] H. H. Hung, _The amalgamation property for $G$-metric spaces_, Proc. Amer. Math. Soc. 37 (1973), 53–58.
  * [26] Y. Ishiki, _An embedding, an extension, and an interpolation of ultrametrics_ , $p$-Adic Numbers Ultrametric Anal. Appl. 13 (2021), no. 2, 117–147.
  * [27] by same author, _On dense subsets in spaces of metrics_ , (2021), preprint arXiv:2104.12450, to apper in Colloq. Math.
  * [28] T. Jech, _Set theory_ , Springer Monographs in Mathematics, Springer-Verlag, Berlin, 2003, The third millennium edition, revised and expanded.
  * [29] I. Juhász, _Untersuchungen über $\omega_{\mu}$-metrisierbare Räume_, Ann. Univ. Sci. Budapest. Eötvös Sect. Math. 8 (1965), 129–145.
  * [30] J. L. Kelley, _General topology_ , Graduate Texts in Mathematics, No. 27, Springer-Verlag, New York-Berlin, 1975, Reprint of the 1955 edition [Van Nostrand, Toronto, Ont.].
  * [31] Nguyen Van Khue and Nguyen To Nhu, _Two extensors of metrics_ , Bull. Acad. Polon. Sci. 29 (1981), 285–291.
  * [32] J. Kąkol, A. Kubzdela, and W. Śliwa, _A non-Archimedean Dugundji extension theorem_ , Czechoslov. Math. J. 63(138) (2013), no. 1, 157–164.
  * [33] Y. Kodama, _On ${\rm LC}^{n}$ metric spaces_, Proc. Japan Acad. 33 (1957), 79–83.
  * [34] A. Kucia and W. Kulpa, _Spaces having uniformities with linearly ordered bases_ , Uniw. Śląski w Katowicach—Prace Mat. 3 (1973), 45–50.
  * [35] S. Kuhlmann, _Ordered exponential fields_ , Fields Institute Monographs, vol. 12, American Mathematical Society, Providence, RI, 2000.
  * [36] E. Michael, _Selected Selection Theorems_ , Amer. Math. Monthly 63 (1956), no. 4, 233–238.
  * [37] V. Niemytzki and A. Tychonoff, _Beweis des satzes, dass ein metrisierbarer raum dann und nur dann kompakt ist, wenn er in jeder metrik vollständig ist_ , Fund. Math. 12 (1928), 118–120.
  * [38] K. Nomizu and H. Ozeki, _The existence of complete Riemannian metrics_ , Proc. Amer. Math. Soc. 12 (1961), 889–891.
  * [39] P. Nyikos and H. C. Reichel, _On uniform spaces with linearly ordered bases. II. ( $\omega_{\mu}$-metric spaces)_, Fund. Math. 93 (1976), no. 1, 1–10.
  * [40] H. Ochsenius and E. Olivos, _A comprehensive survey of non-Archimedean analysis in Banach spaces over fields with an infinite rank valuation_ , Advances in ultrametric analysis, Contemp. Math., vol. 596, Amer. Math. Soc., Providence, RI, 2013, pp. 215–236.
  * [41] C. Perez-Garcia and W. H. Schikhof, _Locally convex spaces over non-Archimedean valued fields_ , Cambridge Studies in Advanced Mathematics, vol. 119, Cambridge University Press, Cambridge, 2010.
  * [42] P. Pongsriiam and I. Termwuttipong, _Remarks on ultrametrics and metric-preserving functions_ , Abstr. Appl. Anal. (2014), Art. ID 163258, 9.
  * [43] S. Priess-Crampe, _Generalized Keller spaces_ , Bull. Belg. Math. Soc. Simon Stevin 14 (2007), no. 5, 979–991.
  * [44] R. Sikorski, _Remarks on some topological spaces of high power_ , Fund. Math. 37 (1950), no. 1, 125–136.
  * [45] D. J. Souppouris, _Generalized metric spaces are paracompact_ , Math. Proc. Cambridge Philos. Soc. 77 (1975), 325–326.
  * [46] I. S. Stares and J. E. Vaughan, _The Dugundji extension property can fail in $\omega_{\mu}$-metrizable spaces_, Fund. Math. 150 (1996), no. 1, 11–16.
  * [47] I. Stasyuk and E. D. Tymchatyn, _A continuous operator extending ultrametrics_ , Comment. Math. Univ. Carolin. 50 (2009), no. 1, 141–151.
  * [48] F. W. Stevenson and W. J. Thorn, _Results on $\omega_{\mu}$-metric spaces_, Fund. Math. 65 (1969), no. 3, 317–324.
  * [49] E. D. Tymchatyn and M. Zarichnyi, _A note on operators extending partial ultrametrics_ , Comment. Math. Univ. Carolin. 46 (2005), no. 3, 515–524.
  * [50] E. K. van Douwen, _Simultaneous extension of continuous functions_ , Ph.D. thesis, Vrije Universiteit Amsterdam, 1975.
  * [51] S. -T. Wang, _Remarks of $\omega_{\mu}$-additive spaces_, Fund. Math. 55 (1964), 101–112.
  * [52] S. Willard, _General topology_ , Dover Publications, 2004; originally published by the Addison-Wesley Publishing Company in 1970.