# MonoGraspNet: 6-DoF Grasping with a Single RGB Image

Guangyao Zhai1,∗, Dianye Huang1,∗, Shun-Cheng Wu1, Hyunjun Jung1,
Yan Di1,†, Fabian Manhardt2, Federico Tombari1,2, Nassir Navab1,3 and Benjamin
Busam1 * Contribute equally. ${\dagger}$ Corresponding author.1 Technical
University of Munich (TUM), Munich, Germany.2 Google Inc.3 Johns Hopkins
University, Baltimore, MD, USA.

###### Abstract

6-DoF robotic grasping is a long-lasting but unsolved problem. Recent methods
utilize strong 3D networks to extract geometric grasping representations from
depth sensors, demonstrating superior accuracy on common objects but perform
unsatisfactorily on photometrically challenging objects, e.g., objects in
transparent or reflective materials. The bottleneck lies in that the surface
of these objects can not reflect back accurate depth due to the absorption or
refraction of light. In this paper, in contrast to exploiting the inaccurate
depth data, we propose the first RGB-only 6-DoF grasping pipeline called
MonoGraspNet that utilizes stable 2D features to simultaneously handle
arbitrary object grasping and overcome the problems induced by photometrically
challenging objects. MonoGraspNet leverages keypoint heatmap and normal map to
recover the 6-DoF grasping poses represented by our novel representation
parameterized with 2D keypoints with corresponding depth, grasping direction,
grasping width, and angle. Extensive experiments in real scenes demonstrate
that our method can achieve competitive results in grasping common objects and
surpass the depth-based competitor by a large margin in grasping
photometrically challenging objects. To further stimulate robotic manipulation
research, we additionally annotate and open-source a multi-view and multi-
scene real-world grasping dataset, containing 120 objects of mixed photometric
complexity with 20M accurate grasping labels.

##  I Introduction

Humans visually perceive the world with passive RGB views and are able to
perform sophisticated interactions with objects even if the objects are unseen
previously, translucent, reflective, or transparent. Robotic grasping mostly
relies on RGB cameras [1], or active depth sensors such as ToF [2], LiDAR [3],
and active stereo [4], to perform arbitrary object grasping using the
simplified grasping representation in $SE(2)$ or $SE(3)$ with no prior
knowledge such as explicit object models or category information. Existing
learning–based approaches can be categorized in two directions, namely planar
grasping and 6-DoF grasping. Planar grasping [5, 6, 7, 8] relies on a simple
but effective grasping representation, which defines grasps as oriented
bounding boxes. Although such a low DoF grasp representation reduces the task
to a simple detection problem, it limits the performance in 3D manipulation
tasks. 6-DoF grasping enjoys more dexterity than planar grasping, which is
suitable for handling complex scenarios. Most 6-DoF grasping methods [9, 10,
11, 12, 13] extract geometric information from dense input and use it to score
the grasp candidates. Despite the satisfactory grasping performance on common
objects even if the shape is unknown, the reliance on the dense 3D information
makes them vulnerable when the input sensing is unstable, especially on
transparent or highly reflective surfaces [14]. Cross-modality fusion
approaches for perception tasks can be used to combine RGB images with other
sensor inputs [15, 16, 17]. This allows to deal with the aforementioned
limitation and shows promising results [18, 19]. Nevertheless, these methods
still rely on external depth information as indispensable input, which makes
the methods less convienient and more expensive. Motivated by this, we aim to
fully explore the ability of the easily accessible and stable RGB modality on
the 6-DoF robotic grasping task without any depth input.

Figure 1: A Franka robot mounting an RGB camera driven by MonoGraspNet is
performing transparent object grasping of a glas in a household scenario. The
crop on the left shows the custom mount for the RGB camera on the gripper and
the color overlays (green-blue-red) illustrate the grasping process for this
photometrically challenging object.

In this paper, we propose _MonoGraspNet_ , the first deep learning 6-DoF
grasping pipeline that requires only a single RGB image to estimate accurate
grasp poses. MonoGraspNet stacks two parallel networks to detect 2D keypoints
and predict the scene normal map respectively. Then it further adopt a
regression network to estimate keypoint depths, grasping directions, widths,
and angles. The sparse setup further encourages accurate grasping property
estimation on the cropped region of interest, as show in Fig. 2. Compared with
dense per-pixel estimation, our method is thus economical in training and more
lightweight in deployment.

Furthermore, despite existing datasets [20, 21, 22, 23, 24, 25, 18, 19], we
notice an absence of objects of varying photometric challenges and perfect
annotation labels, preventing current methods from being used for arbitrary
object grasping. Most importantly, previous datasets are not applicable to
high-level tasks, like Robot-Environment Interaction, due to their limited
perceptual views and single-scenario configurations. To serve both basic and
advanced manipulation tasks, we provide multi-view and multi-scene annotations
with large-scale grasping labels based on two indoor datasets [26, 14], which
comprise challenging objects of different photometric complexity. Along with
other off-the-shelf dense annotations, we believe the extended dataset can
serve better for diverse grasping purposes.

In summary, our main contributions are the followings:

  * •

We propose the first deep learning pipeline for 6-DoF grasping from a single
RGB image, designing a novel grasp representation.

  * •

We overcome grasping limitations for photometrically challenging objects (see
Fig. 1 for a first impression). Our experiments show that MonoGraspNet can
achieve competitive results in grasping common objects and surpass the depth-
based competitor by a large margin in grasping photometrically challenging
objects.

  * •

We open-source a large-scale grasping dataset comprising objects of varying
photometric complexity. The data includes 20M grasping labels for 120
household objects under multiple views in multiple scenes.

##  II Related Work

The purpose of arbitrary/universal grasping is to extract shared
representations for all objects in different shapes and categories without
prior knowledge. Methods belonging to this field can be roughly divided by the
sensor modality and grasp dimensions they use.

RGB-based 3-DoF Grasping: Previous methods using RGB as the input source are
called planar grasping. They usually set RGB cameras in a top-down view and
treat pose regression as a detection problem [27, 28, 29, 30, 31]. They
formulate grasping points in fixed-height oriented rectangles whose width
defines the grasping width. Thus, robust and developed 2D detection networks
can be used to solve the problem. Oriented rectangles can also be transferred
into other formats. GKNet [32] regards one rectangle as three keypoints, which
can then be regressed by keypoint-related networks such as CenterNet [33] and
CornerNet [34]. They do not introduce depth into the pipeline, so their grasp
representation is limited to 3-DoF. In contrast, our method can recover
grasping poses in $SE(3)$, although we only require RGB input.

Depth-based 3-DoF Grasping: Methods in this direction represented by Dex-Net
2.0 [20] go further by directly deploying 2D-based networks, like GQ-CNN [20],
on depth image processing to perform the grasping task. As the representation
is aligned with planar grasping, they still belong to 3-DoF grasping and share
the same issue with it.

Depth-based 6-DoF Grasping: 6-DoF grasping plays a more critical role in the
grasping community, allowing robots to plan higher dexterous grasps. Two ways
exist in the depth processing: one represented by [35] directly uses a depth
image and leverages both 2D detection and accurate depth to estimate the
rotated angles and gripper depth to recover 6-DoF poses. However, due to the
top-down view and eye-on-hand requirement, some constraints arising from the
pose representation persist. This makes it unsuitable in practice for some
positions. Consistent with planar grasping, it is enough for the bin-picking
task, but the performance is limited for more complex manipulations. The other
way which has surged to solve the problem is based on point cloud processing
represented by a large body of method [13, 36, 10, 9, 37] using 3D backbones
[38, 39, 40]. They aim to achieve so-called “AnyGrasp” performance. The
success rate of these approaches is high when the depth is reliable. Due to
the strong dependency on the depth sensor, however, the performance drops
severely for photometrically challenging objects.

RGB-D based 6-DoF Grasping: Fusing RGB with depth in a completion setup [41]
is used to deal with extreme situations and photometric challenging objects.
ClearGrasp [18] masks out the transparent objects in the scene, and uses
estimated normals and edges to reconstruct missing depth. TransCG [19]
collects depth ground truth using a 3D scanner and IR system, which is used to
supervise the network taking fused features of inaccurate depth and RGB image
as input. The improved performance lays the fact that RGB can alleviate the
depth-missing problem. Our work takes the idea to a further level, which we
explore the ability of RGB on 6-DoF grasping without bringing in the depth
map.

Figure 2: Schematic overview over our MonoGraspNet pipeline. Given a monocular
image, our Keypoint-Network and Normal-Network predict a keypoint heatmap and
a normal map. After 2D keypoint selection and local region cropping, the DWA-
Network regresses the rest grasping instructions. Exemplary for one keypoint,
given the detected keypoint location, we crop the same regions in RGB image
and normal map using an adjustable radius $r$ to obtain a
$2\times(2r+1)\times(2r+1)\times 3$ feature map, which we then reshape to
$(2r+1)\times(2r+1)\times 6$. Besides, we employ RoI Align to aggregate the
features and bring them to a size of $112\times 112\times 6$, same as for the
other crops. The DWA-Network utilizes three branches for regressing depth,
width, and angle associated with the estimated keypoint. Finally, the visible
grasping point (3D Keypoint1) and invisible grasping point (3D Keypoint2) can
be derived. Figure 3: Our grasp representation. $d$ is the depth value of 2D
keypoint $p$. $\phi$ is the dihedral angle of gripper plane (green) and
platform plane (grey). $w$ is the grasping width calculated as
$w=\|P_{1}-P_{2}\|$.

##  III Methodology

In this work, we propose a novel method, which we dub MonoGraspNet, for
estimating the 6-DoF grasp poses from a single monocular image, and a new
dataset with multi-view, multi-scene, and large-scale grasping labels.

###  III-A MonoGraspNet

Given an RGB image, our method estimates a set of 6-DoF grasping
representations, which are then used to recover 6-DoF poses leveraging a
simple conversion. The input image is fed to Keypoint- and Normal-Network to
estimate a set of keypoints represented by a heatmap and a normal map. Those
keypoints are then used to crop RGB and normal maps into joint patches, which
are used as the input to our DWA-Network for estimating depth, width, and
angle (DWA). The system pipeline is illustrated in Figure 2.

Grasping Representation: Since we are the first method to regress 6-DoF
grasping poses from monocular data alone, we first introduce our novel
representation, better suited for our setup. In the core, we combine and
extend the different representations from Contact-GraspNet and L2G to derive
our new formulation (TABLE I). Contact-GraspNet [10] uses the visible grasping
point $P_{1}$, grasping axis $\mathbf{n_{x}}$, approaching axis
$\mathbf{n_{z}}$ and grasping width $w$ to represent the 6-DoF grasp, whereas
L2G [37] relies on two 3D grasping keypoints $P_{1},P_{2}$ together with the
angle between gripper plane and platform plane $\phi$. As recovering sensitive
3D information from 2D input is a difficult ill-posed problem, the pattern we
propose in this work is to leverage 2D features to a large extent.
Specifically, we split $P_{1}$ into a 2d keypoint $p$ aligned with planar
grasping represented by [32] and a further depth value $d$, ending up with
$\\{\mathbf{n_{x}},p,d,w,\phi\\}$ as our final parameterization, as shown in
Fig. 3 (see Sec. III-Bfor 6-DoF pose recovery).

TABLE I: Grasp Representation Comparison with others Methods | Input | #Format
---|---|---
C-GraspNet [10] | Point Cloud | $P_{1},\mathbf{n_{x}},\mathbf{n_{z}},w$
L2G [37] | Point Cloud | $P_{1},P_{2},\phi$
MonoGraspNet (Ours) | RGB | $\mathbf{n_{x}},p,d,w,\phi$

Direction Regression:  Robust regression of the grasping axis
$\mathbf{n_{x}}$, which describes the gripper closure direction, is an
important component in obtaining our grasp pose. It can be calculated by
antipodal sampling through force closure inspection, assuming the availability
of an object mesh. For parallel-jaw grippers, force closure depends on the
friction cone and the direction of the line connecting two grasping points.
Furthermore, the friction cone is determined by the object surface normal
$\mathbf{v}\in\mathbb{R}^{3}$ and the friction coefficient $\mu$, which are
agnostic in the real-world implementation. Nonetheless, the smaller the cosine
distance between $\mathbf{n_{x}}$ and $-\mathbf{v}$, the higher the grasping
success rate, regardless of $\mu$. Thus, we can treat
$\mathbf{n_{x}}=-\mathbf{v^{\ast}}\approx-\mathbf{v}$, with
$\mathbf{v^{\ast}}$ being the estimated surface normal. We employ our normal
estimation network (Normal-Network), shown in Fig. 2, to infer the normal map.
Since the normal consistency around the edges tends to be lower than other
parts, grasping can become very complicated for the robot. To alleviate this
problem, we use detected 2D keypoints to search for the corresponding normal,
as discussed below.

Keypoint Detection:  Successful grasping is highly dependent on accurate
prediction of the underlying 3D geometry. Unfortunately, leveraging a state-
of-the-art depth estimator is not a promising direction due to the bleeding-
out effect around depth discontinuities. It is proved in Sec. IV. Therefore,
we instead try to focus on the utilization of 2D features. In particular, we
propose to detect 2D keypoints $p$ first and subsequently recover the
associated visible 3D keypoints $P_{1}$, as shown in Sec. III-B. Notice that
we formulate keypoint detection as heatmap regression using our Keypoint-
Network, as depicted in Fig. 2. Thanks to the dense and accurate annotations
in our dataset, we observe that keypoint ground truth spread on the smooth
surfaces of objects instead of sharp edges. By well training the network and
using estimated keypoints to query surface normal and additionally inspecting
the normal consistency around these keypoints, we are able to overcome the
aforementioned issue in the normal selection part.

DWA Regression:  In order to obtain the final pose, after having estimated the
normal, we still need to predict depth, width, and angle (DWA). NeWCRFs [42],
a Conditional Random Field (CRF) based method, has shown that (1) the value to
be estimated coming from RGB relies on cues such as colors and pixel
positions. Further, (2) the depth of a pixel is usually not determined by
distant pixels but by pixels that are only within a certain distance of the
target pixel. For a single pixel, too many pairwise connections result in
similar performance but require a lot of redundant computations. Based on (1),
we additionally improve the idea by adding learned surface normal into
consideration:

$\displaystyle
E(\mathbf{q})=\sum_{i}E_{u}\left(q_{i}\right)+\sum_{ij}E_{p}\left(q_{i},q_{j}\right)$
(1) $\displaystyle
E_{p}=f\left(q_{i},q_{j}\right)g\left(I_{i},I_{j}\right)h\left(p_{i},p_{j}\right)k\left(\mathbf{v_{i}},\mathbf{v_{j}}\right),$

where $E(\mathbf{q})$ is the energy function for the pixels in the image,
$q_{i}$ is the feature value of pixel $i$, and $j$ represents the other
pixels. The unary potential function $E_{u}$ uses features of pixel $i$ to
calculate its associated energy. Further, the pairwise potential function
$E_{p}$ computes the energy for pairs of pixels and
$f\left(q_{i},q_{j}\right)$ is the pairwise weight. Finally, $I_{i}$ denotes
the RGB value of pixel $i$. $p_{i}$ is the coordinate of pixel $i$ and
$\mathbf{v_{i}}$ is the normal vector at $p_{i}$.

The motivation is that surface normal can provide strong shape prior of
objects. For extracting features from photometrically challenging objects,
especially for transparent objects, color and position are not enough, as
background and transparent parts can look similar in RGB. However, their
orientations represented by the surface normals are different. Introducing the
normal into the function can make $f\left(q_{i},q_{j}\right)$ reweighted and
optimized. Inspired by (2), we crop local regions centered on each keypoint to
let the feature encoder only focus on the respective regions, as our task does
not care about depth estimation for the whole map. Besides, we further extend
(2) to the grasping width and angle regression. In practice, we achieve this
by concatenating RGB normalized crops and normal crops along the feature
channel. $f\left(q_{i},q_{j}\right)$ can be represented by various existed
feature encoder, such as [43, 44, 45]. Before sending feature crops into the
encoder, we perform RoI Align [6] on each crop, as some of the keypoints may
appear near the boundaries of RGB images, resulting in these crops with
different sizes. Moreover, RoI Align allows flexible feature output
adjustment. After obtaining the high-dimension feature map, we employ fully
connected layers to regress depth $d$, grasping width $w$, and rotated angle
$\phi$ for each 2D keypoint.

Figure 4: Examples from one scene in the proposed dataset. The first picture
shows the dense annotations in the image. The second one demonstrates 6-DoF
grasps in the 3D view. Grasps are downsampled for a better look.

Network Selections: In this work, we leverage the state-of-the-art approach
from [46] for normal prediction and adopt the same loss functions originally
proposed, NLL loss. For the Keypoint-Network, we use the center-point
regression branch in GKNet [32] with DLA [47] as the backbone. We use a
variant of the focal loss as in [34]. Finally, we employ an attention-based
backbone Swin-Transformer Base [45] as the backbone for our DWA-Network, as it
has been proven in several regression works to be capable of acting as a very
strong feature extractor. To be consistent with Swin-Transformer, we set the
input after RoI Align as $112\times 112\times 6$ and utilize a patch size of
$2$. The output feature map is of size $7\times 7\times 1024$, consistent with
the original paper. We use L2 loss for all three branches in our DWA-Network.

###  III-B 6-DoF Pose Recovery

The visible and invisible 3D keypoints $P_{1}$ and $P_{2}$ can be obtained by

$P_{1}=dK^{-1}p,\quad P_{2}=P_{1}-w\mathbf{v}^{\ast},$ (2)

where $K$ is the camera intrinsic matrix. Finally, using $P_{1}$, $P_{2}$, and
$\phi$, we can now calculate the 6-DoF grasp pose. Note that $\phi$ is
collected in the robot base frame, with $P_{1}$ and $P_{2}$ are in the camera
frame. So we first transform $P_{1}$ and $P_{2}$ to robot base and calculate
the center point $P_{c}^{\ast}$ according to

$\displaystyle P_{c}^{\ast}$ $\displaystyle=(P_{1}^{\ast}+P_{2}^{\ast})/2$ (3)
$\displaystyle[P_{1}^{\ast},P_{2}^{\ast}]$
$\displaystyle=\mathbf{T}_{base\leftarrow{cam}}[P_{1},P_{2}].$

We can also obtain each component vector in the rotation matrix for the grasp
by :

$\displaystyle\mathbf{n_{x}}=-\mathbf{a^{*}}$ (4)
$\displaystyle\left\\{\begin{array}[]{l}\mathbf{n_{y}}\cdot\mathbf{n_{x}}=0\\\
\phi=\arccos{(\mathbf{n_{y}}\cdot\mathbf{n})}\\\
\left\|\mathbf{n_{y}}\right\|=1\end{array}\right.$ (5)
$\displaystyle\mathbf{n_{z}}=\mathbf{n_{x}}\times\mathbf{n_{y}}$ (6)

where $\mathbf{n}$ is the normal of the platform plane $[0,0,1]^{\text{T}}$.
From (5), two $\mathbf{n_{y}}$ solutions are obtained by solving a quadratic
equation. We always take the one whose accompanying $\mathbf{n_{z}}$ points to
the platform, since it is safer and easier for robot to reach. Then the grasp
rotation matrix can be represented by
$\mathbf{R}=[\mathbf{n_{x}},\mathbf{n_{y}},\mathbf{n_{z}}]$. The final 6-DoF
pose $\mathbf{G}={\left[{{\mathbf{R}},P_{c}^{\ast};{\mathbf{0}},1}\right]}$ in
the robot base frame.

###  III-C Dataset Collection

To explore RGB-only information in this challenging grasping task, we need to
learn 3D information from a dataset that contains pixel-perfect depth rather
than the measurement coming from a regular depth camera, as it does not work
when facing photometrically challenging objects. Moreover, to avoid creating
any synthetic-to-real domain gap, the data should preferably originate from
the real world.

Given this, we choose [26] and [14] as our base dataset, which contains
various perfect ground truth. Meanwhile, As both datasets were collected by
the robot arm in real-world scenarios, the perception field and workspace are
exactly consistent with the grasping task. By taking advantage of this, we
label grasping poses for each mesh in the dataset by antipodal sampling and
inspect poses with physical simulators, which is a well-developed routine in
[22, 48]. Then we reproject these grasps back into the scene in the robot
frame based on 6-DoF object poses $\mathbf{T}_{{base}\leftarrow obj}$, and
perform collision inspection with background meshes. After these procedures,
we can obtain grasp labels in each camera frame $\mathbf{G}_{cam}$ by recorded
camera poses relative to the robot base $\mathbf{T}_{cam\leftarrow{base}}$:

$\displaystyle\mathbf{G}_{cam}$ $\displaystyle=\mathbf{T}_{cam\leftarrow
obj}\mathbf{G}_{obj}$ (7) $\displaystyle\mathbf{T}_{cam\leftarrow obj}$
$\displaystyle=\mathbf{T}_{cam\leftarrow{base}}\mathbf{T}_{{base}\leftarrow
obj},$

An example is shown in Fig. 4. This extended dataset will be available with
original PhoCaL and HAMMER for the community to research.

As described in Sec. III-A, our pipeline calculates grasps by using visible 2D
features from which we extract invisible information. To serve this purpose
better, We further search for the nearest 3D points with respect to the
grasping points in the point cloud, which can be obtained from depth ground
truth with intrinsic camera parameters, and then reproject these points to the
image plane to obtain corresponding pixels. These pixels are treated as the
ground truth of visible 2D keypoints. Moreover, we record the correspondence
between each keypoint and the corresponding grasping point pair to attain the
ground truth of grasping width and grasping angle for each grasp.

##  IV Experimental Evaluation

In this section, we first provide our utilized implementation details. Then we
introduce our experimental setup and present our results compared with the
depth-based state-of-the-art grasping method Contact-GraspNet in various real-
world experiments, defined as the ultimate test of the grasp performance in
[11].

###  IV-A Implementation Details

Training Settings: All the experiments are conducted on a single NVIDIA RTX
3090 GPU with Adam optimizer. The Keypoint-Network is trained for 30 epochs
with the initial learning rate of 1e-4 and batch size of $2$. For Normal-
Network training, we follow the instructions in the original paper [46]. The
proposed DWA-Network is trained for 30 epochs with the initial learning rate
to be 1.25e-4 and batch size to be $32$. We decrease the learning rate by $10$
times at the $15$th and the $20$th epoch. Our input image size is $832\times
1088$.

Robot Hardware: We use a 7-DoF Franka Panda robot with a parallel-jaw gripper
as the end-effector. The RGB camera mounted on the gripper base is a Phoenix
5.0 MP Polarization camera, the same as the one used in [26, 14] shown in Fig.
1. For testing Contact-GraspNet, we install a Azure Kinect depth camera beside
the robot. All cameras are hand-eye calibrated.

TABLE II: Success rate (%) of familiar single object grasping experiments in (Un-)Seen scenes 2 Method | Avg. | (1) Cock-glass | (2) Mug | (3) Liquid | (4) Spoon | (5) Monster | (6) Biscuit |
---|---|---|---|---|---|---|---|---
S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S |
[42]+C-GraspNet [10] | 23.3 | 10.0 | 0 | 0 | 0 | 13.3 | 33.3 | 26.7 | 6.7 | 0 | 26.7 | 0 | 73.3 | 20.0 |
C-GraspNet [10] | 62.2 | 60.0 | 6.7 | 0 | 93.3 | 93.3 | 60.0 | 66.7 | 20.0 | 6.7 | 93.3 | 93.3 | 100 | 100 |
MonoGraspNet (Ours) | 83.9 | 72.2 | 80.0 | 60.0 | 86.7 | 86.7 | 60.0 | 53.3 | 86.7 | 80.0 | 86.7 | 73.3 | 93.3 | 80.0 |
2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
TABLE III: Success rate (%) of unfamiliar single object grasping experiments in (Un-)Seen scenes 2 Method | Avg. | (8) Narrow glass | (9) Red kettle | (10) Trans-kettle | (11) Glass | (12) Fanta | (13) Gel
---|---|---|---|---|---|---|---
S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S | S | Un-S
[42]+C-GraspNet [10] | 26.2 | 11.9 | 0 | 0 | 30.7 | 13.3 | 0 | 0 | 13.3 | 6.7 | 40.0 | 20.0 | 73.3 | 40.0
C-GraspNet [10] | 42.2 | 42.2 | 0 | 0 | 86.7 | 80.0 | 6.7 | 0 | 0 | 0 | 93.3 | 100 | 66.7 | 73.3
MonoGraspNet (Ours) | 75.5 | 72.2 | 60.0 | 40.0 | 80.0 | 86.7 | 60.0 | 53.3 | 80.0 | 80.0 | 86.7 | 93.3 | 80.0 | 80.0
2 |  |  |  |  |  |  |  |  |  |  |  |  |  |
TABLE IV: Completion rate of clutter removal experiments 2 Type | C-GraspNet [10] | MonoGraspNet (Ours)
---|---|---
1st trial | 2nd trial | 1st trial | 2nd trial
4-object scene | 3/4 | 3/4 | 3/4 | 3/4
5-object scene | 4/5 | 4/5 | 4/5 | 5/5
2 |  |  |  |
Figure 5: a) Objects used for testing. 1-6 are familiar objects (blue block).
8-13 are unfamiliar objects (red block). Objects in white block are used for
clutter removal. b) Recovered point cloud from monocular depth estimation is
out of the shape, resulting in the biscuit box vanished from the scene.

###  IV-B Experimental Details

Evaluation Protocols: We select 14 objects shown in Fig. 5.a) to compare our
method with Contact-GraspNet and report the success rate and completion rate.
Success rate is used for testing single object grasping in (un-)seen
scenarios. We let the robot execute 15 grasps by putting the objects at three
random positions in the scene, and then we calculate successful grasps.
Completion rate is the percentage of objects removed from the clutter, which
can be a robust metric when testing multiple object grasping. We execute the
grasping task in four scenarios with different configurations sorted by
ascending difficulty: 1) familiar objects in seen scenes, 2) familiar objects
in unseen scenes, 3) unfamiliar objects in seen scenes, and 4) unfamiliar
objects in unseen scenes. Additionally, we set an experiment by combining
Contact-GraspNet and NeWCRFs [42] to support our claim in Sec. III-A-Keypoint
Detection, which is that the path of estimating dense depth map first and then
grasping objects using the recovered point cloud results in bad performance.

Running Pipeline: For the motion planning in the experiment, we adopt the way
of directly controlling the joints, as no 3D information is input. To avoid
the influence that different algorithm may bring, which is also out of scope
in this research, the same planning method is used for Contact-GraspNet. Since
there is no segmentation module inside MonoGraspNet, we also remove the
relevant module [49] accompaning with Contact-GraspNet, which is allowed by
the instruction from [10].

Results TABLE II and TABLE III show the comparison results under the four
situations. Familiar objects: trained objects, but in untrained views.
Unfamiliar objects: untrained objects. S: scenes that were seen in the
training set but in a novel view. Un-S: unseen scenes. As shown in Row 1,
directly feeding the point cloud from [42] into Contact-GraspNet yields the
worst results, with only $23.3\%$ success rate for familiar objects in seen
scenes, as the recovered point cloud is distorted in a unstructured shape. An
example is shown in Fig. 5.b). In TABLE II, it can be seen that Contact-
GraspNet is slightly superior to MonoGraspNet when grasping normal objects,
i.e. objects (2, 5, 6), with an average of $6.6\%$ leap forward. This
seemingly discouraging outcome may be expected, since depth from sensors is
typically more accurate than predicted depth from RGB images. However, when
dealing with photometrically challenging objects (1, 3, 4), MonoGraspNet
surpasses Contact-GraspNet to a large margin, with an average success rate of
$78.9\%$ against $28.9\%$. In this situation, inaccurate depth limits the
performance of grasp pose estimation of Contact-GraspNet. We also verified the
generalization ability of MonoGraspNet. By comparing the results in unseen
scenes with the ones in seen scenes, we can see that the success rate remains
stable and consistent, with only maximum $11.7\%$ decrease in average. It
proves that MonoGraspNet can generalize to other real-world scenarios without
pre-training or fine-tuning. For unfamiliar object grasping in TABLE III, even
though the textures or shapes of these objects are distinct from the trained
ones in the dataset, MonoGraspNet can still yield adequate success rate, as it
uses local 2D features to infer grasps, which remains stable across different
objects and different scenes.

We also perform a more challenging clutter removal task. As we do not have
collision check module, we always let the robot execute the nearest grasp. We
random choose four or five objects with one challenging object inside from the
white block of Fig. 5. Compared to Contact-GraspNet, we still show a
competitive performance when moving simple clutter with photometrically
challenging objects inside.

TABLE V: Success rate (%) comparison with different depth estimation 2 Method | Normal | Challenging
---|---|---
Fanta | Biscuit | Spoon | Glass
DWA-Net w/o normal | 80.0 | 93.3 | 86.7 | 66.7
DWA-Net | 86.7 | 93.3 | 86.7 | 80.0
2 |  |  |  |

###  IV-C Ablation Study

To prove the motivation that introducing the normal map can help depth
estimation in DWA-Network. We re-train a DWA-Network with RGB crops as the
only input. TABLE V shows that the success rate performance of grasping normal
objects and even the reflective object (Spoon as an example here) are similar,
but the improvement is obvious for grasping transparent objects (unfamiliar
Glass).

##  V Conclusions

In contrast to previous conventional routines which rely on accurate depth,
this paper proposes the first RGB-only 6-DoF grasping pipeline called
MonoGraspNet. It utilizes stable 2D features to simultaneously handle
arbitrary object grasping and overcome the problems induced by photometrically
challenging objects. Our method leverages keypoint heatmap and normal map to
recover the 6-DoF grasping poses represented by our proposed novel
representation. The experiments demonstrate that our method can achieve
competitive results on grasping common objects with simple textures while
surpassing the depth-based competitor significantly on grasping
photometrically challenging objects. We additionally annotate a large-scale
real-world grasping dataset, containing 120 objects of mixed photometric
complexity with 20M accurate grasping labels.

## References

  * [1] D. Guo, T. Kong, F. Sun, and H. Liu, “Object discovery and grasp detection with a shared convolutional neural network,” in _ICRA_ , 2016.
  * [2] A. Maldonado, U. Klank, and M. Beetz, “Robotic grasping of unmodeled objects using time-of-flight range data and finger torque information,” in _IROS_ , 2010.
  * [3] K. M. Popek, M. S. Johannes, K. C. Wolfe, R. A. Hegeman, J. M. Hatch, J. L. Moore, K. D. Katyal, B. Y. Yeh, and R. J. Bamberger, “Autonomous grasping robotic aerial system for perching (agrasp),” in _IROS_ , 2018.
  * [4] R. B. Rusu, A. Holzbach, R. Diankov, G. Bradski, and M. Beetz, “Perception for mobile manipulation and grasping using active stereo,” in _Humanoids_ , 2009\.
  * [5] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _NeurIPS_ , 2015.
  * [6] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _ICCV_ , 2017.
  * [7] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _CVPR_ , 2018.
  * [8] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _CVPR_ , 2016.
  * [9] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang, F. Sun, and J. Zhang, “Pointnetgpd: Detecting grasp configurations from point sets,” in _ICRA_ , 2019.
  * [10] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox, “Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes,” in _ICRA_ , 2021.
  * [11] A. Mousavian, C. Eppner, and D. Fox, “6-dof graspnet: Variational grasp generation for object manipulation,” in _ICCV_ , 2019.
  * [12] C. Wu, J. Chen, Q. Cao, J. Zhang, Y. Tai, L. Sun, and K. Jia, “Grasp proposal networks: an end-to-end solution for visual learning of robotic grasps,” in _NeurIPS_ , 2020.
  * [13] H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large-scale benchmark for general object grasping,” in _CVPR_ , 2020.
  * [14] H. Jung, P. Ruhkamp, G. Zhai, N. Brasch, Y. Li, Y. Verdie, J. Song, Y. Zhou, A. Armagan, S. Ilic, A. Leonardis, and B. Busam, “Is my depth ground-truth good enough? hammer – highly accurate multi-modal dataset for dense 3d scene regression,” 2022. [Online]. Available: https://arxiv.org/abs/2205.04565
  * [15] A. Lopez-Rodriguez, B. Busam, and K. Mikolajczyk, “Project to adapt: Domain adaptation for depth completion from noisy and sparse sensor data,” in _ACCV_ , 2020.
  * [16] S. Gasperini, P. Koch, V. Dallabetta, N. Navab, B. Busam, and F. Tombari, “R4dyn: Exploring radar for self-supervised monocular depth estimation of dynamic scenes,” in _3DV_ , 2021.
  * [17] Y. Verdié, J. Song, B. Mas, B. Busam, A. Leonardis, and S. McDonagh, “Cromo: Cross-modal learning for monocular depth estimation,” in _CVPR_ , 2022.
  * [18] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, J. Lee, A. Zeng, and S. Song, “Clear grasp: 3d shape estimation of transparent objects for manipulation,” in _ICRA_ , 2020.
  * [19] H. Fang, H.-S. Fang, S. Xu, and C. Lu, “Transcg: A large-scale real-world dataset for transparent object depth completion and a grasping baseline,” _RA-L_ , vol. 7, no. 3, 2022.
  * [20] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” in _RSS_ , 2017.
  * [21] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _IJRR_ , vol. 37, no. 4-5, 2018.
  * [22] C. Eppner, A. Mousavian, and D. Fox, “Acronym: A large-scale grasp dataset based on simulation,” in _ICRA_ , 2021.
  * [23] A. Depierre, E. Dellandréa, and L. Chen, “Jacquard: A large scale dataset for robotic grasp detection,” in _IROS_ , 2018.
  * [24] (2017) Cornell grasping dataset. [Online]. Available: http://pr.cs.cornell.edu/grasping/rect˙data/data.php
  * [25] G. Zhai, Y. Zheng, Z. Xu, X. Kong, Y. Liu, B. Busam, Y. Ren, N. Navab, and Z. Zhang, “Da2 dataset: Toward dexterity-aware dual-arm grasping,” _RA-L_ , vol. 7, no. 4, 2022.
  * [26] P. Wang, H. Jung, Y. Li, S. Shen, R. P. Srikanth, L. Garattoni, S. Meier, N. Navab, and B. Busam, “Phocal: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects,” in _CVPR_ , 2022.
  * [27] D. Guo, F. Sun, H. Liu, T. Kong, B. Fang, and N. Xi, “A hybrid deep architecture for robotic grasp detection,” in _ICRA_ , 2017.
  * [28] S. Kumra and C. Kanan, “Robotic grasp detection using deep convolutional neural networks,” in _IROS_ , 2017.
  * [29] U. Asif, J. Tang, and S. Harrer, “Graspnet: An efficient convolutional neural network for real-time grasp detection for low-powered devices.” in _IJCAI_ , 2018.
  * [30] J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” in _ICRA_ , 2015.
  * [31] X. Zhou, X. Lan, H. Zhang, Z. Tian, Y. Zhang, and N. Zheng, “Fully convolutional grasp detection network with oriented anchor box,” in _IROS_ , 2018.
  * [32] R. Xu, F.-J. Chu, and P. A. Vela, “Gknet: grasp keypoint network for grasp candidates detection,” _IJRR_ , 2022.
  * [33] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _ICCV_ , 2019.
  * [34] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _ECCV_ , 2018.
  * [35] X. Zhu, L. Sun, Y. Fan, and M. Tomizuka, “6-dof contrastive grasp proposal network,” in _ICRA_ , 2021.
  * [36] M. Breyer, J. J. Chung, L. Ott, S. Roland, and N. Juan, “Volumetric grasping network: Real-time 6 dof grasp detection in clutter,” in _CoRL_ , 2020.
  * [37] A. Alliegro, M. Rudorfer, F. Frattin, A. Leonardis, and T. Tommasi, “End-to-end learning to grasp via sampling from object point clouds,” _RA-L_ , vol. 7, no. 4, 2022.
  * [38] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _CVPR_ , 2017.
  * [39] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _NeurIPS_ , 2017.
  * [40] A. Alliegro, D. Valsesia, G. Fracastoro, E. Magli, and T. Tommasi, “Denoise and contrast for category agnostic shape completion,” in _CVPR_ , 2021.
  * [41] H. Jung, N. Brasch, A. Leonardis, N. Navab, and B. Busam, “Wild tofu: Improving range and quality of indirect time-of-flight depth with rgb fusion in challenging environments,” in _3DV_ , 2021.
  * [42] W. Yuan, X. Gu, Z. Dai, S. Zhu, and P. Tan, “Newcrfs: Neural window fully-connected crfs for monocular depth estimation,” in _CVPR_ , 2022.
  * [43] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [44] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2020.
  * [45] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021.
  * [46] G. Bae, I. Budvytis, and R. Cipolla, “Estimating and exploiting the aleatoric uncertainty in surface normal estimation,” in _ICCV_ , 2021.
  * [47] F. Yu, D. Wang, E. Shelhamer, and T. Darrell, “Deep layer aggregation,” in _CVPR_ , 2018.
  * [48] C. Eppner, A. Mousavian, and D. Fox, “A billion ways to grasps - an evaluation of grasp sampling schemes on a dense, physics-based grasp data set,” in _ISRR_ , 2019.
  * [49] Y. Xiang, C. Xie, A. Mousavian, and D. Fox, “Learning rgb-d feature embeddings for unseen object instance segmentation,” in _CoRL_ , 2021.