# Double-Dot Network for Antipodal Grasp Detection

Yao Wang, Yangtao Zheng, Boyang Gao and Di Huang∗ Yao Wang, Yangtao Zheng and
Di Huang are with the State Key Laboratory of Software Development
Environment, School of Computer Science and Engineering, Beihang University,
Beijing 100191, China; Boyang Gao is with the Geometry Robotics.*Corresponding
author<EMAIL_ADDRESS>
###### Abstract

This paper proposes a new deep learning approach to antipodal grasp detection,
named Double-Dot Network (DD-Net). It follows the recent anchor-free object
detection framework, which does not depend on empirically pre-set anchors and
thus allows more generalized and flexible prediction on unseen objects.
Specifically, unlike the widely used 5-dimensional rectangle, the gripper
configuration is defined as a pair of fingertips. An effective CNN
architecture is introduced to localize such fingertips, and with the help of
auxiliary centers for refinement, it accurately and robustly infers grasp
candidates. Additionally, we design a specialized loss function to measure the
quality of grasps, and in contrast to the IoU scores of bounding boxes adopted
in object detection, it is more consistent to the grasp detection task. Both
the simulation and robotic experiments are executed and state of the art
accuracies are achieved, showing that DD-Net is superior to the counterparts
in handling unseen objects.

##  I Introduction

Grasping, an essential and important operation for robotic manipulation, aims
to grab and lift objects by grippers using inputs of imaging sensors. For the
potential to significantly reduce labor costs, it has been widely attempted
and adopted in various applications, such as manufacturing, logistics, and
services, in recent years. Unfortunately, grasping is a much more difficult
and complicated task to robots than to humans, and it still remains challenge
to handle unseen instances of arbitrary poses with diverse textures and
shapes.

Robotic grasping has been extensively discussed for years and typically
addressed in the 3D space. Early approaches [1] [2] [3] [4] regard it as a
synthesis problem where the input is fitted to the templates (_e.g._ CAD
models) whose grasps are assumed available. They only work on known objects,
thus impeding its spread in real-world scenarios. To generalize to unknown
objects, some approaches employ their 3D point-clouds reconstructed from depth
data and estimate grasps by analyzing shape characteristics based on hand-
crafted or deep learned geometry features [5] [6] [7] [8] [9]. They deliver
promising results, but are usually criticized for low running speed. There
also exist several studies that consider grasping as a motion planing process
and model it by Reinforcement Learning (RL) [10] [11] [12]. While attractive,
they generally suffer from long and unstable training phases.

|  |
---|---|---
|  |
|  |
(a) [13] | (b) Ours | (c) Ground Truth
Figure 1: Visualization of predictions and ground-truths. (a) and (b) show the
grasp results achieved by [13] in bounding boxes and the proposed method in
double-dots, respectively, and (c) shows the corresponding ground-truths.

Recently, due to the great demand in applications, robotic grasping is
simplified to the 2D domain, _i.e._ views and lifts are perpendicular to
support surfaces. In this way, the searching space of grasps becomes smaller
and the acquirement of successful grasps becomes easier. Compared to the
investigations that adapt the RL idea from 3D to 2D [14] [15], much more
efforts are made to the detection framework, where the successful grasp is
treated as a special kind of targets within the given image, and it samples
grasps and determines their validity. The sampling efficiency and
classification accuracy of the preliminary detection methods are limited by
the hand-designed features [16] [17] [18] [19]. Such drawbacks are overcome in
the descendant alternatives through deep CNN features and end-to-end learning
mechanisms. For example, [20] [21] represent a planar grasp as a 5-dimensional
gripper configuration, formed by an oriented rectangle, and apply the advanced
object detection pipeline that first generate the anchors, then simultaneously
regress and evaluate the grasp candidates, and finally select the best grasp
configuration.

In fact, object detection and grasp detection are not exactly the same, and
direct extension from the former to the latter does not well address their
differences, thereby leaving room for improvement. As in [17] [22] [23], they
modify the gripper configuration in terms of dimensionality or pattern so that
it better accommodates to grasp detection in different data modalities (_i.e._
RGB images or depth maps). However, the gap is not eliminated. As we know,
object detection methods [24] [25] [26] commonly exploit Intersection over
Union (IoU) to evaluate the quality of the returned rectangle (bigger IoU is
better), but this measurement is too coarse for grasp detection. Even though a
grasp rectangle has a good IoU, a small error in rotation or translation may
force the gripper to collide with the object, incurring execution failure, as
in Fig. 1. More importantly, current state of the art detectors are anchor-
based and their accuracies are sensitive to pre-defined anchor scales and
aspect ratios, which makes them problematic to generalize to unseen objects
with large shape differences.

In this paper, we propose a new deep learning approach, namely Double-Dot
Network (DD-Net), to antipodal robotic grasping, which dedicates to alleviate
the gap between object detection and grasp detection. It follows the anchor-
free concept of CornerNet [27] and CenterNet [28] that detects an object as
very few keypoints, which does not require empirically pre-set anchors, thus
allowing more generalized and flexible prediction on unseen objects.
Specifically, in contrast to current detection based grasping methods
delivering bounding boxes (or other shapes, _e.g._ circles [23]), DD-Net
generates the gripper configuration as a pair of fingertips. An effective CNN
architecture is introduced to localize such fingertips in a one-stage manner,
and with the help of auxiliary centers for refinement, DD-Net accurately and
robustly infers grasp candidates. In addition, a loss function is designed to
evaluate grasps, which measures the quality of the fingertips rather than the
IoU scores of the bounding boxes, making the optimization consistent to the
task. Simulation experiments are executed on the major public grasping
platform, Jacquard, reporting the state of the art SGT accuracy and very
competitive results are also achieved on robotic experiments compared to the
counterparts in terms of accuracy and efficiency.

##  II Related Work

In the literature, planar grasping is generally formulated in two ways. One is
motion planning, mainly handled by RL. Breyer _et al._ [14] model this problem
as a Markov Decision Process, where a neural network is designed to predict
the actions and finger joint positions and Trust Region Policy Optimization is
used to update policy parameters.

Another is detection, where candidate sampling and graspability scoring are
conducted in series. To generate grasp proposals, Le _et al._ [29] represent
contact points by a set of hand-crafted features, including depth and pixel
intensity variations, gradient angles, distances, _etc._. They further
evaluate those candidates by an SVM classifier. Mahler  _et al._ [17] select
the grasp by iteratively sampling a set of candidates and re-fitting the grasp
distribution according to the grasp quality given by a pre-trained deep neural
network (_i.e._ GQ-CNN) based scorer. This method shows promising results, but
suffers from a low running speed. Gariépy _et al._ [30] train an STN (Spatial
Transformer Network) to accelerate sampling; however, STN is supervised by GQ-
CNN, which is not accurate enough, thus limiting the performance.

Figure 2: Illustration of double-dot grasps. Dots marked in red are fingertips
of a parallel-jaw gripper.

More recent studies follow the object detection pipeline. [31] [13] [32] [33]
adapt the two-stage detectors, such as Faster-RCNN [24] and RRPN [34] to grasp
detection. Considering the difference between the two tasks, the major issue
is to predict the orientation of the bounding box. [31] [32] discretize the
angle and convert its prediction to a classification problem, while [13] [33]
employ anchor boxes with different orientations. Those solutions indeed take a
step to grasp detection, but due to the inheritance of IoU based optimization
and anchor based mechanism in object detection, there exists much space for
improvement. [13] [35] replace IoU minimization by point metric or grasp path;
unfortunately, the methods are still too rough. [36] [37] attempt to generate
better grasp configuration from pixel-wise prediction. While attractive, their
models are very preliminary.

##  III Problem Formulation

In this work, we target the task of planar grasping, which localizes two
fingertips of the parallel-jaw gripper equipped on the robotic arm in the
image coordinates where it has the maximum possibility to successfully lift
the object laying on the plane.

Formally, a grasp is defined as double-dot representation:

$\bm{g}=\left\\{\bm{c}_{1}=(x_{1},y_{1}),\bm{c}_{2}=(x_{2},y_{2})\right\\}\,,$
(1)

where $\bm{c}_{i}$ denotes the center of a fingertip of a parallel-jaw
gripper. Fig. 2 illustrates the fingertips relative to the object.

Figure 3: Architecture of DD-Net. It consists of three major parts: an
Hourglass network, a fingertip prediction branch and a center prediction
branch. Orientation matching and center matching only work for inference.

##  IV Method

###  IV-A Grasp Detection Revisit

Recently, a number of object detection methods have been adapted to grasp
detection, but as displayed in Fig. 1, correct detection does not always
guarantee a success of grasp. This gap between object detection and grasp
detection is mainly caused by the definition of grasp representation.

As we know, in the majority of the existing public datasets, to match the
concept of the anchor box in object detection, a grasps is annotated as a
5-dimensional rectangle:

$\bm{g}^{\prime}=\left\\{x,y,h,w,\theta\right\\}\,,$ (2)

where $(x,y)$ is the center of the rectangle; $w$ and $h$ represent the width
and the height of the rectangle, _i.e._ , the opening size and the jaw size of
the end effector; and $\theta$ denotes the orientation of the rectangle with
regard to the horizontal axis. As the jaw size is fixed, $h$ can be ignored in
practice.

The grasp quality is decided by the Intersection over Union (IoU) between the
detected rectangle and the corresponding Ground-Truth (GT), where a bigger IoU
indicates a better grasp. However, such a measurement is too coarse in grasp
detection, as a small error of the grasp rectangle in terms of rotation or
translation tends to make the gripper touch the object, leading to execution
failure, although it still holds a good IoU. On the other side, anchor-based
detectors currently dominate the community, but their accuracies are sensitive
to the anchor setting, _i.e._ scales as well as aspect ratios. This mechanism
is problematic in grasp detection as it is expected to handle unseen objects
even with big differences in shape.

|
---|---
(a) Double-dot labels | (b) Orientation labels
Figure 4: Samples: (a) to increase double-dot labels, pixels around GT
fingertips are annotated according to an unnormalized 2D Gaussian distribution
and (b) the orientation is defined as the angle towards the center point.

The analysis above motivates us to re-define the grasp by double-dot
representation as in Sec. III and develop anchor-free grasp detectors.

###  IV-B Data Labelling

To make full use of existing datasets, where the labels are provided as the
common rectangle representation, for model building, we convert rectangle
representations to double-dots representation as in Sec. III.

Considering that rectangle labels are sparse, we assume that the locations
nearer to the GT fingertips converted from the bounding boxes are more likely
to be positive samples, which increases double-dot labels. In this case, we
label the pixel $\bm{p}$ around GT fingertips according to an unnormalized 2D
Gaussian distribution, which is defined as:

$s(\bm{p})=\min\left(1,\sum_{g\in\bm{G}}\sum_{i=1,2}\frac{\exp(-QR(\theta)\Sigma^{-1}R^{T}(\theta)Q)}{\sigma^{(g)}_{x}\sigma^{(g)}_{y}}\right)\,,$
(3)

where $Q=\bm{p}-\bm{c}^{(g)}_{i}$,
$\Sigma=diag(\sigma^{(g)}_{x},\sigma^{(g)}_{y})^{2}$, $R(\theta)$ is the
rotation matrix and $\bm{G}$ is the ensemble of grasp annotations within an
image. In this study, $\sigma^{(g)}_{x}$ and $\sigma^{(g)}_{y}$ are
empirically set as 1 and 0.75$h$ respectively so that fingertips are not on
objects.

The orientation of each fingertip $\theta_{\bm{c}_{i}}$ is defined as the
angle towards the center point:

$\theta_{\bm{c}_{i}}=\arctan\frac{(y_{i}-y_{i}^{center})}{(x_{i}-x_{i}^{center})}\,.$
(4)

The illustration of the fingertips and corresponding orientations are shown in
Fig. 4.

###  IV-C Network Architecture

Inspired by the latest advances in general object detection, an object can be
detected as very few (two or three) fiducial points [27] [28], we propose a
novel single-stage approach to grasp detection, namely Double-Dot Network (DD-
Net). Instead of predicting the traditional 5-dim grasp rectangle, we directly
detect two fingertips as a grasp. Meanwhile, as the grasp is more stable when
it is nearer to the center of the object, we also localize the gripper center
as an auxiliary point for refinement.

Specifically, DD-Net includes three key components: image encoding, fingertip
prediction and center prediction, as shown in Fig. 3. We adopt an off-the-
shelf convolutional neural network, _i.e._ Hourglass [38], to extract image
features, conveying local and global clues for comprehensive description.
These features then enter into two branches: one for fingertip prediction and
another for center prediction. In the fingertip branch, we compute the score,
offset and orientation, while in the center branch, we only calculate the
score and offset. The focal loss $L_{det}$ is used to train the network to
score for each pixel to be a fingertip or a center point, defined as:

$L_{det}=-\sum\limits_{h,w}\begin{cases}(1-q_{\bm{p}})^{\alpha}\log(q_{\bm{p}}),&\text{if
$s_{\bm{p}}=1$}\\\\[7.74997pt]
(1-s_{\bm{p}})^{\beta}(q_{\bm{p}})^{\alpha}\log(1-q_{\bm{p}}),&\text{otherwise}\end{cases}\,,$
(5)

where $q_{\bm{p}}$ and $s_{\bm{p}}$ are the predicted score and the GT label
at pixel $\bm{p}$, respectively. $\alpha$ and $\beta$ are the hyper-parameters
that weight the contribution of each pixel.

To remap the locations from heatmaps to the input image, an offset is
predicted. The offset is defined as
$\bm{o}_{g}=(\frac{x_{g}}{n}-\lfloor\frac{x_{g}}{n}\rfloor,\frac{y_{g}}{n}-\lfloor\frac{y_{g}}{n}\rfloor)$
where $n$ is the down-sampling factor of the network. The offset loss is
defined as

$L_{off}=\sum_{g\in\bm{G}}SmoothL1\left(\bm{o}_{g},\bm{\hat{o}}_{g}\right)\,,$
(6)

To predict the orientation of the plate, we apply the smooth L1 Loss to
regress the $\sin$ and $\cos$ value of the angle.

$L_{ori}^{sin}=\sum_{\bm{g}\in\bm{G}}\sum_{i=1,2}SmoothL1\left(\eta,\sin(\theta_{\bm{c}_{i}^{(g)}})\right)\,,$
(7)

where $\eta$ is the $\sin$ value predicted by the network. $L_{ori}^{cos}$ can
be similarly defined.

We define our final loss function as follows:

$\displaystyle
L=L_{det}^{con}+L_{det}^{cen}+L_{off}^{con}+L_{off}^{cen}+L_{ori}^{cos}+L_{ori}^{sin}$
(8)

where $L_{det}^{con}$ and $L_{det}^{cen}$ denote the focal losses, which are
used to train the network to detect fingertips and center points respectively.
$L_{off}^{con}$ and $L_{off}^{cen}$ are used to train the network to predict
their offsets. $L_{ori}^{cos}$ and $L_{ori}^{sin}$ are used to train the
network to predict the plate orientation of each fingertip.

###  IV-D Fingertip Grouping

During inference (see Algorithm. 1), we first sample the fingertip candidates
from the heatmaps generated by the network and then group them for refinement.
Two grouping principles, _i.e._ , orientation matching and center matching,
are designed to filter the candidates, illustrated in Fig. 5.

Orientation matching groups fingertips with the prediction of the plate
orientation and the non-antipodal pairs are rejected. A pair of fingertips is
filtered, if one of the predicted orientation of a fingertip deviates that of
the fingertip pair over a threshold. Center matching defines a central region
as a circle centered at the midpoint of the two fingertips, similar to
CenterNet [28]. If the central region of a fingertip pair does not contain any
center point, the fingertip pair is rejected. In our experiments, the radius
of a central region is set as $\frac{1}{3}$ of the distance between
fingertips.

After filtering the candidates, the score of the remaining grasp is defined as
the sum of the ones of the three points, _i.e._ two fingertips and its center.
Finally, we select the grasp with the highest score as the final prediction to
execute.

Algorithm 1 Inference process

Input: input image $I$
Output: grasp prediction ${\bm{g}^{*}}$

1:Generate heatmap, offset and orientation of fingertips by DD-Net.

2:Generate heatmap and offset of centers by DD-Net.

3:Select top-$k$ fingertip and center candidates on heatmaps.

4:Adjust their positions with the predicted offsets.

5:Group $k$ fingertips as grasp candidates and filter them using the
orientation and center matching strategies.

6:Compute the grasp score by the sum of the three points. Rank and select the
grasp with the highest score as final prediction ${\bm{g}^{*}}$.

7:return ${\bm{g}^{*}}$

|
---|---
(a) Orientation Matching | (b) Center Matching
Figure 5: Two principles to filter candidates: (a) orientation matching
accepts the paired fingertips if the angle is below a threshold and (b) center
matching accepts the ones if the central region contains centers (green
points).

##  V Experiments

To evaluate the proposed method, we carry out both simulation and robotic
experiments. The databases, metrics, protocols, implementation details, and
results are described in the subsequent.

###  V-A Datasets and Metrics

The simulation experiments are conducted on the Jacquard and Cornell
benchmarks and the robotic ones are on a self-built database.

The Jacquard dataset contains 54k images of 11k objects with more than one
million unique grasp locations labeled by simulation trials. It supports the
evaluation by Simulated Grasp Trails (SGT). Depierre _et al._ [39] release a
web interface where the scene index and the corresponding grasp prediction can
be uploaded. The scene is rebuilt in the simulation environment and the grasp
is performed by the simulated robot. A grasp is considered successful if the
object is correctly lifted, moved away and dropped at a given location.

The Cornell Grasp dataset is one of the most common datasets used in grasp
detection. It includes 885 images, and each image is labeled by a few
graspable rectangles. The evaluation is based on the rectangle metric with the
standard threshold values of $30^{\circ}$ for the angle and 25% for IoU.

The self-built dataset consists of 43 objects with different textures, shapes,
materials, and scales, as shown in Fig. 6. Success rate is used as the
evaluation metric.

|
---|---
(a) hardware set-up | (b) objects
Figure 6: Device and dataset of robotic experiments

###  V-B Protocols

Both the Jacquard and Cornell datasets are divided into a training set and a
testing set with a ratio of 4:1. There are two ways to split the dataset:
image-wise split and object-wise split. Image-wise split means the training
and testing sets are divided randomly, while object-wise split means objects
in the test set do not appear in training. For fair comparison, as in previous
work, we split the Jacquard dataset by object-wise and the Cornell Grasp
dataset both by image-wise and by object-wise.

To study the generalization ability on unknown objects in the real-world case,
we directly deploy DD-Net (trained on Jacquard without fine-tuning) to an UR5
collaborative robot with a Backyard E140F gripper and a fixed Intel RealSense
D435 camera. We coarsely estimate the height by averaging the values around
the fingertip position predicted in the depth channel. All the experiments are
launched on a desktop with an AMD Ryzen 5 2600 Six-Core Processor and a
graphics card of NVIDIA GeForce 1080. Each object is placed in three different
stable poses on a table. A grasp is considered successful if the robot lifts
the object and drops it to the appointed area.

TABLE I: Comparison with state of the arts in terms of SGT and rectangle metric by object-wise on the Jacquard database. Algorithm | Accuracy (%)
---|---
SGT | Rectangle
Depierre _et al._ [39] | 72.42 | 74.2
Zhou _et al._[13] | 81.95 | 92.8
Zhang _et al._[40] | - | 93.6
Depierre _et al._[41] | 85.74 | -
Kumra _et al._ [42] | - | 94.6
DD-Net | 89.41 | 97.0
TABLE II: Comparison in terms of rectangle metric by image-wise (IW) and object-wise (OW) on the Cornell Grasp database. Algorithm | Input | Accuracy (%)
---|---|---
IW | OW
SAE, struct. reg. Two stage [21] | RGB-D | 73.9 | 75.6
MultiGrasp [43] | RGB-D | 88.0 | 87.1
Multi-model [44] | RGB | 88.8 | 87.7
ZF-net [31] | RGB-D | 93.2 | 89.1
STEM-CaRFs [45] | RGB-D | 88.2 | 87.5
GraspPath [35] | RGD | 86.4 | 84.7
GraspNet [36] | RGB-D | 90.6 | 90.2
Multiple, M=10 [37] | RGB | 91.5 | 90.1
VGG-16 model [32] | RGB-D | 95.5 | 91.7
ResNet-50 model [32] | RGB-D | 96.0 | 96.1
ResNet-50 FCGN [13] | RGB | 97.7 | 94.9
ResNet-101 FCGN [13] | RGB | 97.7 | 96.6
ROI-GD, ResNet-101 [40] | RGB | 93.6 | 93.5
ROI-GD, ResNet-101 [40] | RGD | 92.3 | 91.7
GR-ConvNet [42] | RGB | 96.6 | 95.5
GR-ConvNet [42] | RGBD | 97.7 | 96.6
DD-Net, ResNet-50 | RGB | 96.1 | 95.5
DD-Net, ResNet-101 | RGB | 96.6 | 96.0
DD-Net, Hourglass-104 | RGB | 97.2 | 96.1
|
---|---
Figure 7: Good grasps (red rectangles) by DD-Net are misjudged by Cornell GTs (green rectangles). TABLE III: Results of robotic experiment. The object index is shown in Fig. 6. Object index | #1 | #2 | #3 | #4 | #5 | #6 | #7 | #8 | #9 | #10 | #11 | #12 | #13 | #14 | #15
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
DD-Net | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3
Dex-Net | 2/3 | 2/3 | 3/3 | 3/3 | 2/3 | 3/3 | 0/3 | 2/3 | 2/3 | 3/3 | 2/3 | 3/3 | 2/3 | 3/3 | 2/3
Object index | #16 | #17 | #18 | #19 | #20 | #21 | #22 | #23 | #24 | #25 | #26 | #27 | #28 | #29 | #30
DD-Net | 3/3 | 3/3 | 2/3 | 2/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3
Dex-Net | 2/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 2/3 | 3/3 | 2/3 | 3/3 | 3/3 | 2/3 | 1/3 | 1/3 | 3/3
Object index | #31 | #32 | #33 | #34 | #35 | #36 | #37 | #38 | #39 | #40 | #41 | #42 | #43 | Overall
DD-Net | 3/3 | 2/3 | 2/3 | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 93.0%
Dex-Net | 2/3 | 0/3 | 1/3 | 2/3 | 2/3 | 3/3 | 3/3 | 2/3 | 2/3 | 2/3 | 2/3 | 3/3 | 3/3 | 75.2%

###  V-C Implementation Details

DD-Net is implemented in PyTorch, with the training and inference accomplished
on GTX1080Ti GPUs.

In training, the model is randomly initialized under the default setting
without pre-training on any external dataset. The resolution of the input
image is 511$\times$511 and that of the output is 128$\times$128\. The images
in Jacquard are directly resized to 511$\times$511\. Due to the difference in
aspect ratio, the images in Cornell are first cropped to 480$\times$480 and
then rescaled to 511$\times$511\. The Adam algorithm is used for optimization.
The learning rate is set at 2.5$\times$10-4, and the training mini-batch size
is set as 10. During training, we conduct data augmentation including random
horizontal flipping, random color jittering, and random rotating to avoid
over-fitting.

In inference, we take the top 70 fingertips suggested by the heatmaps. The
grasp with an opening size greater than 70 pixels or smaller than 2 pixels is
rejected so that it is executable. To make the results compatible to those
using 5-dim parameters to represent grasp configurations, the plate length is
set at 10 pixels and 35 pixels for Jacquard and Cornell respectively.

###  V-D Evaluation on Jacquard

Table I summarizes the results of different methods on Jacquard in terms of
both the criteria of SGT and rectangle metric. It can be seen that the
proposed DD-Net outperforms the state of the art counterparts [39] [13] [41]
by a large margin, with an improvement of 3.7% in SGT using RGD images as
input, which demonstrates its superiority in simulation evaluation. Meanwhile,
we can also notice that DD-Net reaches the best score using the standard
rectangle metric, with a gain of 2.4% over the second best reported by [42],
showing its good compatibility. Finally, as the experiments are conducted in
an object-wise way, this performance clearly indicates the effectiveness of
DD-Net in handling unseen objects.

###  V-E Evaluation on Cornell

Table II displays the comparative results using the rectangle metric on the
Cornell Grasp database. We can see that DD-Net delivers very competitive
scores, comparable to the top ones in both the image-wise and object-wise
scenarios. The reason that it does not show obvious increase in accuracy lies
in that the GT annotations on Cornell are not as complete as those in Jacquard
and some good grasps are not correctly judged as shown in Fig. 7. On the other
side, we evaluate the proposed approach with different backbones, and it can
be seen that DD-Net with the light and popular ResNet-50 still reaches good
results and that with stronger backbones, _i.e._ ResNet-101 and Hourglass-104,
slightly increase the performance.

###  V-F Robotic Evaluation

The results of robotic experiments are displayed in Table III, where DD-Net
demonstrates its strong ability in dealing with unseen objects. Refer to the
video attached for more examples. We take the reputed open-source Dex-Net as
the counterpart and we can see that DD-Net significantly outperforms Dex-Net
by 17.8% in success rate. Meanwhile, its planning time is 4 times faster
(0.26s _vs._ 1.0s).

TABLE IV: Ablation study of orientation matching and center matching in fingertip grouping using SGT on Jacquard. Orien. Match. |  Cent. Match. | Accuracy (%)
---|---|---
✗ | ✗ | 52.5
✗ | ✓ | 55.1
✓ | ✗ | 85.6
✓ | ✓ | 89.4
|  |
---|---|---
|  |
|  |
(a) | (b) | (c)
Figure 8: Result visualization of DD-Net: (a) the fingertip heatmap and top-k
candidates; (b) the center heatmap and top-k candidates; and (c) the fingertip
pairs refined by the two matching strategies (blue line indicates final grasp
prediction).

###  V-G Further Analysis

(1) Ablation study: To better understand the major components of DD-Net, we
launch an additional experiment to validate the necessity of the orientation
matching and center matching strategies in fingertip grouping. The SGT scores
are reported as in Table IV, and we can see that both the two schemes
contribute to performance gain and orientation matching plays a dominant role.

(2) Visualization: Intermediate results are shown in Fig. 8. We can see that
the locations of fingertip candidates predicted by the network distribute at
the surrounding of the objects, which coincide with the expectation of human
beings. With the help of the grasp orientation and center, the network selects
the best fingertip pair from all the candidates as shown in Fig. 8 (c).

##  VI Conclusion

This paper addresses the issue of antipodal grasp detection and formulates it
as predicting a pair of fingertips to improve the performance on unseen
objects. A novel deep network, DD-Net, is proposed to accurately and robustly
localize such points. Furthermore, a specialized loss is designed to measure
the grasp quality in training and two matching strategies are introduced to
filter fingertip candidates in inference. Both the simulation and robotic
experiments are carried out and state-of-the-art results are reached,
demonstrating the effectiveness of the proposed approach.

## ACKNOWLEDGMENT

This work is partly supported by the National Natural Science Foundation of
China (No. 62022011), the Research Program of State Key Laboratory of Software
Development Environment (SKLSDE-2021ZX-04), and the Fundamental Research Funds
for the Central Universities.

## References

  * [1] V.-D. Nguyen, “Constructing force-closure grasps,” _The International Journal of Robotics Research_ , vol. 7, no. 3, pp. 3–16, 1988.
  * [2] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” in _IEEE International Conference on Robotics and Automation_ , 2000, pp. 348–353.
  * [3] C. Ferrari and J. F. Canny, “Planning optimal grasps,” in _IEEE International Conference on Robotics and Automation_ , 1992, pp. 2290–2295.
  * [4] K. B. Shimoga, “Robot grasp synthesis algorithms: A survey,” _The International Journal of Robotics Research_ , vol. 15, no. 3, pp. 230–266, 1996\.
  * [5] A. Mousavian, C. Eppner, and D. Fox, “6-dof graspnet: Variational grasp generation for object manipulation,” in _IEEE International Conference on Computer Vision_ , 2019, pp. 2901–2910.
  * [6] C. Dune, E. Marchand, C. Collowet, and C. Leroux, “Active rough shape estimation of unknown objects,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2008, pp. 3622–3627.
  * [7] Z.-C. Marton, D. Pangercic, N. Blodow, J. Kleinehellefort, and M. Beetz, “General 3d modelling of novel objects from a single view,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2010, pp. 3700–3705.
  * [8] D. Kraft, N. Pugeault, E. BAŞESKI, M. POPOVIĆ, D. KRAGIĆ, S. Kalkan, F. Wörgötter, and N. Krüger, “Birth of the object: Detection of objectness and extraction of object shape through object–action complexes,” _International Journal of Humanoid Robotics_ , vol. 5, no. 02, pp. 247–265, 2008.
  * [9] M. Popović, G. Kootstra, J. A. Jørgensen, D. Kragic, and N. Krüger, “Grasping unknown objects using an early cognitive vision system for general scene understanding,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2011, pp. 987–994.
  * [10] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _The International Journal of Robotics Research_ , vol. 37, no. 4-5, pp. 421–436, 2018.
  * [11] T. Lampe and M. Riedmiller, “Acquiring visual servoing reaching and grasping skills using neural reinforcement learning,” in _The International Joint Conference on Neural Networks_ , 2013, pp. 1–8.
  * [12] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _IEEE International Conference on Robotics and Automation_ , 2017, pp. 3389–3396.
  * [13] X. Zhou, X. Lan, H. Zhang, Z. Tian, Y. Zhang, and N. Zheng, “Fully convolutional grasp detection network with oriented anchor box,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2018, pp. 7223–7230.
  * [14] M. Breyer, F. Furrer, T. Novkovic, R. Siegwart, and J. Nieto, “Flexible robotic grasping with sim-to-real transfer based reinforcement learning,” _arXiv preprint arXiv: 1803.04996_ , 2018.
  * [15] J. Mahler and K. Goldberg, “Learning deep policies for robot bin picking by simulating robust grasping sequences,” in _Conference on Robot Learning_ , 2017, pp. 515–524.
  * [16] E. Chinellato, R. B. Fisher, A. Morales, and A. P. Del Pobil, “Ranking planar grasp configurations for a three-finger hand,” in _IEEE International Conference on Robotics and Automation_ , 2003, pp. 1133–1138.
  * [17] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” in _Robotics: Science and Systems_ , 2017.
  * [18] D. L. Bowers and R. Lumia, “Manipulation of unmodeled objects using intelligent grasping schemes,” _IEEE Transactions on Fuzzy Systems_ , vol. 11, no. 3, pp. 320–330, 2003.
  * [19] A. Morales, E. Chinellato, P. Sanz, A. Del Pobil, and A. H. Fagg, “Learning to predict grasp reliability for a multifinger robot hand by using visual features,” in _International Conference AI Soft Computing_ , 2004.
  * [20] Y. Jiang, S. Moseson, and A. Saxena, “Efficient grasping from rgbd images: Learning using a new rectangle representation,” in _IEEE International Conference on Robotics and Automation_ , 2011, pp. 3304–3311.
  * [21] I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” _The International Journal of Robotics Research_ , vol. 34, no. 4-5, pp. 705–724, 2015.
  * [22] Z. Wang, Z. Li, B. Wang, and H. Liu, “Robot grasp detection using multimodal deep convolutional neural networks,” _Advances in Mechanical Engineering_ , 2016.
  * [23] Y. Xu, L. Wang, A. Yang, and L. Chen, “Graspcnn: Real-time grasp detection using a new oriented diameter circle representation,” _IEEE Access_ , vol. 7, pp. 159 322–159 331, 2019.
  * [24] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 91–99.
  * [25] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European Conference on Computer Vision_ , 2016, pp. 21–37.
  * [26] S. Liu, D. Huang, and Y. Wang, “Receptive field block net for accurate and fast object detection,” in _European Conference on Computer Vision_ , 2018, pp. 385–400.
  * [27] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” in _European Conference on Computer Vision_ , 2018, pp. 734–750.
  * [28] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _IEEE International Conference on Computer Vision_ , 2019, pp. 6569–6578.
  * [29] Q. V. Le, D. Kamm, A. F. Kara, and A. Y. Ng, “Learning to grasp objects with multiple contact points,” in _IEEE International Conference on Robotics and Automation_ , 2010, pp. 5062–5069.
  * [30] A. Gariépy, J.-C. Ruel, B. Chaib-draa, and P. Giguère, “Gq-stn: Optimizing one-shot grasp detection based on robustness classifier,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2019, pp. 3996–4003.
  * [31] D. Guo, F. Sun, H. Liu, T. Kong, B. Fang, and N. Xi, “A hybrid deep architecture for robotic grasp detection,” in _IEEE International Conference on Robotics and Automation_ , 2017, pp. 1609–1614.
  * [32] F.-J. Chu, R. Xu, and P. A. Vela, “Real-world multiobject, multigrasp detection,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3355–3362, 2018.
  * [33] H. Karaoguz and P. Jensfelt, “Object detection approach for robot grasp detection,” in _IEEE International Conference on Robotics and Automation_ , 2019, pp. 4953–4959.
  * [34] J. Ma, W. Shao, H. Ye, L. Wang, H. Wang, Y. Zheng, and X. Xue, “Arbitrary-oriented scene text detection via rotation proposals,” _IEEE Transactions on Multimedia_ , vol. 20, no. 11, pp. 3111–3122, 2018\.
  * [35] L. Chen, P. Huang, and Z. Meng, “Convolutional multi-grasp detection using grasp path for rgbd images,” _Robotics and Autonomous Systems_ , vol. 113, pp. 94–103, 2019.
  * [36] U. Asif, J. Tang, and S. Harrer, “Graspnet: An efficient convolutional neural network for real-time grasp detection for low-powered devices,” in _International Joint Conferences on Artificial Intelligence_ , 2018, pp. 4875–4882.
  * [37] G. Ghazaei, I. Laina, C. Rupprecht, F. Tombari, N. Navab, and K. Nazarpour, “Dealing with ambiguity in robotic grasping via multiple predictions,” in _Asian Conference on Computer Vision_ , 2018, pp. 38–55.
  * [38] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in _European Conference on Computer Vision_ , 2016, pp. 483–499.
  * [39] A. Depierre, E. Dellandréa, and L. Chen, “Jacquard: A large scale dataset for robotic grasp detection,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2018, pp. 3511–3516.
  * [40] H. Zhang, X. Lan, S. Bai, X. Zhou, Z. Tian, and N. Zheng, “Roi-based robotic grasp detection for object overlapping scenes,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2019, pp. 4768–4775.
  * [41] A. Depierre, E. Dellandréa, and L. Chen, “Optimizing correlated graspability score and grasp regression for better grasp prediction,” _arXiv preprint arXiv:2002.00872_ , 2020.
  * [42] S. Kumra, S. Joshi, and F. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020.
  * [43] J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” in _IEEE International Conference on Robotics and Automation_ , 2015, pp. 1316–1322.
  * [44] S. Kumra and C. Kanan, “Robotic grasp detection using deep convolutional neural networks,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2017, pp. 769–776.
  * [45] U. Asif, M. Bennamoun, and F. A. Sohel, “Rgb-d object recognition and grasp detection using hierarchical cascaded forests,” _IEEE Transactions on Robotics_ , vol. 33, no. 3, pp. 547–564, 2017.