[style=english] [style=english] [style=english] [style=english] [1]

[1]

[cor1]Corresponding author

# Physics-Aware Iterative Learning and Prediction of Saliency Map for Bimanual
Grasp Planning

Shiyao Wang<EMAIL_ADDRESS>Xiuping Liu<EMAIL_ADDRESS>Charlie
C.L. Wang<EMAIL_ADDRESS>Jian Liu<EMAIL_ADDRESS>School of Mathematics Sciences, Dalian University of Technology, Dalian 116024
School of Engineering, The University of Manchester, United Kingdom  School of
Software, Tsinghua University, Beijing 100084, P. R. China

###### Abstract

Learning the skill of human bimanual grasping can extend the capabilities of
robotic systems when grasping large or heavy objects. However, it requires a
much larger search space for grasp points than single-hand grasping and
numerous bimanual grasping annotations for network learning, making both data-
driven or analytical grasping methods inefficient and insufficient. We propose
a framework for bimanual grasp saliency learning that aims to predict the
contact points for bimanual grasping based on existing human single-handed
grasping data. We learn saliency corresponding vectors through minimal
bimanual contact annotations that establishes correspondences between grasp
positions of both hands, capable of eliminating the need for training a large-
scale bimanual grasp dataset. The existing single-handed grasp saliency value
serves as the initial value for bimanual grasp saliency, and we learn a
_saliency adjusted score_ that adds the initial value to obtain the final
bimanual grasp saliency value, capable of predicting preferred bimanual grasp
positions from single-handed grasp saliency. We also introduce a physics-
balance loss function and a physics-aware refinement module that enables
physical grasp balance, capable of enhancing the generalization of unknown
objects. Comprehensive experiments in simulation and comparisons on dexterous
grippers have demonstrated that our method can achieve balanced bimanual
grasping effectively.

###### keywords:

bimanual grasp planning human grasp preference grasping balance deep learning
grasp saliency

##  1 Introduction

In manual operation environments, bimanual interactions with household objects
are required for many daily tasks. In comparison to the single-handed
grasping, grasping with both hands has obvious advantages in terms of
stability and balance for big or heavy objects. For example, when grasping an
empty kettle, people prefer to grasp the handle with a single hand. However,
when the kettle is full of water, people typically choose to grasp it with
both hands, which is more stable than grasping it with only one hand. Learning
from this bimanual grasping experience can significantly extend the capability
of bimanual robotic system for completing the tasks of bimanual grasping in
similar scenarios.

Although there are many works on robotic grasping, the problem of bimanual
grasp planning is still a challenge. Planning a feasible bimanual grasp needs
to search contact points of both hands on the object surface using physics-
based analyses such as force-closure Dai et al. (2015), which results in a
search space much larger than that for contact points of single hand. To solve
this problem, previous works conduct the bimanual grasping either by regarding
two hands of the robot as two fingers connected to a wrist and then solve it
with single-grasp planners Vahrenkamp et al. (2011); Berenson and Srinivasa
(2008); Caraza and Yun (1991), or separately computing potential contact
points of each hand for grasp candidate selection Seo and Kumar (2012); Saut
et al. (2010); Przybylski et al. (2011); Hayakawa et al. (2021). Furthermore,
to compute a grasp using both hands simultaneously, de Silva and Suárez
(2016); Rojas-de Silva and Suárez (2017) approaches split the object surface
into two slices from which the contact points of each hand are selected. While
these methods strive to narrow down the search space, they overly reduce the
feasibility of choosing more stable bimanual contact points.

On the other aspect, human demonstrations have been exploited to improve
robotic grasping and 3D grasp synthesis in data-driven methods Amor et al.
(2012); Yu et al. (2021); Zhu et al. (2021); Brahmbhatt et al. (2020); Song et
al. (2016). However, it is challenging to collect a large number of human
grasp trials that requires a significant cost. Some recent works replace human
grasp demonstration with semantic hand-object contact representation such as
contact map Brahmbhatt et al. (2019a), saliency map Lau et al. (2016); Zhang
et al. (2020) and affordence map Mandikal and Grauman (2021); Deng et al.
(2021); Xu et al. (2022); Song et al. (2016) to synthesize the grasps.
ContactGrasp Brahmbhatt et al. (2019b) involved robotic grasp synthesis for
dexterous hands from the contact maps recorded by ContactDB Brahmbhatt et al.
(2019a). Lau et al. (2016) collected the dataset of human grasping preference
through crow-sourcing and utilized a deep ranking framework to predict grasp
saliency map. However, these methods can’t directly address the challenging
problem of bimanual grasping since they mainly focus on the single-handed
grasping task and no existing large-scale dataset of human annotations of
bimanual contact points. These defects raise a challenge for the network to
predict specific contact areas that humans prefer to grasp with both hands in
the absence of adequate training data.

Based on the observation of human grasping behavior, two intuitive
correlations between single-handed grasping and bimanual grasping we founded
may help to tackle the challenge. To illustrate the correlation, we invited 30
participants and asked them to annotate grasps for 8 categories of household
objects with only one and both hands, respectively. As shown in figure 1,
humans usually choose bimanual contact points based on object’s geometry and
affordence. It can be observed that the contact position of right hand is
generally located in the functional area of object, while the corresponding
contact position of left hand would be selected to assist in grasping balance
or support. We further compute the grasp coverage for each object, which is
the degree of overlap between bimanual contact areas and single-handed contact
areas. It shows that the grasp coverage is above 70% for most categories of
objects, indicating that humans usually still choose the specific grasping
area similar to single-hand grasping when conducting bimanual grasping.

\begin{overpic}[width=433.62pt,tics=10]{user_study} \end{overpic} Figure 1:
The figure shows the average grasp coverage of 8 categories of objects and
examples of single-handed grasp poses and bimanual grasp poses. The grasp
coverage is defined as the percentage of users who labeled single-handed
grasping positions are covered by bimanual grasping positions.

Inspired by the aforementioned intuitive correspondence and similarity between
single-handed grasping and bimanual grasping, we advocate to predict bimanual
contact points relying on human single-handed grasping preference, rather than
relying on large-scale data with bimanual contact annotations. To this end, we
propose a bimanual grasp saliency learning framework for bimanual contact
point detection. The existing human single-handed grasp saliency map is served
as initiation of bimanual saliency map, and we build a small number of manual
annotations of bimanual contact points to learn correspondence between grasp
positions of both hands. Therefore, our method learns saliency adjusted score
that adjusts the initial value to bimanual saliency value. In addition, to
make the initial value of bimanual saliency be fine-tuned, we also design an
iterative learning strategy so that the initial value can be updated. In order
to further achieve the generalization when human single-handed grasping
experience fail in bimanual grasping for unseen objects that are not included
in the training dataset since the lack of large-scale dataset, our method
takes the physical factors of grasping into account for both training and
inference stages. Finally, we train a segmentation network to predict bimanual
contact points from the bimanual grasp saliency map and generate bimanual
grasp poses relying on the existing 3D grasp synthesis approach Brahmbhatt et
al. (2019b).

Given an input object point cloud, we propose a physics-aware bimanual
saliency iterative learning framework for bimanual grasp planning. Our method
does not rely on a large number of bimanual contact annotation, but only
adjusts the single-handed saliency map to the bimanual saliency map through
minimal human annotations and physical criterion, so as to obtain the contact
points of both hands.

We conducted comprehensive experiments on a variety of household objects and
evaluated our approach in simulation. Results have demonstrated that our
method can generate stable grasps with predicted bimanual contact points
effectively. In summary, our work makes the following contributions:

  * •

We exploit the correlations between single-handed grasping and bimanual
grasping and propose a neural network-based method to predict the bimanual
grasp saliency map only relying on existing human single-handed grasping
preference saliency data and a small dataset of manual annotations.

  * •

We propose a bimanual saliency iterative learning strategy so that the initial
value of bimanual saliency can be efficiently updated during both training and
inference stages.

  * •

We take the constraint of physical balance in grasping into account and design
a physical balance based loss function and a physics-aware refinement to
enhance the generalization for bimanual grasping of objects that have not been
‘seen’ in the training dataset.

##  2 Related work

###  2.1 Bimanual grasp Planners.

Planning bimanual grasp is an important field of robotic grasping in which
several studies and practical applications have been found Smith et al.
(2012). Many relevant works focus on the force-closure methods to search
contact points on the object surface and synthesize feasible grasps of both
hands. To plan the bimanual grasping, Berenson and Srinivasa (2008) formalized
both hands of the robot as an abstraction of two fingers connected to a wrist
and converted the bimanual grasping problem into a single-handed grasping
problem. Seo and Kumar (2012) used the whole body to grasp polyhedral objects
where both arms are represented by chain grippers and the feasible contact
configurations are selected. Vahrenkamp et al. (2009) proposed to address the
bimanual grasping problem by choosing grasp candidates for each hand
separately. To avoid suffering from manipulability of bimanual grasp, the
approach Vahrenkamp et al. (2011) computed the reachable grasp candidates of
each end-effector by inverse kinematics and then found force-closure grasp
configurations performed by the grasp wrench space analysis. Unlike the
aforementioned discussed works, de Silva and Suárez (2016) computed a
kinematically reachable grasp of two dexterous hands simultaneously. In this
process, the object observed as the point cloud is divided into two parts
where the contact points of both hands are selected, and then the combinations
of the two-handed contact points are evaluated to satisfy the force closure
condition. After potential contact points are determined, the suitable grasp
configuration is performed by computing inverse kinematics and checking
collision. Although these grasp planners can be used to compute bimanual
grasps by simplifying the search space of bimanual contacts, they overly
reduce the feasibility of choosing more stable bimanual contact points. In
contrast to these previous works, we begin to shift focus towards using human
grasp preference to plan contact points for bimanual grasp, which helps in
efficiently finding feasible bimanual grasps.

###  2.2 Grasp synthesis from hand-object contact.

The research of grasp synthesis recently focus on the semantic hand-object
representation (contact map, grasp saliency, grasp affordance, etc.) to
synthesize task-oriented grasp poses of robot grippers. Brahmbhatt et al.
(2019a) first annotated contact maps on the object surfaces that represent
human-like functional grasp regions of the human hand. Based on that,
ContactGrasp Brahmbhatt et al. (2019b) synthesized functional robotic grasps
by optimizing stable grasp configurations computed by a state-of-art grasp
planner to align with the contact maps. ContactPose Brahmbhatt et al. (2020)
further recorded both human hand poses and contact points to learn functional
grasps of the dexterous hands. Some relevant works also used contact map as an
intermediate variable to constrain the grasp generation Li et al. (2023a) or
to refine generated grasps during inference Jiang et al. (2021). However,
different grasps would sometimes result in the same contact map because of
each part of the hand does not correspond to the specific part of the contact
map. To overcome this issue caused by the contact map, Zhu et al. (2021)
present a high-level semantic touch code that described which part of the hand
contact the object that are manually pre-segmented into the functional parts
for grasping and train a deep architecture to generate human-like grasps Wu et
al. (2023); Zhang et al. (2023). In contrast to the contact map represented
from human demonstration, the semantic hand-object representation based on
human visual perception is also widely used for grasp detection. Chen et al.
(2022) present an affordance-based network to predict grasp affordance map
given an input point cloud and select candidate grasps based on it. In
addition, Lau et al. (2016) first proposed to predict grasp saliency, which is
an important tool to explore human grasp skills. This approach first defined
the grasp salient points on the mesh model that human would like to grasp and
then instructed online user annotators to crowd-source such saliency data.
After that, a deep ranking technique is developed to measure grasp saliency.
However, the grasp saliency prediction heavily relies on sufficient grasp
annotations, which requires an expensive acquisition cost. To overcome this
challenge, Zhang et al. (2020) developed a transfer learning methodology to
augment saliency data via deep shape correspondence. While these approaches
contribute to robotic analyses of task-oriented human-like grasping, they
cannot directly specify bimanual grasping, only relying on human grasp data of
the single hand. Different from these methods, our solution is able to predict
the bimanual grasp saliency map for bimanual grasp planning relying on human
single-handed grasping preference without depending on large-scale data with
bimanual contact annotations.

\begin{overpic}[width=433.62pt,tics=10]{pipeline} \end{overpic} Figure 2:  The
overview of our physics-aware iterative learning and prediction of bimanual
saliency map for novel objects. We first use a pre-trained SSPN model to
compute the single-handed saliency map given an input point cloud. Next, the
proposed BSPN network is leveraged to predict bimanual saliency map, which
takes both point cloud and the single-handed saliency map, and the bimanual
contact points are computed by applying BCPN network given both point cloud
and the result of BSPN. After that, we refine bimanual saliency map to conform
the physical stability using physics-aware refinement module.

##  3 Method

This work concentrates on predicting a physics-aware saliency map of object
for bimanual grasp planning, which implies predicting bimanual grasp contact
points that are physically plausible for grasping balance and amenable to
human grasping preferences. In order to predict bimanual saliency map in
absence of training data, we propose a novel framework that exploits physics-
aware bimanual saliency map from the single-handed saliency map with minimal
bimanual grasp contact annotations, as shown in Figure 2. It generates maps
for bimanual saliency grasp areas by the bimanual saliency map prediction
network (Sec. 3.2), named as BSPN, computes the contact points of both hands
from bimanual saliency map by the bimanual contact point prediction network
(Sec. 3.3), named as BCPN, refines the predicted result of bimanual saliency
map by the physics-aware refinement (Sec. 3.4), and synthesizes grasp
configurations from computed contact points by ContactGrasp Brahmbhatt et al.
(2019b), a grasp optimizer that computes grasp poses (Sec. 3.5).

###  3.1 Problem Statement

Given an input object point cloud $P=\\{x_{i}\\}^{N}_{i=1}$, we define its
bimanual saliency map $B=\\{b_{i}\\}^{N}_{i=1}$ and single-handed saliency map
$S=\\{s_{i}\\}^{N}_{i=1}$, where $b_{i}\in[0,1]$ and $s_{i}\in[0,1]$ denote
the bimanual saliency value and single-handed saliency value of the point
$x_{i}$ respectively. The closer the saliency value of point $x_{i}$ is to 1,
the more likely it is to be grasped. Moreover, to annotate bimanual grasp
contact points, we also define $L=\\{l_{i}\\}^{N}_{i=1}$ as a set of bimanual-
contact labels, where $l_{i}$ is formula as follow:

$l_{i}=\begin{cases}1&if\ x_{i}\ \in\ right-hand,\\\ 2&elif\ x_{i}\ \in\ left-
hand,\\\ 0&otherwise.\end{cases}$

$x_{i}\in\ right-hand$ denotes that $x_{i}$ is annotated as the grasp contact
point of right hand and $x_{i}\in\ left-hand$ denotes that $x_{i}$ is
annotated as the grasp contact point of left hand. More details about building
the bimanual-contact label set is described in Sec. 4.1. Notice that we only
use a small number of bimanual annotations in our approach, as the extensive
annotation is time-consuming.

Based on the observation of human grasping behavior with both hands, we
discover that two intuitive correlations between single-handed grasping and
bimanual grasping. Firstly, the contact position of the right hand is
generally located in the functional area of the object, while the
corresponding contact position of the left hand would be selected to assist in
grasping balance and support. Secondly, the grasp contacts chosen by one hand
are covered by the grasp contacts of both hands. These correlations give us an
important insight about predicting bimanual saliency map by taking the single-
handed saliency map as initialization. Motivated by this, we compute the
bimanual saliency map $B$ using single-handed saliency map $S$ and the spatial
relationship between the grasping positions of both hands. We define the
relationships as correspondences between two kinds of grasping areas: the
correspondences between the points labeled left-hand and the points labeled
right-hand. Specifically, for a pair of points $x_{i}$ and $x_{j}$ that are
annotated as left-hand and right-hand respectively, we define their saliency
corresponding vector $v_{ij}\subset V$ as a displacement from $x_{i}$ to
$x_{j}$. Thus, the corresponding relationship could be presented as:

$x_{j}=x_{i}+v_{ij}.$ (1)

Our basic idea is, for a pair of points ($x_{i}$,$x_{j})$ that are annotated
as left-hand and right-hand respectively, their bimanual saliency values
$b_{i}$ and $b_{j}$ should be as similar as possible. Therefore, we aim to
find a mapping function $F_{\theta}:(S|P)\rightarrowtail\Delta S$ to predict a
saliency adjusted score $\Delta S\in R^{N\times 1}$ which adjusts single-
handed saliency map to bimanual saliency map based on the saliency
corresponding vector. Thus, the bimanual saliency map can be formulated as
$B=S+\Delta S$. In particular, learning the adjusted score should not only
make the bimanual saliency map amenable to the human grasping perception, but
also satisfy the physical balance and stability for grasping.

Although we can obtain the bimanual saliency map by adjusting the single-
handed saliency map, it may not adjust the saliency value of both hands
simultaneously if we only rely on a fixed single-handed saliency map, which is
due to the bimanual saliency value of the points labeled right-hand are
unchanged during the process. Hence, we develop an iterative learning strategy
to update the single-handed saliency map.

In order to further ensure the generalization for novel objects when human
single-handed grasping experience fail in bimanual grasping, we also take the
grasping balance into account for both training and inference stages. On the
one hand, we add the physics-balance loss function to bimanual saliency map
learning. On the other hand, we design a physics-aware refinement module to
refine the predicted result of the bimanual saliency map so that it conforms
to physical stability.

\begin{overpic}[width=433.62pt,tics=10]{network} \end{overpic} Figure 3: The
network architecture of BSPN. The encoder is consist of 3-layer MLP. And the
decoder is consist of 4-layer MLP, which takes the feature concatenated by
64-dimensional and 1024-dimensional features as input. Finally, the decoder
outputs $m-$dimensional prediction. For the saliency corresponding vector $V$
and the saliency adjusted score $\Delta S$, $m$ is $3$ and $1$ respectively.
The total training loss function consists of three items: Correspondence Loss
$L_{c}$, Adjustment Loss $L_{a}$ and Physics-aware Balance Loss $L_{p}$.

###  3.2 Bimanual Saliency Map Prediction Network

Figure 3 demonstrates the architecture of BSPN, which takes both an object
point cloud $P$ and the single-handed saliency map $S$ as inputs and outputs
the bimanual saliency map $B$. We first generate the single-handed saliency
map as the initialization of the bimanual saliency map. Then, we adjust the
single-handed saliency map based on the bimanual grasp contact annotations and
the physical criterion to generate a bimanual saliency map. Finally, we also
propose an iterative learning strategy that updates the single-handed saliency
map both in the training and inference stages. Next, we introduce the single-
handed saliency map generation, the network structure and the iterative
learning strategy of BSPN.

####  3.2.1 Single-handed saliency map

The single-handed saliency map is generated based on SSPN Zhang et al. (2020)
which takes a local patch centered on a point $x_{i}$ as input to predict the
single-handed saliency value $s_{i}^{o}$ of $x_{i}$. We take the patch for
each point as input to SSPN, then traverse all points of the point cloud and
finally output the single-handed saliency map $S^{o}$.

It is worth notice that since the SSPN is pre-trained on the existing training
data of single-handed grasp saliency, the single-handed saliency areas
predicted by SSPN may sometimes incompletely coincide with the position of
points labeled right-hand in our bimanual grasp contact annotation. Therefore,
towards addressing this issue, we design and train a correction module (CM)
for correcting the single-handed saliency map generated by SSPN. Specifically,
it takes both an object point cloud and the single-handed saliency map $S^{o}$
as inputs and outputs the corrected single-handed saliency map $S^{c}$. Thus,
the single-handed saliency map after correction can be written as
$S=S^{o}+S^{c}$. The CM network also follows encoder-decoder architecture that
is built on PointNet. In the training stage, we force the single-handed
saliency value of points labeled right-hand to be more saliency by making
$L_{correct}$ close to 0. The $L_{correct}$ is formulated as follow:

$L_{correct}=\frac{1}{N}(\sum_{i=1}^{N}E_{g}(x_{i})\cdot(s_{i}-1)^{2}+\sum_{i=1}^{N}E_{n}(x_{i})\cdot
s_{i}^{2}),$ (2)

where $\displaystyle E_{g}(x_{i})=\begin{cases}1&if\ l_{i}\ =1,\\\
0&otherwise.\end{cases}$ and $\displaystyle E_{n}(x_{i})=\begin{cases}1&if\
l_{i}\ \in\\{0,2\\}\\\ 0&otherwise.\end{cases}$. Hence, for points labeled
right-hand, we encourage their single-handed saliency values to be close to 1,
and for other points, we encourage their single-handed saliency values to be
close to 0. After obtaining the single-handed saliency map, we take it as the
initial value of the bimanual saliency map.

####  3.2.2 Network structure

As shown in Figure 3, the BSPN is a supervised network, which is composed of
two encoder-decoder architectures. Both two encoders
$E_{\theta_{1}}:(P)\rightarrowtail f_{t}$ and
$E_{\theta_{2}}:(P)\rightarrowtail f_{s}$ are built on PointNet Qi et al.
(2016) architecture. The encoder $E_{\theta_{1}}$ takes a point cloud as input
to extract the spatial location feature $f_{s}\in\mathbb{R}^{N\times 1024}$
associated with the saliency corresponding vectors. The encoder
$E_{\theta_{2}}$ takes a point cloud as input to extract the saliency
transformation feature $f_{t}\in\mathbb{R}^{N\times 1024}$ associated with the
saliency adjusted score. For each encoder, according to the PointNet
architecture, the feature representation for each point is computed by 5-layer
MLP architecture. The features of each point are then aggregated into the
global feature representation of the input point cloud by max-pooling. After
that, we replicate the global feature $N$ times as a global feature
representation for each point. Considering adding local geometric information,
we need to concatenate the features $f_{s}^{1}$ and $f_{t}^{1}$ of the first
layer of MLP architecture with the features output from the last layer,
respectively.

Both decoders $D_{\theta_{3}}:(f_{t},f_{t}^{1})\rightarrowtail V$ and
$D_{\theta_{4}}:(f_{s},f_{s}^{1})\rightarrowtail\Delta S$ are regressors to
predict the saliency corresponding vectors and the saliency adjusted score,
respectively. They use 4-layer MLP architecture. The decoder $D_{\theta_{3}}$
takes both the saliency transformation features $f_{t}$ and $f_{t}^{1}$ as
inputs and outputs the saliency corresponding vectors $V$. The decoder
$D_{\theta_{4}}$ takes both the spatial location features $f_{s}$ and
$f_{s}^{1}$ as inputs and outputs the saliency adjusted score $\Delta S$. In
the inference stage, only $E_{\theta_{2}}$ and $D_{\theta_{4}}$ are used. The
encoder $E_{\theta_{2}}$ takes point cloud $P$ to output the spatial location
features. With these features, the decoder $D_{\theta_{4}}$ outputs the
saliency adjusted score $\Delta S$, and then we generate the bimanual saliency
map by adding the saliency adjusted score to the single-handed saliency map.

####  3.2.3 Loss function

To enable our BSPN network to generate reasonable bimanual saliency map by
adjusting the single-handed saliency, we design the loss functions based on
the single-handed saliency map, bimanual grasp contact annotation and physical
criterion. First, we use two loss functions to constrain the saliency
corresponding vectors $V$ and saliency adjusted score $\Delta S$. The first is
the correspondence loss $L_{c}$, which is used to constrain the spatial
correspondences between the points labeled left-hand and the points labeled
right-hand. The second is the adjustment loss $L_{a}$, which makes the
adjustment of the single-handed saliency map well. Besides that, to ensure
that the generated bimanual saliency map is physically plausible for grasp
balance, we also propose a physics-balance loss $L_{p}$. Overall, the total
loss function consists of three items: correspondence loss, adjustment loss
and physics-balance loss.

_Correspondence Loss_ The correspondence loss makes the errors between the
predicted corresponding vectors and ground truth corresponding vectors as
small as possible. For a given predicted saliency corresponding vector $v_{i}$
and the ground truth $\hat{v_{i}}$, the loss can be formulated as the mean
squared error between $v_{i}$ and $\hat{v_{i}}$:

$L_{c}=\frac{1}{N}(\sum_{i=1}^{N}\phi(x_{i})\cdot(v_{i}-\hat{v_{i}})^{2}),$
(3)

where $\phi(x_{i})=\begin{cases}1&if\ l_{i}\ \in\ \\{1,2\\},\\\
0&otherwise.\end{cases}$ $\phi(x_{i})$ is an indicative function. If $x_{i}$
is annotated as either left-hand or right-hand, $\phi(x_{i})$ is equal to $1$
otherwise $\phi(x_{i})$ is equal to $0$. This means that we only need to focus
on the accuracy of the saliency corresponding vector prediction of points in
annotation.

_Adjustment Loss_ The points labeled left-hand and the points labeled right-
hand are the potential contact points of bimanual grasping, which means that
their bimanual saliency values seem to be almost identical. Therefore, we use
corresponding vector to constrain bimaunal saliency values of a pair of points
labeled left-hand and labeled right-hand to be as similar as possible. Given
$b_{i}$ and $b_{j}$, the goal is achieved by optimizing the mean squared
error:

$L_{a_{1}}=\frac{1}{N}(\sum_{i=1}^{N}\phi(x_{i})\cdot(b_{i}-b_{j})^{2}).$ (4)

Furthermore, for the points labeled right-hand and points labeled neither
left-hand nor right-hand, that is, the points with $l_{i}\in\\{0,1\\}$, we
believe that their bimanual saliency values do not need to change in the
adjustment process. Their bimaunal saliency values should be consistent with
the single-handed saliency value. Given the bimanual saliency value $b_{i}$
and the single-handed saliency value $s_{i}$, the loss formula is as follow:

$L_{a_{2}}=\frac{1}{N}(\sum_{i=1}^{N}\psi(x_{i})\cdot(b_{i}-s_{i})^{2}),$ (5)

where $\displaystyle\psi(x_{i})=\begin{cases}1&if\ l_{i}\ \in\\{0,1\\},\\\
0&otherwise.\end{cases}$. $\psi(x)$ is also an indicator. Therefore, the
adjustment loss function $L_{a}$ can be written as:

$L_{a}=\lambda_{1}\cdot L_{a_{1}}+\lambda_{2}\cdot L_{a_{2}},$ (6)

where $\lambda_{1}=1$ and $\lambda_{2}=1.5$ is the corresponding weights.

_Physics-balance Loss_ We force the midpoint between the left-hand contact
point and the right-hand contact point to approach the direction of gravity,
which ensures the physical balance for grasping guided by the bimanual
saliency map. Specifically, we select a pair of contact points $x_{l}$ and
$x_{r}$ with the highest bimanual saliency value among the points labeled
left-hand and the points labeled right-hand. Assuming the mass of the object
is uniform, we set the object’s geometric center $p_{center}$ as the center of
gravity and select the vertical downward direction as the direction of
gravity. $x_{c}$ is the midpoint between $x_{l}$ and $x_{r}$, and we calculate
the Euclidean distance from the relative coordinates of midpoint
$x_{c}-p_{center}$ to the straight line $G$ where the gravity direction is
located, and set it as the physical balance loss, which could be presented as:

$L_{p}=D(x_{c}-p_{center},G),$ (7)

where $D(\cdot,\cdot)$ is the Euclidean distance from $x_{c}-p_{center}$ to
the straight line $G$.

The final loss function is the linearly combination of the correspondence
item, the adjustment item and the physics-balance loss item, which could be
presented as:

$L_{total}=w_{1}\cdot L_{c}+w_{2}\cdot L_{a}+w_{3}\cdot L_{p}.$ (8)

where $w_{1}=1$, $w_{2}=1$ and $w_{3}=2$ denote the corresponding weights.

####  3.2.4 Iterative Learning Strategy

We also need to update the single-handed grasp saliency map in the adjustment
process. Particularly, for the points annotated as right-hand, their single-
handed saliency values need to change since we use a pre-generated single-
handed saliency map as initialization. To this end, we employ an iterative
learning strategy in BSPN. Specifically, we update the single-handed saliency
map $S$ when the training epoch $>=K$ and every $M$ epochs. It updates $m$
times in total, which can be formula as $S^{t}=B^{t-1},t=1,2,\cdots,m$, where
$B^{0}$ is equal to the initial single-handed saliency map. Meanwhile, we also
need to re-formulate the adjustment loss $L_{a_{2}}$ since the original item
but only constrain the points labeled right-hand and but also constrain the
points labeled neither left-hand nor right-hand. In the iteration phase, we
only need to force $L_{a_{2}}$ to constrain the points labeled neither left-
hand nor right-hand by representing the indicator $\psi(x)$ as:
$\displaystyle\psi(x_{i})=\begin{cases}1&if\ l_{i}\ =0,\\\
0&otherwise.\end{cases}$.

Besides that, we also need to give a stop condition for the iteration. Instead
of terminating the iteration by manually setting the iteration number, which
is hard to control the reasonable of results, we design an automatic mechanism
for iteration termination based on the generated bimanual saliency map and the
physics-balance loss function. Specifically, we define two stop conditions.
One condition is the average bimanual saliency value of all the points labeled
left-hand and the points labeled right-hand. The other condition is the
physics-balance distance that is used in Eq 7. Therefore, the iteration is
automatically terminated when the average bimanual saliency value reaches to a
threshold $\sigma_{s}$ and the physics-balance distance is less than a
threshold $\sigma_{p}$ during iteration. In practice, $\sigma_{s}$ and
$\sigma_{p}$ are set as $0.8$ and $0.12$, respectively.

###  3.3 Bimanual Contact Point Prediction Network

We compute bimanual contact points based on the object point cloud, predicted
bimanual saliency map and the bimanual contact label. We develop a neural
network, named bimanual contact point prediction network (BCPN), to compute
the bimanual contact points.

The BCPN is built on the PointNet segmentation network which takes the
concatenation of the point cloud and predicted bimanual saliency map as input
and outputs the bimanual contact labels. We used it as a classifier
$C_{\theta}:(B,P)\rightarrowtail L$ for every point $x_{i}\in P$ to classify
whether it labels right-hand, left-hand or not. A small number of bimanual-
contact labels we captured is represented as the ground truth for training. To
classify each point, we adopt the cross-entropy loss function. We train the
BCPN with the BSPN architecture together, which is due to the prediction of
the saliency corresponding vector in BSPN and the classification of contact
points both require the geometric information of the bimanual grasping
position. Therefore, we an additional BSPN branch to assist the training of
BCPN. The final loss function is defined as:

$L_{classify}=w_{4}\cdot\frac{1}{N}(-\sum_{i=1}^{N}\hat{l_{i}}log(l_{i}))+L_{total},$
(9)

where $l_{i}$ is the predicted bimanual contact label, $\hat{l_{i}}$ is the
ground truth of the bimanual contact label and $w_{4}=1.5$ is constant for
balancing the loss terms. Note that the BSPN is only employed in training and
discarded in inference.

After predicting the bimanual contact labels, we compute bimanual contact
points by combing with the bimanual saliency map. Specifically, we threshold
bimanual saliency map at $\tau$ and define a bimanual saliency mask:
$\displaystyle m_{i}=\begin{cases}1&if\ b_{i}>=\tau,\\\
0&otherwise.\end{cases}$. Therefore, we can use $l_{i}=l_{i}\cdot m_{i}$ to
classify whether $x_{i}$ is a contact point or not.

###  3.4 Physics-aware Refinement

Although the BSPN network works well for the objects of the same category as
the training data, it is not generalized enough for novel objects since the
small amount of training data. Hence, we propose a physics-aware refinement
module to further ensure generalization ability across objects.

The physics-aware refinement aims to refine the predicted bimanual saliency
map to conform to the physical stability. Specifically, the refinement module
takes an object point cloud, bimanual saliency map $B$ predicted by BSPN and
bimanual contact labels predicted by BCPN as input and utilizes the MLP
architecture to compute the refined adjustment $R$ of per point’s bimanual
saliency value. The refined bimanual saliency map is formula as: $B_{r}=B+R$.
We hope that a pair of the grasp contact points, having the highest refined
bimanual saliency value, of which one is labeled right-hand and the other is
not labeled right-hand, conforms to the physical stability. Therefore, the
distance between their midpoint and the straight line where the gravity
direction is located should be as small as possible. The objective function
for the physics-aware refinement is similar with the Physics-balance loss
function Eq 7. We iteratively refine the bimanual saliency map until the
objective function is smaller than the threshold $w_{r}$, where $w_{r}=0.12$.

To compute the final bimanual contact points, we concatenate the point cloud
and the refined bimanual saliency map as input and apply K-means clustering
Krishna and Murty (1999) based on the concatenation. We set $K=3$ and the
points are divided into three categories: left-hand contact point, right-hand
contact point and others. In this way, we can obtain the contact points of
both hands, which could be used to plan the bimanual grasping.

###  3.5 Application: bimanual grasp synthesis

We synthesize grasp poses of both hands based on the generated bimanual
contact points by utilizing ContactGrasp Brahmbhatt et al. (2019b).
ContactGrasp is a grasp optimizer for functional grasp generation from an
object model and a contact map. It seeks a hand configuration coincided with a
contact map by encouraging the hand to approach the contact points and
discouraging to touch the other points.

In our implementation, the MANO Romero et al. (2017) hand model is adopted for
optimization. First, we import MANO into GraspIt! Miller and Allen (2004) to
initialize the hand configuration. Then, we construct left and right-hand
contact maps based on the contact points computed by BCPN, respectively.
Finally, we compute the grasp configuration for each hand. Based on the right-
hand contact map and the right MANO hand model, we optimize the right hand’s
grasp by ContactGrasp and choose the optimal grasp. Similarly, we can also
obtain the optimal grasp of the left hand. After optimizing the grasps
separately, we combine the grasps of bimanual hands as the final output of our
application.

##  4 Experimental Results

In this section, we describe experiments to demonstrate the effectiveness,
robustness and generalization of our proposed framework. We first introduce
the implementation details of our network architecture and then perform
ablation studies to verify the advantages of our network design. We also
conduct qualitative and comparative experiments to evaluate the performance of
the proposed network in terms of saliency map prediction, robustness to
manually labeling and generalization to novel objects, and bimanual grasp
synthesis. Finally, we assess the grasping capacity of our prediction on
virtual synthetic object models in simulation and give the statistics for
grasp success rates.

\begin{overpic}[width=433.62pt,tics=10]{ablation} \end{overpic} Figure 4:
Ablation Study. We compare our method to some possible alternatives to
demonstrate the rationality of our model and loss function. We show the
predicted saliency maps with various baseline variants and evaluate their
plausibility by BCACR metric.

###  4.1 Implementation details

We use the existing single-handed grasp saliency dataset Lau et al. (2016) to
train SSPN. All the $15$ household objects from this dataset, including Mug,
Pot, Pan and Tackle, were held out as a training set. We also select 40
objects from ShapeNet object set Chang et al. (2015) for these object
categories as a testing set. The training data of saliency corresponding
vectors are collected from human annotation and the input point clouds are
captured from the object models present in Lau et al. (2016) using the Blue
noise Sampling method Xu et al. (2012). We randomly sample 5000 surface points
for each object during the training of BSPN and BCPN. In the process of
iterative learning, we set up the hyperparameters $K=2000$ and $M=200$ for Mug
and Tackle, and $K=1500$ and $M=250$ for Pot and Pan. The neural network is
trained on a 2080Ti GPU.

The single-handed grasp saliency dataset. The dataset in Lau et al. (2016)
contains $15$ household object models and their single-handed grasp saliency
maps. This dataset is collected by asking online users to choose a point which
they prefer to grasp from a pair of points sampled on the model. The method
Lau et al. (2016) takes a learning-to-rank method to convert the ranking data
to the single hand grasp saliency value for each point.

Contact points annotation. We conducted a manual annotation of bimanual
contact points on the object models present in Lau et al. (2016).
Specifically, we invited $30$ users and ask them to utilize the CloudCompare
software D (2020) to mark the points where their both left and right hands
will contact when grasping the model. Specifically, in the CloudCompare
software, the user marks the contact area of the left and right hands on the
surface of the model mesh with a rectangular frame. After that, we sample 5000
points on the mesh, and the points located in the contact area are the contact
points.

Saliency corresponding vector. The ground truth of saliency corresponding
vectors are built using the labels of contact points $L$. In practice, for a
point $x_{i}$ labeled right-hand, we compute the vector $y_{j}-x_{i}$ with
each point $y_{j}$ labeled left-hand as saliency corresponding vector. Thus,
we can obtain a list of candidate vectors for $x_{i}$, and then we randomly
select $1000$ vectors from them to build the ground truth set of saliency
corresponding vectors. Similarly, we can also obtain the ground truth of
saliency corresponding vectors for each point $y_{i}$ labeled left-hand.

Grasp configuration. For grasp synthesis, we adopt MANO Romero et al. (2017)
hand models as robotic hands to generate grasp configuration based on our
bimanual saliency map. MANO hand is a parametric model of a human hand which
grasp is 16-DoF. We import it into GraspIt! Miller and Allen (2004) and make
it available for ContactGrasp method Brahmbhatt et al. (2019b).

###  4.2 Ablation study

To justify the rationality of the design choice of each component in our
network architecture, we compare our full pipeline against the following
baseline variants.

_Baseline1:_ The network only includes $E_{\theta_{1}}-D_{\theta_{3}}$
component, and it is trained with the saliency corresponding vector predicted
loss $L_{c}$. In the test phase, the saliency value of each point in the point
cloud is updated to the saliency value of its corresponding point directed by
the predicted saliency corresponding vector.

_Baseline2:_ The set of saliency corresponding vectors of all points labeled
right-hand is only adopted as the ground truth, instead of the original ground
truth used in the training phase of our network.

_Baseline3:_ The network contains both $E_{\theta_{1}}-D_{\theta_{3}}$ and
$E_{\theta_{2}}-D_{\theta_{4}}$ components, and it is trained without the
proposed loss function $L_{a_{2}}$.

_Baseline4:_ The calculation of the loss function $L_{a_{2}}$ dose not contain
$\psi(x)$ in the training phase.

_Baseline5:_ The network trained without the iterative learning strategy.

_Baseline6:_ The network is trained iteratively with correspondence loss
adjustment loss $L_{c}$ but without physics-balance loss $L_{p}$.

_Baseline7:_ The network is trained with all loss functions and the physics-
aware refinement module is removed during the testing phase.

To evaluate the plausibility of the predicted saliency maps in the ablation
studies, we propose the bimanual contact annotation coverage rate (BCACR),
measures the degree to which the bimanual grasp points derived from the
saliency map are included within the bimanual grasp regions annotated by human
observers. Specifically, we first count all annotated grasp contact points $C$
and then filter out grasp saliency points from the bimanual saliency map by a
given threshold $\tau_{c}$, where $\tau_{c}=0.7$. We count the number of grasp
saliency points $C_{cover}$ that have a bimanual-contact label of $1$ or $2$.
Thus, BCACR is defined as the percentage of grasp saliency points which are
covered by the bimanual contact annotated area among all annotated grasp
contact points:

$\text{BCACR}=\frac{C_{cover}}{C}\times 100\%$ (10)

The higher value of BCACR, the more plausible the result of saliency map.
Figure 4 shows the results of ablation studies quantitatively and
qualitatively, respectively.

As shown in Figure 4, our full method achieves better performance than the
other baselines, which is fit to both human grasp preference and physical
stability. Figure 4(a) indicates that the network cannot predict reasonable
bimanual saliency results only from the original single grasp saliency and
without the training process of saliency adjusted score. Furthermore, from
Figure 4(b) we can see that the insufficient training data of the saliency
corresponding vector has a negative impact on the prediction accuracy of
saliency corresponding vector, leading to the result is undesirable. Figure
4(c-d) show that when the loss function $L_{a_{2}}$ is turned off in the
network training stage, the $L_{a_{1}}$ becomes invalid and the saliency
values of points that outside the labeled contact regions become increase,
resulting in a large deviation with human grasp position selection. Notice
that our iterative training strategy plays a critical role in learning
bimanual saliency adjustment from original single hand saliency values. As
illustrated in Figure 4(e), the initial saliency value is not updated in the
training phase when the iteration strategy is not removed, which means that
the grasp saliency of both hands is not adjusted simultaneously. Furthermore,
Figure 4(f) shows that the results predicted by training without physics-
balance loss only conform to manual annotations, which cannot satisfy physical
balance. Figure 4(g) shows that for novel objects whose shapes is different
significantly from objects in the training set, physically plausible results
can’t be computed without physics-aware refinement, even if with the guidance
of the physics-balance loss at the training stage.

\begin{overpic}[width=433.62pt,tics=10]{qualitativeResults_new} \end{overpic}
Figure 5: Gallery of examples of grasp saliency maps and contact points
predicted by our approach. The points colored by blue and red denote the
predicted contact points of both hands, respectively.

###  4.3 Qualitative results and comparisons

Gallery. Our network can perform better prediction performance of both grasp
saliency map and contact positions of both hands. In Figure 5, we show a
gallery of results of household objects predicted by our network. For each
model, the contact points consist of a point set of left-hand contacts
(denoted by red spheres) and a point set of right-hand contacts (denoted by
blue spheres), which are predicted from the saliency map.

\begin{overpic}[width=433.62pt,tics=10]{Robustness_to_manual_annotation}
\put(6.0,-2.0){\small Left} \put(20.0,-2.0){\small Right}
\put(31.0,-2.0){\small Left} \put(41.0,-2.0){\small Right}
\put(56.0,-2.0){\small Left} \put(69.5,-2.0){\small Right}
\put(80.5,-2.0){\small Left} \put(90.5,-2.0){\small Right} \end{overpic}
Figure 6: Our method is robust for interference in manual annotation. We show
two object models with both left and right views respectively. The left
columns are the results of training network neither using physics-balance loss
nor physics-aware refinement module. The right columns are the results of
training network both using physics-balance loss and physics-aware refinement
module.

Manual annotation of bimanual contact points is a key step in influencing the
accuracy of the predicted corresponding vectors when training BSPN. In
general, an accurate annotation helps to improve the capacity for bimanual
saliency map prediction, but it is vulnerable to interference, such as human
misoperation in annotation. To prove the robustness of our method to manual
annotation, we show comparison results under annotation with noisy in Figure
6. Specifically, we randomly scramble the user’s bimanual contact point
annotations on the object surface as disturbed annotation. The ratio of
disturbed labeling to accurate labeling is 3:1. We train the models without
and with physics-balance loss on the noisy dataset, respectively. Moreover,
for the model which is trained without the physics-balance loss, we also do
not use the physics-aware refinement module during testing.

The results show that when the annotations are inaccurate, the model cannot
predict a reasonable bimanual saliency map without using physics-balance loss
and physics-aware refinement. However, when the physics-balance loss and the
physics-aware refinement are added, our method not only relies on annotations
but also considers physical criterion. Therefore, as illustrate in Figure 6,
our method can predict physically plausible bimanual saliency map even if the
manual annotations are under interference.

\begin{overpic}[width=433.62pt,tics=10]{unknown_object} \end{overpic} Figure
7: Our approach achieves physically plausible results even if when human
experience is not valid for novel objects. The two columns on the left show
the saliency maps before physics-aware refinement. The two columns on the
right show the results after physics-aware refinement.

Generalization to novel objects. Additionally, we also compared the
performance of our approach on the physics-aware refinement to evaluate
generalization to novel objects. To this end, we trained our model on the data
of mug category and tested it on other object categories. Figure 7 shows the
qualitative results before and after executing the physics-aware refinement.
We can see that the physics-aware refinement can notably improve
generalization ability to novel objects, which is mainly due to it guarantees
the physical balance for bimanual saliency map. In other word, our network is
able to generate physically plausible result, even if when human experience is
not valid for novel objects.

\begin{overpic}[width=433.62pt,tics=10]{qualitativeResults_3} \end{overpic}
Figure 8: Comparison of our method to the single-handed grasp saliency
analysis method on pan and teapot models. The two columns on the left show
saliency maps computed by the single-handed grasp saliency analysis method and
the right columns show grasp saliency maps computed by our method.

Comparison with the single-handed grasp saliency method. To verify the
necessary of jointly learning the grasping saliency of both hands, we compare
with a single-handed saliency analysis method SSPN Zhang et al. (2020).
Compared with SSPN, we consider the bimanual grasp saliency prediction as a
stacking problem of grasp saliency of left and right hands. We use SSPN to
compute both the right-handed and left-handed grasp saliency map.
Specifically, we build a small dataset to train SSPN network so as to compute
the left-handed grasp saliency map. We ask users to label contact points in
the point cloud where their left hands prefer to grasp when grasping with
single hand. Thus, we collect a total of 30 users’ annotations and compute the
probability that each point in the point cloud is selected by the user as a
left-handed saliency value. Using this small dataset, we train SSPN to obtain
the left-handed grasp saliency map. Therefore, we combine the right-handed
grasp saliency map with the left-handed grasp saliency map and normalize it as
the final result. Figure 8 shows the comparison results both on pan and teapot
models. Two columns on the left in Figure 8 show the saliency maps computed by
the SSPN and the right show the results computed by our method. In the Figure
8, the bimanual saliency map computed by SSPN is the same as that of the
single hand. It proves that grasping objects with the left or right hand in
isolation that are far from the human preference of grasping objects
cooperatively with both hands. In contrast, our method jointly predicts the
saliency values of both hands and can obtain more reasonable saliency maps.

\begin{overpic}[width=433.62pt,tics=10]{qualitativeResults_2}
\put(5.0,-2.0){\small(a) Vahrenkamp et al} \put(30.0,-2.0){\small(b) de Silva
et al} \put(55.0,-2.0){\small(c) GendexGrasp} \put(81.0,-2.0){\small(d) Ours}
\end{overpic} Figure 9: Bimanual grasps of the pot, vase, kitchenpot and pan
models. The first two column (a) and (b) show the results generated by
Vahrenkamp et al. (2011); de Silva and Suárez (2016). The column (c) show
results computed by GendexGrasp Li et al. (2023b). The last column (d) show
the bimanual grasp computed by method Brahmbhatt et al. (2019b) from our
predicted contact points of both hands.

Comparison with bimanual grasp synthesis. The existing literatures on bimanual
grasp planning only consider force-closure contact points on the object
surface. Therefore, it is difficult for them to synthesize grasp that is
amenable to human grasp experience. We compare our method with sampling-based
grasp optimization proposed by methods Vahrenkamp et al. (2011); de Silva and
Suárez (2016) and contact-based grasp synthesis proposed by GendexGrasp method
Li et al. (2023b), and evaluate the grasp quality by computing the generalized
Q1 analytic measure for grasp stability proposed by method Liu et al. (2020).
Figure 9 shows the comparison of bimanual grasp synthesis for four target
objects. The first two column (a) and (b) show the results generated by
Vahrenkamp et al. (2011); de Silva and Suárez (2016). The third column (c)
show results computed by GendexGrasp Li et al. (2023b). The last column (d) in
Figure 9 show the bimanual grasp computed by method Brahmbhatt et al. (2019b)
from our predicted contact points of both hands. It demonstrates that our
approach is helpful in generating grasps that are notably closer to the human
grasp preference than the physics-based analytic method.

\begin{overpic}[width=433.62pt,tics=10]{simulation} \end{overpic} Figure 10:
Progressive demonstrations of grasping objects with single dexterous hand and
both dexterous hands in simulation. The first row show the single-handed
grasping results generated by method Brahmbhatt et al. (2019b). The last two
row are the stages before and after using the physics-balance loss and
physics-aware refinement module.

###  4.4 Simulation experiments with dexterous hands

We further carry out our experiments in real-time physical simulator Coumans
and Bai (2016–2019) and quantitatively evaluate our approach with MANO hands
and the single-handed grasp planning method Brahmbhatt et al. (2019b) in terms
of grasping success rate. 80 shape models from 8 object categories (including
4 novel object categories) are selected for evaluation. Firstly, we compute
bimanual grasp candidates for each object and select the optimal one according
to the metric of ContactGrasp. As illustrated in Figure 10, we test if a grasp
is successful in the simulation environment by applying the generated bimanual
grasp and measuring the movement of object. We test each grasp by moving the
hands upwards to 15 cm above the ground and evaluate if the object tilts or
drops out during translation. We repeat this process for each category of
object and measure their grasping success rates.

We can see that if the physics-balance loss and physics-aware refinement
module are not used, the grasp will tilt or even fall, but both relying on
both the human experience and the physical criterion, the result achieves more
stable and reasonable. We also observe that when the object size is larger
relative to one hand, such as the pan in Figure 10, it’s very difficult to
perform successful grasp either by grasping the handle or body of the pan with
the single hand. However, the bimanual grasping is more stable than single-
handed grasping.

As table 1 illustrated, the generated grasps from our saliency map, on
average, achieves 92.5% success rate for all object categories, compared to an
average success rate of only 70.62% for single-handed grasping generated by
the method Brahmbhatt et al. (2019b), which suggests that the proposed method
can not only improve grasp quality in bimanual grasping, but also has notable
superiority than single-handed grasping.

Table 1: Grasping success rate.

Target Object’s Category | Mug | Pot | Pan | Tool | Kitchen Pot | Vase | Kettle | CAD Models
---|---|---|---|---|---|---|---|---
Single-handed Grasping Success Rate | 80% | 70% | 75% | 70% | 50% | 80% | 70% | 70%
Ours Grasping Success Rate | 100% | 100% | 100% | 80% | 90% | 100% | 90% | 80%

##  5 Conclusion and Discussion

In this work, we present a physics-aware iterative learning and prediction of
saliency map for bimanual grasp planning that can generate bimanual contact
points that are physically plausible for grasping balance and amenable to
human grasping preferences. Without relying on large training data of bimanual
grasp annotations, our method can predict bimanual grasp saliency from single-
handed grasp saliency, through learning saliency corresponding vectors that
conduct the relationships of grasp contacts between both hands by a small
amount of bimanual grasp annotations. This facilitates bimanual grasp
synthesis for household objects. Extensive experimental results on dexterous
hands exhibit the effectiveness of our method.

Our current solution still has some limitations. We leverage the optimizer to
synthesize grasps of both hands guided by bimanual saliency map. However, the
grasp optimizer cannot simultaneously generate grasps of both hands, which is
due to it needs a more complex grasp initialization. As a result, we have to
synthesize the grasp of each hand separately. Hence, in the future work, we
plan to integrate the bimanual grasp generation into the network for
performing bimanual grasp planning in end-to-end fashion.

## Acknowledgement

We thank the anonymous reviewers for their valuable comments. This work was
supported by NSFC programs (62272082), Emerging Interdisciplinary Cultivation
Project of Jiangxi Academy of Sciences (2022YXXJC0101).

## References

  * Amor et al. (2012) Amor, H.B., Kroemer, O., Hillenbrand, U., Neumann, G., Peters, J., 2012.  Generalization of human grasping for multi-fingered robot hands.  2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2043–2050.
  * Berenson and Srinivasa (2008) Berenson, D., Srinivasa, S.S., 2008.  Grasp synthesis in cluttered environments for dexterous hands.  Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots , 189–196.
  * Brahmbhatt et al. (2019a) Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J., 2019a.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 8701–8711.
  * Brahmbhatt et al. (2019b) Brahmbhatt, S., Handa, A., Hays, J., Fox, D., 2019b.  Contactgrasp: Functional multi-finger grasp synthesis from contact.  2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2386–2393.
  * Brahmbhatt et al. (2020) Brahmbhatt, S., Tang, C., Twigg, C.D., Kemp, C.C., Hays, J., 2020.  Contactpose: A dataset of grasps with object contact and hand pose, in: ECCV.
  * Caraza and Yun (1991) Caraza, J.A.N., Yun, X., 1991.  Two-handed grasping with two-fingered hands.  Fifth International Conference on Advanced Robotics ’Robots in Unstructured Environments , 597–602 vol.1.
  * Chang et al. (2015) Chang, A.X., Funkhouser, T.A., Guibas, L.J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F., 2015.  Shapenet: An information-rich 3d model repository.  ArXiv abs/1512.03012.
  * Chen et al. (2022) Chen, W., Liang, H., Chen, Z., Sun, F., Zhang, J.W., 2022.  Learning 6-dof task-oriented grasp detection via implicit estimation and visual affordance.  2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 762–769.
  * Coumans and Bai (2016–2019) Coumans, E., Bai, Y., 2016–2019.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  http://pybullet.org.
  * D (2020) D, G.M., 2020.  3d is here: Point cloud library (pcl).  CloudCompare (version 2.11.3) [GPL software] OpenSource Proj .
  * Dai et al. (2015) Dai, H., Majumdar, A., Tedrake, R., 2015.  Synthesis and optimization of force closure grasps via sequential semidefinite programming, in: International Symposium of Robotics Research.
  * Deng et al. (2021) Deng, S., Xu, X., Wu, C., Chen, K., Jia, K., 2021.  3d affordancenet: A benchmark for visual object affordance understanding.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 1778–1787.
  * Hayakawa et al. (2021) Hayakawa, S., Wan, W., Koyama, K., Harada, K., 2021.  A dual-arm robot that autonomously lifts up and tumbles heavy plates using crane pulley blocks.  IEEE Transactions on Automation Science and Engineering , 1–15.
  * Jiang et al. (2021) Jiang, H., Liu, S., Wang, J., Wang, X., 2021.  Hand-object contact consistency reasoning for human grasps generation.  2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 11087–11096.
  * Krishna and Murty (1999) Krishna, K., Murty, M.N., 1999.  Genetic k-means algorithm.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 29, 433–439.
  * Lau et al. (2016) Lau, M., Dev, K., Shi, W., Dorsey, J., Rushmeier, H.E., 2016.  Tactile mesh saliency.  ACM Transactions on Graphics (TOG) 35, 1 – 11.
  * Li et al. (2023a) Li, H., Lin, X., Zhou, Y., Li, X., Huo, Y., Chen, J., Ye, Q., 2023a.  Contact2grasp: 3d grasp synthesis via hand-object contact constraint arXiv:2210.09245.
  * Li et al. (2023b) Li, P., Liu, T., Li, Y., Geng, Y., Zhu, Y., Yang, Y., Huang, S., 2023b.  Gendexgrasp: Generalizable dexterous grasping , 8068--8074.
  * Liu et al. (2020) Liu, M., Pan, Z., Xu, K., Ganguly, K., Manocha, D., 2020.  Deep differentiable grasp planner for high-dof grippers .
  * Mandikal and Grauman (2021) Mandikal, P., Grauman, K., 2021.  Learning dexterous grasping with object-centric visual affordances.  2021 IEEE International Conference on Robotics and Automation (ICRA) , 6169--6176.
  * Miller and Allen (2004) Miller, A.T., Allen, P.K., 2004.  Graspit! a versatile simulator for robotic grasping.  IEEE Robotics and Automation Magazine 11, 110--122.
  * Przybylski et al. (2011) Przybylski, M., Asfour, T., Dillmann, R., 2011.  Planning grasps for robotic hands using a novel object representation based on the medial axis transform.  2011 IEEE/RSJ International Conference on Intelligent Robots and Systems , 1781--1788.
  * Qi et al. (2016) Qi, C.R., Su, H., Mo, K., Guibas, L.J., 2016.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  arXiv preprint arXiv:1612.00593 .
  * Romero et al. (2017) Romero, J., Tzionas, D., Black, M.J., 2017.  Embodied hands: Modeling and capturing hands and bodies together.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) .
  * Saut et al. (2010) Saut, J.P., Gharbi, M., Cortés, J., Sidobre, D., Siméon, T., 2010.  Planning pick-and-place tasks with two-hand regrasping.  2010 IEEE/RSJ International Conference on Intelligent Robots and Systems , 4528--4533.
  * Seo and Kumar (2012) Seo, J., Kumar, V.R., 2012.  Spatial, bimanual, whole-arm grasping.  2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , 257--264.
  * Rojas-de Silva and Suárez (2017) Rojas-de Silva, A., Suárez, R., 2017.  Contact force computation for bimanual grasps.  2017 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) , 1--6.
  * de Silva and Suárez (2016) de Silva, A.R., Suárez, R., 2016.  Grasping bulky objects with two anthropomorphic hands.  2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 877--884.
  * Smith et al. (2012) Smith, C., Karayiannidis, Y., Nalpantidis, L., Gratal, X., Qi, P., Dimarogonas, D.V., Kragic, D., 2012.  Dual arm manipulation - a survey.  Robotics Auton. Syst. 60, 1340--1353.
  * Song et al. (2016) Song, H.O., Fritz, M., Goehring, D., Darrell, T., 2016.  Learning to detect visual grasp affordance.  IEEE Transactions on Automation Science and Engineering 13, 798--809.
  * Vahrenkamp et al. (2009) Vahrenkamp, N., Berenson, D., Asfour, T., Kuffner, J.J., Dillmann, R., 2009.  Humanoid motion planning for dual-arm manipulation and re-grasping tasks.  2009 IEEE/RSJ International Conference on Intelligent Robots and Systems , 2464--2470.
  * Vahrenkamp et al. (2011) Vahrenkamp, N., Przybylski, M., Asfour, T., Dillmann, R., 2011.  Bimanual grasp planning.  2011 11th IEEE-RAS International Conference on Humanoid Robots , 493--499.
  * Wu et al. (2023) Wu, R., Zhu, T., Peng, W., Hang, J., Sun, Y., 2023.  Functional grasp transfer across a category of objects from only one labeled instance.  IEEE Robotics and Automation Letters 8, 2748--2755.
  * Xu et al. (2022) Xu, C., Chen, Y., Wang, H., Zhu, S.C., Zhu, Y., Huang, S., 2022.  Partafford: Part-level affordance discovery from 3d objects.  ArXiv .
  * Xu et al. (2012) Xu, Y., Hu, R., Gotsman, C., Liu, L., 2012.  Blue noise sampling of surfaces.  Computers & Graphics 36, 232--240.
  * Yu et al. (2021) Yu, Q., Shang, W., Zhao, Z., Cong, S., Li, Z., 2021.  Robotic grasping of unknown objects using novel multilevel convolutional neural networks: From parallel gripper to dexterous hand.  IEEE Transactions on Automation Science and Engineering 18, 1730--1741.  doi:10.1109/TASE.2020.3017022.
  * Zhang et al. (2020) Zhang, L., yao Wang, S., Zhou, J., Liu, J., Zhu, C., 2020.  3d grasp saliency analysis via deep shape correspondence.  Comput. Aided Geom. Des. 81, 101901.
  * Zhang et al. (2023) Zhang, Y., Hang, J., Zhu, T., Lin, X., Wu, R., Peng, W., Tian, D., Sun, Y., 2023.  Functionalgrasp: Learning functional grasp for robots via semantic hand-object representation.  IEEE Robotics and Automation Letters .
  * Zhu et al. (2021) Zhu, T., Wu, R., Lin, X., Sun, Y., 2021.  Toward human-like grasp: Dexterous grasping via semantic representation of object-hand.  2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 15721--15731.