# GRaCE: Optimizing Grasps to Satisfy Ranked Criteria
in Complex Scenarios

Tasbolat Taunyazov1, Kelvin Lin1, and Harold Soh1,2 1All authors are with the
Dept. of Computer Science, National University of Singapore {tasbolat, klin-
zw<EMAIL_ADDRESS>Systems Institute, National University of
Singapore

###### Abstract

This paper addresses the multi-faceted problem of robot grasping, where
multiple criteria may conflict and differ in importance. We introduce Grasp
Ranking and Criteria Evaluation (GRaCE), a novel approach that employs
hierarchical rule-based logic and a rank-preserving utility function to
optimize grasps based on various criteria such as stability, kinematic
constraints, and goal-oriented functionalities. Additionally, we propose
GRaCE-OPT, a hybrid optimization strategy that combines gradient-based and
gradient-free methods to effectively navigate the complex, non-convex utility
function. Experimental results in both simulated and real-world scenarios show
that GRaCE requires fewer samples to achieve comparable or superior
performance relative to existing methods. The modular architecture of GRaCE
allows for easy customization and adaptation to specific application needs.

##  I Introduction

We typically grasp an object with a particular goal in mind and this goal
affects the grasp. For example, it is natural to grasp a scissors by its
handle to cut something, but grasp it by the blade to safely pass it to
another individual. However, this goal is _not_ the only criteria the grasp
has to satisfy, _nor the most important_ ; if the blade was obstructed or
inaccessible (e.g., in Fig. 1), we would grasp the scissors by its handle even
if our intention was to hand it over. This simple example illustrates that
there are (i) _multiple criteria_ that affects the determination of a grasp
and (ii) there is an underlying _priority_ over these criteria. Here, the
grasp should be stable, reachable, and not result in a collision with other
objects (the mug): these criteria are more important than the aforementioned
goal (functional) objective.

In this work, we are motivated by the problem of generating grasps that
satisfy multiple criteria of differing importance. We apply hierarchical rule-
based logic to robot grasping [1] and introduce a grasp utility function that
is _rank-preserving_ [2], i.e., it assigns larger utility values to grasps
that satisfy higher ranked constraints. For example, robots are bound by their
kinematic and dynamic constraints, which limits whether a proposed grasp can
be performed. A stable grasp that satisfies these constraints should have
larger utility than one that sacrifices these criteria for a functionally
appropriate (but non-executable) grasp. Here, we take a probabilistic approach
and optimize the _utility_ of a grasp, where the probability of a grasp
satisfying a specific criteria is given by a classifier. Additional
classifiers can be added (or removed) depending on the precise requirements of
the application. This modular approach — which we call Grasp Ranking and
Criteria Evaluation (GRaCE) — enables a robot to trade-off multiple
conflicting criteria in complex contexts where not all desired objectives can
be satisfied.

Figure 1: Illustration of the expected grasp utility, $U$, used in GRaCE. The
blue region indicates higher utility values that are collision free and
stable.

Although the utility function enables scoring of grasps, it is a complicated
non-convex function to optimize, especially when the classifiers are
themselves complex (e.g., a deep neural network). Inspired by progress in
gradient-free methods (e.g., [3]), we propose GRaCE-OPT — a _hybrid_
optimization method that combines gradient-based and gradient-free
optimization. Specifically, GRaCE-OPT applies gradient-free optimization, an
Evolutionary Strategy (ES) [4], to conduct a more “diverse” exploration over
the landscape and prevent the optimization process from getting stuck at local
optima. However, on its own, this gradient-free method can be slow to
converge. As such, we use gradient-based optimization on a _lower-bound_ of
the utility to improve convergence speed. Experiments in complex environments
show that GRaCE requires significantly fewer samples and less time to achieve
comparable (or better) performance to a filtering method used in prior works
[5, 6]. Our evaluations involved two simulated grasping scenarios — shelf and
diner (Fig. 2) — in IsaacGym [7] and two real-world scenarios; these test
scenarios are designed to be challenging (cluttered with small optimal
grasping regions) and where the probability of satisfying multiple criteria
may be traded-off.

To summarize, this paper contributes GRaCE which comprises a utility function
that assigns higher values following user-specified hierarchical rules and an
optimization method that uses both gradient-free and gradient-based
optimization of the expected utility. Code and implementation details can be
found online at https://github.com/clear-nus/GRaCE.

##  II Background and Related Work

GRaCE is a modular framework for optimizing 6-DoF grasps. It builds upon
related work on 6-DoF grasp candidate generation and prior work on the
optimization of multiple criteria specified via rule/constraint hierarchies.
We briefly review these two topics in this section.

6-DoF Grasp Filtering and Refinement. Generating appropriate 6-DoF grasping
remains an active area of research. One common approach is to first _sample_ a
set of grasp, either through data-driven methods [8, 9], heuristics [5] or a
combination of both [6], then _filter_ the grasps using evaluators to select
the most promising candidates for execution. This sample-and-filter approach
is common and can be very effective in practice [5]. However, it can be time-
consuming in complex environments even with state-of-the-art samplers,
especially the optimal grasp regions are small.

An alternative approach to optimize grasps directly. Early work on multi-
fingered end-effector grasping [10] demonstrated that a scoring function for
grasp quality (a pre-trained classifier) can be used to optimize grasps toward
high quality regions. More recent work have applied optimization together with
sample-and-filter methods, e.g., GraspNet [8] optimizes/refines grasp samples
using the quality evaluator. These methods focus on a single quality
criterion, where else our work addresses the problem of trading-off multiple
conflicting criteria.

GRaCE can also be seen as a contrasting approach to “end-to-end” data-driven
6-DoF grasping [11, 12] where the sampler is trained to generate grasps that
satisfy multiple criteria. However, these methods require retraining when a
new criterion is added/removed, which is computationally expensive. GRaCE
enables the inclusion and removal of grasp criterion “on-the-fly”, which we
believe is more practical for many real-world applications. This aspect is
similar to very recent work [9] that refines grasps using gradient flows, but
GRaCE enables the ranking of multiple criteria and proposes a hybrid
optimization technique.

Hierarchical Optimization of Multiple Criteria. A key component of our
framework is a utility function, which leverages a rule hierarchy. Rule
hierarchies have a long history in optimization, with early works dating back
to 1967 [13]. More recent methods encode rule hierarchies using temporal logic
[14, 2]. Unlike these methods, our framework is differentiable and we do not
have to rely on external SAT solvers for optimization. Our work is closely
related to very recent research on planning with a rank-preserving reward
function for autonomous vehicles [15]. Our grasp utility function has a
similar structure to their proposed reward function, but our approach is
probabilistic; we optimize the expected rank of the grasp via a hybrid
optimization method.

##  III Ranking Grasps via Utility Functions

In this section, we present our approach for trading-off criteria for grasp
generation. A grasp $\mathbf{g}$ is typically defined as a set of contact
points with an object which restricts movement when external forces are
applied [16]. For simplicity, we will refer to end-effector poses as grasps
(or grasp candidates) and denote them as $\mathbf{g}$, even if they do not
satisfy the definition above (e.g., the pose does not make contact with the
object). We first discuss how grasp criteria can be ranked, followed by our a
utility function, and, finally, formulate an optimization based grasp
generation method.

TABLE I: Formulas and Grasp Criteria with Associated Probability. Priority | Rule |  Probability
---|---|---
1 | $\phi^{(1)}=\bigwedge_{j=1}^{M_{1}}c_{j}^{(1)}$ | $\prod_{j=1}^{M_{1}}p_{j}^{(1)}$
$\vdots$ | $\vdots$ | $\vdots$
$N$ | $\phi^{(N)}=\bigwedge_{j=1}^{M_{N}}c_{j}^{(N)}$ |  $\prod_{j=1}^{M_{N}}p_{j}^{(N)}$

Criteria, Priority, and Rules. We define a grasp criterion as a predicate
$c^{(i)}_{j}(\mathbf{g})$ where $i\in\\{1,...,N\\}$ is the criterion’s
priority (with descending importance) for a grasp $\mathbf{g}$ and $j$ is an
index of criterion, $j=1,\dots,M_{i}$. $M_{i}$ is a number of criteria with
the same priority $i$. A rule $\phi^{(i)}(\mathbf{g})$ is defined as a
conjunction of criteria
$\phi_{i}(\mathbf{g})=\bigwedge_{j}^{M_{i}}c_{j=1}^{(i)}(\mathbf{g})$.

Let $p_{j}^{(i)}:=P(c_{j}^{(i)}(\mathbf{g})|\mathbf{o})$ be the probability
that criterion $c^{(i)}_{j}(\mathbf{g})$ is satisfied under observed context
$\mathbf{o}$. For notational simplicity, we will drop the explicit dependence
of $p_{j}^{(i)}$, $c_{j}^{(i)}$ and $\phi^{(i)}$ on $\mathbf{g}$ and
$\mathbf{o}$. We assume that criteria are conditionally independent given the
grasp and context. As such, the probability that a rule $\phi^{(i)}$ is
satisfied is given by $\prod_{j=1}^{M_{i}}p_{j}^{(i)}$. Table I shows a list
of priorities, rules, and their associated probabilities.

Rule Hierarchy and Rank of a Grasp. A rule hierarchy $\psi$ is defined as a
sequence of rules $\psi:=\\{\phi^{(i)}\\}_{i=1}^{N}$. The rule hierarchy
induces a total order on the grasps, enabling us to rank grasps. A grasp that
satisfies all the rules has the highest rank, i.e., rank 1. A grasp that
satisfies all the rules except the lowest priority rule has rank 2. This
continues on, with grasps satisfying none of the rules having the lowest rank.
Formally, we define a rank of a grasp as:

###### Definition 1

Let $\psi$ to be rule hierarchy with $N$ rules. Let
$\emph{eval}:\phi^{(i)}\mapsto\\{0,1\\}$ be a function that evaluates rule
$\phi^{(i)}$ to be 1 if the rule is satisfied, 0 otherwise. Then the rank of
the grasp $r:\operatorname{\mathbf{\mathcal{G}}}\mapsto\mathbb{R}$ is defined
as:

$r(\mathbf{g}):=2^{N}-\sum_{i=1}^{N}2^{N-i}\emph{eval}(\phi^{(i)})$

TABLE II: Rank-Preserving Grasp Utility $r(\mathbf{g})$ |  Satisfied Rules |  Probability
---|---|---
$1$ | $\bigwedge_{i=1}\phi^{(i)}$ | $\prod_{j=1}^{M_{1}}p_{j}^{(1)}\cdots\prod_{j=1}^{M_{N}}p_{j}^{(N)}$
$\vdots$ | $\vdots$ | $\vdots$
$2^{N}$ | $\bigwedge_{i=1}\neg\phi^{(i)}$ |  $(1-\prod_{j=1}^{M_{1}}p_{j}^{(1)})\cdots(1-\prod_{j=1}^{M_{N}}p_{j}^{(N)})$

Table II summarizes grasp ranks for the rule hierarchy and our utility is
defined as the negative expected rank,

$U(\mathbf{g})=-\mathbb{E}_{\psi}[r(\mathbf{g})]=\sum_{i=1}^{N}2^{N-i}\prod_{j=1}^{M_{i}}p_{j}^{(i)}-2^{N}$
(1)

This simplified form can be obtained by observing that
$\text{eval}(\phi^{(i)})$ is a Bernoulli variable with probability
$\prod_{j=1}^{M_{i}}p_{j}^{(i)}$,

$\displaystyle U(\mathbf{g})$
$\displaystyle=-\mathbb{E}_{\psi}[r(\mathbf{g})]$
$\displaystyle=-\mathbb{E}_{\psi}[2^{N}-\sum_{i=1}^{N}2^{N-i}\text{eval}(\phi^{(i)})]$
(by definition)
$\displaystyle=-2^{N}+\sum_{i=1}^{N}2^{N-i}\mathbb{E}_{\psi}[\text{eval}(\phi^{(i)})]$
$\displaystyle\text{ (by linearity of }\mathbb{E}\text{)}$
$\displaystyle=-2^{N}+\sum_{i=1}^{N}2^{N-i}\prod_{j=1}^{M_{i}}p_{j}^{(i)}$

Problem Statement. We seek to find a grasp that maximizes the utility
function:

$\mathbf{g}^{*}=\arg\max_{\mathbf{g}}U(\mathbf{g})$ (2)

The key challenge is that Eq. (2) is a non-convex function of the grasps with
local optima that can trap standard gradient-based methods. Moreover, the
multiplication of probabilities leads to numerical instabilities with
vanishing gradients when used with neural classifiers [17, 18]. In the next
section, we describe how to optimize this function using GRaCE.

##  IV Hybrid Optimization of Grasps

In this section, we introduce GRaCE-OPT, a hybrid optimization technique that
leverages both gradient-free and gradient-based methods to optimize Equation
(2). As an initial step, we considered optimizing a lower-bound of Eq. 2 using
Jensen’s inequality:

$\displaystyle\log U(\mathbf{g})$
$\displaystyle=\log\left(\sum_{i=1}^{N}2^{N-i}\prod_{j=1}^{M_{i}}p_{j}^{(i)}\right)$
$\displaystyle\geq\sum_{i=1}^{N}\left(\log\left(2^{N-i}\prod_{j=1}^{M_{i}}p_{j}^{(i)}\right)\right)$
$\displaystyle=\sum_{i=1}^{N}\sum_{j=1}^{M_{i}}\log p_{j}^{(i)}\triangleq
L(\mathbf{g})$ (3)

Empirically, we find $L(\mathbf{g})$ to be easier to optimize and numerically
stable, but inspection of its form clearly shows that it is no longer rank
preserving since the utilities are factored out.

As such, we only use this gradient-based optimization as an inner-loop within
a gradient-free ES setup, shown in Algorithm 1 below. We assume that we have
access to a grasp sampler $q_{0}$ from which we can sample initial grasps
$\mathbf{G}_{0}$ from (line 1). In practice, $q_{0}$ can be any grasp
candidate sampler, e.g., GraspNet-VAE [8] or a heuristic sampler such as GPD
[5]. We then optimize these grasps over $T$ outer gradient-free iterations
(lines 2-10). In detail: new batches of grasps are sampled using a
multivariate Gaussian distribution with mean $\mathbf{G}_{t}$ and covariance
matrix $\Sigma$. The covariance $\Sigma$ manually selected in our work but can
also be adaptive [19]. Lines 4 to 6 optimizes the lower bound $L(\mathbf{g})$.
Line 8 and 9 assesses grasps using $U(\mathbf{g})$ and selects the top $Q$
grasps. In preliminary experiments, we found GRaCE-OPT to be superior to using
either gradient-based or gradient-free methods alone.

Algorithm 1 GRaCE-OPT

1:grasp sampler $q_{0}$, utility $U(\mathbf{g})$, lower bound for utility
$L(\mathbf{g})$, number of update steps ($T$), covariance matrix ($\Sigma$),
step size ($\eta$), number of update steps for lower bound ($K$), selection
size ($Q$).

2:$\mathbf{G}_{0}\sim q_{0}$

3:for $t\leftarrow 1$ to $T$ do

4: $\mathbf{\tilde{\mathbf{G}}}_{0}\sim\mathcal{N}(\mathbf{G}_{t},\Sigma)$

5: for $k\leftarrow 1$ to $K$ do // Optimize new samples

6:
$\tilde{\mathbf{G}}_{t}=\tilde{\mathbf{G}}_{t-1}+\eta\nabla_{\tilde{\mathbf{G}}_{t-1}}L(\tilde{\mathbf{G}}_{t-1})$

7: end for

8: $\mathbf{G}^{\text{combined}}$ = concat[$\mathbf{G}_{t}$,
$\mathbf{\tilde{G}}_{L}$]

9:
$\mathbf{G}^{\text{sorted}}=\operatorname*{arg\,sort}_{\mathbf{G}}U(\mathbf{G}^{\text{combined}})$

10: $\mathbf{G}_{t}=(\mathbf{G}^{\text{sorted}})_{1:Q}$ // Select top $Q$
grasps

11:end for

12:return $\mathbf{G}_{T}$ // Optimized grasp

##  V Criteria for successful 6-DoF grasps

In this section, we describe different grasp criteria used in our experiments.
We assume a setup where a human user is asking the robot to perform a task,
e.g., to “handover the scissors”. The robot has access to the natural language
utterance from the human as well as observations of the environment (a point
cloud).

As previously mentioned, we assume a probabilistic setup where the probability
of criteria satisfaction is given by a classifier
$P(c^{(i)}_{j}(\mathbf{g})|\mathbf{o})$. We used four different classifiers
that capture different quality aspects of a grasp: stability, executability,
collision-free, and functional. We discuss these classifiers at a high-level
and leave implementation details to the online supplementary
material111https://github.com/clear-nus/GRaCE

Stability Classifier (S). We use the stability evaluator in [9]. The
classifier takes as inputs a grasp pose and a point cloud of the object, and
outputs a prediction of grasp stability.

Execution Classifier (E). Our execution classifier captures two important
aspects of robot poses: reachability map [20] and kinematic singularity [21]:

We calculate the manipulability score for a given grasp:

$\omega(\bm{\theta})=\sqrt{\det{\mathbf{J(\bm{\theta})J(\bm{\theta})^{T}}}}\geq
0$ (4)

where $\mathbf{J(\bm{\theta})}$ is the Jacobian matrix and $\bm{\theta}$ is
the Inverse Kinematics (IK) solution. Then, we define the predicted grasp
pose, $\tilde{\mathbf{g}}$, using the IK solution:

$\tilde{\mathbf{g}}=\text{FK}(\bm{\theta})$

Finally, we combine this two quantities to yield,

$p(\text{eval}(c_{e}(\mathbf{g}))=1)=\begin{cases}\sigma(-C_{m}d(\mathbf{g},\tilde{\mathbf{g}})),&\text{if
}\mathbf{g}\neq\tilde{\mathbf{g}}\\\
\sigma(C_{w}(\omega({\bm{\theta}})-\omega_{\text{th}})),&\text{otherwise}\end{cases}$
(5)

where $C_{m}$ and $C_{w}$ are scaling coefficients, $\omega_{\text{th}}$ is a
lowest manipulability threshold that allows safe grasp execution, and
$d(\cdot,\cdot)$ is a distance function between predicted grasp pose from the
IK solution and current pose calculated in SE(3) [22].

Collision Detection Classifier (C). The backbone of our Collision Detection
Classifier is the 3-D Signed Distance Function (SDF) [23]. For simplicity, we
use the original version of SDF that is designed for convex objects. Let
${\mathcal{X}}\in{\mathbb{R}}^{3}$ represent a point cloud represented with
respect to the world frame and $\mathbf{x}_{k}\in{\mathcal{X}}$ be a point
within ${\mathcal{X}}$. The SDF for the box $R_{i}$ is defined as

$d_{R_{i}}=\frac{1}{|{\mathcal{X}}|}\sum_{k=1}^{k=|{\mathcal{X}}|}\|\max(|\mathbf{x}|-\mathbf{H},0)\|_{2}$
(6)

where $\mathbf{H}\in{\mathbb{R}}^{3}$ is the half-extent of the box in
Cartesian coordinates. We decompose the gripper into three boxes $R_{1}$,
$R_{2}$ and $R_{3}$.The SDF is differentiable and we use it to create our
collision detection classifier:

$P(\text{eval}(c_{c}(\mathbf{g}))=1|\mathbf{o})=\sigma\left(C_{c}(d_{\text{th}}-\frac{1}{3}\sum_{i=1}^{i=3}d_{R_{i}})\right)$
(7)

where $d_{\text{th}}$ is a user-defined threshold and $C_{c}$ is a scale
coefficient.

Intention Classifier (N). Our intention classifier outputs the probability
that the grasp location coincides with their intent.

We first extract the user’s intent (e.g., “Handover”) from their utterance
(e.g., “Hand over the knife”) using JointBERT [24]. Our JointBERT model is
trained on a curated dataset of programmatically generated queries and
evaluated on sentences surveyed from test users. To evaluate if the grasp
matches the intention, we use TaskGrasp [25] as it can identify affordance-
rich and affordance-poor regions of objects. TaskGrasp evaluates grasps with
respect to the point cloud and task, and outputs
$P(\text{eval}(c_{n}(\mathbf{g}))=1|\mathbf{o})$. As TaskGrasp inherently
assumes that all grasps are stable before inference, we lower the score to
zero if the grasp is more than 3cm away from the nearest point in the point
cloud; we find that this modification helps to reduce false positives.

Summary and Ranking. The above classifiers are all differentiable and
gradients can be obtained using modern auto-differention libraries such as
PyTorch [26]. In our experiments, we rank the criteria as follows: the
S-classifier has rank 1, the E-classifier and C-classifier have rank 2, and
the N-classifier has rank 3.

##  VI Simulation Experiment

Figure 2: Shelf and Diner benchmark environments with sample grasps with high
utility.

The goal of our experiments is to establish if GRaCE is able to (i)
effectively generate suitable grasps in a complex environment, and (ii) trade-
off multiple criteria. In particular, GRaCE is more computationally expensive
compared to a simple sample-and-filter approach (principally due to gradient
computation). Is this added cost justified? Moreover, are the multiple
criteria necessary for finding successful grasps and if so, can they be
traded-off effectively? To answer these questions, we conducted the following
experiments.

###  VI-A Simulated Environments

GRaCE was evaluated in simulation using IsaacGym from NVIDIA [7]. IsaacGym is
a state-of-the-art simulator, capable of simulating robot movement and object
grasping. We engineered two simulation environments, namely a Shelf module and
a Diner module:

  * •

Shelf consists of a two-layered shelf with common everyday items placed on
both layers. The Shelf module is designed to be cluttered and complex. Hence,
the optimal grasping region for each object is confined to a small area,
reducing the effectiveness of sampling-based methods.

  * •

Diner consists of items that may be present in a typical dining setup, such as
bowls, forks, a pan, and a spatula.

The graspable objects in these environments are from the ShapeNet dataset, and
the shelves and tables were created using Blender. We packaged the Shelf and
Diner modules as a set of OBJ files that can be loaded into any simulator
capable of importing OBJ meshes.

Figure 3: Results on Experiments on the Shelf (top) and Diner (bottom)
Environments. The bar graphs show averages with standard deviation as error-
bars. Using 50 samples, GRaCE outperforms Filter (5000 samples) and takes less
computational time.

###  VI-B Experiment Process

Perception. To carry out GRaCE, we first record point cloud data through
IsaacGym’s simulated depth and segmentation cameras from multiple views, and
segment out the target object from the environment.

Grasp Sampling and Optimization. GraspNet is then used to sample grasps and
optimized with GRaCE. The resulting output is a list of grasp poses, along
with their utility scores.

Grasp Planning, Execution, and Evaluation. The optimized grasps are passed to
Moveit! [27] to generate trajectory plan. To minimize collision, instead of
planning to the grasp pose, we plan the trajectory to a pre-grasp
configuration 5cm linearly behind the actual grasp pose. The robot performs
the grasping by moving the end-effector towards the object and closing its
grippers. To execute the plan, we use a configuration-space controller to
closely mimic and execute the planned trajectories. As IsaacGym is
deterministic across sessions, only one execution attempt of each trajectory
was performed. A grasp was termed as _successful_ if the target object is held
by the gripper fingers after the trajectory was executed. Note that this
measure of success excludes the intention criteria (which is subjective and
handled separately).

Baseline Methods. We compared the following methods:

  * •

GRaCE as described with all four classifers.

  * •

GRaCE with only the S-classifier. This ablation uses a single criteria
(stability) and is similar to the refinement used in GraspNet [8] and its
variants.

  * •

Ablations of GRaCE by removing criteria, e.g., SE denotes that only the
stability and execution classifier were used. These ablations enable us to see
if excluding important criteria leads to more failures.

  * •

Sample-and-Filter, termed as “Filter”, is a popular approach due to its
simplicity and ease of implementation.

###  VI-C Results

In this section, we summarize our main findings. In general, we find GRaCE to
be superior to filtering on both the Shelf and Diner scenarios. Moreover, it
is able to prioritize important criteria to find higher utility grasps.

Is optimization really necessary? Does GRaCE outperform Sample-and-Filter?  We
evaluated GRaCE (SEC) against sample-and-filter with different sample sizes
(10, 50, 100). Fig. 3 shows the average number of successes per object for the
top-10 grasps across the different objects in the Shelf and Diner environments
(seven and five objects, respectively). Note that the intention criterion was
excluded as compliance with user intent involves subjective evaluation.

Focusing on the subplots in Fig. 3(a) and 3(f), we observe that GRaCE
outperforms Filter across the same sampling sizes (10, 50, 100). We further
ran Filter with larger sample sizes (1000 and 5000), which enabled it to
achieve attain similar performance to GRaCE. At 5000 samples, Filter performs
similarly to GRaCE using 50 samples. However, this also resulted in it
requiring longer almost 2x longer compute times as shown in Figs. 3(b) and
3(g). In short, although GRaCE is more expensive _per sample_ , it is able to
achieve better grasps with fewer samples.

Can GRaCE optimize multiple criteria to find successful grasps? The results of
our GRaCE ablations are shown in Figs. 3(c) and 3(h). We observe that the
using all three classifiers (SEC) resulted in the best performance. The marked
increase in performance from SE to SC may be attributed to the cluttered
nature of Shelf and Diner, where many candidate grasp poses can collide with
other objects.

Figure 4: Selected objects for intention evaluation.

Does GRaCE with the intention classifier generate successful functional
grasps? More precisely, we sought to evaluate if (i) GRaCE would generate
grasps in regions matching the user intent if the higher-ranked criteria can
be satisfied, and (ii) prioritize the higher-ranked criteria, even if the
resulting grasp has violates the the functional criteria. To that end, we
selected four objects (shown in Fig. 4) and paired with pan and spatula with
the “Use” intention, and the scissors and fork with the “handover” intention.

Figure 5: Incorporating the intention classifier (SECN) shifts grasps towards
functional regions.

Fig. 5 shows the grasps generated with and without the intention criteria. To
elaborate, the spatula can be separated into two regions: handle, which is
ideal to grasp for “use”, and the head, which should not be grasped for this
purpose. Notably, both of these regions satisfy the stability, executable, and
collision-free criteria. We see that GRaCE using only SEC generated grasps in
both regions, while GRaCE with SECN produced grasps only at the handle.
Similar grasps can be observed for the pan. Turning our attention to the
“handover” intent, the scissors and fork are in placements that limit access
to regions that have coincide with the “handover” intention. In this case, we
observe GRaCE (with SECN) to forgoes these regions and instead produces grasps
that satisfy the other, more highly ranked, criteria (examples in Fig. 6).

Figure 6: GRaCE produces grasps that prioritize the higher ranked criteria,
sacrificing functional regions for stable, executable, and collision free
grasps.

##  VII Real-world Experiments

Thus far, we have discussed GRaCE in simulation settings, but does GRaCE’s
performance carry over to the real world? We conducted real-world tests
comparing GRaCE against the filter baseline, similar to the simulation setup.
We tasked a real-world Franka Emika Panda robot (with an RGB-D camera) to
grasp objects in two different scenarios (Fig. 7):

  * •

Box, where the robot was tasked to generate grasps for 10 different items in a
clutter. Here, there is no intention criteria and the goal was to execute a
stable grasp and lift the object.

  * •

Bowl, where the robot attempted to grasp one of three different items (a
wooden spoon, a knife, or a screwdriver) with the intent to handover the
object. A grasp was successful if the robot managed to lift the object out of
the bowl and hand it over to the experimenter.

Both these settings are challenging due to (i) noisy perception and (ii) the
feasible grasp region for each object was generally small due to the clutter.
In each experiment, we conducted 10 trials for each object to be grasped and
recorded the number of grasp successes; in total, our experiment involved 260
real-world grasps. We set GRaCE to use 50 samples, while Filter used 1000
samples. Both methods have comparable timings; GRaCE took an average of $14$
seconds to obtain a grasp, while Filter took $12$ seconds. Note that these
timings are for un-optimized Python implementations and future work can look
into reducing this computation time.

Does GRaCE find successful multi-criteria grasps in real-world scenarios? Our
results, summarized in Table III, show that GRaCE outperforms Filter in both
domains by a significant margin. In both cases, GRaCE achieves approximate
double the success rate of Filter. Qualitatively, we found GRaCE to more
reliably return a feasible grasp; in contrast, Filter failed to return _any_
suitable grasp in 26 out of the 130 trials (20%). Other failures in both cases
were commonly due to perception errors and robot trajectories executed near
singular configurations, leading to grasp offsets, collisions, and robot
errors. Overall, our findings affirm that GRaCE sustains its performance in
real-world conditions.

Figure 7: Real world items, along with the box and bowl scenarios. TABLE III: Real-World Grasp Experiments: Average Success Rates with Standard Deviation in Brackets. Method | Box | Bowl
---|---|---
GRaCE | 65% (0.10) | 57% (0.06)
Filter | 31% (0.12) | 33% (0.06)

##  VIII Conclusions and Future Work

In this study, we introduced GRaCE, a modular framework designed for
optimizing robotic grasps based on multiple, often conflicting, criteria. Our
experimental evaluations show GRaCE’s efficacy in generating high-quality
grasps in complex, cluttered environments. Several avenues for improvement
emerge. First, extending GRaCE to incorporate additional criteria, such as
tactile data for improved grasping of soft or deformable objects, holds
promise. Second, the computational efficiency of GRaCE can be further
enhanced; specifically, the computational burden of gradient calculations may
be mitigated through the use of optimized classifiers.

##  IX Acknowledgements

This research is supported by the National Research Foundation Singapore and
DSO National Laboratories under the AI Singapore Programme (AISG Award No:
AISG2-RP-2020-017).

## References

  * [1] M. Wilson and A. Borning, “Hierarchical constraint logic programming,” _The Journal of Logic Programming_ , vol. 16, no. 3-4, pp. 277–318, 1993.
  * [2] J. Tuumova, L. I. R. Castro, S. Karaman, E. Frazzoli, and D. Rus, “Minimum-violation ltl planning with conflicting specifications,” in _2013 American Control Conference_. IEEE, 2013, pp. 200–205.
  * [3] A. Pourchot and O. Sigaud, “Cem-rl: Combining evolutionary and gradient-based methods for policy search,” _arXiv preprint arXiv:1810.01222_ , 2018.
  * [4] H.-G. Beyer and H.-P. Schwefel, “Evolution strategies–a comprehensive introduction,” _Natural computing_ , vol. 1, pp. 3–52, 2002.
  * [5] A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,” _The International Journal of Robotics Research_ , vol. 36, no. 13-14, pp. 1455–1473, 2017.
  * [6] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang, F. Sun, and J. Zhang, “Pointnetgpd: Detecting grasp configurations from point sets,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3629–3635.
  * [7] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021.
  * [8] A. Mousavian, C. Eppner, and D. Fox, “6-dof graspnet: Variational grasp generation for object manipulation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2901–2910.
  * [9] T. Taunyazov, H. Zhang, J. P. Eala, N. Zhao, and H. Soh, “Refining 6-dof grasps with context-specific classifiers,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2023.
  * [10] Y. Zhou and K. Hauser, “6dof grasp planning by optimizing a deep learning scoring function,” in _Robotics: Science and systems (RSS) workshop on revisiting contact-turning a problem into a solution_ , vol. 2, 2017, p. 6.
  * [11] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox, “Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [12] H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large-scale benchmark for general object grasping,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 444–11 453.
  * [13] F. Waltz, “An engineering approach: hierarchical optimization criteria,” _IEEE Transactions on Automatic Control_ , vol. 12, no. 2, pp. 179–180, 1967.
  * [14] R. Dimitrova, M. Ghasemi, and U. Topcu, “Maximum realizability for linear temporal logic specifications,” in _Automated Technology for Verification and Analysis: 16th International Symposium, ATVA 2018, Los Angeles, CA, USA, October 7-10, 2018, Proceedings 16_. Springer, 2018, pp. 458–475.
  * [15] S. Veer, K. Leung, R. Cosner, Y. Chen, and M. Pavone, “Receding horizon planning with rule hierarchies for autonomous vehicles,” _arXiv preprint arXiv:2212.03323_ , 2022.
  * [16] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” in _Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065)_ , vol. 1. IEEE, 2000, pp. 348–353.
  * [17] S. Hochreiter, “The vanishing gradient problem during learning recurrent neural nets and problem solutions,” _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ , vol. 6, no. 02, pp. 107–116, 1998.
  * [18] B. Hanin, “Which neural net architectures give rise to exploding and vanishing gradients?” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [19] C. Igel, N. Hansen, and S. Roth, “Covariance matrix adaptation for multi-objective optimization,” _Evolutionary computation_ , vol. 15, no. 1, pp. 1–28, 2007.
  * [20] A. Makhal and A. K. Goins, “Reuleaux: Robot base placement by reachability analysis,” in _2018 Second IEEE International Conference on Robotic Computing (IRC)_. IEEE, 2018, pp. 137–142.
  * [21] M. Rubagotti, T. Taunyazov, B. Omarali, and A. Shintemirov, “Semi-autonomous robot teleoperation with obstacle avoidance via model predictive control,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2746–2753, 2019.
  * [22] C. Belta and V. Kumar, “Euclidean metrics for motion generation on se (3),” _Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science_ , vol. 216, no. 1, pp. 47–60, 2002.
  * [23] T. Chan and W. Zhu, “Level set based shape prior segmentation,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 2. IEEE, 2005, pp. 1164–1170.
  * [24] Q. Chen, Z. Zhuo, and W. Wang, “Bert for joint intent classification and slot filling,” 2019. [Online]. Available: https://arxiv.org/abs/1902.10909
  * [25] A. Murali, W. Liu, K. Marino, S. Chernova, and A. Gupta, “Same object, different grasps: Data and semantic knowledge for task-oriented grasping,” in _Conference on Robot Learning_ , 2020.
  * [26] S. Imambi, K. B. Prakash, and G. Kanagachidambaresan, “Pytorch,” _Programming with TensorFlow: Solution for Edge Computing Applications_ , pp. 87–104, 2021.
  * [27] S. Chitta, I. Sucan, and S. Cousins, “Moveit! [ros topics],” _IEEE Robotics & Automation Magazine_, vol. 19, no. 1, pp. 18–19, 2012.