# Research Challenges and Progress in Robotic Grasping and Manipulation
Competitions

Yu Sun, Joe Falco, Máximo A. Roa, and Berk Calli Accepted for publication by
IEEE Robotics and Automation Letters.Yu Sun is with the University of South
Florida, Joe Falco is with the National Institute of Standards and Technology
(NIST), Máximo A. Roa is with the German Aerospace Center (DLR), and Berk
Calli is with Worcester Polytechnic Institute (WPI). E-mails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper discusses recent research progress in robotic grasping and
manipulation in the light of the latest Robotic Grasping and Manipulation
Competitions (RGMCs). We first provide an overview of past benchmarks and
competitions related to the robotics manipulation field. Then, we discuss the
methodology behind designing the manipulation tasks in RGMCs. We provide a
detailed analysis of key challenges for each task and identify the most
difficult aspects based on the competing teams’ performance in recent years.
We believe that such an analysis is insightful to determine the future
research directions for the robotic manipulation domain.

###### Index Terms:

Grasping; Dexterous Manipulation; Performance Evaluation and Benchmarking

##  I Introduction

Robotic grasping and manipulation as a research field had tremendous growth in
the last decade. Researchers have made significant progress in different areas
that prevented robots from reliably handling household items, mechanical
parts, and packages. The progress in robotic grasping and manipulation has
shown new application promises that led to a renewed interest in robotics from
the general public, industry, and government agencies. Nevertheless, the
growth and progress have not been even. Some challenges receive a great deal
of well-deserved attention because they are either obvious or standing in the
way of big commercialization potential. Some challenges might not be as
popular and remain unsolved for decades, but they could be crucial for many
applications. Some challenges may have changed or vanished because a new kind
of hardware becomes available or an engineering solution became adequate. A
comprehensive overview of the major challenges not only helps us analyze the
history of the robotic grasping and manipulation field, but also allows us to
determine future research directions. We should keep track of the challenges,
their changes, and the progress made to solve them. Competitions rooted in
real-life applications could be an ideal vehicle for this purpose.

This paper discusses recent progress in the robotic manipulation field based
on the recent Robotic Grasping and Manipulation Competitions (RGMC)111Certain
commercial entities and items are identified in this paper to foster
understanding. Such identification does not imply recommendation or
endorsement by the National Institute of Standards and Technology, nor does it
imply that the materials or equipment identified are necessarily the best
available for the purpose.. For defining the RGMCs, we designed over 40
grasping and manipulation tasks that reflect realistic scenarios in service
robotics and industry. Each task has a detailed setup and requirement
description and scoring rules. The task details can be found on the
competition webpages [1, 2, 3, 4] and [5, 6]. This paper presents the
challenges and the progress of the competing teams, identifies critical areas
preventing better performance in robotic manipulation, and provides
observations regarding future research directions.

##  II Background

###  II-A Related Robotic Grasping and Manipulation Benchmarks

Establishing experimentation methodologies that allow comparison across
different research groups is still a pending challenge in robotics. The large
variety in robotic platforms, setups, and software implementations poses
numerous difficulties to achieve common experiment protocols. Even the
pressure to publish plays a role in this lack of comparison, as applying the
benchmarking protocols is a thorough and time-consuming process, which is
sometimes neglected under the pressure of publication deadlines. In this
sense, initiatives such as the Reproducible Articles (R-articles) of the IEEE
Robotics & Automation Magazine (RAM) [7] aim to encourage easy reproduction of
results via adopting and/or presenting detailed experiment protocols. Several
initiatives of benchmarks for grasping and manipulation have been proposed,
focusing on different levels of the manipulation system:

TABLE I: Robotic competitions involving manipulation Competition/Challenge |  Years | Tasks* | Focus** |  Description
---|---|---|---|---
|  | Fi | Fr | Gr | Ma | As | Mm |
RoboCup@Home [8] |  2006-2021 | X | X | X | X |  | X |  Domestic service tasks, different fixed challenges each year
DARPA Robotics Challenge [9] |  2012-2015 | X |  | X | X |  | X |  Disaster response scenarios
RoCKIn [10] |  2014, 2015 | X |  | X | X |  | X |  Two tracks: RoCKIn@Home (domestic environment) and RoCKIn@Work (factory environment)
European Robotic Challenge (EuRoC) [11] |  2014-2017 |  | X | X | X |  | X |  Reconfigurable manufacturing cells, shop floor logistics and manipulation, and plant servicing and inspection
Amazon Picking Challenge (APC) [12] |  2015-2016 | X |  | X | X |  |  |  Logistics scenario, retrieving items
IROS Robotic grasping and Manipulation Competition (RGMC) [5, 6] |  2016-2017, 2019-2020 | X |  | X | X | X |  |  Different tracks: service, manufacturing, and logistics
Cybathlon arm prosthesis race [13] |  2016, 2020 | X |  | X | X |  |  |  Solution of everyday tasks using arm prostheses
Amazon Robotics Challenge [14] |  2017 | X |  | X | X |  |  |  Logistic environment, as evolution of APC [12], pick and stow items
ICRA Mobile Manipulation Challenge [15],[16] |  2017, 2019 | X |  | X |  |  | X |  Navigate, pick and place items
World Robot Challenge (WRC) [17] |  2018, 2021 | X |  | X | X | X |  |  Three categories: industrial scenarios (agile manufacturing), service robotics (home and convenience stores), and disaster robotics (inspection and maintenance in a plant)
IROS Fan Robotic Challenge [18] |  2018 | X |  | X | X |  |  |  Pick up and manipulate a Spanish fan
IROS Mobile Manipulation Hackathon [19] |  2018 |  | X | X | X |  | X |  Freely-chosen application to showcase both mobility and manipulation
RoboSoft Competition [20] |  2018-2021 | X |  | X | X |  |  |  Industrial, surgical or domestic scenarios in the manipulation track
Smart City Robotics Challenge (SciRoc) [21] |  2019, 2021 | X |  | X | X |  |  |  Different episodes, or challenges, in domestic and logistic scenarios
OCRTOC, Open Cloud Robot Organization Challenge [22] |  2020, 2021 | X |  | X | X |  |  |  Table reorganization problem, tested in a remote lab
Real Robot Challenge [23] |  2020-2021 | X |  | X | X |  |  |  Grasping and in-hand manipulation tasks in a remote platform
*Tasks: Fi: Fixed, Fr: Free
**Focus: Gr: Grasping, Ma: Manipulation, As: Assembly, Mm: Mobile manipulation

Mechanism level: In these types of benchmarks, the intrinsic capabilities of
the mechanisms (often times end-effectors) are considered and measured. A
benchmarking protocol for assessing and comparing the grasping abilities of
robotic hands is presented in [24]. The benchmark in [25] assesses the
robustness and resilience of the robotic hands by determining the impulsive
conditions that break their grasp and their mechanism itself. A detailed list
of procedures for analyzing the mechanical properties of the robotic hands is
given in [26]. A specific benchmark for compliant hands is provided in [27].

Algorithm level: These benchmarks evaluate the performance of a specific
algorithm in the robotic manipulation pipeline. The benchmarks in [28, 29]
assess the performance of grasp planning algorithms. In [30], the force
control capabilities of the system are analyzed. A platform-independent method
for quantifying the motion planning performance is presented in [31]. Object
segmentation and pose estimation data sets and benchmarks are commonly used in
the robotics and computer vision community, with recent examples in [32, 33].

System level: These benchmarks consider the task performance of a robotic
system as a whole, fully integrated autonomous solution (perception, planning,
control). The box and blocks test in [34] assesses the pick-and-place
performance of a robotic system. Similarly, pick-and-place abilities in
logistics scenarios are evaluated in [35]. Inspired by the Amazon Picking
Challenge, a shelf picking benchmark is presented in [36]. In-hand
manipulation performance is quantified in [37]. Assessment for various
challenging manipulation tasks such as cloth [38], bimanual [39] and aerial
[40] are also provided as system-level benchmarks.

###  II-B Related Competitions and Challenges

The robotics community has a long history in organizing competitions and
challenges. Over the last decade, there have been multiple competitions that
involve robotic manipulation in different degrees of complexity, pushing
forward the research in the field. The competitions focus on different
aspects, such as pure grasping, manipulation, assembly, or even mobile
manipulation, as summarized in Table I. In terms of tasks, those competitions
adopt two approaches:

  * •

Completing a fixed set of tasks: the environment, objects, and rules are
defined in advance. Thus, the tasks provide an objective measure of progress
intra-edition, i.e., comparison of performance of different teams, and inter-
edition, i.e., measuring progress across different years on the same set of
tasks. The disadvantage of this approach is that teams usually over-engineer
their solution (or work around the rules) to fulfill the intended task.

  * •

Demonstration of a freely-chosen application: this open format allows the
teams to demonstrate their strengths in a self-selected scenario, usually
following very general constraints, e.g., using a common robotic platform or
demonstrating tasks that include prescribed components. The disadvantage is
that demonstrations are hardly comparable among them.

The Robotic Grasping and Manipulation Challenge (RGMC) uses system level
benchmarks and defines a fixed set of tasks in each track to be solved using a
robot manipulator. The descriptions of the tasks are provided in the next
section.

##  III Task Designs

To evaluate robotic systems’ capability, we have designed over 40 real-life
tasks based on our knowledge of the research challenges in robotic grasping
and manipulation. All these tasks have detailed protocols, rules, and scoring
policies. For instance, the task descriptions and rules of the 2016 and 2017
RGMCs can be found in [5]. Each competition comprises challenges that span a
large set of robotic manipulation capabilities with varying difficulty levels.
Easier tasks are expected to be solved by most teams, while some challenging
tasks are not expected to be fully accomplished by any of the teams. Such
variety in difficulty serves multiple purposes: Easier tasks encourage
participation and constitute a starting point for new robotics researchers.
Challenging tasks aim to differentiate the most successful teams while
encouraging robot/algorithm design and integration innovation. Limited by
natural constraints of the competition settings (e.g., timeline, space, and
equipment), the tasks are carefully composed to appeal to a large group of
researchers and research interests. Therefore, some of the tasks that were
selected in a certain year might be replaced by others in the following years,
even though the teams did not fully accomplish them.

Here we present the tasks in the RGMCs according to the research challenges
associated with them. There are plenty of unsolved research challenges in
perception, grasping, manipulation, and mechanical design trade-offs for both
real-time service tasks as well as for manufacturing/assembly tasks [41].

###  III-A Research Challenges in Perception

Perception is critical to most robotic grasping and manipulation tasks. A
significant amount of work has been dedicated to solving various challenges in
robotic perception and, over the years, we have seen great improvement,
especially after depth cameras became widely accessible and high-precision
tactile sensors became less expensive. However, many challenges remain
unsolved, and while designing our competitions we highlight these challenges
to gauge progress.

Figure 1: Task setups based on perception challenges. Description of the tasks
is provided in Table II.

####  III-A1 Objects with shiny surfaces

Computer vision has a long-standing difficulty in dealing with shiny objects.
Because of the reflection, some shiny surface areas cannot be well-perceived
by vision sensors. The missing areas make segmentation and pose estimation
difficult, and drop the success rate of grasping and manipulation algorithms.
In RGMCs, we have several objects with shiny surfaces, such as the silverware
sets, 2017 RGMC (Fig. 1a) and spoons and tongs, 2019/2020 RGMCs (Fig. 1b).
Also, parts such as the plastic assembly base (task board) and metallic
components in manufacturing tasks (Fig. 1g) can present perception
difficulties.

TABLE II: Tasks reflecting perception challenges Research challenges |  Tasks
---|---
Shiny objects |  -Pick up silverware (Fig. 1a)
|  -Pick up a a polished spoon for stirring (Fig. 1b)
|  -Pick up metal tongs to get ice cubes (Fig. 1b)
|  -Pick up metal pegs for insertion
|  -Locate insertion holes on task board
Translucent objects |  -Pick up transparent cup lid (Fig. 1c)
|  -Pick up ice cubes (Fig. 1d)
|  -Pour a certain amount of water into a cup (Fig. 1e)
Precision/accuracy |  -Insert electrical connectors (Fig. 1f)
|  -Insert pegs, fasteners (Fig. 2f)
|  -Insert gears (Fig. 3b)
|  -Insert/route belts (Fig. 1g)

####  III-A2 Translucent or transparent objects

Translucent objects are also very challenging for robotic vision. In all four
RGMCs, we have translucent to-go cups and lids (Fig. 1c). We have also
introduced a pile of ice cubes for a picking task in the last two RGMCs. In
general, the sensing aspect of the ice cube picking task is more challenging
than the sensing for picking up a to-go cup and a lid, since the to-go cups
and lids are standalone objects on the table, and they occupy a significant
space to approximate their pose. On the other hand, the ice cubes are randomly
piled up in an ice bucket (Fig. 1d), and determining the pose of an individual
ice cube among other (also translucent) ice-cubes within a translucent ice
bucket is a very challenging task. In 2017 RGMC, the competition requested to
pour a certain amount of water into a cup. Estimating the amount of water in
the cup also provides a significant challenge since it is transparent (Fig.
1e). There are currently no translucent or transparent objects used for the
manufacturing tasks.

####  III-A3 Tasks requiring high precision or accuracy

Tasks with tight tolerances usually require the perception part of the system
to be precise. For typical peg-in-hole problems such as plugging a Universal
Serial Bus (USB) light into a socket (Fig. 1f), a robot would need to use
vision to precisely localize the socket. To compensate for the inaccuracies of
the localization, an exploration algorithm or a failure recovery approach is
often times needed. Using force sensors to guide corrective motions is a
common example of such exploration strategies.

####  III-A4 Other challenges

Service track objects are provided to teams prior to competitions depending on
type and difficulty. This time can range from hours to days. The teams usually
utilize this time to collect data and learn models for object identification,
segmentation, and planning. While developing techniques to model the objects
in a short amount of time under a semi-controlled environment is a challenge,
obtaining prior models of the objects significantly reduces the perception and
planning difficulties of the task. If the objects are unknown (no learning
data or models), the tasks could be much more difficult. However, for real
service applications, learning object models is typically not feasible. In the
case of the manufacturing track, design data is typically known, so all
components and Computer Aided Design (CAD) data are provided to teams weeks in
advance to the competition. At the start of the competition all teams are
given updated CAD data that reflects changes in the challenge tasks from the
practice tasks, which imitates challenges associated with product changes in
batch type production runs.

Table II summarizes the tested perception challenges and the tasks reflecting
them in the RGMCs.

###  III-B Research Challenges in Grasping

Figure 2: Tasks reflecting grasp challenges. Description of the tasks is
provided in Table III.

####  III-B1 Grasping with imperfect perception

Perception errors cause most failures in grasping. If the object’s pose is
estimated seriously wrong, the robot could knock over the object or push it
away, end up with an unstable grasp, or completely miss the object. For end-
to-end approaches, even if there is no separated pose estimation step, the
noise in perception layers still dramatically affects the grasp results.
Unfortunately, it is almost impossible to have “perfect” perception. Noise can
always find a way to creep in. So grasping approaches should be prepared to
deal with perception uncertainties. In RGMC, most tasks require robots to
handle a bad grasp caused by perception errors since many tasks have
challenging objects.

####  III-B2 Objects with challenging shapes and surfaces

Even with perfect vision, some objects can still be challenging to pick up or
manipulate. It could be because there is a minimal good surface area for
grasping. Curved and narrow surfaces are challenging to grasp. For example,
the curved finger rings of a pair of scissors pose a significant challenge to
a robotic hand (Fig. 2a). Holding and manipulating scissors with them are both
very difficult. Another challenging task is to pick up a saucer with a cup on
it with one hand, since it is difficult to hold the edge of the saucer and
balance the unknown torque generated by the cup’s weight (Fig. 2b).

####  III-B3 Grasping objects in clutter

Picking up an object in clutter is quite challenging. Its neighbor objects
could block the robotic hand and pose planning difficulties, since usually a
gripper would need to grasp on two sides of the object, and the hand
approaches from another side. One task that reflects this challenge is picking
objects from a shopping basket and placing them in defined areas (Fig. 2c).
Grasping objects in clutter for manipulation is more challenging since the
object would need to be held in a certain way to provide needed manipulability
[5] and interactive force [42]. Therefore, the grasping points and
orientations for manipulation are more limited than for picking. When objects
are in clutter, a robotic hand may not be able to reach and grasp the object
in the desired way. For example, in the stirring water task (Fig. 2d), and
pea-picking using a spoon task (Fig. 2e), the spoons are in a silverware
organizer and they are difficult to pick up.

####  III-B4 Regrasping

A robot may pick up an object in one grasp but may need to change to a
different grasp for manipulation. Then the robot would need to regrasp the
object after it is picked up. As mentioned before, in both stirring and pea-
picking tasks, the grasp of the spoon should be adjusted after it is picked
up. In the task of putting on or removing a bolt from a nut with a nut driver,
since multiple turns are required, a robotic hand would need to regrasp the
nut driver to overcome the rotation limit of the wrist (Fig. 2f). Due to the
number of fasteners, making regrasp operations efficient is challenging.

TABLE III: Tasks reflecting grasp challenges Research challenges |  Tasks
---|---
Objects with challenging shapes and surfaces |  -Cut a piece of paper using a pair of scissors (Fig. 2a)
-Transfer a cup on its saucer (Fig. 2b)
Grasp in clutter |  -Pick and place objects from a shopping basket (Fig. 2c)
|  -Pick up silverware from a silverware organizer (Fig. 1a)
Regrasp |  -Assemble fasteners
|  -Use a spoon to stir water in a cup (Fig. 2d)
|  -Use a spoon to pick up peas (Fig. 2e)
Grasp for manipulation |  -Hammer a nail (Fig. 2g)
-Stir water with a spoon (Fig. 2d)
|  -Assemble/disassemble electrical connectors (Fig. 2i)
|  -Insert pegs (Fig. 2j)
|  -Insert/screw fasteners (Fig. 2f)
|  -Insert/mesh gears (Fig. 3b)
|  -Insert/rout belts/wires (Fig. 2k,l)
Grasp for in-hand manipulation |  -Fully extend and fully press syringe (Fig. 2h)
-Grasp and use scissors to cut a piece of paper to half along a line (Fig. 2a)
|  -Pick up ice cubes using tongs (Fig. 1d)

####  III-B5 Grasping for manipulation

For some manipulations, an object should be grasped in a certain way so that
the manipulation can be performed efficiently and the object has less chance
of being dropped during the operation [43, 44]. We have designed nail
hammering (Fig. 2g) and water stirring tasks to gauge if the team considers
the manipulation requirements in grasp planning. Performing force-based
manipulation for insertions requires adequate grasping force to avoid movement
of the object within grasp and ultimate insertion failure (Fig. 2j). In
addition, grasping non-rigid objects to effectively control their shape, as in
threading belts on pulleys (Fig. 2k) and wire routing (Fig. 2l), is a very
challenging problem.

####  III-B6 Grasping for in-hand manipulation

In-hand manipulation is still a very challenging research area. We have
designed several tasks that would require some in-hand manipulation after
grasping. These include extending and pressing a syringe (Fig. 2h), cutting
paper with a pair of scissors (Fig. 2a), and using tongs to pick up ice cubes
(Fig. 1d), but they have not been satisfactorily solved so far.

Table III summarizes the tested grasp challenges and the tasks reflecting them
in the RGMCs.

Figure 3: Tasks reflecting manipulation challenges. Description of the tasks
is provided in Table IV.

###  III-C Research Challenges in Manipulation

Manipulation motions can usually be generated with motion planning and motion
control. In general, the challenges in manipulation are caused by imperfect
perception and the lack of capability to predict a motion’s outcome. Both can
be extremely challenging. In RGMCs, since the objects are provided to the
teams beforehand (at least for several hours), they can model the objects and
their behaviors to a certain extent, which dramatically reduces the
challenges. However, several tasks remain challenging because either the
tolerance is tight or the behaviors are difficult to model. For instance, peg-
in-hole tasks and tightening or loosening bolts (Fig. 3d) require purposeful
motions. Tearing up a paper towel (Fig. 3a) and pouring water (Fig. 3b)
require the robot to predict the kitchen roll motion and water flowing speed
in response to a motion. Opening a bottle with a locking safety cap (Fig. 3c)
is a task that requests the robot to predict a pressing outcome. Since it is
repeatable, it is easier to model than water and cloth. Assembling flexible
belts and routing wires, while difficult, becomes more achievable with
practiced routines and the use of CAD data. Table IV summarizes the tasks
reflecting those challenges in the RGMCs.

TABLE IV: Tasks reflecting manipulation challenges Research challenges |  Tasks
---|---
Manipulation with imperfect perception |  -Solve peg in hole with a wood board
-Assemble/disassemble electrical connectors
|  -Insert pegs (Fig. 2j)
|  -Insert/screw fasteners (Fig. 3d)
|  -Insert/mesh gears (Fig. 3b)
Manipulation that needs accurate prediction |  -Open a water bottle cap and pour water into a to-go cup
-Pour water from a pitcher into a cup (Fig. 1e)
-Tear a paper towel (Fig. 3a)
|  -Open a bottle with a locking safety cap (Fig. 3c)
|  -Threading flexible belts (Fig. 3e)
|  -Routing wires (Fig. 3f)

##  IV Results and Progress

Each year, seven to ten teams passed the initial screening and participated in
the competition. Teams are from world-renowned research groups in
universities, prominent startups, and large corporations. We can see that
their improvement in performances aligns with the progress made by the
robotics research community over the same period of time. Though their
performance might not represent the absolute best solutions available at the
time, their successes and struggles in the RGMCs reflect research progress and
remaining challenges.

###  IV-A Overview

In the four RGMCs, twenty-three tasks have been used to measure the
contestant’s performance in the service track, some used in multiple years
with stricter requirements toward achieving real-world application. In the
beginning, in 2016, the teams were allowed to predefine the locations of the
objects with little to no randomness. Then in 2017 and afterward, the teams
were only allowed to define object regions based on the robot’s workspace and
the organizers randomly placed the objects in those regions. In 2016 and 2017,
only the best possible result for every task was counted, while in 2019 and
2020, the total score of five trials was counted. In 2019, the total allowed
time was 90 minutes, while in 2020, the total allowed time was 60 minutes. The
manufacturing track was first introduced in 2017, and teams were allowed to
set the position of the task board and kit layout, and optional random
placement provided bonus points. The dominant solution was the use of lead-
through programming. To encourage the use of perception, in 2019 and 2020
randomized placement was required and CAD data was supplied as an option for
extracting locations of assembly components on both the task board and kit
layout relative to a coordinate frame. While the 2017 task board contained
only insertion and fastening operations, the 2019 and 2020 task boards
introduced wire routing and belt on pulley operations. The Appendix provides
details of the tasks, the best scores, unsolved tasks, and performances of the
RHGM teams based on the data and organizers’ observation.

###  IV-B Research Progress and Challenges in Perception

####  IV-B1 Objects with shiny surfaces

In 2016, teams were struggling with silverware due to their shiny surfaces. In
2020, several teams could reliably segment a shiny spoon from other spoons and
estimate its pose for grasping. They managed the reflection by controlling the
lighting using multi-flash, similar to the technique in [45]. Objects with
shiny surfaces may not be a significant challenge if the object model is known
and its location variation is limited.

####  IV-B2 Translucent or transparent objects

Translucent objects still pose a significant challenge to perception,
especially for randomly stacked objects such as ice cubes in a bucket. No team
was able to finish the picking ice cube task. For the pouring task, all teams
completed it using hard-coded pouring motions. We do not think any team could
estimate the water level in the receiving cup when the water is transparent.
Even though several promising pouring approaches are available, none of the
teams used them. So, when an object is truly transparent, alternative
perception other than vision could be used. On the other hand, randomly
stacked translucent objects are still very challenging.

####  IV-B3 Tasks requiring high precision or accuracy

In plugging a USB light into a socket task, several teams were able to finish
the task. Some teams are slower than others. But in general, the peg-in-hole
problem is not a significant challenge if both models are known.

###  IV-C Research Progress and Challenges in Grasping

####  IV-C1 Grasp with imperfect perception

Teams in RGMCs have gradually incorporated approaches to handle perception
uncertainties, since they found that it was one of several major reasons that
make their solutions slow and unreliable. It is a significant challenge that
requires both research and engineering work.

####  IV-C2 Objects with challenging shapes and surfaces

Teams have explored several options and made good progress. Several teams
tried an automatic tool-change approach that allows them to swap grippers.
Several grippers were designed and made to deal with challenging shapes and
surfaces. Some others developed a gripper that incorporates multiple fixtures.
They can flip or rotate so that the right fixture is in contact with the
object. Many smart and inspiring designs have come out, but they are tailored
and calibrated on known objects. It is a significant challenge for cases with
unknown objects or under unseen situations.

####  IV-C3 Grasp objects in clutter

RGMCs only have several tasks with objects in clutter, and the level of
cluttering is moderate. On the one hand, we observed the teams made progress
over the years in dealing with closely lying objects. But on the other hand,
we also observed that cluttered environments still slowed down their
performance dramatically. So it is still a challenge, and we would expect it
is more difficult if the objects are unknown and the situations are new.

####  IV-C4 Regrasp

Many tasks require the robot to adjust the grasp after picking up an object.
We have observed many teams put down the object and used the setup to position
or orient the object in a certain way and then regrasped it. Many of the pick-
place-regrasp routines are well crafted and impressive. However, teams have
been avoiding in-air regrasp even it would be more efficient. Adjusting a
grasp or regrasping in the air is still very challenging.

####  IV-C5 Grasp for manipulation

Several tasks require grasp planning while considering manipulation
requirements. We have mostly seen that teams predefine several grasp points
based on their experiences and simulations. As far as we know, teams have
avoided computing and searching for proper grasp points on the fly. The
approach would fall apart if the objects are unseen or different from the
provided models. For known objects, the predefined grasp point approach seems
sufficient.

####  IV-C6 Grasp for in-hand manipulation

In-hand manipulation remains the biggest challenge, where teams have been
unable to complete tasks that utilize scissors, syringes, and tongs. These
tasks, in most cases have been removed from RGMC due to their high difficulty.

###  IV-D Research Progress and Challenges in Manipulation

####  IV-D1 Manipulation with imperfect perception

Many teams have no problem dealing with tasks testing manipulation with
imperfect perception. In general, the available object models and setups and
tactile/force sensors allow teams to complete those tasks successfully.

####  IV-D2 Manipulation that needs accurate prediction

Teams did little in predicting the outcomes of a manipulation action. Most
manipulation motions were generated based on obstacle avoidance. As far as we
know, they have avoided modeling dynamics. They usually created a list of
possible outcomes and matched a set of predefined motion strategies to those
outcomes. This approach is not sufficient when dealing with flexible objects
and fluids, since a finite enumeration of the states of those objects is
usually not feasible. Even though the teams have not provided satisfactory
solutions, we believe that recent research progress could be used to solve
some of the challenges [46, 47]. Table V summarizes the tested challenges and
the progress in the RGMCs.

TABLE V: Perception challenges and progress Research challenges |  Progress
---|---
Shiny objects |  Mostly solved for standalone objects if their models are known.
Translucent objects |  Still very challenging for randomly stacked translucent objects.
Requiring high precision or accuracy |  Largely manageable if the objects are known. Implementations are incorporating force control and CAD data.
Grasp with imperfect perception |  It is the major reason that solutions are slow and unreliable. It is a significant challenge that requires research and a lot of engineering work.
Objects with challenging shapes and surfaces |  It remains a significant challenge even though special-purpose grippers and fingers are designed.
Grasp objects in clutter |  It is still a challenge, and we would expect it is more difficult if the objects are unknown and the situations are new.
Regrasp |  Many teams developed impressive pick-place-regrasp routines, but adjusting a grasp or regrasping in the air is still very challenging.
Grasp for manipulation |  For manipulating known objects in a known condition, the predefined grasp point approach seems to work well. However, this approach would fall apart if the objects are unseen or the conditions are new.
Grasp for in-hand manipulation |  In-hand manipulation remains the most difficult challenge, as it requires accurate prediction of the motion effects and suitable control of the end effector.
Manipulation with imperfect perception |  The available object models, design data, and tactile/force sensors allow teams to complete those tasks successfully.
Manipulation that needs accurate prediction |  Teams have avoided modeling dynamics for prediction, even though recent research progress could be used to solve some of the challenges.

##  V Discussion and Future Directions

This paper provided an overview of the tasks and challenges proposed in the
RGMCs. Different from many evaluation setups in the literature, competitions
such as RGMC usually require participants to run demos at a certain time and
to complete tasks in a limited amount of time. In addition, participants
usually do not have total control of the setup, making the competition setup
realistic to real-world applications.

Over the years, we have seen that tasks that were initially deemed as hard,
have been solved using different ingenious approaches. Especially in the first
editions, teams tried to solve the most difficult challenges with highly-
engineered solutions. But with progress made in research, we have seen less
and less hard-coded routines, less use of predefined grasp points, which have
led to solutions becoming more effective and reliable. For software, the most
significant challenges identified by the teams are related to perception. The
arrival of learning-based approaches has facilitated solving tasks that use
known objects, as they can be readily available for system training. The
generality of such approaches to be applicable to familiar and fully unknown
objects is still a large challenge. Learning-based approaches have been also
lately used for defining grasping configurations, thus avoiding the reliance
on predefined grasp poses. For motion planning, most teams relied on open
libraries, such as the Open Motion Planning Library (OMPL) [48] or MoveIt
[49], and using Robot Operating System (ROS) [50] in most cases as middleware.
These openly-available resources enable teams to more efficiently create
solutions.

We saw that some of the RGMC tasks still represent a significant challenge.
The most clear example are tasks requiring some degree of in-hand
manipulation, which were effectively removed from the last RGMC editions. This
reflects the state of industry, where most end-effectors are two-finger
grippers that are used for pick-and-place tasks. Industrial applications
usually prefer to design special fingertips for grasping different types of
objects, or using a robot accompanied by a tool changer that endows the robot
with the ability to switch end-effectors for performing multiple tasks. The
promised generic dexterity of multi-finger end-effectors still seems more a
research topic rather than a viable commercial solution, mainly due to the
higher mechatronic and control complexity and higher cost associated with
multi-finger grippers.

There has been a significant increase in manufacturing track solutions that
utilize CAD data. Typically, perception or force solutions are used to
localize the task board with subsequent localization of part assembly points
on the task board using CAD data. Improvements to the CAD pipeline could
result in improved data formats to transmit dimensions, geometric features,
relative part positions, mating descriptions, and tolerances from CAD systems.
In addition, the inclusion of force-based assembly parameters within a CAD
system, which could be best specified by a designer who is most familiar with
the mechanical properties of the parts to be assembled, should be considered.
Methods for automatic robot program generation using this data should also be
considered.

In terms of robotic devices, participants have used a full range, from
experimental robotic hands and arms to off-the-shelf robot arms and simple
parallel grippers. Hardware for perception is nowadays relatively standard,
with depth cameras dominating the landscape, although different teams have
been using other types of sensors (stereo cameras, laser-based perception, in-
hand cameras). Integration of different hardware and software components
remains, however, a large challenge, requiring hours of dedication to reach a
stable and robust execution during the demonstrations. In most RGMC editions,
teams had to transport their own equipment to the competition locations. The
pandemic required the 2020 edition to be a fully online competition, which
greatly alleviated the logistics for participation of teams around the world.
Moving toward a standard remote lab for testing different approaches for
solving a set of tasks seems like a reasonable approach to enhance
comparability and reproducibility of results in robotics in the near future.

New robot designs provide improved methods for fast lead-through programming
using direct interactions between the operator and the robot. Teams leveraging
these solutions typically score well by programming the most difficult high
scoring tasks first, to accumulate as many points as possible within a given
time frame. Competition format changes to discourage solutions that only use
lead-through programming could include: a required part assembly order, less
tasks per board, and unknown task board offset and part variations.

Several successes have ensued RGMC teams. Dorabot [51], for instance,
participated in the 2016 and 2017 RGMC competitions with prototypes of their
dexterous modular robotic three-fingered hand, which is now a product. Focused
on robotic logistic solutions, the company now has products including a five-
fingered hand, and mobile manipulators. Robotic Materials [52] emerged as a
startup out of the University of Boulder following the RGMC 2016/2017
competitions, and participated in RGMC 2019/2020 competition using their
prototype smart gripper, which integrated position, force, and depth camera
sensing modalities. Their work on this gripping system continues with the
development of easy-to-use programming interfaces aimed at small- and medium-
sized manufacturers. Southern Denmark University [53] research focuses on the
development of a flexible workstation for automated assembly tasks that can be
used commercially. Their generalized solution is proving to be robust and
streamlined, achieving the first ever perfect score in the 2020 RGMC
Manufacturing Track.

## Acknowledgment

We would like to acknowledge all RGMC co-organizers. In addition to the
authors, they are Nadia Cheng (2016), Hyouk Ryeol Choi (2016), Zoe Doulgeri
(2016, 2017), Erik D. Engeberg (2016, 2017), Kris Hauser (2016), Nancy Pollard
(2016), Zeyang Xia (2016), Yasuyoshi Yokokohji (2017, 2019), Zoe Doulgeri
(2017), Yunjiang Lou (2017), Hyungpil Moon (2017), Juxi Leitner (2019), Rong
Xiong (2019), and Adam Norton (2020). We also acknowledge Kenneth Kimble of
NIST for numerous detailed competition task board designs.

## References

  * [1] “Robotic grasping and manipulation competition 2016,” http://www.rhgm.org/activities/competition_iros2016/.
  * [2] “Robotic grasping and manipulation competition 2017,” http://www.rhgm.org/activities/competition_iros2017/.
  * [3] “Robotic grasping and manipulation competition 2019,” https://rpal.cse.usf.edu/competition_iros2019/.
  * [4] “Robotic grasping and manipulation competition 2020,” https://rpal.cse.usf.edu/competition_iros2020/.
  * [5] Y. Sun, J. Falco, N. Cheng, H. Choi, E. Engeberg, N. Pollard, M. Roa, and Z. Xia, “Robotic grasping and manipulation competition: task pool,” in _Robotic Grasping and Manipulation Challenge_. Springer, 2016, pp. 1–18.
  * [6] J. Falco, Y. Sun, and M. Roa, “Robotic grasping and manipulation competition: competitor feedback and lessons learned,” in _Robotic Grasping and Manipulation Challenge_. Springer, 2016, pp. 180–189.
  * [7] F. Bonsignorio, “A new kind of article for reproducible research in intelligent robotics [from the field],” _IEEE Robotics & Automation Magazine_, vol. 24, no. 3, pp. 178–182, 2017.
  * [8] “Robocup@home,” https://athome.robocup.org.
  * [9] “DARPA robotics challenge (DRC),” https://www.darpa.mil/program/darpa-robotics-challenge.
  * [10] “Rockin - robot competitions kick innovation in cognitive systems and robotics,” http://rockinrobotchallenge.eu/competitions.php.
  * [11] B. Siciliano _et al._ , “EuRoC - the challenge initiative for european robotics,” in _Proc. Int. Symp. Robotics (ISR)_ , 2014, pp. 1–7.
  * [12] C. Eppner, S. Höfer, R. Jonschkowski, R. Martín-Martín, A. Sieverling, V. Wall, and O. Brock, “Lessons from the Amazon Picking Challenge: Four aspects of building robotic systems,” in _Proc. Int. Joint Conf. Artificial Intelligence (IJCAI)_ , 2017, pp. 4831–4835.
  * [13] “Cybathlon,” https://cybathlon.ethz.ch/de.
  * [14] “Aussies win Amazon Robotics Challenge,” https://spectrum.ieee.org/automaton/robotics/industrial-robots/aussies-win-amazon-robotics-challenge.
  * [15] “DJI robomaster mobile manipulation challenge,” https://www.robomaster.com/en-US/resource/pages/announcement/673.
  * [16] “FetchIt! the mobile manipulation challenge,” https://opensource.fetchrobotics.com/competition.
  * [17] “World Robot Summit,” https://worldrobotsummit.org/en/.
  * [18] “IROS 2018 fan robotic challenge,” https://www.iros2018.org/fan-robotic-challenge.
  * [19] “IROS 2018 mobile manipulation hackathon,” http://iros18-mmh.pal-robotics.com/.
  * [20] “Robosoft 2021 competition rulebook,” https://softroboticsconference.org/pdf/RoboSoft2021-Competition-Rulebook.pdf.
  * [21] “Smart city robotics challenge,” https://sciroc.org/.
  * [22] Z. Liu, W. Liu, Y. Qin, F. Xiang, M. Gou, S. Xin, M. A. Roa, B. Calli, H. Su, Y. Sun, and P. Tan, “Ocrtoc: A cloud-based competition and benchmark for robotic grasping and manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 1, pp. 486–493, 2022.
  * [23] “Real robotics challenge,” https://real-robot-challenge.com/.
  * [24] I. Llop-Harillo, A. Pérez-González, J. Starke, and T. Asfour, “The anthropomorphic hand assessment protocol (AHAP),” _Robotics and Autonomous Systems_ , vol. 121, p. 103259, 2019.
  * [25] F. Negrello, W. Friedl, G. Grioli, M. Garabini, O. Brock, A. Bicchi, M. A. Roa, and M. G. Catalano, “Benchmarking hand and grasp resilience to dynamic loads,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1780–1787, 2020.
  * [26] J. Falco, K. Van Wyk, and E. Messina, “Performance metrics and test methods for robotic hands,” _NIST, Tech. Rep._ , vol. 1227, 2018.
  * [27] P. Sotiropoulos, M. A. Roa, M. F. Martins, W. Fried, H. Mnyusiwalla, P. Triantafyllou, and G. Deacon, “A benchmarking framework for systematic evaluation of compliant under-actuated soft end effectors in an industrial context,” in _Proc. IEEE-RAS Int. Conf. on Humanoid Robots_ , 2018, pp. 650–657.
  * [28] Y. Bekiroglu, N. Marturi, M. A. Roa, K. J. M. Adjigble, T. Pardi, C. Grimm, R. Balasubramanian, K. Hang, and R. Stolkin, “Benchmarking protocol for grasp planning algorithms,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 315–322, 2019.
  * [29] F. Bottarel, G. Vezzani, U. Pattacini, and L. Natale, “GRASPA 1.0: GRASPA is a robot arm grasping performance benchmark,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 836–843, 2020.
  * [30] J. Falco, J. Marvel, R. Norcross, and K. Van Wyk, “Benchmarking robot force control capabilities: Experimental results,” _National Institute of Standards and Technology (NIST)_ , vol. 100, 2016.
  * [31] F. Lagriffoul, N. T. Dantam, C. Garrett, A. Akbari, S. Srivastava, and L. E. Kavraki, “Platform-independent benchmarks for task and motion planning,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3765–3772, 2018\.
  * [32] N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, B. Price, S. Cohen, and T. Huang, “Youtube-vos: Sequence-to-sequence video object segmentation,” in _Proc. Europ. Conf. on Computer Vision (ECCV)_ , 2018, pp. 585–601.
  * [33] T. Grenzdörffer, M. Günther, and J. Hertzberg, “YCB-M: a multi-camera RGB-D dataset for object recognition and 6DoF pose estimation,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2020, pp. 3650–3656.
  * [34] A. S. Morgan, K. Hang, W. G. Bircher, F. M. Alladkani, A. Gandhi, B. Calli, and A. M. Dollar, “Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 454–461, 2019.
  * [35] H. Mnyusiwalla, P. Triantafyllou, P. Sotiropoulos, M. A. Roa, W. Friedl, A. Sundaram, D. Russell, and D. Deacon, “A bin-picking benchmark for systematic evaluation of robotic pick-and-place systems,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1389–1396, 2020.
  * [36] J. Leitner, A. Tow, N. Sünderhauf, J. Dean, J. Durham, M. Cooper, M. Eich, C. Lehnert, R. Mangels, C. McCool, P. Kujala, L. Nicholson, T. Pham, J. Sergeant, L. Wu, F. Zhang, B. Upcroft, and P. Corke, “The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2017, pp. 4705–4712.
  * [37] S. Cruciani, B. Sundaralingam, K. Hang, V. Kumar, T. Hermans, and D. Kragic, “Benchmarking in-hand manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 588–595, 2020.
  * [38] I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Alenya _et al._ , “Benchmarking bimanual cloth manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1111–1118, 2020.
  * [39] K. Chatzilygeroudis, B. Fichera, I. Lauzana, F. Bu, K. Yao, F. Khadivar, and A. Billard, “Benchmark for bimanual robotic manipulation of semi-deformable objects,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2443–2450, 2020.
  * [40] A. Suarez, V. M. Vega, M. Fernandez, G. Heredia, and A. Ollero, “Benchmarks for aerial manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2650–2657, 2020.
  * [41] K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokokohji, “Benchmarking protocols for evaluating small parts robotic assembly systems,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 883–889, 2020.
  * [42] Y. Lin and Y. Sun, “Task-oriented grasp planning based on disturbance distribution,” in _Robotics Research_. Springer, 2016, pp. 577–592.
  * [43] ——, “Task-based grasp quality measures for grasp synthesis,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2015, pp. 485–490.
  * [44] ——, “Grasp planning to maximize task coverage,” _Int. J. Robotics Research_ , vol. 34, no. 9, pp. 1195–1210, 2015.
  * [45] A. Agrawal, Y. Sun, J. Barnwell, and R. Raskar, “Vision-guided robot system for picking objects by casting shadows,” _Int. J. Robotics Research_ , vol. 29, no. 2-3, pp. 155–173, 2010.
  * [46] T. Chen, Y. Huang, and Y. Sun, “Accurate pouring using model predictive control enabled by recurrent neural network,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2019, pp. 7688–7694.
  * [47] Y. Huang, J. Wilches, and Y. Sun, “Robot gaining accurate pouring skills through self-supervised learning and generalization,” _Robotics and Autonomous Systems_ , vol. 136, p. 103692, 2021.
  * [48] “Open Motion Planning Library,” https://ompl.kavrakilab.org/.
  * [49] “MoveIt,” https://moveit.ros.org/.
  * [50] “Robot Operating System,” https://www.ros.org/.
  * [51] “Dorabot,” https://www.dorabot.com/.
  * [52] “Robotic Materials,” http://www.roboticmaterials.com/.
  * [53] “University of Southern Denmark,” https://www.sdu.dk/en/forskning/sdurobotics.

See pages 1-last of appendix.pdf