# Probing Velocity Dispersion inside CMEs in Inner Corona: New Insights on CME
Initiation

Satabdwa Majumdar Aryabhatta Research Institute of Observational Sciences,
Nainital, 263001, India  Austrian Space Weather Office, Geosphere Austria,
Graz, Austria  Elke D’ Huys Solar-Terrestrial Centre of Excellence - SIDC,
Royal Observatory of Belgium, Avenue Circulaire 3, Brussels 1180, Belgium
Marilena Mierla Solar-Terrestrial Centre of Excellence - SIDC, Royal
Observatory of Belgium, Avenue Circulaire 3, Brussels 1180, Belgium  Institute
of Geodynamics of the Romanian Academy, Bucharest, Romania  Nitin Vashishtha
Aryabhatta Research Institute of Observational Sciences, Nainital, 263001,
India  Dana-Camelia Talpeanu Solar-Terrestrial Centre of Excellence - SIDC,
Royal Observatory of Belgium, Avenue Circulaire 3, Brussels 1180, Belgium
Dipankar Banerjee Aryabhatta Research Institute of Observational Sciences,
Nainital, 263001, India  Indian Institute of Astrophysics,2nd Block,
Koramangala, Bangalore, 560034, India  Center of Excellence in Space Science,
IISER Kolkata, Kolkata 741246, India  Martin A. Reiss Community Coordinated
Modeling Center, NASA Goddard Space Flight Center, 8800 Greenbelt Rd.,
Greenbelt, MD 20771, USA

###### Abstract

This work studies the kinematics of the leading edge and the core of 6 Coronal
Mass Ejections (CMEs) in the combined field of view of Sun Watcher using
Active Pixel System detector and Image Processing (SWAP) on-board PRoject for
On-Board Autonomy (PROBA-2) and the ground-based K-Cor coronagraph of the
Mauna Loa Solar Observatory (MLSO). We report, for the first time, on the
existence of a critical height hc, which marks the onset of velocity
dispersion inside the CME. This height for the studied events lies between
1.4-1.8 R⊙, in the inner corona. We find the critical heights to be relatively
higher for gradual CMEs, as compared to impulsive ones, indicating that the
early initiation of these two classes might be different physically. We find
several interesting imprints of the velocity dispersion on CME kinematics. The
critical height is strongly correlated with the flux-rope minor radius and the
mass of the CME. Also, the magnitude of velocity dispersion shows a reasonable
positive correlation with the above two parameters. We believe these results
will advance our understanding of CME initiation mechanisms and will help
provide improved constraints to CME initiation models.

††facilities: PROBA-2/SWAP, MLSO/K-Cor, SOHO/LASCO-C2††software: IDL, R.

##  1 Introduction

The discovery of Coronal Mass Ejections (or CMEs) dates back to the early
1970s (Hansen et al., 1971), which marks a major milestone in the context of
the Sun-Earth connection. Since their discovery (for a historical review, see
Gopalswamy, 2016), these violent, yet fascinating eruptions of plasma and
magnetic field attracted the attention of the scientific community. This
fascination to study CMEs arises from a two fold perspective. The first comes
from scientific curiosity due to the wide range of properties exhibited by
these eruptions (Webb & Howard, 2012; Chen, 2017), and the second arises from
an economical and technological perspective. These eruptions are the major
drivers of space weather, having the potential to create strong geomagnetic
storms that can disrupt communication systems and pose threats to astronauts
in space (for a recent review, see Temmer, 2021), and thus a clear
understanding of them is essential.

From describing CMEs as “discrete, bright, white-light features” (as quoted in
Hundhausen et al., 1984), we have come a long way towards having a better and
improved understanding of these eruptions. This has been possible thanks to
the wealth of observational resources and models that have helped us probe
into the different aspects of CMEs. Often observable properties of CMEs like
height, width, etc. are estimated based on their morphological properties, and
this morphological classification of CMEs is mostly done based on their
appearance in the white-light coronagraph images (see Munro et al., 1979;
Cremades & Bothmer, 2004). In this context, two particular morphologies stand
out from the rest: the loop-like CME (Crifo et al., 1983) and the three-part
structured CME (Illing & Hundhausen, 1985). The latter is considered evidence
for a flux-rope structure of a CME (Song et al., 2022), and such a three part
structure can be statistically found in one out of every three CMEs (Vourlidas
et al., 2013).

Ever since the start of CME observations, understanding the kinematics of
these eruptions has been of significant interest and importance. CMEs often
show three phases of evolution in their kinematic profile, with an initial
slow rise phase, followed by an impulsive acceleration phase and then the
propagation phase where it experiences very little or almost no acceleration
or deceleration (Zhang et al., 2001). Out of these three phases, the first two
phases are usually confined in the inner coronal heights ($\sim 3$R⊙) where
the CME generally experiences the main acceleration phase (Chen & Krall, 2003;
Majumdar et al., 2020, 2022). However, we are yet to arrive at a clear
understanding of the first two phases of evolution, and these initial phases
are important as they hold clues about the eruption process of the CME.

Since the three-part structured CMEs provide a unified morphological picture
in the form of a flux-rope, it is important to track the different parts of
these three-part CMEs. Such an attempt will shed more light on the relative
kinematics of these different parts, and the role played by one on the
evolution of the other (if any at all). Unfortunately, such studies (see;
Schmahl & Hildner, 1977; Wood et al., 1999; Srivastava et al., 2000; Krall et
al., 2001; Maričić et al., 2004; Koutchmy et al., 2008; Maričić et al., 2009;
Chifu et al., 2012) are rare, as it is extremely difficult to identify such
structured CMEs in the lower heights. Nevertheless, these studies are crucial
as they can provide exceptionally important views as inputs to different CME
initiation models (for a review, see Forbes et al., 2006). These studies have
shown that the CME leading edge (hereafter LE) travels with a higher velocity
than the CME core, giving rise to a velocity dispersion inside a CME.
Particularly, Maričić et al. (2009) reported on the different times of onset
of acceleration in the LE and the core. This indicates the possibility of the
existence of a time and height at which the dispersion in velocities of the
two structures starts. Also, recently, Bemporad et al. (2018) showed that a
radial gradient of speed from the inner core to the LE can get manifested into
a radial gradient of electron temperature distribution. This difference in
temperature across different fronts might manifest into different expansion
rates along the different parts of a CME, as recently reported by Majumdar et
al. (2022). Hence, even from the thermodynamic perspective, this dispersion of
velocity across the core and the LE seems to play an important role. Thus, it
is extremely important to study this velocity gradient within the CME,
particularly in the lesser explored inner coronal heights, to better
understand its effect on the different aspects of CME early evolution.

CMEs can be grouped into two dynamical classes, impulsive and gradual CMEs
(Moon et al., 2002). The impulsive CMEs tend to originate from the active
regions on the Sun, while the gradual CMEs are connected to erupting
prominences. This distinction is reflected in different aspects of the
kinematics of these CMEs (see Pant et al., 2021; Majumdar et al., 2021) and in
various statistical properties as recently pointed out based on a CME source
region catalogue by Majumdar et al. (2023). Hence, whether any difference in
the velocity dispersion is seen for impulsive and gradual CMEs is worth
studying. In this work, we aim to probe this velocity dispersion of CMEs in
the inner corona, and the influence of it on the overall kinematic evolution
of CMEs. In Section 2, we describe the data sources and the working method,
followed by the main results of this work in Section 3. We finally summarize
the main conclusions in Section 4.

##  2 Data and Method

###  2.1 Data Source

To track the CMEs in the inner corona, we use the combined observations in the
17.4 nm bandpass of the Sun Watcher using Active Pixel system detector and
image processing (SWAP; Halain et al., 2013; Seaton et al., 2013) on-board
PRoject for On-Board Autonomy (PROBA-2; Santandrea et al., 2013) and the 2
minute cadence level 2 data processed with Normalized Radially Graded Filter
(NRGF; Morgan et al., 2006) from the ground based coronagraph K-Cor of the
Mauna Loa Solar Observatory. We track the CME further out in the corona using
the data from Large Angle Spectroscopic COronagraph C2 (LASCO; Brueckner et
al., 1995) on-board the Solar and Heliospheric Observatory (SOHO; Domingo et
al., 1995). We also identify the source regions of the CMEs using the
different passbands of the Atmospheric Imaging Assembly (AIA; Lemen et al.,
2011) on-board the Solar Dynamics Observatory (SDO; Pesnell et al., 2012) and
Extreme Ultraviolet Imager (EUVI; Howard et al., 2008) on-board the Solar
Terrestrial Relations Observatory (STEREO; Kaiser et al., 2008).

###  2.2 Data Preparation

To increase the visibility of the off-limb structures in SWAP images, the
intensity is re-scaled using the standard BYTSCL.PRO routine in IDL, which
simply scales the values of an array lying in the range (min,max) to
(0,new$\\_$max), and then the disk of the Sun was masked. Running difference
images are then created for the purpose of removing the static features. In
order to further enhance the coronal intensity for outward propagating
structures, a radial filter, called the Multi-Scale Gaussian Normalization
(MGN; Morgan & Druckmüller, 2014) has been applied. A resulting image is shown
in Figure 1(a), where the eruption front for the CME on 5 November 2014, can
be identified in the SWAP field of view (FOV). We also apply a gamma
transformation in the form of $I^{\gamma}$ (where $I$ is the image array and
$\gamma$ lies between 0.3 to 0.5) to the above images, to further enhance the
brightness and the contrast of the eruption features with respect to the
background. To remove the static features in the off-limb corona in K-Cor
images, running difference images are created, where each image is subtracted
by the previous image and so on. A resulting image is shown in Figure 1(b),
where a three-part structure CME can be seen. To understand the spatial
overlap of the CME as seen in SWAP and K-Cor, in Figure 1(c), a running
difference image in SWAP FOV (created using JHelioviewer (Müller et al.,
2017)) is shown, with blue lines tracing the eruption front. The same blue
lines are then overlaid on the CME observed in K-Cor (running difference
images created using JHelioviewer) in Figure 1(d), showing that SWAP
approximately observes the core of the CME in K-Cor. It is worth noting that
the blue outline lies slightly ahead of the core in K-Cor. This is partly due
to the time difference between the two images and the fact that Extreme
Ultraviolet and white light images might not be looking at the exact same
physical feature, which is also seen as an outcome in the form of the height
difference between the SWAP front and the K-Cor core as shown in Figure 2.
However, if the K-Cor core height measured at 19:30 UT (which was 1.33 R⊙), is
translated forward in time with the instantaneous speed of the core at that
time, taken from Figure 3(a) (which was 320 km/s), the estimated height comes
to be 1.36 R⊙, which leads to a higher overlap with the overlaid blue
contours.

Date | Time (UT) | Latitude (∘) | Longitude (∘) | Source Region | hc (R⊙) | Vdisp (km/s)
---|---|---|---|---|---|---
2014 - 11 - 05 | 19:30:16 | 16 | -80 | AR | 1.54 | 115
2014 - 12 - 21 | 01:57:58 | 10 | -82 | AP | 1.42 | 152
2020 - 11 - 26 | 20:50:56 | 32 | -85 | AR | 1.59 | 140
2021 - 05 - 07 | 19:02:48 | 19 | -83 | AR | 1.55 | 166
2021 - 06 - 10 | 18:07:54 | 22 | -135 | AP | 1.82 | 217
2022 - 05 - 24 | 22:41:48 | 35 | -77 | PE | 1.7 | 159
Table 1: The table shows the summary of the studied events. The date and time
of every event (derived from K-Cor images) is provided, followed by the
latitude and longitude (in Stonyhurst heliographic coordinates) of the source
region of the CME. The source region column (AR - Active Regions, AP - Active
Prominences and PE - Prominence Eruptions) provides the class of the
identified source region. The last two columns provide the critical height and
the strength of velocity dispersion.

###  2.3 Working Method

For this study, only CMEs that showed clear three part structure in K-Cor and
LASCO images were selected, which strongly restricted the event selection. It
was further ensured that the CME was observed sideways in the plane of the
sky, with a distinct leading edge that remained un-diluted in successive
frames. K-Cor being a ground-based facility, data acquisition is restricted to
only a few hours during the day, which further constrained the event
selection. To identify the source regions of the CMEs, we follow the
methodology of Pant et al. (2021) and Majumdar et al. (2023) on the SWAP and
AIA images, and the identified source regions are shown in Table 1. The Active
Regions (AR) are the usual flaring regions showing bright emission in extreme
ultraviolet wavelengths, the Prominences that erupt from the quiet Sun regions
are denoted as PEs and the prominences whose foot-points are rooted in an
Active Region are classified as APs. It can be seen from the values of the
longitudes that almost all the CMEs studied in this work are limb events,
(which is also an outcome of our event selection, as we selected events that
show the three-part structure) except the one that occurred on June 10, 2021.

Figure 1: A view of the eruption front for the CME that occurred on 5
November, 2014 in (a) SWAP FOV (up to 1.7 R⊙ and in (b) the three-part
structured CME in K-Cor FOV (1.05 - 3 R⊙). A circle fitted to the cross-
section of the CME is shown in (b) with the corresponding minor radius. (c)
Running difference image in SWAP FOV (created using JHelioviewer) with blue
lines tracing the eruption front, and (d) running difference image in K-Cor
FOV (created using JHelioviewer) with Blue lines corresponding to (c) overlaid
on the K-Cor FOV.

##  3 Results

###  3.1 Observations

We manually tracked the LE and the core by eye, and in Figure 2(a), we plotted
the height of the LE (in blue data points) and core (in brown data points) of
the CME on November 5, 2014 as seen in K-Cor and LASCO-C2 FOV and the height
of the eruption front in SWAP FOV (in black data points). We find that the
height of the eruption front in SWAP FOV is very similar to the core height in
K-Cor, thus confirming that SWAP is indeed observing the core of the CME
(which was also noted in O’Hara et al. (2019)). From Figure 2(a), a curvature
is clearly seen in the data points corresponding to K-Cor and SWAP
observations, thus indicating the presence of acceleration at these heights.
However, for an initial rough estimate of the linear speeds of the three
different parts of the CME, we fit a straight line to the different height-
time data and get the linear speeds of the K-Cor LE as 404 km s-1, SWAP front
as 318 km s-1 and K-Cor core as 289 km s-1. Thus, we see a radial gradient in
velocity inside a CME from core to the LE, and hence a presence of velocity
dispersion. It is indeed interesting to see that the speed of the eruption
front in SWAP (that approximately corresponds to the core in K-Cor) is higher
than the speed of the core in K-Cor. We believe this is an outcome of the fact
that the extreme ultraviolet and white-light images do not necessarily capture
the same physical feature, and as the SWAP front is tracked at the outer edge
of the SWAP field of view, lack of proper signal to noise ratio in those
regions would also contribute to the errors in height measurements. A more
detailed investigation of this behavior in a future study in this regard, with
higher resolution data that would enable tracking multiple structures inside a
three part structured CME, and with better signal to noise ratio, would be
crucial and helpful. In Table 1, a measure of the magnitude of dispersion
($\mathrm{V_{disp}}$) is provided as the difference between the linear
velocities of the LE and core in the K-Cor and LASCO FOV as shown in Figure
2(a). We do not include the velocity in the SWAP FOV, since SWAP does not
observe the three-part structure of the CME. To have a better understanding of
this dispersion in velocities, we plot the difference between the heights of
the K-Cor LE and K-Cor core (in blue) and between SWAP front and K-Cor core
(in black) versus time in Figure 2(b), with the corresponding height of the
K-Cor LE on the top axis. It can be seen that initially, the gap between the
tracked features remains constant, thus indicating that velocity dispersion
has not yet started. We find that this regime continues until a critical
height (hc) of 1.54 R⊙, corresponding to an approximate duration of 14 minutes
since their first appearance in the SWAP FOV, after which, the gap starts
increasing. This is the height of commencement of velocity dispersion . Thus
it seems that as the flux-rope starts evolving, initially the entire structure
moves with the same speed. It is only after a finite time that the change in
speeds sets in, which marks the start of velocity dispersion. For the other
five CMEs studied in this work, further examples of the snapshots of the CMEs
as seen in the K-Cor and SWAP FOV, are included in the appendices, in Figure 6
and 7, and a plot of the difference in the LE and core heights for each of
those CMEs is shown in Figure 8.

Figure 2: (a) Height-time plot of the LE and core in blue and brown data
points, for the CME on November 5, 2014 as tracked in the K-Cor FOV, and in
black data points, the eruption front as tracked in the SWAP FOV. (b) Plot of
the difference in height of LE and core for the same CME in the K-Cor FOV and
LASCO FOV (in blue), and the difference in height of the SWAP front and K-Cor
core (in black). The corresponding height of the LE is plotted on the top
axis. Figure 3: A plot of the velocity profile of the LE and core in the
combined K-Cor and LASCO FOV for the CME on November 5, 2014 in (a) and for
the CME on May 24, 2022 in (b). The solid lines represent a smooth spline
fitted to the data points, and the shaded region indicate the uncertainty in
the fitted spline.

In the context of velocity dispersion and the presence of a critical height,
instead of relying on the linear speeds, it is also important to look into the
velocity profiles of the two tracked features in K-Cor and LASCO FOV. The
instantaneous velocity data points and profiles are calculated using smooth
splines (for details, see Majumdar et al., 2020). So, in Figure 3(a) and (b),
we plot the velocity profiles of the LE and the core of two CMEs (in blue and
brown data points and lines respectively). The shaded region denote the
uncertainty in the smooth spline fitting. In Figure 3(a) for the CME on 5
November 2014, a distinct difference is noted in the two velocity profiles. We
find that the LE velocity profile initially rises and then decreases, showing
the presence of the typical impulsive acceleration phase that is known to
occur in the inner corona (Majumdar et al., 2020, 2022). On the other hand,
the velocity profile of the CME core shows similar behaviour but on a smaller
scale of magnitude. This indicates the role of the injected Lorentz force in
accelerating impulsive CMEs suddenly to very high speeds, while the core
appears to experience a weaker acceleration, leading to a much lesser speed,
thereby initiating the dispersion in velocity. However, when analyzing the
velocity profiles of the LE and core for a different CME (24 May 2022) in
Figure 3(b), a different scenario is observed. The LE does not show any signs
of impulsiveness, but a slow yet steady rise in the velocity is noted, which
is a signature of gradual CMEs that experience relatively weaker acceleration
but have a longer acceleration duration. The core, in this case, shows
resemblance with the LE, as a steady increase in the core speed is also noted,
although the speeds of the core and the LE are different. A plot of the
velocity profiles of the LE and core for the other CMEs studied in this paper
is included in the appendices, please see Figure 9.
We find that the critical heights (i.e. the height at which the velocity
dispersion starts) of all the investigated CMEs lie in the range of
$1.42-1.82$ R⊙ (see Table 1), which is in the inner corona. We also note that
CMEs originating from erupting prominences (PE or AP), tend to have higher
critical heights as compared to CMEs originating from ARs, however, since this
conclusion is based on very few events, in future it would be important to
test this on a larger data set.

###  3.2 Simulation

We also conducted a 2.5D numerical simulation of a breakout CME (van der Holst
et al., 2007; Zuccarello et al., 2012; Hosteaux, S. et al., 2018; Talpeanu,
D.-C. et al., 2020, 2022) using Message Passing Interface - Adaptive Mesh
Refinement Versatile Advection Code (MPI-AMRVAC) (Porth et al., 2014; Xia et
al., 2018) under a magnetohydrodynamic (MHD) regime. The initiation of the CME
in the breakout scenario involves reconnection between the multipolar
(quadrupolar) magnetic field and the overlying background magnetic field
(Antiochos, 1998; Antiochos et al., 1999). To trigger the eruption, we induced
shearing motions at the base of the central arcade, resulting in an upward
movement of the arcade and reconnection with the overlying magnetic field. We
tracked the leading edge and the core of the flux rope (shown with two arrows
in Figure 4(a,b) for two different time frames), and the evolution of the gap
between these two features is plotted in Figure 4(c). For the ease of
visualization, the Color-map in Figure 4(a,b) represents the relative running
difference density given by
$\rho_{\text{rel}}=\frac{\rho(t_{i})-\rho(t_{i-1})}{\rho_{0}}$,where the
running difference frames in density, created by subtracting two successive
frames at times $t_{i}$ and $t_{i-1}$, is divided by $\rho_{0}$, which
represents the density of the equilibrium state before applying the shear. We
again find that the gap between the two features remains constant initially,
and a critical height of start of velocity dispersion is thereby noted at a
height of 2.84 R⊙, which lies in the inner corona (as found for the six CMEs
from the observations). The average speed of the LE of the simulated CME was
determined to be $\sim$ 200 km/s, which is also consistent with the earlier
discussion that gradual CMEs tend to exhibit higher critical heights compared
to impulsive CMEs.

Figure 4: (a,b) A simulated breakout CME for two different time frames, with
the two arrows denoting the leading edge and core of the flux-rope. The
referenced time is measured from the initiation of shearing. The color bar
signifies the relative running difference density, denoted as
$\rho_{\text{rel}}=\frac{\rho(t_{i})-\rho(t_{i-1})}{\rho_{0}}$, where
$\rho_{0}$ represents the density of the equilibrium state before applying the
shear, and $\rho(t_{i})$ represents the density in the $i$th time frame. (c) A
plot of the difference in the height of the leading edge and flux-rope centre
for the breakout CME simulated in (a,b) versus time, with the corresponding
leading edge height plotted along the top axis.

###  3.3 Dependence on Flux-rope Cross-section

The phenomenon of velocity dispersion is also influenced by certain
geometrical and morphological properties of CMEs. In Figure 5(a) we plot the
variation of critical height (along the left hand vertical axis) and the
magnitude of velocity dispersion, calculated by taking the difference between
the average speeds of the LE and core, (along the right hand vertical axis)
for all the CMEs, with respect to the CME flux-rope minor radius. The latter
is calculated by fitting a circle to the flux-rope cross section (as shown in
Figure 1(a)) and the corresponding radius of curvature is noted. The circle
was fitted to the CME assuming a flux-rope morphology of the observed CME,
where the flux-rope is embedded in the CME cavity, and a three part structure
with the bright LE and core indicates a rough estimate of the extent of the
flux-rope minor radius of curvature along the CME cross-section (see
Gopalswamy & Yashiro, 2011, for reference). We find that for the CMEs studied
in this work, the critical height is higher for CMEs with larger flux-rope
minor radius, with a strong positive correlation of $0.96$. A regression line
is also fitted which relates the two quantities through the following
equation:

$\mathrm{h_{c}}\,=\,0.8\mathrm{r_{min}}\,+\,1.4$ (1)

where $\mathrm{h_{c}}$ and $\mathrm{r_{min}}$ are in R⊙. Thus a CME with a
larger flux-rope cross-section is expected to experience the onset of
dispersion at a higher height. We also find a positive correlation of the
magnitude of dispersion with the flux-rope minor radius, with a correlation
coefficient of $0.73$, indicating that once the dispersion sets in, the larger
the flux-rope cross-section, the more the dispersion in velocity. This is
expected, since the magnetic tension has an inverse dependence on the radius
of curvature, and hence, the greater the flux-rope minor radius, the lower is
the magnetic tension experienced, leading to greater dispersion in the
velocities.

Figure 5: A plot of the critical height (left-side scale) and magnitude of
velocity dispersion (right-side scale) for all the CMEs, versus (a) flux-rope
minor radius, (b) CME mass.

###  3.4 Dependence on CME Mass

In Figure 5(b), we similarly look into the dependence of critical height and
magnitude of velocity dispersion on the mass of the CME. The mass of the CME
is calculated using the SCC_CALC_CME_MASS.PRO procedure (which is a part of
the STEREO Science Center Software111https://stereo-
ssc.nascom.nasa.gov/software.shtml) in the Solarsoft package of IDL, which
further allows the user to specify the angle of propagation of the CME with
respect to the plane of the sky. In this regard, it should be noted that the
mass of the CME is expected to increase, as the CME is known to accrete mass
in the initial heights. Thus, we measured the mass in successive heights, and
the final maximum mass was taken. We find that the critical height is strongly
and positively correlated with the mass of the CME, with a correlation
coefficient of $0.96$, with the two quantities related by the following
equation,

$\mathrm{h_{c}}\,=\,1.5\,\times\,10^{-16}\mathrm{M}\,-\,1.5$ (2)

where $\mathrm{h_{c}}$ is in R⊙ and M in g. Thus, from this we note that the
more mass a CME has, the later is the onset of dispersion, or the more
reluctant it is towards dispersion. Thus, it seems the velocity dispersion is
an inertial property of the CME, as the higher the mass of a body, the greater
the inertia, and hence the more reluctance towards exhibiting dispersion. We
also find for these six CMEs, a positive correlation of $0.60$ between the
dispersion magnitude and the mass of the CME, which further confirms the above
idea, as once the dispersion sets in, the more massive CMEs are prone to more
dispersion than the lighter ones.

##  4 Discussion and conclusion

In this work, using the combined observations from SWAP, K-Cor and LASCO-C2,
the phenomenon of velocity dispersion inside CMEs is probed. Velocity
dispersion has been studied in very few studies earlier and a crucial missing
element was the implications of this dispersion on the kinematics in the inner
coronal heights. For example, Wood et al. (1999); Koutchmy et al. (2008) study
the relative kinematics of the different parts of a CME, but capture the late
evolutionary phase beyond the inner coronal heights. Our analysis tries to
address this particular missing element in our understanding of CMEs by
studying the imprints of velocity dispersion on the kinematics in the inner
corona. Prior to this work, there have been a few studies on velocity
dispersion in the lower heights (see Schmahl & Hildner, 1977; Srivastava et
al., 2000; Krall et al., 2001; Maričić et al., 2004; Chifu et al., 2012).
However, although Schmahl & Hildner (1977) successfully captured the early
evolution of the associated prominence, they failed to capture the evolution
of the LE at the lower heights. This restricted them from comparing the
relative evolution of the LE and core simultaneously in the lower heights.
Krall et al. (2001) despite studying the velocity profiles in the lower
heights for the different parts of the CMEs, were unable to arrive at clear
conclusions due to challenges with the large noise in the data. The current
work, with the help of the data from K-Cor, SWAP and LASCO-C2, successfully
captures the height-time and velocity-time profiles for both the LE and the
core in a combined field of view from 1.1 - 6 R⊙, thus including the
kinematics of CMEs happening in the inner corona as well as the middle corona
(West et al., 2023). Now, although the above studies focussed on the height-
time profiles, our work extended their findings by probing the height-time
evolution of the separation between the two structures, thereby finding the
presence of a critical height ($\mathrm{h_{c}}$) which marks the onset of
velocity dispersion. Srivastava et al. (2000) studied a gradual CME and found
that the acceleration of the core was delayed with respect to the LE, while
Maričić et al. (2009) noted variations in the onset of acceleration of the two
parts for a few CMEs, while the majority of the CMEs showed well synchronised
accelerations of the LE and core. Now, whether the different critical heights
reported in this work are an outcome of these variations in the acceleration
phases can be explored in future studies. Our results show that the speeds of
the core is less than the LE speeds for all the CMEs, which is also in
agreement with the previous studies. Another important aspect of this study is
the connection of the kinematics with the source region information. Although
we do not find any clear imprint of the source regions on the estimated
critical heights, yet we do find that the critical heights lie relatively
closer to the Sun for CMEs coming from ARs, but the study needs to be extended
to a larger sample set of CMEs to arrive at a strong conclusion on the effect
of source regions. A set of six CMEs were analysed in this work. In the
following, we briefly summarize our main results:

  * •

A clear evidence of velocity dispersion through a radial velocity gradient is
noted inside CMEs from the inner core to the LE in the inner corona. We report
for the first time on the existence of a certain critical height
$\mathrm{h_{c}}$ that marks the onset of velocity dispersion inside CMEs
(Figure 2(b)). This is further supported by the results from a breakout CME
simulated using MPI-AMRVAC, where a critical height is observed as well.

  * •

A look into the speed profiles of the LE and the core in K-Cor for two CMEs
provided two distinct pictures. For the impulsive CME (Figure 3(a)), the LE
speed profile shows the presence of impulsive acceleration phase, while the
core does show a similar profile, but to a lesser extent. On the other hand,
for the gradual CME (Figure 3(b)), we find that both the LE and the core
experience a small gradual acceleration which is reflected into a steady rise
in the speeds.

  * •

We find that the critical height shows a strong positive correlation with the
CME minor radius (Figure 5(a)), indicating that the larger the minor radius,
the more the CME resists dispersion, and hence the higher the critical height.
The magnitude of dispersion is also positively correlated with the minor
radius. Thus, it seems a larger minor radius of CME resists dispersion for a
longer time, but once the dispersion sets in, the strength of the dispersion
is higher for larger minor radius CMEs.

  * •

A strong positive correlation is also found between the mass of the CMEs and
the critical height (Figure 5(b)), indicating the fact that the more massive a
CME is, the later the dispersion starts. Furthermore, a positive correlation
between the mass of the CME and the dispersion strength shows that the more
massive the CME, the greater is the dispersion experienced once the CME
crosses the critical height. Thus from these two factors, it is clear that
velocity dispersion is an inertial property of a CME, as it seems that
initially the mass resists the start of dispersion by increasing the critical
height for more massive CMEs, but once the critical height is reached, the
more massive CMEs experience more severe dispersion in the speeds of their LE
and core.

Thus, it is clear from the above results that the early kinematics of CMEs
lead to a velocity dispersion inside CMEs, that leaves imprints on different
kinematic properties of CMEs. However, it should be noted that the above
conclusions are based on six events, hence a more elaborate statistical study
will be important to have a better understanding of this phenomenon. Also,
since this work uses single vantage point observations of CMEs (which
introduces projection effects), in future, a study of velocity dispersion in
the 3D kinematic profiles (found from 3D CME reconstruction techniques) would
be a crucial follow up work along these lines. Also, recent and upcoming solar
missions like Solar Orbiter (Müller et al., 2020), the recently launched
Aditya-L1 (Prasad et al., 2017) and the upcoming PROBA-3 (Shestov et al.,
2021) missions (will) have coronagraphs that (will) observe the inner corona
with overlapping fields of view and with varied and improved image cadence
(for a comparison, see; Vashishtha et al., 2023). These observations could be
employed to study velocity dispersion in 3D by using different reconstruction
techniques. We believe this work will improve our understanding of the early
kinematics of CMEs in the lower heights, and would motivate more studies along
these lines in the near future. Apart from that, these results will also
provide crucial inputs to models that study CME initiation.

We thank the anonymous reviewer for the insightful comments and suggestions
which have greatly helped in improving the manuscript. SM and DB acknowledge
the PROBA-2 Guest Investigator program grant and the Royal Observatory of
Belgium for all the cooperation throughout this project. SM acknowledges
Vaibhav pant and Ritesh Patel for the very fruitful discussions regarding this
work. The authors acknowledge NCAR for making the K-Cor (DOI:
10.5065/D69G5JV8) data available. This research was funded in whole, or in
part by the Austrian Science Fund (FWF)[P34437]. SWAP is a project of the
Centre Spatial de Liège and the Royal Observatory of Belgium funded by the
Belgian Federal Science Policy Office (BELSPO). Courtesy of the Mauna Loa
Solar Observatory, operated by the High Altitude Observatory as part of the
National Center for Atmospheric Research (NCAR). NCAR is supported by the
National Science Foundation. The SOHO/LASCO data used here are produced by a
consortium of the Naval Research Laboratory (USA), Max-Planck-Institut fuer
Aeronomie (Germany), Laboratoire d’Astronomie (France), and the University of
Birmingham (UK). SOHO is a project of international cooperation between ESA
and NASA. The authors would like to acknowledge the support of Aryabhatta
Research Institute of Observational Sciences (ARIES) and their Computer
Section for facilitation and access to the “Surya” High-Performance Computing
(HPC) Facility in the completion of this work. The ROB team thanks the Belgian
Federal Science Policy Office (BELSPO) for the provision of financial support
in the framework of the PRODEX Programme of the European Space Agency (ESA)
under contract numbers 4000120800, 4000134474, and 4000136681. The authors
acknowledge the benefits from the discussion of the ISSI-BJ Team “Solar
eruptions: preparing for the next generation multi-wavelength coronagraph”.

##  Appendix A Supplementary Images

###  A.1 SWAP and K-Cor Images

In Figure 6 and 7, the corresponding SWAP and K-Cor images for the other five
CMEs out of the six CMEs studied in this work are shown. Figure 6 shows the
eruption front for the CME on 21 December 2014 in SWAP FOV in (a) and the
three part structured CME in K-Cor FOV in (b). Similarly, for the CME on 26
November 2020, the eruption front in SWAP is shown in (c) and the CME in K-Cor
in (d), and for the CME on 7 May 2021, the SWAP observation is shown in (e)
and the K-Cor image in (f). Figure 7 shows the eruption front for the CME on
10 June 2021 in SWAP FOV in (a) and the CME in (b), and for the CME on 24 May
2022, the eruption front in SWAP is shown in (c) and the CME in K-Cor in (d).

Figure 6: Snapshots of the CMEs occurring on 21 December 2014 (top row), 26
November 2020 (middle row) and 7 May 2021 (bottom row) for the eruption front
as observed by SWAP (left column) and for the three part structured CMEs as
seen in the K-Cor (right column) field of view.  Figure 7: Snapshots of the
CMEs occurring on 10 June 2021 (top row), and 24 May 2022 (bottom row) for the
eruption front as observed by SWAP (left column) and for the three part
structured CMEs as observed in K-Cor (right column) field of view.

###  A.2 Kinematic Profiles

In Figure 9, the velocity profiles for the CME LE and core for the other four
CMEs studied in this work are shown. It can be seen again that although both
the LE and core experience acceleration, but the magnitude of the experienced
accelerations are different, as the LE is clearly seen to travel with much
higher speeds than the core. The evolution of the difference in heights
between the LE and core in K-Cor and LASCO images and the difference in
heights between the eruption front in SWAP and core in K-Cor are shown in
Figure 8 for the other five CMEs studied in this work. It can be seen from
Figure 8, that the evolution of the height difference indicates the height of
start of velocity dispersion for all the events. Please note that the
evolution of the gap between the eruption front in SWAP and the K-Cor core is
not shown for the CME on 26 November 2020, as for that CME, the corresponding
SWAP observations were not available. It is worth pointing out that with
better image cadence in the inner coronal observations, it would be possible
in future studies by combining data from Aditya-L1, PROBA-3, to better capture
and understand the initial phase before the start of velocity dispersion (for
a comparison of the different image cadence of different upcoming missions,
see Vashishtha et al., 2023).

Figure 8: Plot of the difference in the height of the LE and core in the
combined K-Cor and LASCO FOV, and the difference between SWAP eruption front
and K-Cor core in the SWAP FOV for the CMEs on 21 December, 2014 (a), 26
November, 2020 (b), 7 May, 2021 (c), 10 June, 2021 (d) and 24 May, 2022 (e).
Please note that in (b), the corresponding SWAP observations were not
available for this particular CME.  Figure 9: Velocity profiles of the LE and
core in the combined K-Cor and LASCO FOV for the CMEs on 21 December, 2014
(a), 26 November, 2020 (b), 7 May, 2021 (c) and 10 June, 2021 (d).

## References

  * Antiochos (1998) Antiochos, S. K. 1998, The Astrophysical Journal, 502, L181, doi: 10.1086/311507
  * Antiochos et al. (1999) Antiochos, S. K., DeVore, C. R., & Klimchuk, J. A. 1999, The Astrophysical Journal, 510, 485, doi: 10.1086/306563
  * Bemporad et al. (2018) Bemporad, A., Pagano, P., & Giordano, S. 2018, A&A, 619, A25, doi: 10.1051/0004-6361/201833058
  * Brueckner et al. (1995) Brueckner, G. E., Howard, R. A., Koomen, M. J., et al. 1995, Sol. Phys., 162, 357, doi: 10.1007/BF00733434
  * Chen (2017) Chen, J. 2017, Physics of Plasmas, 24, 090501, doi: 10.1063/1.4993929
  * Chen & Krall (2003) Chen, J., & Krall, J. 2003, Journal of Geophysical Research (Space Physics), 108, 1410, doi: 10.1029/2003JA009849
  * Chifu et al. (2012) Chifu, I., Inhester, B., Mierla, M., Chifu, V., & Wiegelmann, T. 2012, Sol. Phys., 281, 121, doi: 10.1007/s11207-012-0107-5
  * Cremades & Bothmer (2004) Cremades, H., & Bothmer, V. 2004, A&A, 422, 307, doi: 10.1051/0004-6361:20035776
  * Crifo et al. (1983) Crifo, F., Picat, J. P., & Cailloux, M. 1983, Sol. Phys., 83, 143, doi: 10.1007/BF00148249
  * Domingo et al. (1995) Domingo, V., Fleck, B., & Poland, A. I. 1995, Sol. Phys., 162, 1, doi: 10.1007/BF00733425
  * Forbes et al. (2006) Forbes, T. G., Linker, J. A., Chen, J., et al. 2006, Space Sci. Rev., 123, 251, doi: 10.1007/s11214-006-9019-8
  * Gopalswamy (2016) Gopalswamy, N. 2016, Geoscience Letters, 3, 8, doi: 10.1186/s40562-016-0039-2
  * Gopalswamy & Yashiro (2011) Gopalswamy, N., & Yashiro, S. 2011, ApJ, 736, L17, doi: 10.1088/2041-8205/736/1/L17
  * Halain et al. (2013) Halain, J. P., Berghmans, D., Seaton, D. B., et al. 2013, Sol. Phys., 286, 67, doi: 10.1007/s11207-012-0183-6
  * Hansen et al. (1971) Hansen, R. T., Garcia, C. J., Grognard, R. J. M., & Sheridan, K. V. 1971, PASA, 2, 57, doi: 10.1017/S1323358000012856
  * Hosteaux, S. et al. (2018) Hosteaux, S., Chané, E., Decraemer, B., Talpeanu, D.-C., & Poedts, S. 2018, A&A, 620, A57, doi: 10.1051/0004-6361/201832976
  * Howard et al. (2008) Howard, R. A., Moses, J. D., Vourlidas, A., et al. 2008, Space Sci. Rev., 136, 67, doi: 10.1007/s11214-008-9341-4
  * Hundhausen et al. (1984) Hundhausen, A. J., Sawyer, C. B., House, L., Illing, R. M. E., & Wagner, W. J. 1984, J. Geophys. Res., 89, 2639, doi: 10.1029/JA089iA05p02639
  * Illing & Hundhausen (1985) Illing, R. M. E., & Hundhausen, A. J. 1985, J. Geophys. Res., 90, 275, doi: 10.1029/JA090iA01p00275
  * Kaiser et al. (2008) Kaiser, M. L., Kucera, T. A., Davila, J. M., et al. 2008, Space Sci. Rev., 136, 5, doi: 10.1007/s11214-007-9277-0
  * Koutchmy et al. (2008) Koutchmy, S., Slemzin, V., Filippov, B., et al. 2008, A&A, 483, 599, doi: 10.1051/0004-6361:20078311
  * Krall et al. (2001) Krall, J., Chen, J., Duffin, R. T., Howard, R. A., & Thompson, B. J. 2001, ApJ, 562, 1045, doi: 10.1086/323844
  * Lemen et al. (2011) Lemen, J., Title, A., Boerner, P., et al. 2011, Solar Physics, 275, 17, doi: 10.1007/s11207-011-9776-8
  * Majumdar et al. (2020) Majumdar, S., Pant, V., Patel, R., & Banerjee, D. 2020, ApJ, 899, 6, doi: 10.3847/1538-4357/aba1f2
  * Majumdar et al. (2022) Majumdar, S., Patel, R., & Pant, V. 2022, ApJ, 929, 11, doi: 10.3847/1538-4357/ac5909
  * Majumdar et al. (2021) Majumdar, S., Patel, R., Pant, V., & Banerjee, D. 2021, ApJ, 919, 115, doi: 10.3847/1538-4357/ac1592
  * Majumdar et al. (2023) Majumdar, S., Patel, R., Pant, V., et al. 2023, ApJS, 268, 38, doi: 10.3847/1538-4365/aceb62
  * Maričić et al. (2009) Maričić, D., Vršnak, B., & Roša, D. 2009, Sol. Phys., 260, 177, doi: 10.1007/s11207-009-9421-y
  * Maričić et al. (2004) Maričić, D., Vršnak, B., Stanger, A. L., & Veronig, A. 2004, Sol. Phys., 225, 337, doi: 10.1007/s11207-004-3748-1
  * Moon et al. (2002) Moon, Y. J., Choe, G. S., Wang, H., et al. 2002, ApJ, 581, 694, doi: 10.1086/344088
  * Morgan & Druckmüller (2014) Morgan, H., & Druckmüller, M. 2014, Sol. Phys., 289, 2945, doi: 10.1007/s11207-014-0523-9
  * Morgan et al. (2006) Morgan, H., Habbal, S. R., & Woo, R. 2006, Sol. Phys., 236, 263, doi: 10.1007/s11207-006-0113-6
  * Müller et al. (2017) Müller, D., Nicula, B., Felix, S., et al. 2017, A&A, 606, A10, doi: 10.1051/0004-6361/201730893
  * Müller et al. (2020) Müller, D., St. Cyr, O. C., Zouganelis, I., et al. 2020, A&A, 642, A1, doi: 10.1051/0004-6361/202038467
  * Munro et al. (1979) Munro, R. H., Gosling, J. T., Hildner, E., et al. 1979, Sol. Phys., 61, 201, doi: 10.1007/BF00155456
  * O’Hara et al. (2019) O’Hara, J. P., Mierla, M., Podladchikova, O., D’Huys, E., & West, M. J. 2019, ApJ, 883, 59, doi: 10.3847/1538-4357/ab3b08
  * Pant et al. (2021) Pant, V., Majumdar, S., Patel, R., et al. 2021, Frontiers in Astronomy and Space Sciences, 8, 73, doi: 10.3389/fspas.2021.634358
  * Pesnell et al. (2012) Pesnell, W. D., Thompson, B. J., & Chamberlin, P. C. 2012, Sol. Phys., 275, 3, doi: 10.1007/s11207-011-9841-3
  * Porth et al. (2014) Porth, O., Xia, C., Hendrix, T., Moschou, S. P., & Keppens, R. 2014, ApJS, 214, 4, doi: 10.1088/0067-0049/214/1/4
  * Prasad et al. (2017) Prasad, B., Banerjee, D., Singh, J., et al. 2017, Current Science, 113, 613, doi: 10.18520/cs/v113/i04/613-615
  * Santandrea et al. (2013) Santandrea, S., Gantois, K., Strauch, K., et al. 2013, Sol. Phys., 286, 5, doi: 10.1007/s11207-013-0289-5
  * Schmahl & Hildner (1977) Schmahl, E., & Hildner, E. 1977, Sol. Phys., 55, 473, doi: 10.1007/BF00152588
  * Seaton et al. (2013) Seaton, D. B., Berghmans, D., Nicula, B., et al. 2013, Sol. Phys., 286, 43, doi: 10.1007/s11207-012-0114-6
  * Shestov et al. (2021) Shestov, S. V., Zhukov, A. N., Inhester, B., Dolla, L., & Mierla, M. 2021, A&A, 652, A4, doi: 10.1051/0004-6361/202140467
  * Song et al. (2022) Song, H., Li, L., & Chen, Y. 2022, ApJ, 933, 68, doi: 10.3847/1538-4357/ac7239
  * Srivastava et al. (2000) Srivastava, N., Schwenn, R., Inhester, B., Martin, S. F., & Hanaoka, Y. 2000, ApJ, 534, 468, doi: 10.1086/308749
  * Talpeanu, D.-C. et al. (2020) Talpeanu, D.-C., Chané, E., Poedts, S., et al. 2020, A&A, 637, A77, doi: 10.1051/0004-6361/202037477
  * Talpeanu, D.-C. et al. (2022) Talpeanu, D.-C., Poedts, S., D´Huys, E., Mierla, M., & Richardson, I. G. 2022, A&A, 663, A32, doi: 10.1051/0004-6361/202243150
  * Temmer (2021) Temmer, M. 2021, Living Reviews in Solar Physics, 18, 4, doi: 10.1007/s41116-021-00030-3
  * van der Holst et al. (2007) van der Holst, B., Jacobs, C., & Poedts, S. 2007, The Astrophysical Journal, 671, L77, doi: 10.1086/524732
  * Vashishtha et al. (2023) Vashishtha, N., Majumdar, S., Patel, R., Pant, V., & Banerjee, D. 2023, Frontiers in Astronomy and Space Sciences, 10, 1232197, doi: 10.3389/fspas.2023.1232197
  * Vourlidas et al. (2013) Vourlidas, A., Lynch, B. J., Howard, R. A., & Li, Y. 2013, Sol. Phys., 284, 179, doi: 10.1007/s11207-012-0084-8
  * Webb & Howard (2012) Webb, D. F., & Howard, T. A. 2012, Living Reviews in Solar Physics, 9, 3, doi: 10.12942/lrsp-2012-3
  * West et al. (2023) West, M. J., Seaton, D. B., Wexler, D. B., et al. 2023, Sol. Phys., 298, 78, doi: 10.1007/s11207-023-02170-1
  * Wood et al. (1999) Wood, B. E., Karovska, M., Chen, J., et al. 1999, ApJ, 512, 484, doi: 10.1086/306758
  * Xia et al. (2018) Xia, C., Teunissen, J., Mellah, I. E., Chané, E., & Keppens, R. 2018, The Astrophysical Journal Supplement Series, 234, 30, doi: 10.3847/1538-4365/aaa6c8
  * Zhang et al. (2001) Zhang, J., Dere, K. P., Howard, R. A., Kundu, M. R., & White, S. M. 2001, ApJ, 559, 452, doi: 10.1086/322405
  * Zuccarello et al. (2012) Zuccarello, F. P., Meliani, Z., & Poedts, S. 2012, The Astrophysical Journal, 758, 117, doi: 10.1088/0004-637X/758/2/117


#  A Robust Controller for Stable 3D Pinching Using Tactile Sensing

Efi Psomopoulou1, Nicholas Pestell1, Fotios Papadopoulos2, John Lloyd1, Zoe
Doulgeri3 and Nathan F. Lepora1 Manuscript received: February, 24, 2021;
Revised April, 27, 2021; Accepted June, 24, 2021.This paper was recommended
for publication by Editor Dan Popa upon evaluation of the Associate Editor and
Reviewers’ comments. This work was supported by a Leverhulme Leadership Award
on an “A biomimetic forebrain for robot touch” (RL-2016-39). (Corresponding
author: Efi Psomopoulou.)1EP, NP, JL and NFL are with the Department of
Engineering Mathematics and Bristol Robotics Laboratory, University of
Bristol, Bristol BS8 1QU, U.K<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>is
with the Shadow Robotics Company, London NW5 1LP, U.K.
<EMAIL_ADDRESS>is with the Department of Electrical and Computer
Engineering, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece.
<EMAIL_ADDRESS>Object Identifier (DOI): see top of this page.

###### Abstract

This paper proposes a controller for stable grasping of unknown-shaped objects
by two robotic fingers with tactile fingertips. The grasp is stabilised by
rolling the fingertips on the contact surface and applying a desired grasping
force to reach an equilibrium state. The validation is both in simulation and
on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with
custom-built optical tactile sensors (based on the BRL TacTip). The controller
requires the orientations of the contact surfaces, which are estimated by
regressing a deep convolutional neural network over the tactile images.
Overall, the grasp system is demonstrated to achieve stable equilibrium poses
on various objects ranging in shape and softness, with the system being robust
to perturbations and measurement errors. This approach also has promise to
extend beyond grasping to stable in-hand object manipulation with multiple
fingers.

###### Index Terms:

Force and Tactile Sensing; Grasping.

##  I INTRODUCTION

Achieving human-like dexterity with robot hands has been a major goal of
robotics for many years. Various novel robot hands have been built, often
focusing on their morphology to resemble the human hand and imitate its
functionality with use of soft materials and underactuation [1, 2, 3, 4].
However, our understanding of how signals interact between the brain, the
sensory system and the fingers is still limited. Several review papers
summarising progress on multi-finger robot grasping and dexterous object
manipulation have been published in the last decade [5, 6]. A central message
is that it remains a challenge to combine all of the mechanical and tactile
sensing components necessary to achieve the performance of the human hand with
robots.

Although robotic hands with integrated soft tactile sensors are becoming more
common, almost all of the research focuses on object recognition, slip
detection and grasp success prediction [7, 8, 9], with little work on applying
tactile sensing to achieve or improve grasp stability. However, grasp
stability and manipulation dexterity appear to be closely connected to the
rolling ability of human fingertips, which allows for fine adjustment of the
contact positions with an object [10, 11]. Therefore, being able to robustly
control fingertip rolling via proprioceptive and tactile feedback could bridge
the gap between human and robot grasping/manipulation in unstructured and
unknown environments.

---
Figure 1: The Shadow Modular Grasper, with two tactile fingertips, stably
grasping a brain-shaped stress toy. The controller has rolled the fingertips
so that the two lines connecting the fingertips and the contact points are
parallel. Tactile images from the two fingertips are also shown.

The present work makes progress in this direction by proposing and
implementing a control law that makes use of tactile feedback to roll the
fingertips of two robotic fingers on an object’s surface into a stable grasp.
We focus on the grasp itself, assuming that the robot hand has already reached
the object in a ready-to-grasp configuration. Our previous work focused on the
planar case where no tactile sensing was needed [12]. Here we extend the
controller to 3D where tactile information is necessary to achieve grasp
stability. The proposed control law allows pinching of an arbitrary-shaped
object, using tactile information about the contact orientation to attain a
stable equilibrium state by fingertip-rolling motions. No knowledge of the
system dynamics is needed nor any trajectory planning performed for the finger
movement. A preset desired grasping force is further achieved and the control
law can be extended to achieve in-hand manipulation using touch.

The main contributions of this work concern a robust controller for stable
pinch grasping using tactile sensing:
1) We implement a low-complexity controller that robustly achieves grasp
stability with two fingers, using only proprioceptive and contact information
from tactile sensing.
2) The controller is validated by both simulations and experiments on novel
objects with a fully-actuated hand (Shadow Modular Grasper) with custom
tactile-sensing fingertips.
3) We show that the system dynamically achieves an equilibrium with a pre-set
grasping force from varied initial configurations, assuming it is reachable by
a rolling motion.

##  II BACKGROUND AND RELATED WORK

Early research work on grasp control has focused on detailed grasp analysis
and planning of form and force closure grasps involving the accurate planning
of fixed contact locations and contact forces [13, 14, 15]. More recently, the
availability of grasp planning simulators has helped data-driven methods
become popular. These approaches rely on sampling grasp candidates from some
knowledge base and ranking them according to a specified metric [16, 17]. All
the above approaches are characterised by static analysis.

Although force closure implies the existence of an equilibrium, this is not
sufficient for ensuring grasp stability [13]. The need for further study of
grasp dynamics and the development of analytical models that better resemble
reality is identified in Bohg et al. [5]. One way to work towards this is to
design model-free grasp controllers that dynamically achieve a stable grasp
equilibrium state. Previous research work in this direction includes feedback
control laws of low complexity based on rolling contacts [18, 11].

Grasp control needs state information about the contact, which is provided by
tactile sensing at the fingertips. There has been much work on tactile sensing
in robot hands using capacitive sensors, tactile skin, BioTac sensors,
barometric sensors [19, 20, 21] or optical tactile sensors, such as the
GelSight [22] and the BRL TacTip [23, 24]. This work focuses on the TacTip
because it has been integrated recently into a fully-actuated hand, the Shadow
Modular Grasper [25]. The TacTip is a biomimetic optical tactile sensor based
on the dermal papillae structure in human tactile skin, and is fabricated by
3D-printing pinlike structures in a compliant skin imaged with a camera.
Estimating contact state information, such as the pose of an object’s surface,
has been recently demonstrated using deep learning on the tactile images [26].

This paper focuses on controllers that dynamically achieve a stable grasp
equilibrium state. It extends previous work that focused on a planar problem
of achieving a stable grasp and desired relative finger orientation by rolling
fingertips on the object’s surface, the direction of which is singular [12].
This work considers the full 3D case where system dynamics are more complex
and rolling motion on the contact surface has two degrees of freedom. As the
complexity of the problem increases, the need for tactile information arises.
By using a tactile sensor which accurately perceives the orientation of the
contacted surface, such as the TacTip, we propose a robust controller that
dynamically achieves grasp stability.

##  III METHODS

###  III-A System modelling

Consider two robotic fingers of $n_{i}$ degrees of freedom (DOF), labelling
the fingers $i=1,2$, with revolute joints and rigid hemispherical tips of
radius $r_{1}=r_{2}=r$ in contact with a rigid object of arbitrary shape. This
paper makes the following assumptions:
i) An equilibrium state is assumed reachable by fingertip rolling motion on
the object surface.
ii) Friction at the contacts is sufficient to sustain tangential contact
forces so that a rolling constraint holds at all times.
iii) Both fingertips and the objects are assumed rigid.
iv) The mass of the object is small enough to ignore gravity.

The vector
$\mathbf{q}_{i}=\begin{bmatrix}q_{i1}&q_{i2}&\ldots&q_{in_{i}}\end{bmatrix}^{T}$
denotes the joint angles for the $i_{th}$ finger and $\mathbf{p}_{o}$
$\in\mathbb{R}^{3}$ is the object’s position vector. Let $\mathbf{p}_{c_{i}}$
$\in\mathbb{R}^{3}$ describe the position vector of each finger’s contact
point with the object. The contact frame’s orientations are defined by the
normal unit vector pointing inwards $\mathbf{n}_{z_{i}}$ $\in\mathbb{R}^{3}$
and the tangential unit vectors to the contact surface,
$\mathbf{t}_{x_{i}},\mathbf{t}_{y_{i}}$ $\in\mathbb{R}^{3}$ (Fig. 2).

The system is modelled under the following contact and rolling constraints:

$\displaystyle\begin{bmatrix}D_{ii}&D_{i3}\end{bmatrix}\begin{bmatrix}\mathbf{\dot{q}}_{i}\\\
\mathbf{\dot{p}}_{o}\\\
\boldsymbol{\omega}_{o}\end{bmatrix}=0,\,\,\begin{bmatrix}A_{ii}&A_{i3}\end{bmatrix}\begin{bmatrix}\mathbf{\dot{q}}_{i}\\\
\mathbf{\dot{p}}_{o}\\\ \boldsymbol{\omega}_{o}\end{bmatrix}=0,$ (1)

where $\mathbf{\dot{q}}_{i}\in\mathbb{R}^{n_{i}}$ are the joint velocities of
the $i_{th}$ finger, $\mathbf{\dot{p}}_{o}$, $\boldsymbol{\omega}_{o}$ are the
object’s translational and rotational velocities, and

$\displaystyle D_{ii}$ $\displaystyle=\mathbf{n}_{z_{i}}^{T}J_{v_{i}},\\!\\!$
$\displaystyle
D_{i3}=\begin{bmatrix}-\mathbf{n}_{z_{i}}^{T}&\mathbf{n}_{z_{i}}^{T}\hat{p}_{{oc}_{i}}\end{bmatrix}$
$\displaystyle A_{ii}$
$\displaystyle=\begin{bmatrix}\mathbf{t}_{x_{i}}^{T}{J_{v_{i}}}+r\mathbf{t}_{y_{i}}^{T}J_{\omega_{i}}\vspace{+1mm}\\\
\mathbf{t}_{y_{i}}^{T}{J_{v_{i}}}-r\mathbf{t}_{x_{i}}^{T}J_{\omega_{i}}\end{bmatrix},\\!$
$\displaystyle
A_{i3}=\begin{bmatrix}-\mathbf{t}_{x_{i}}^{T}&\mathbf{t}_{x_{i}}^{T}\hat{p}_{oc_{i}}\vspace{+1mm}\\\
-\mathbf{t}_{y_{i}}^{T}&\mathbf{t}_{y_{i}}^{T}\hat{p}_{oc_{i}}\end{bmatrix}$

with $J_{v_{i}}=J_{v_{i}}(\mathbf{q}_{i})\in\mathbb{R}^{3\times n_{i}}$,
$J_{\omega_{i}}=J_{\omega_{i}}(\mathbf{q}_{i})\in\mathbb{R}^{3\times n_{i}}$
being the translational and rotational Jacobian matrices,
$\mathbf{p}_{oc_{i}}=\mathbf{p}_{c_{i}}-\mathbf{p}_{o}$ and
$\hat{p}_{{oc}_{i}}$ is the skew-symmetric matrix formed by the elements of
$\mathbf{p}_{{oc}_{i}}$. The symbol “$\ \ \widehat{}\ \ $” over a vector
denotes the above diffeomorphism.

The first equation in (1) is the contact constraint implying that the
fingertip cannot leave the object’s surface. This is true as the system is
considered after initial contact with the object is established and as long as
the proposed controller applies contact forces inwards the object. The second
equation in (1) is the rolling constraint denoting that the velocity of the
contact point on the fingertip surface is equal to the velocity of the contact
point on the object surface; i.e. that the friction at the contact is
sufficient to sustain the tangential contact forces for the rolling motion.

The system dynamics under the contact and rolling constraints (1) is described
by the following equations for both fingers and the object:

$\displaystyle M_{i}(\mathbf{q}_{i})\mathbf{\ddot{q}}_{i}$
$\displaystyle+C_{i}(\mathbf{q}_{i},\mathbf{\dot{q}}_{i})\mathbf{\dot{q}}_{i}+{D_{ii}}^{T}f_{i}+{A_{ii}}^{T}\boldsymbol{\lambda}_{i},$
$\displaystyle+J_{\omega_{i}}^{T}K_{s_{i}}Q_{s}\boldsymbol{\omega}_{rel_{i}}=\mathbf{u}_{i}$
(2) $\displaystyle M\begin{bmatrix}\mathbf{\ddot{p}}_{o}\\\
\boldsymbol{\dot{\omega}}_{o}\end{bmatrix}$
$\displaystyle+{D_{13}}^{T}f_{1}+{D_{23}}^{T}f_{2}+{A_{13}}^{T}\boldsymbol{\lambda}_{1}+{A_{23}}^{T}\boldsymbol{\lambda}_{2}$
$\displaystyle-\begin{bmatrix}0_{3\times 3}\\\
K_{s_{1}}\end{bmatrix}Q_{s}\boldsymbol{\omega}_{rel_{1}}-\begin{bmatrix}0_{3\times
3}\\\ K_{s_{2}}\end{bmatrix}Q_{s}\boldsymbol{\omega}_{rel_{2}}=0,$ (3)

where $M_{i}(\mathbf{q}_{i})\\!\\!\in\\!\\!\mathbb{R}^{n_{i}\times n_{i}}\\!$,
$C_{i}(\mathbf{q}_{i},\mathbf{\dot{q}}_{i})\mathbf{\dot{q}}_{i}\in\mathbb{R}^{n_{i}}$
denote the positive definite inertia matrix and the vector of Coriolis and
centripetal forces of the $i^{\rm th}$ finger respectively. The lagrange
multipliers $f_{i}$ and
$\boldsymbol{\lambda}_{i}=\begin{bmatrix}\lambda_{y_{i}}&\lambda_{z_{i}}\end{bmatrix}^{T}$are
associated with the contact and rolling constraints, respectively.
$\mathbf{u}_{i}\in\mathbb{R}^{n_{i}}$ is the vector of applied joint torques,
$M\\!\\!=\\!\\!{\rm diag}\left(m_{o}I_{3},I_{o}\right)$ is the object’s
positive definite inertia matrix with object mass $m_{o}$ and moment of
inertia matrix $I_{o}$, with $I_{3}$ the identity matrix of size 3. The
relative rotational velocity between a fingertip and the object is
$\boldsymbol{\omega}_{rel_{i}}=\boldsymbol{\omega}_{t_{i}}-\boldsymbol{\omega}_{o}\in\mathbb{R}^{3}$,
with rotational velocities $\boldsymbol{\omega}_{t_{i}}\in\mathbb{R}^{3}$. The
friction coefficients $K_{s_{i}}$ are a diagonal matrix associated with the
object’s spinning motion which may occur around the interaction line
$\dfrac{\mathbf{p}_{c_{1}}-\mathbf{p}_{c_{2}}}{\|\mathbf{p}_{c_{1}}-\mathbf{p}_{c_{2}}\|}\triangleq\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$
and $Q_{s}$ is the projection matrix on the interaction line which
$Q_{s}=\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}^{T}$.

###  III-B Controller for stable pinching

The rolling motion of the fingertips on the contact surface can be separated
into two tangential components. The two tangential directions
$\mathbf{t}_{x_{i}}$, $\mathbf{t}_{y_{i}}$ and their corresponding contact
normals $\mathbf{n}_{z_{i}}$ define the contact frames (see Fig. 2).

(a)

(b)

Figure 2: Contact point $\\{c_{i}\\}$ and fingertip $\\{t_{i}\\}$ frames at
the equilibrium state of grasping a trapezoidal object. The lines connecting
the contact points and the fingertips are parallel.

The rolling motion of each fingertip on the contact surface can be analysed
into its two components as follows:

  * •

Rolling in the $\mathbf{t}_{x_{i}}$ direction, around $\mathbf{t}_{y_{i}}$ by
angle $\phi_{i}$.

  * •

Rolling in the $\mathbf{t}_{y_{i}}$ direction, around $\mathbf{t}_{x_{i}}$ by
angle $\psi_{i}$.

These angles $\phi_{i}$, $\psi_{i}$ represent the rolling distances covered by
each fingertip $i=1,2$:
$\dot{\phi}_{1}=\mathbf{t}_{y_{1}}^{T}\boldsymbol{\omega}_{t_{1}}$,
$\dot{\phi}_{2}=-\mathbf{t}_{y_{2}}^{T}\boldsymbol{\omega}_{t_{2}}$,
$\dot{\psi}_{1}=\mathbf{t}_{x_{1}}^{T}\boldsymbol{\omega}_{t_{1}}$,
$\dot{\psi}_{2}=\mathbf{t}_{x_{2}}^{T}\boldsymbol{\omega}_{t_{2}}$.

We propose the following grasping controller for each finger’s joint torques:

$\displaystyle\mathbf{u}_{i}=$ $\displaystyle-
K_{vi}\mathbf{\dot{q}}_{i}-(-1)^{i}f_{d}J_{vi}^{T}\frac{\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}}{\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|}$
$\displaystyle-rf_{d}J_{\omega
i}^{T}\big{[}(-1)^{i+1}\mathbf{t}_{x_{i}}\sin\psi-\mathbf{t}_{y_{i}}\sin\phi\big{]},$
(4)

where

$\displaystyle\psi=\psi_{1}-\psi_{2},$ $\displaystyle\phi=\phi_{2}-\phi_{1}$
(5)

and $\mathbf{p}_{t_{i}}$ is the fingertip position vector.

The controller’s tunable parameters are $f_{d}$ which is a positive constant
setting the desired grasping force magnitude at each contact and
$K_{vi}\in\mathbb{R}^{n_{i}\times n_{i}}$ which is a positive definite
diagonal matrix denoting the damping gain of each joint and its values are
chosen empirically such that, given a desired $f_{d}$, the system’s
performance is as smooth as possible.

The first term of (4) is introduced for joint damping. The second term
represents applied forces of magnitude $f_{d}$ at the direction of the line
connecting the fingertips
$\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}\triangleq\dfrac{\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}}{\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|}$
and the third term expresses the tangential contact forces at equilibrium.

The closed-loop system under the proposed controller as proved in the Appendix
is passive and asymptotically converges to an equilibrium state manifold that
satisfies the following conditions:
1) The line connecting the centre of the fingertips
$\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$ is parallel to the line
connecting the contact points $\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$.
2) The contact forces that are applied along the
$\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$ direction have a magnitude of
$f_{d}$ $N$.
3) At equilibrium $\psi=\beta_{\psi}$ and $\phi=\beta_{\phi}$ with
$\sin\beta_{\psi}=\mathbf{t}_{y_{i}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$,
$\sin\beta_{\phi}=(-1)^{i+1}\mathbf{t}_{x_{i}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$.

Notice that the equilibrium state manifold describes antipodal grasps along
$\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$ of $f_{d}$ magnitude. Hence,
the proposed controller achieves convergence to an equilibrium state in this
manifold without presetting or measuring positions of contacts $\\{c_{1}\\}$,
$\\{c_{2}\\}$. We remark that as there are multiple equilibrium states that
satisfy the above conditions, different initial states lead to different
equilibrium states in the manifold. In consequence, the initial state of the
system does not necessarily correspond to an equilibrium. The closed-loop
system’s passivity means that after any non-persistent disturbance the system
will always converge to a state in the equilibrium manifold.

Further notice that the proposed control law in equations (4,5) assumes
knowledge of the tangential directions at the contacts; therefore an estimate
of the contact surface orientation is needed, which is here achieved by using
tactile sensing. This is in contrast to our previous work where tactile
sensing was not utilised [12]. However, we will show that the controller is
robust to measurement errors of the contact surface orientation. Other
quantities are calculated using only the robotic finger forward kinematics and
the radius of the hemispherical tips.

The proposed control law dynamically stabilises the grasp of an object of any
shape via fingertip rolling on the contact surfaces with a desired grasp force
$f_{d}$. The $f_{d}$ parameter should be set so that assumption (ii) holds and
contact forces stay within the friction cone. The controller does not assume
solving the inverse kinematics of the system for trajectory planning neither
does it depend on the knowledge of its dynamic parameters.

###  III-C Tactile data acquisition and processing for the Shadow Modular
Grasper

The Shadow Modular Grasper is a fully-actuated three-fingered robotic hand
with 9 DoF (3 DoF per finger). The hand has a payload of 2 kg, with each
finger applying up to 10 N of normal continuous force and each joint has a
dedicated torque sensor for a 10 kHz closed-loop control [2]. The hand is
fully integrated with ROS and is provided with open APIs for grasping control.

Tactile sensing is enabled on the hand by replacing the fingertips of the
Modular Grasper with custom-built tactile sensors [25] (see Fig 1). The
tactile fingertips are adapted from an optical biomimetic tactile sensor
developed in Bristol Robotics Laboratory, the BRL TacTip [23]. Deformation of
the tactile sensing pad is imaged with the internal camera at its native
resolution of 1920 x 1080, then adaptively thresholded with a Gaussian filter
(width 27, mean 0 pixels) and subsampled/cropped to 240x135-pixel greyscale
images (Fig. 1). All image acquisition and processing was carried out in
Python OpenCV.

Hyperparameters | Optimised values
---|---
$\\#$ convolutional hidden layers, $N_{\rm conv}$ | 5
$\\#$ convolutional kernels, $N_{\rm filters}$ | 256
$\\#$ dense hidden layers, $N_{\rm dense}$ | 2
$\\#$ dense hidden layer units, $N_{\rm unit}$ | 64
hidden layer activation function | eLU
dropout coefficient | 0.01
L1-regularisation coefficient | 0.0001
L2-regularisation coefficient | 0.01
batch size | 16
TABLE I: Neural network and learning hyperparameters.
---
Figure 3: Data collection set-up with tactile fingertip mounted as an end-
effector on a ABB robot arm. The pose components collected are roll $\phi$,
pitch $\theta$ and depth $z$ relative to the sensor frame. Pose ranges used as
labels to train/test the neural network are shown in the table.

The tactile image data was used to estimate the pose from a contacted surface
relative to the sensor (Fig. 3) and thus estimate the contact frame
orientation of each finger with respect to the hand base. The tactile image
data was also used for contact deformation measurement. A simple yet robust
measure of the difference in tactile images can be found from the Structural
Similarity Index Measure (SSIM) [27], which can be used to measure contact
deformation by comparing a tactile image against a non-deformed reference
image [9]. Here we use $e_{\rm SSIM}(I)=1-{\rm SSIM}(I,I_{\rm ref})$ as a
measure of the deformation of image $I$ compared with the reference image
$I_{\rm ref}$, with SSIM implemented using Python SciKit-Image and computed
from the local means, variances and cross-covariance of the two images [28].
The SSIM-based deformation measure changed gradually as the contact
intensified, making it suitable for use in a feedback controller.

The fingertip is mounted as an end-effector on a 6 DoF robot arm (IRB120, ABB
Robotics). The fingertip then repeatedly contacted a flat surface to gather
labelled contact data to train, validate and test a deep neural network. For
details of the deep learning method used, we refer to [26]. Each of the 5000
samples of data had a random labelled pose (ranges in Table inside Fig. 3).
The network hyperparameters were optimised (Table I) with the training
implemented in the Python Keras library using a GeForce GTX 1660 GPU. This
work considers a two-fingered grasp; thus, the above model procedure was
repeated for both tactile fingertips.

---
Figure 4: Block diagram of the proposed controller with feedback derived from
the tactile images.

###  III-D System Integration

For the purpose of this study, we use the predicted contact orientation from
the tactile sensor (Section III-C) inside the controller of Section III-B. To
that end, the predicted Euler angles are transformed into the unit vectors
$\mathbf{t}_{x_{i}}$ and $\mathbf{t}_{y_{i}}$ and expressed in the Shadow
Modular Grasper’s base frame. The predicted sensor deformation $z$ is used to
detect contact with an object. Two Python models, one per sensor, run on the
host PC and interact with the grasp controller (in C++) which runs in a Docker
container via a ROS-network (Fig. 4).

Each finger’s joint can be controlled in either position or torque mode. The
control is implemented within a 1 kHz update loop running inside the Docker
container. The two fingertip poses are predicted with an update rate of
$\sim$100 ms and then broadcasted to the grasping controller inside the Docker
container via ROS topics.

(a) Initial Configuration
---
(b) Final Configuration
Figure 5: Initial configuration and system equilibrium state for the pinching
controller in simulation for three different objects of mass $0.0021$ kg. A
cube of side length $0.048$ m, a trapezoidal object (height $0.048$ m, small
base $0.0277$ m side angles $30$ & $15$ $\deg$) and a sphere of radius $0.024$
m (zoom-in detail of the parallel lines)

A grasp is comprised of two distinct phases:
(i) Closing phase During this phase all joints are controlled in position mode
and commanded to a set of target angles via a PID which is an implementation
of the rosControl ROS-package. The hand controller switches to the next phase
once contact is detected using the SSIM (Section III-C).
(ii) Grasping phase After both sensors have detected contact, all joints
switch to torque mode and the grasping controller (Section III-B) is
activated, which leads to the hand stably grasping the object.

Figure 6: Finger joint and object velocities for the trapezoidal object as
well as grasping force response.
Figure 7: Equilibrium configuration without (brown fingers) and with (grey
fingers) tactile measurement errors of $30^{\circ},15^{\circ}$ per contact
surface orientation.

##  IV RESULTS

###  IV-A Simulation of stable pinching

Two identical robotic fingers with rigid fingertips (radii $r=0.015$ m) are
simulated in MATLAB. Three different object shapes are used; a cube, a
trapezoidal object and a sphere. They are considered rigid and their
parameters are given in the caption of Fig. 5. The system is simulated under
the proposed controller with damping gains $K_{v_{i}}=0.07I_{4}$ for both
fingers $i=1,2$ and desired grasping force $f_{d}=4\,$N.

The initial system configuration is shown in the top row of Fig. 5. The bottom
row shows the system’s equilibrium state where it is clear that the lines
connecting the fingertips and the contact points are parallel. System time
responses are shown indicatively for the trapezoidal object in Fig. 6 and are
consistent with theoretical expectations. Joint and object velocities converge
to zero and the grasping force converges to the desired magnitude $f_{d}=4$ N.

---
Figure 8: Tactile sensor’s training and validation loss as well as performance
of the trained models for depth, roll and pitch. (a) Initial Configuration
---
(b) Final Configuration
Figure 9: Initial configuration and system equilibrium state for the pinching
controller on the tactile Shadow Modular Grasper for three different objects,
a stack of post-it notes, an empty cardboard box and a plastic lemon.

The only external measurements used by the proposed controller come from the
tactile sensing. Therefore, we investigate the robustness to measurement
errors in the contact surface orientation. To demonstrate this, we simulate a
measurement error of $30^{\circ},15^{\circ}$ for each contact surface
orientation respectively. This is implemented for the trapezoidal object, but
practically it is like the fingers ‘feel’ they are holding a cube. Simulation
results show that the system converges to a different equilibrium state of the
manifold with respect to the final finger-object pose (Fig. 7, equilibrium
configurations with (grey fingers) and without (brown fingers) errors are
overlayed).

###  IV-B Tactile perception of contact surface orientation

Pose component | Sensor 1 MAE | Sensor 2 MAE | range
---|---|---|---
vertical $z$ |  $0.05$ mm |  $0.08$ mm |  $2.5$ mm
roll $\phi$ | $0.22^{\circ}$ | $0.45^{\circ}$ | $16^{\circ}$
pitch $\theta$ | $0.18^{\circ}$ | $0.38^{\circ}$ | $12^{\circ}$
TABLE II: Pose prediction performance.

Two deep convolutional neural networks were trained to regress pose over the
labelled tactile images (Section III-C). The training involved Bayesian
optimisation over the network architecture and hyperparameters [26] (optima
reported in Table I). Pose estimation was then assessed on distinct test sets
for each sensor. Plots of the predictions versus ground truth show good
performance for all pose components ($z$, $\phi$, $\theta$) (Fig. 8). As is
evident from errors due to small vertical pose component (Fig. 8 colourmap
according to $z$), the outliers are caused by little or no contact with the
surface. In the scope of this paper, information from the sensors is only
needed when there is contact with the object. The overall accuracy is similar
for both sensors and typically below 0.1 mm and 0.5∘ (Table II). Small
differences between the two sensors are negligible with respect to their
ranges and can be attributed to variations in their fabrication and optics,
such as the amount of injected gel and the camera placement. These variations
are also a reason for training a network for each sensor.

###  IV-C Stable pinching experimental results on the tactile Shadow Modular
Grasper

We consider four different objects of different shapes and softness (Figs 1,
9-10). The system runs under the proposed controller with damping gains
$K_{v_{i}}=0.07I_{4}$, desired grasping force $f_{d}=10$ N and fingertip
radius $r=0.015$ m.

The controller achieves a stable grasp by rolling the fingertips on the
contact surfaces. The finger configuration is kept as close as possible to the
initial one. Figure 9 shows the initial configuration and the system’s
equilibrium state for households objects whose shapes resemble those used in
simulation (Section IV-A). Note the empty cardboard box in the middle deforms
under grasping into a trapezoid. The experiments in this section show that the
controller achieves stability even in the case of soft materials.

Figure 1 shows the system’s equilibrium state for the brain-shaped object
which is the most challenging to control because it is soft and has an uneven
texture. The fingertip and contact frames are captured from the online
rendition of the frames in RVIZ, a 3D visualisation tool of ROS, and are
overlaid on the experimental setup photo.

In both Figs 1 and 9, it is clear that the lines connecting the fingertips and
contact points are parallel, which matches the theoretical results from
Section III-B. To further test the stability of the system, two external
perturbations were applied by pushing the right finger just before $t=235$ sec
and $t=245$ sec. Fig. 10 shows the finger joint angle velocities which return
to zero after perturbations; hence, the system converges to an equilibrium
state.

Figure 10: Joint angle velocities with external perturbation while grasping an
object. Initial grasp at $220$ sec, large perturbation at $235$ sec, smaller
perturbation at $245$ sec.

##  V DISCUSSION

In this paper, we have presented a dynamically stable controller for pinching
an arbitrary-shaped object with two robotic fingers. The novelty with respect
to our previous work in this area [12] lies in the use of tactile sensing to
address grasp stability in a 3D setting. This was attained by using a
convolutional neural network to extract the orientation of the contact frames
for use in the controller. We showed that the system is robust to tactile
measurement errors, in that a systematic error in estimating the contact
orientation leads to a new equilibrium nearby the original system pose. While
our tactile measurements were accurate on test data (to within 0.5∘),
systematic measurement errors could occur on objects that are distinct from
the planar surface used in training, such as for very curved or soft regions
of the object. That objects of these types (a squidgy brain and lemon) were
stably grasped indicates the method’s robustness. The control also appeared to
be robust to other sources of error. In particular, the theoretical analysis
of the system was based on the assumptions that the fingertips and objects are
rigid and that the mass of the object is small enough to ignore gravity.
However, the controller performed well with soft, curved tactile fingertips
and the objects ranged in softness, curvature and weight.

The controller’s performance depends however on the assumption of rolling
fingertips. In practice, this means that the friction at each contact is such
that the contact force remains in the friction cone during the grasp so that
there is no slippage in the fingertip-object contact and hence rolling is
possible. The value of $f_{d}$ as well as soft contacts by the deformable
fingertips play a role in satisfying this constraint. If a non-satisfaction of
assumptions causes slippage (eg. heavy object, persistent external
perturbation, $\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$ outside of the
friction cones at the contacts), the grasp will fail.

Finally, although the scope of the present work was stable pinch grasping, we
would like to emphasise that the controller should extend to a broader range
of capabilities, including stable in-hand manipulation of unknown objects
using touch. By introducing additional control parameters in the rolling
angles $\phi$ and $\psi$ of (5), the rolling of the fingertips on the contact
surface can be controlled to manipulate the object’s in-hand orientation. In
consequence, the controller could be used to adjust the hand-object composite
to the environment’s geometry, for example to place an object inside a
constrained space. It could also be used to adjust the internal force to
ensure the safety of the grasp by increasing the grasping force in case of
slippage [29]. These topics will be examined in future work, along with
extending the controller to use multiple fingers of the robot hand.

## Appendix: System Equilibrium and Stability Analysis

Substituting (4) into (2), the closed loop system can be written as:

$\displaystyle M_{i}\mathbf{\ddot{q}}_{i}$
$\displaystyle+(C_{i}+K_{vi})\mathbf{\dot{q}}_{i}+J_{\omega_{i}}^{T}K_{s_{i}}Q_{s}\boldsymbol{\omega}_{rel_{i}}+D_{ii}^{T}\Delta
f_{i}$ $\displaystyle+A_{ii}^{T}\begin{bmatrix}\Delta\lambda_{x_{i}}\\\
\Delta\lambda_{y_{i}}\end{bmatrix}+rJ_{\omega_{i}}^{T}\Delta\mathbf{N}_{i}=0$
(6) $\displaystyle M_{o}\mathbf{\ddot{p}}_{o}$
$\displaystyle-\sum_{i=1}^{2}(\mathbf{n}_{z_{i}}\Delta
f_{i}+\mathbf{t}_{x_{i}}\Delta\lambda_{x_{i}}+\mathbf{t}_{y_{i}}\Delta\lambda_{y_{i}})=0$
(7) $\displaystyle I_{o}\boldsymbol{\dot{\omega}}_{o}$
$\displaystyle+\sum_{i=1}^{2}\hat{p_{oc_{i}}}^{T}(\mathbf{n}_{z_{i}}\Delta
f_{i}+\mathbf{t}_{x_{i}}\Delta\lambda_{x_{i}}+\mathbf{t}_{y_{i}}\Delta\lambda_{y_{i}})$
$\displaystyle-\sum_{i=1}^{2}K_{s_{i}}Q_{s}\boldsymbol{\omega}_{rel_{i}}+\mathbf{S}_{N}=0$
(8)

where $\Delta
f_{i}=f_{i}-(-1)^{i+1}f_{d}\mathbf{n}_{z_{i}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$,
$\Delta\lambda_{x_{i}}=\lambda_{x_{i}}-(-1)^{i+1}f_{d}\mathbf{t}_{x_{i}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$,
$\Delta\lambda_{y_{i}}=\lambda_{y_{i}}-(-1)^{i+1}f_{d}\mathbf{t}_{y_{i}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$,
$\Delta\mathbf{N}_{i}=(-1)^{i+1}f_{d}(\mathbf{t}_{y_{i}}\mathbf{t}_{x_{i}}^{T}-\mathbf{t}_{x_{i}}\mathbf{t}_{y_{i}}^{T})\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}+f_{d}\big{[}(-1)^{i+1}\mathbf{t}_{x_{i}}\sin\psi-\mathbf{t}_{y_{i}}\sin\phi\big{]}$
and
$\mathbf{S}_{N}=f_{d}(\hat{p}_{oc_{1}}^{T}-\hat{p}_{oc_{2}}^{T})\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$.

The system equilibrium is found by setting velocities and accelerations to
zero in (6-8). This leads to: $\Delta
f_{i\infty}=\Delta\lambda_{y_{i\infty}}=\Delta\lambda_{z_{i\infty}}=\Delta\mathbf{N}_{i\infty}=S_{N\infty}=0$,
which then gives:

$\displaystyle\big{(}\hat{p}_{oc_{2\infty}}^{T}-\hat{p}_{oc_{1\infty}}^{T}\big{)}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}_{\infty}$
$\displaystyle=0$ (9)
$\displaystyle\psi_{1\infty}-\psi_{2\infty}=\beta_{\psi\infty},$
$\displaystyle\ \ \ \phi_{2\infty}-\phi_{1\infty}=\beta_{\phi\infty}$ (10)

with
$\sin\beta_{\psi\infty}\\!=\mathbf{t}_{y_{i\infty}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}_{\infty}$,
$\sin\beta_{\phi\infty}\\!=(-1)^{i+1}\mathbf{t}_{x_{i\infty}}^{T}\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}_{\infty}$,
and the subscript ∞ denoting equilibrium values.

Notice that
$\mathbf{p}_{oc_{2}}-\mathbf{p}_{oc_{1}}\\!=\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$
is the interaction line vector; hence (9) indicates a zero outer product of
$\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$ with
$\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}$, which implies that these
lines are parallel to each other at equilibrium. Notice also that $\Delta
f_{i\infty}=\Delta\lambda_{y_{i\infty}}=\Delta\lambda_{z_{i\infty}}=0$ satisfy
the object’s translational motion equation (7) at equilibrium. The physical
meaning is that, at equilibrium, since
$\overrightarrow{\mathbf{t}_{1}\mathbf{t}_{2}}||\overrightarrow{\mathbf{c}_{1}\mathbf{c}_{2}}$,
contact forces lie on the interaction line with magnitude $f_{d}$.

To facilitate the stability analysis, we rewrite the closed loop system
equation in a compact form, collecting all Lagrange multipliers in the vector
$\boldsymbol{\lambda}=[f_{1}\ f_{2}\ \boldsymbol{\lambda}_{1}^{T}\
\boldsymbol{\lambda}_{2}^{T}]^{T}$ and all system position variables in
$\mathbf{x}=[\mathbf{q}_{1}^{T}\ \mathbf{q}_{2}^{T}\ \mathbf{p}_{o}^{T}\
\boldsymbol{\omega}_{o}^{T}]^{T}$:

$\displaystyle M_{s}\mathbf{\ddot{x}}\\!$
$\displaystyle+\\!(C_{s}+K_{v})\mathbf{\dot{x}}\\!+\\!A\boldsymbol{\lambda}\\!+\\!B_{J_{\omega}}Q\boldsymbol{\omega}_{rel}\\!+\\!f_{d}{J_{v}}^{T}\frac{\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}}{\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|}$
$\displaystyle+rf_{d}{J_{\omega}}^{T}\begin{bmatrix}\mathbf{t}_{x_{1}}\sin\psi-\mathbf{t}_{y_{1}}\sin\phi\\\
-\mathbf{t}_{x_{2}}\sin\psi-\mathbf{t}_{y_{2}}\sin\phi\\\ 0_{6\times
1}\end{bmatrix}=0$ (11)

with $M_{s}={\rm diag}\left(M_{1},M_{2},M\right)$, $C_{s}={\rm
diag}\left(C_{1},C_{2},0_{6\times 6}\right)$, $K_{v}={\rm
diag}\left(K_{v_{1}},K_{v_{2}},0_{6\times 6}\right)$,
$J_{v}=\begin{bmatrix}-J_{v1}&J_{v2}&0_{3\times 6}\end{bmatrix}$,
$J_{\omega}=\begin{bmatrix}J_{\omega 1}&J_{\omega 2}&0_{3\times
6}\end{bmatrix}$, $Q={\rm diag}\left[K_{s_{1}}Q_{s},K_{s_{2}}Q_{s}\right]$,
$\boldsymbol{\omega}_{rel}=\begin{bmatrix}\boldsymbol{\omega}_{rel_{1}}^{T}&\boldsymbol{\omega}_{rel_{2}}^{T}\end{bmatrix}^{T}$,
$A\\!=\\!\\!\left[\begin{smallmatrix}{D_{11}}^{T}&0_{n_{1}\times
1}&{A_{11}}^{T}&0_{n_{1}\times 2}\\\ 0_{n_{2}\times
1}&{D_{22}}^{T}&0_{n_{2}\times 2}&{A_{22}}^{T}\\\
{D_{13}}^{T}&{D_{23}}^{T}&{A_{13}}^{T}&{A_{23}}^{T}\end{smallmatrix}\right]$,
$B_{J_{\omega}}\\!\\!=\\!\\!\left[\begin{smallmatrix}J_{\omega_{1}}^{T}&0_{5\times
3}\\\ 0_{4\times 3}&J_{\omega_{2}}^{T}\\\ 0_{3\times 3}&0_{3\times 3}\\\
-I_{3}&-I_{3}\end{smallmatrix}\right]$. Similarly, the constraints can be
written as: $A^{T}\mathbf{\dot{x}}=0$. Left-multiplying (11) by
$\mathbf{\dot{x}}^{T}$ yields $\dfrac{dV}{dt}+W=0$, where:

$\displaystyle
V=\frac{1}{2}\mathbf{\dot{x}}^{T}M_{s}\mathbf{\dot{x}}+f_{d}\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|+rf_{d}\big{(}z_{1}(t)+z_{2}(t)\big{)}$
$\displaystyle
W=\sum_{j=1}^{n_{1}}k_{v_{1j}}\dot{q}_{1j}^{2}+\sum_{j=1}^{n_{2}}k_{v_{2j}}\dot{q}_{2j}^{2}+\sum_{i=1}^{2}\boldsymbol{\omega}_{rel_{i}}^{T}K_{s_{i}}Q_{s}\boldsymbol{\omega}_{rel_{i}}$

where $z_{1}(t)=\int_{0}^{\phi}\sin\xi_{1}\mathrm{d}\xi_{1}$,
$z_{2}(t)=\int_{0}^{\psi}\sin\xi_{2}\mathrm{d}\xi_{2}$ and $k_{v_{ij}}$ are
the diagonal elements of the matrix $K_{v_{i}}$. Clearly $V$ is positive
definite with respect to $\mathbf{\dot{x}}$,
$\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|$, $z_{1}(t)$ and $z_{2}(t)$ for
$-\frac{\pi}{2}<\phi<\frac{\pi}{2}$ and $-\frac{\pi}{2}<\psi<\frac{\pi}{2}$ in
the constraint manifold defined by
$\mathcal{M}_{c}(\mathbf{x})=\\{\mathbf{x}\in\mathbb{R}^{n_{1}+n_{2}+6}:A^{T}\mathbf{\dot{x}}=0\\}$.
Given $\dot{V}\\!=\\!-W\leq 0$, it is clear that $V(t)\leq V\left(0\right)$
holds and consequently $\mathbf{\dot{x}}$,
$\|\mathbf{p}_{t_{2}}-\mathbf{p}_{t_{1}}\|$, $z_{1}(t)$ and $z_{2}(t)$ are
bounded. Following the proof line in [12], it is proved that the system
velocities and accelerations converge to zero
$\mathbf{\dot{x}},\mathbf{\ddot{x}}\rightarrow 0$ and that $\mathbf{\dot{x}}$
converges to zero exponentially as $t\rightarrow\infty$.

## References

  * [1] C. Piazza, G. Grioli, M. Catalano, and A. Bicchi, “A century of robotic hands,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 2, no. 1, pp. 1–32, 2019.
  * [2] Shadow Robot Company. (2021) Modular grasper documentation. [Online]. Available: https://modular-grasper.readthedocs.io/en/latest/
  * [3] R. Deimel and O. Brock, “A novel type of compliant and underactuated robotic hand for dexterous grasping,” _The International Journal of Robotics Research_ , vol. 35, no. 1-3, pp. 161–185, 2016.
  * [4] M. Catalano, G. Grioli, E. Farnioli, A. Serio, C. Piazza, and A. Bicchi, “Adaptive synergies for the design and control of the Pisa/IIT softhand,” _The International Journal of Robotics Research_ , vol. 33, no. 5, pp. 768–782, 2014.
  * [5] J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-Driven Grasp Synthesis – A Survey,” _IEEE Transactions on Robotics_ , vol. 30, no. 2, pp. 289–309, 2013.
  * [6] A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” _Science_ , vol. 364, no. 6446, 2019.
  * [7] Z. Kappassov, J.-A. Corrales, and V. Perdereau, “Tactile sensing in dexterous robot hands – Review,” _Robotics and Autonomous Systems_ , vol. 74, pp. 195–220, 2015.
  * [8] H. Yousef, M. Boukallel, and K. Althoefer, “Tactile sensing for dexterous in-hand manipulation in robotics – A review,” _Sensors and Actuators A: Physical_ , vol. 167, no. 2, pp. 171–187, 2011.
  * [9] J. W. James, A. Church, L. Cramphorn, and N. F. Lepora, “Tactile Model O: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand,” _Soft Robotics_ , 2020.
  * [10] S. Arimoto, _Control Theory of Multi-fingered Hands: A Modelling and Analytical-mechanics Approach for Dexterity and Intelligence_. Springer-Verlag London Limited, 2008.
  * [11] R. Ozawa and K. Tahara, “Grasp and dexterous manipulation of multi-fingered robotic hands: a review from a control view point,” _Advanced Robotics_ , vol. 31, no. 19-20, pp. 1030–1050, 2017.
  * [12] E. Psomopoulou, D. Karashima, Z. Doulgeri, and K. Tahara, “Stable pinching by controlling finger relative orientation of robotic fingers with rolling soft tips,” _Robotica_ , vol. 36, no. 2, p. 204–224, 2018.
  * [13] D. Prattichizzo and J. C. Trinkle, “Grasping,” in _Springer Handbook of Robotics_. Springer Berlin Heidelberg, 2008, pp. 671–700.
  * [14] T. Wimbock, C. Ott, a. Albu-Schaffer, and G. Hirzinger, “Comparison of object-level grasp controllers for dynamic dexterous manipulation,” _The International Journal of Robotics Research_ , vol. 31, no. 1, pp. 3–23, Sep. 2011.
  * [15] D. Prattichizzo, M. Malvezzi, M. Gabiccini, and A. Bicchi, “On the manipulability ellipsoids of underactuated robotic hands with compliance,” _Robotics and Autonomous Systems_ , vol. 60, no. 3, pp. 337 – 346, 2012.
  * [16] C. Goldfeder, P. K. Allen, C. Lackner, and R. Pelossof, “Grasp planning via decomposition trees,” in _IEEE International Conference on Robotics and Automation_ , April 2007, pp. 4679–4684.
  * [17] M. T. Ciocarlie and P. K. Allen, “Hand Posture Subspaces for Dexterous Robotic Grasping,” _The International Journal of Robotics Research_ , vol. 28, no. 7, pp. 851–867, June 2009.
  * [18] M. Grammatikopoulou, E. Psomopoulou, L. Droukas, and Z. Doulgeri, “A controller for stable grasping and desired finger shaping without contact sensing,” in _IEEE International Conference on Robotics and Automation_ , May 2014, pp. 3662–3668.
  * [19] R. Kõiva, M. Zenker, C. Schürmann, R. Haschke, and H. Ritter, “A highly sensitive 3D-shaped tactile sensor,” in _IEEE/ASME International Conference on Advanced Intelligent Mechatronics_ , 2013.
  * [20] F. Veiga, B. B. Edin, and J. Peters, “In-hand object stabilization by independent finger control,” _CoRR_ , vol. abs/1806.05031, 2018.
  * [21] L. U. Odhner, L. P. Jentoft, M. R. Claffee, N. Corson, Y. Tenzer, R. R. Ma, M. Buehler, R. Kohout, R. D. Howe, and A. M. Dollar, “A compliant, underactuated hand for robust manipulation,” _The International Journal of Robotics Research_ , vol. 33, no. 5, pp. 736–752, 2014.
  * [22] W. Yuan, S. Dong, and E. Adelson, “GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force,” _Sensors_ , vol. 17, no. 12, p. 2762, 2017.
  * [23] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora, “The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies,” _Soft Robotics_ , vol. 5, no. 2, pp. 216–227, 2018.
  * [24] N. F. Lepora, “Soft biomimetic optical tactile sensing with the TacTip: A review,” _arXiv preprint arXiv:2105.14455_ , 2021.
  * [25] N. Pestell, L. Cramphorn, F. Papadopoulos, and N. F. Lepora, “A sense of touch for the shadow modular grasper,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 2220–2226, April 2019.
  * [26] N. F. Lepora and J. Lloyd, “Optimal deep learning for robot touch: Training accurate pose models of 3d surfaces and edges,” _IEEE Robotics Automation Magazine_ , vol. 27, no. 2, pp. 66–77, 2020.
  * [27] S. Luo, W. Yuan, E. Adelson, A. G. Cohn, and R. Fuentes, “ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition,” in _IEEE International Conference on Robotics and Automation_ , 2018, pp. 2722–2727.
  * [28] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [29] J. W. James and N. F. Lepora, “Slip detection for grasp stabilization with a multifingered tactile robot hand,” _IEEE Transactions on Robotics_ , pp. 1–14, 2020.