††thanks: Formerly Dana Vaknin

# Leggett-Garg-like Inequalities from a Correlation Matrix Construction

Dana Ben Porath  Faculty of Engineering and the Institute of Nanotechnology
and Advanced Materials, Bar Ilan University, Ramat Gan, Israel  Eliahu Cohen
<EMAIL_ADDRESS>Faculty of Engineering and the Institute of
Nanotechnology and Advanced Materials, Bar Ilan University, Ramat Gan, Israel

###### Abstract

The Leggett-Garg Inequality (LGI) constrains, under certain fundamental
assumptions, the correlations between measurements of a quantity $Q$ at
different times. Here we analyze the LGI, and propose similar but somewhat
more elaborate inequalities, employing a technique that utilizes the
mathematical properties of correlation matrices, which was recently proposed
in the context of nonlocal correlations. We also find that this technique can
be applied to inequalities that combine correlations between different times
(as in LGI) and correlations between different locations (as in Bell
inequalities). All the proposed bounds include additional correlations
compared to the original ones and also lead to a particular form of
complementarity. A possible experimental realization and some applications are
briefly discussed.

##  I Introduction

Leggett and Garg, in their seminal work legget-prl1985 , provided constraints
on the correlations between measurements of a single quantity at different
times. They showed that given the definition $C_{ij}\equiv\langle
Q_{i}Q_{j}\rangle$ for the correlation between two measurements of a quantity
$Q$ at times $t_{i}$ and $t_{j}$, the sum $|C_{12}+C_{23}+C_{34}-C_{14}|$ is
bounded by 2 in any scenario that maintains “macrorealism” and “non-invasive
measurability” emary-iop2013 . Such a scenario represents the classical
physics view of a macroscopic system, as a system that cannot be in two or
more states at the same time, and in which it is possible to measure the state
with only an arbitrarily small perturbation of it. Determining whether
Leggett-Garg Inequality (LGI) holds for a given system, assists in
distinguishing between systems that obey this classical view and those that
exhibit non-classical behavior (in the particular sense linked to
macrorealism). If a system violates the LGI it necessarily exhibits non-
classical behavior. However, recent works have shown that the contrary is not
always true, i.e., a system can satisfy the LGI but still violate
macroreaslism majidy-pra2019 ; majidy-pra2021 . The initial motivation for the
LGI was using this method to determine whether quantum coherence appears even
in macroscopic systems. Later works focused on experimentally finding LGI
violations in microscopic quantum systems using various measurement types such
as ideal negative measurements knee-naturecomm2012 . A recent work by Shenoy
et al. shenoy-pra2017  presents the applicability of LGI in the area of
quantum cryptography. They showed how the amount of violation of the LGI
indicates that a hacking attempt was made during the quantum key distribution
protocol bb84-arxiv2020 .

The LGI relates to bounds on correlations of a single quantity measured
typically on the same system at different times by the same party, in a manner
which mathematically resembles the well-known Clauser-Horne-Shimony-Holt
(CHSH) inequalities, providing bounds on correlations between quantities
measured typically on a bipartite system at two different locations by two
different parties bell-physics1964 ; clauser-prl1969 ; Cirel'son-letters1980 .
Similarly to the “classical” LGI bound of 2, previous works have shown that
under quantum assumptions, the same correlations are bounded by $2\sqrt{2}$
budroni-prl2013 . Here we find more informative LGI bounds, tighter than
$2\sqrt{2}$, by studying a more general definition for the correlations, which
under certain conditions coincides with the common LGI correlations for
quantum systems fritz-njp2010 . We apply a mathematical method that was
recently proposed for finding richer bounds for CHSH and other novel
inequalities carmi2018significance ; carmi-scienceadvances2019 ; carmi-njp2019
; te2019multiplicative .

We show that maximal violations of the classical LGI bound in the newly found
inequalities, directly follow from the absence of correlations between an
operator and itself at certain times. On the practical level, our results may
allow one to better design quantum temporal correlations, but fundamentally
this highlights the necessity of certain non-commutativity between the
measured operator and the Hamiltonian (see also cohen2020praise  for the
broader consequences of non-commutativity and uncertainty).

In addition to their theoretical merits, we suggest that these tighter bounds
may be beneficial for applied purposes as well, e.g. for devising quantum key
distribution protocols (similarly to shenoy-pra2017 ) or analyzing quantum
metrological schemes (similarly to frowis-prl2016 ).

For completeness, we show in the Appendix that a different definition of the
temporal correlation leads to other bounds.

##  II Materials and Methods

We define the generalized correlation function $C(X,Y)$ — for any two
Hermitian operators $X$ and $Y$ — as

$C(X,Y)=\frac{\frac{1}{2}\langle\left\\{X,Y\right\\}\rangle-\langle
X\rangle\langle Y\rangle}{\Delta_{X}\Delta_{Y}},$ (1)

where $\\{X,Y\\}$ denotes the anti-commutator and $\Delta_{X}=\sqrt{\langle
X^{2}\rangle-\langle X\rangle^{2}}$, $\Delta_{Y}=\sqrt{\langle
Y^{2}\rangle-\langle Y\rangle^{2}}$ are the standard deviations. From the
Schrödinger Robertson uncertainty relation robertson-pr1929 ; schrodinger-1930
, it is easy to prove that this correlation is bounded between -1 to 1. If $X$
and $Y$ represent projective quantum measurements of values $\pm 1$ with an
expected value of 0, this correlation coincides with the symmetric correlation
$\frac{1}{2}\langle\left\\{X,Y\right\\}\rangle$, which Fritz proposed as a
quantum analog of the LGI correlation fritz-njp2010 .

The motivation behind this generalized correlation definition is twofold:
except for the aforementioned fact that in the case of projective measurements
it generalizes the standard symmetric correlation used for LGI, it is also
widely used in quantum optics adesso-jpa2007 ; pirandola-pra2009  and can be
therefore readily generalized to the case of continuous variables.

After constructing the correlations matrix we will employ semi-positive
definiteness conditions, similar to those in carmi2018significance ; carmi-
scienceadvances2019 ; carmi-njp2019 ; te2019multiplicative , to derive our
Leggett-Garg-like inequalities.

##  III Results

Here we present four theorems and corresponding proofs entailing analytical
bounds for correlations between measurements of a quantum system at different
times. Theorems 1, 2, and 17 discuss elaborate Leggett-Greg-like inequalities
describing constraints on generalized correlations of measurements at four
consecutive measurement times. The measurement of a quantity $Q$ at time
$t_{i}$ is described by the Hermitian operator $Q_{i}$. Theorem 1 is
significant mainly because it provides a tighter bound than $2\sqrt{2}$, which
is the known bound under typical quantum assumptions. Theorem 2 resembles the
TLM inequality which is a significant bound that was derived independently by
Tsirelson, Landau and Masanes tsirelson-jsm1987 ; landau-
foundationsofphysics1988 ; masanes-arxiv2003 . The structure of TLM inequality
is characterized by the fact it bounds products of correlations and not merely
their sums as in other inequalities. Theorem 17 demonstrates a complementarity
relation between all pairs of correlations of the four measurements.

Theorem 4 relates to a combination of CHSH and LGI dressel-pra2014 ; white-
npj2016 , where the constraints are on generalized correlations between
quantities measured at two different times $t_{1}$ and $t_{2}$ by two parties,
Alice and Bob, each in a different location. The two consecutive measurements
of Alice (Bob) are represented by the Hermitian operators $A_{1}$ and $A_{2}$
($B_{1}$ and $B_{2}$).

######  Theorem 1.

Elaborate Leggett-Garg-like inequality.

Given four consecutive measurements, define the generalized LGI parameter as

$L=|C(Q_{1},Q_{2})+C(Q_{2},Q_{3})+C(Q_{3},Q_{4})-C(Q_{1},Q_{4})|.$ (2)

The following holds

$L\leq
2\sqrt{1+\sqrt{1-\max{\left\\{C(Q_{1},Q_{3})^{2},C(Q_{2},Q_{4})^{2}\right\\}}}}.$
(3)

###### Proof of Theorem 1.

Let $C$ be the following correlation matrix

$\displaystyle
C=\begin{pmatrix}C(Q_{2i},Q_{2i})&C(Q_{2i},Q_{1})&C(Q_{2i},Q_{3})\\\
C(Q_{2i},Q_{1})&C(Q_{1},Q_{1})&C(Q_{1},Q_{3})\\\
C(Q_{2i},Q_{3})&C(Q_{1},Q_{3})&C(Q_{3},Q_{3})\end{pmatrix}.$ (4)

where $i=1,2$. $C$ is a positive semi-definite matrix, i.e., $C\succeq 0$ (see
carmi-njp2019 ; carmi-scienceadvances2019  for more details regarding the
construction and properties of such matrices). Therefore, by the Schur
complement condition for positive semi-definiteness,

$\displaystyle\begin{pmatrix}1&C(Q_{1},Q_{3})\\\
C(Q_{1},Q_{3})&1\end{pmatrix}\succeq\begin{pmatrix}C(Q_{2i},Q_{1})\\\
C(Q_{2i},Q_{3})\end{pmatrix}\begin{pmatrix}C(Q_{2i},Q_{1})&C(Q_{2i},Q_{3})\end{pmatrix}.$
(5)

Let $v_{j}^{T}=((-1)^{j},1)$. Multiplying by $v_{j}^{T}$ from the left and
$v_{j}$ from the right, the above inequality implies

$2[1+(-1)^{j}C(Q_{1},Q_{3})]\geq[C(Q_{2i},Q_{3})+(-1)^{j}C(Q_{2i},Q_{1})]^{2}.$
(6)

For $j=i-1$,

$\displaystyle\begin{split}&[C(Q_{2},Q_{1})+C(Q_{2},Q_{3})]^{2}\leq
2\left[1+C(Q_{1},Q_{3})\right]\\\ &[C(Q_{4},Q_{3})-C(Q_{4},Q_{1})]^{2}\leq
2\left[1-C(Q_{1},Q_{3})\right].\end{split}$ (7)

Since $C(Q_{i},Q_{j})=C(Q_{j},Q_{i})$ for any $i$ and $j$, then

$\displaystyle\begin{split}&|C(Q_{1},Q_{2})+C(Q_{2},Q_{3})|\leq\sqrt{2\left[1+C(Q_{1},Q_{3})\right]}\\\
&|C(Q_{3},Q_{4})-C(Q_{1},Q_{4})|\leq\sqrt{2\left[1-C(Q_{1},Q_{3})\right]}.\end{split}$
(8)

Using the triangle inequality on the two expressions in the l.h.s of Eq. (8),
we derive the following

$L\leq 2\sqrt{1+\sqrt{1-C(Q_{1},Q_{3})^{2}}}.$ (9)

By repeating the analytical derivation above for the following correlation
matrix

$\displaystyle\tilde{C}=\begin{pmatrix}C(Q_{2i-1},Q_{2i-1})&C(Q_{2i-1},Q_{4})&C(Q_{2i-1},Q_{2})\\\
C(Q_{2i-1},Q_{4})&C(Q_{4},Q_{4})&C(Q_{4},Q_{2})\\\
C(Q_{2i-1},Q_{2})&C(Q_{4},Q_{2})&C(Q_{2},Q_{2})\end{pmatrix},$ (10)

the multiplications by $v_{j}^{T}$ and $v_{j}$ gives rise to:

$2[1+(-1)^{j}C(Q_{4},Q_{2})]\geq[C(Q_{2i-1},Q_{2})+(-1)^{j}C(Q_{2i-1},Q_{4})]^{2}.$
(11)

From Eq. (11), for the cases $i=j=1$ and $i=j=2$, the following inequality is
derived using the triangle inequality

$L\leq 2\sqrt{1+\sqrt{1-C(Q_{2},Q_{4})^{2}}}.$ (12)

Finally, Theorem 1 follows from Eqs. (9) and (12). Thus, we prove that given
our assumptions, the generalized LGI parameter has a bound which is tighter
than $2\sqrt{2}$, and $2\sqrt{2}$ can be reached only if the correlations
$C(Q_{1},Q_{3})$ and $C(Q_{2},Q_{4})$ are equal to $0$. This result
generalizes the known bound of $2\sqrt{2}$ for the LGI with projective
measurements of values $\pm 1$ budroni-prl2013 . ∎

######  Theorem 2.

Leggett-Garg-like inequality in the TLM form.

Given four consecutive measurements,

$\displaystyle\begin{split}&|C(Q_{2},Q_{1})C(Q_{2},Q_{3})-C(Q_{4},Q_{1})C(Q_{4},Q_{3})|\leq\\\
&\sqrt{(1-C(Q_{2},Q_{1})^{2})(1-C(Q_{2},Q_{3})^{2})}+\sqrt{(1-C(Q_{4},Q_{1})^{2})(1-C(Q_{4},Q_{3})^{2})}.\end{split}$
(13)

###### Proof of Theorem 2.

Eq. (5) implies

$\displaystyle\begin{pmatrix}1-C(Q_{2i},Q_{1})^{2}&C(Q_{1},Q_{3})-C(Q_{2i},Q_{1})C(Q_{2i},Q_{3})\\\
C(Q_{1},Q_{3})-C(Q_{2i},Q_{3})C(Q_{2i},Q_{1})&1-C(Q_{2i},Q_{3})^{2}\end{pmatrix}\succeq
0.$ (14)

The determinant of the above matrix is non-negative, and thus,

$|C(Q_{1},Q_{3})-C(Q_{2i},Q_{1})C(Q_{2i},Q_{3})|\leq\sqrt{(1-C(Q_{2i},Q_{1})^{2})(1-C(Q_{2i},Q_{3})^{2})}.$
(15)

For the cases $i=1$ and $i=2$, we obtain the following inequalities,
respectively,

$\displaystyle\begin{split}&|C(Q_{1},Q_{3})-C(Q_{2},Q_{1})C(Q_{2},Q_{3})|\leq\sqrt{(1-C(Q_{2},Q_{1})^{2})(1-C(Q_{2},Q_{3})^{2})}\\\
&|C(Q_{1},Q_{3})-C(Q_{4},Q_{1})C(Q_{4},Q_{3})|\leq\sqrt{(1-C(Q_{4},Q_{1})^{2})(1-C(Q_{4},Q_{3})^{2})}.\end{split}$
(16)

Finally, Theorem 2 is derived from the triangle inequality and Eq. (16). The
resulting inequality presents the TLM criterion for correlations between
measurements at different times. ∎

######  Theorem 3.

Leggett-Garg-like inequality in the form of a complementarity relation.

Given four consecutive measurements,

$\left(\frac{L}{2\sqrt{2}}\right)^{2}+\left(\frac{C(Q_{1},Q_{3})}{2\sqrt{2}}\right)^{2}+\left(\frac{C(Q_{2},Q_{4})}{2\sqrt{2}}\right)^{2}\leq
1.$ (17)

###### Proof of Theorem 3.

From Eq. (9),

$L^{2}\leq 4\left(1+\sqrt{1-C(Q_{1},Q_{3})^{2}}\right).$ (18)

Since $\sqrt{1-a}\leq 1-a/2$ for $a\in[0,1]$, then

$L^{2}+2C(Q_{1},Q_{3})^{2}\leq 8$ (19)

and similarly we can derive from Eq. (12),

$L^{2}+2C(Q_{2},Q_{4})^{2}\leq 8$ (20)

Theorem 17 is derived after summing Eqs. (19) and (20). This inequality
demonstrates a complementarity relation between all six correlations of pairs
from the four measurements. ∎

######  Theorem 4.

Elaborate Bell-Leggett-Garg-like inequality.

For two consecutive measurements $A_{1}$ and $A_{2}$ of Alice, and two
consecutive measurements $B_{1}$ and $B_{2}$ of Bob, define the generalized
Bell-Leggett-Garg inequality parameter as

$BLG=|C(A_{1},A_{2})+C(A_{1},B_{2})+C(B_{1},B_{2})-C(B_{1},A_{2})|.$ (21)

The following holds

$BLG\leq
2\sqrt{1+\sqrt{1-\max{\left\\{C(A_{1},B_{1})^{2},C(A_{2},B_{2})^{2}\right\\}}}}.$
(22)

###### Proof of Theorem 4.

Let $C_{X}$ be the following correlation matrix

$\displaystyle C_{X}=\begin{pmatrix}C(X,X)&C(X,A_{2})&C(X,B_{2})\\\
C(X,A_{2})&C(A_{2},A_{2})&C(A_{2},B_{2})\\\
C(X,B_{2})&C(A_{2},B_{2})&C(B_{2},B_{2})\end{pmatrix},$ (23)

for $X\in\\{A_{1},B_{1}\\}$. Following the analysis in the proof of Theorem 1,
i.e., using the Schur complement condition for positive semi-definiteness and
after multiplying by $v_{j}^{T}=((-1)^{j},1)$ and $v_{j}$, we obtain

$|C(X,B_{2})+(-1)^{j}C(X,A_{2})|\leq\sqrt{2[1+(-1)^{j}C(A_{2},B_{2})]}.$ (24)

For the two cases, $(X=A_{1}~{}\&~{}j=0)$ and $(X=B_{1}~{}\&~{}j=1)$, we
obtain the following inequalities, respectively,

$\displaystyle\begin{split}&|C(A_{1},B_{2})+C(A_{1},A_{2})|\leq\sqrt{2[1+C(A_{2},B_{2})]}\\\
&|C(B_{1},B_{2})-C(B_{1},A_{2})|\leq\sqrt{2[1-C(A_{2},B_{2})]}.\end{split}$
(25)

From the triangle inequality,

$BLG\leq 2\sqrt{1+\sqrt{1-C(A_{2},B_{2})^{2}}}.$ (26)

Similarly, by replacing the variables $A_{2}$ and $B_{2}$ by $A_{1}$ and
$B_{1}$, respectively,

$BLG\leq 2\sqrt{1+\sqrt{1-C(A_{1},B_{1})^{2}}}.$ (27)

Finally, Theorem 4 is derived from Eqs. (26) and (27). ∎

### An example of a system that upholds our new bounds

Here we demonstrate Theorems 1,2 and 17 for a specific spin model emary-
iop2013 ; halliwell-pra2016 , which is defined by the following Hamiltonian
and observable

$\displaystyle\begin{split}&H=\frac{\hbar\omega}{2}\sigma_{x}=\frac{\hbar\omega}{2}\begin{pmatrix}0&1\\\
1&0\end{pmatrix}\\\ &Q=\sigma_{z}=\begin{pmatrix}1&0\\\
0&-1\end{pmatrix}.\end{split}$ (28)

The time evolution of $Q$ is

$\displaystyle Q_{t}=\begin{pmatrix}\cos(\omega t)&-i\sin(\omega t)\\\
i\sin(\omega t)&-\cos(\omega t)\end{pmatrix},$ (29)

and therefore, according to Eq. (1), the generalized correlation between two
measurements of $Q$ at times $t$ and $s$ is

$C(Q_{t},Q_{s})=\cos{(\omega(t-s))}.$ (30)

To demonstrate Theorem 1, for any four consecutive measurement times, define
$D1$ as the difference between our bound (the r.h.s of Eq. (3)) and the LGI
parameter (Eq. (2)). Thus, in our system,

$\displaystyle\begin{split}D1=&~{}2\sqrt{1+\sqrt{1-\max{\left\\{\cos^{2}{(\omega(t_{1}-t_{3}))},\cos^{2}{(\omega(t_{4}-t_{2}))}\right\\}}}}\\\
&-|\cos{(\omega(t_{1}-t_{2}))}+\cos{(\omega(t_{2}-t_{3}))}+\cos{(\omega(t_{3}-t_{4}))}-\cos{(\omega(t_{1}-t_{4}))}|.\end{split}$
(31)

Similarly, to demonstrate Theorem 2, define $D2$ as the difference between the
r.h.s and the l.h.s of Eq.(13), and thus,

$\displaystyle\begin{split}D2=&~{}|\sin(\omega(t_{2}-t_{1}))\sin(\omega(t_{2}-t_{3}))|+|\sin(\omega(t_{4}-t_{1}))\sin(\omega(t_{4}-t_{3}))|\\\
&-|\cos(\omega(t_{2}-t_{1}))\cos(\omega(t_{2}-t_{3}))-\cos(\omega(t_{4}-t_{1}))\cos(\omega(t_{4}-t_{3}))|.\end{split}$
(32)

It can be shown numerically that $D1,D2\geq 0$ for any $t_{1},t_{2},t_{3}$ and
$t_{4}$ (see Figs. 1(a)-(b) for a certain range of parameters). An example of
a measurement time series in which the bound in Theorem 1 and the LGI
parameter are both equal to $2\sqrt{2}$ is $t_{1}=0,t_{2}=\pi/4,t_{3}=\pi/2$
and $t_{4}=3\pi/4$.

In Fig. 1(c), we demonstrate Theorem 17 by showing that the l.h.s of Eq. (17)
is indeed smaller or equal to 1. To do so, we display all possible data points
using the axes $L/2\sqrt{2}$, $C(Q_{1},Q_{3})/2\sqrt{2}$ and
$C(Q_{2},Q_{4})/2\sqrt{2}$, and note that they all reside within the unit
sphere.

(a) Theorem 1 for our example. (b) Theorem 2 for our example. (c) Theorem 17
for our example.

Figure 1: Demonstration of Theorems 1, 2 and 17 for our spin model. (a) - (b),
the values of $D1$ and $D2$ respectively, both of which have a minimal value
of 0, indicating Eqs. (3) and (13) are upheld. (c) The indigo area represents
all data points, while the grey area represents the bounds of half of the unit
sphere. All indigo points are within the gray area, indicating Eq. (17) is
upheld.

##  IV Discussion

In this manuscript, we have studied generalizations of the LGI. We have
utilized a framework based on the quantum correlation matrix
carmi2018significance ; carmi-scienceadvances2019 ; carmi-njp2019  which
yielded now a more detailed version of the LGI by incorporating additional
correlations compared to the standard case. The results emphasize the major
role of the positive-semidefinite correlation matrix in quantum mechanics, not
only in “spatial” scenarios but also in “temporal” ones. They also demonstrate
a type of complementarity — in order for certain correlations to achieve their
maximal values (left hand side of Eq. (3)) others must vanish (those at the
right hand side of Eq. (3)).

We suggest that apart from its foundational and theoretical merits, the
proposed bound may help in designing temporal correlations as well as the
dynamics giving rise to them. They may also have practical implications in the
area of quantum cryptography, as a possible generalization to existing
encryption protocols that are based on LGI shenoy-pra2017  or in the field of
quantum metrology, possibly assisting approaches such as frowis-prl2016 , or
in LGI-based quantum computation assessment vitale-scientificreports2019 ;
santini-pra2022 . In addition, since there are known connections between
correlation matrices and both classical carmi-entropy2018  and quantum liu-
physicsa2020  Fisher information, the bounds derived in this manuscript can
provide analogous bounds on elements of the Fisher information matrix.

Finally, the work presented in this manuscript provides concrete and
measurable predictions and therefore it can be verified in direct experiments.
The system we provide as an example in the Results section is equivalent to
the system which was experimentally measured in athalye-prl2011 , and the
experimental results there are consistent with our bounds. In the Appendix, we
propose an additional definition for the correlations and derive appropriate
LGI-like bounds, which were not directly measured in previous experiments but
can be either calculated or measured, e.g. via weak measurements aharonov-
prl1988 .

## Acknowledgements

E.C. was supported by the Israeli Innovation Authority under Project 73795 and
the Eureka program, by Elta Systems Ltd., by the Pazy Foundation, by the
Israeli Ministry of Science and Technology, and by the Quantum Science and
Technology Program of the Israeli Council of Higher Education. The authors
acknowledge many helpful discussions with Avishy Carmi and Rain Lenny.

## Appendix

In this section, we discuss two subjects — another correlation definition and
the significance of the underlying assumptions.

In principle, one can generalize the above results to the case of non-
Hermitian quantum mechanics moiseyev2011non . By defining the correlation
$C(Q_{i},Q_{j})$ as the following complex correlation coefficient carmi-
njp2019 ; carmi-scienceadvances2019 ,

$C(Q_{i},Q_{j})=\frac{\langle Q_{i}Q_{j}^{\dagger}\rangle-\langle
Q_{i}\rangle\langle Q_{j}\rangle^{\dagger}}{\Delta_{Q_{i}}\Delta_{Q_{j}}},$
(33)

and the complex-valued LGI parameter as

$L=|C(Q_{1},Q_{2})+C(Q_{2},Q_{3})+C(Q_{3},Q_{4})-C(Q_{1},Q_{4})|,$ (34)

we obtain

$\displaystyle\begin{split}&L\leq
2\sqrt{1+\sqrt{1-\max{\left\\{\real^{2}[C(Q_{1},Q_{3})],\real^{2}[C(Q_{4},Q_{2})]\right\\}}}}\\\
&\left(\frac{L}{2\sqrt{2}}\right)^{2}+\left(\frac{\real[C(Q_{1},Q_{3})]}{2\sqrt{2}}\right)^{2}+\left(\frac{\real[C(Q_{4},Q_{2})]}{2\sqrt{2}}\right)^{2}\leq
1,\end{split}$ (35)

if $C(Q_{2},Q_{3})=C(Q_{3},Q_{2})$ or
$[C(Q_{1},Q_{2})=C(Q_{2},Q_{1})~{}\&~{}C(Q_{3},Q_{4})=C(Q_{4},Q_{3})]$. The
proofs of these bounds are similar to the proofs of Theorem 1 and Theorem 17.

As mentioned in the Results section, the analysis of this manuscript yields
tighter LGI bound under the assumption of projective measurements with values
±1. We note that under different assumptions, the LGI parameter can reach the
algebraic bound of 4 budroni-prl2014 .

## References

  * [1] A. J. Leggett and Anupam Garg.  Quantum mechanics versus macroscopic realism: Is the flux there when nobody looks?  Phys. Rev. Lett., 54:857–860, Mar 1985.
  * [2] Clive Emary, Neill Lambert, and Franco Nori.  Leggett-Garg inequalities.  Reports on Progress in Physics, 77(1):016001, dec 2013.
  * [3] Shayan-Shawn Majidy, Hemant Katiyar, Galit Anikeeva, Jonathan Halliwell, and Raymond Laflamme.  Exploration of an augmented set of leggett-garg inequalities using a noninvasive continuous-in-time velocity measurement.  Physical Review A, 100(4):042325, 2019.
  * [4] Shayan Majidy, Jonathan J Halliwell, and Raymond Laflamme.  Detecting violations of macrorealism when the original leggett-garg inequalities are satisfied.  Physical Review A, 103(6):062212, 2021.
  * [5] George C. Knee, Stephanie Simmons, Erik M. Gauger, John J.L. Morton, Helge Riemann, Nikolai V. Abrosimov, Peter Becker, Hans-Joachim Pohl, Kohei M. Itoh, Mike L.W. Thewalt, G. Andrew D. Briggs, and Simon C. Benjamin.  Violation of a Leggett–Garg inequality with ideal non-invasive measurements.  Nature Communications, 3(1):606, Jan 2012.
  * [6] Akshata Shenoy, S Aravinda, R Srikanth, and Dipankar Home.  Can the use of the Leggett-Garg inequality enhance security of the bb84 protocol?  Physics Letters A, 381(31):2478–2482, 2017.
  * [7] Charles H Bennett and Gilles Brassard.  Quantum cryptography: Public key distribution and coin tossing.  arXiv preprint arXiv:2003.06557, 2020.
  * [8] John S Bell.  On the Einstein Podolsky Rosen paradox.  Physics Physique Fizika, 1(3):195, 1964.
  * [9] John F. Clauser, Michael A. Horne, Abner Shimony, and Richard A. Holt.  Proposed experiment to test local hidden-variable theories.  Phys. Rev. Lett., 23:880–884, Oct 1969.
  * [10] B. S. Cirel’son.  Quantum generalizations of Bell’s inequality.  Letters in Mathematical Physics, 4(2):93–100, Mar 1980.
  * [11] Costantino Budroni, Tobias Moroder, Matthias Kleinmann, and Otfried Gühne.  Bounding temporal quantum correlations.  Phys. Rev. Lett., 111(2):020403, 2013.
  * [12] Tobias Fritz.  Quantum correlations in the temporal Clauser–Horne–Shimony–Holt (CHSH) scenario.  New Journal of Physics, 12(8):083055, aug 2010.
  * [13] Avishy Carmi and Eliahu Cohen.  On the significance of the quantum mechanical covariance matrix.  Entropy, 20(7):500, 2018.
  * [14] Avishy Carmi and Eliahu Cohen.  Relativistic independence bounds nonlocality.  Science Advances, 5(4):eaav8370, 2019.
  * [15] Avishy Carmi, Yaroslav Herasymenko, Eliahu Cohen, and Kyrylo Snizhko.  Bounds on nonlocal correlations in the presence of signaling and their application to topological zero modes.  New Journal of Physics, 21(7):073032, jul 2019.
  * [16] Amit Te’eni, Bar Y Peled, Eliahu Cohen, and Avishy Carmi.  Multiplicative Bell inequalities.  Physical Review A, 99(4):040102, 2019.
  * [17] Eliahu Cohen and Avishy Carmi.  In praise of quantum uncertainty.  Entropy, 22(3):302, 2020.
  * [18] Florian Fröwis, Pavel Sekatski, and Wolfgang Dür.  Detecting large quantum Fisher information with finite measurement precision.  Phys. Rev. Lett., 116(9):090801, 2016.
  * [19] Howard Percy Robertson.  The uncertainty principle.  Physical Review, 34(1):163, 1929.
  * [20] Erwin Schrödinger.  Zum heisenbergschen unschärfeprinzip.  Akademie der Wissenschaften, 1930.
  * [21] Gerardo Adesso and Fabrizio Illuminati.  Entanglement in continuous-variable systems: recent advances and current perspectives.  Journal of Physics A: Mathematical and Theoretical, 40(28):7821, 2007.
  * [22] Stefano Pirandola, Alessio Serafini, and Seth Lloyd.  Correlation matrices of two-mode bosonic systems.  Physical Review A, 79(5):052327, 2009.
  * [23] B. S. Tsirel’son.  Quantum analogues of the Bell inequalities. the case of two spatially separated domains.  Journal of Soviet Mathematics, 36(4):557–570, Feb 1987.
  * [24] Lawrence J. Landau.  Empirical two-point correlation functions.  Foundations of Physics, 18(4):449–460, Apr 1988.
  * [25] Ll Masanes.  Necessary and sufficient condition for quantum-generated correlations.  arXiv preprint quant-ph/0309137, 2003.
  * [26] Justin Dressel and Alexander N Korotkov.  Avoiding loopholes with hybrid Bell-Leggett-Garg inequalities.  Physical Review A, 89(1):012125, 2014.
  * [27] T. C. White, J. Y. Mutus, J. Dressel, J. Kelly, R. Barends, E. Jeffrey, D. Sank, A. Megrant, B. Campbell, Yu Chen, Z. Chen, B. Chiaro, A. Dunsworth, I.-C. Hoi, C. Neill, P. J. J. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, A. N. Korotkov, and John M. Martinis.  Preserving entanglement during weak measurement demonstrated with a violation of the Bell–Leggett–Garg inequality.  npj Quantum Information, 2(1):15022, Feb 2016.
  * [28] JJ Halliwell.  Leggett-Garg inequalities and no-signaling in time: A quasiprobability approach.  Physical Review A, 93(2):022123, 2016.
  * [29] V Vitale, G De Filippis, A De Candia, A Tagliacozzo, V Cataudella, and P Lucignano.  Assessing the quantumness of the annealing dynamics via leggett garg’s inequalities: a weak measurement approach.  Scientific reports, 9(1):13624, 2019.
  * [30] Alessandro Santini and Vittorio Vitale.  Experimental violations of leggett-garg inequalities on a quantum computer.  Physical Review A, 105(3):032610, 2022.
  * [31] Avishy Carmi and Daniel Moskovich.  Tsirelson’s bound prohibits communication through a disconnected channel.  Entropy, 20(3):151, 2018.
  * [32] Jing Liu, Haidong Yuan, Xiao-Ming Lu, and Xiaoguang Wang.  Quantum fisher information matrix and multiparameter estimation.  Journal of Physics A: Mathematical and Theoretical, 53(2):023001, 2020.
  * [33] Vikram Athalye, Soumya Singha Roy, and TS Mahesh.  Investigation of the Leggett-Garg inequality for precessing nuclear spins.  Phys. Rev. Lett., 107(13):130402, 2011.
  * [34] Yakir Aharonov, David Z. Albert, and Lev Vaidman.  How the result of a measurement of a component of the spin of a spin-1/2 particle can turn out to be 100.  Phys. Rev. Lett., 60:1351–1354, Apr 1988.
  * [35] Nimrod Moiseyev.  Non-Hermitian quantum mechanics.  Cambridge University Press, 2011.
  * [36] Costantino Budroni and Clive Emary.  Temporal quantum correlations and Leggett-Garg inequalities in multilevel systems.  Phys. Rev. Lett., 113(5):050401, 2014.


# Bio-Inspired Grasping Controller for Sensorized 2-DoF Grippers

Luca Lach1,2, Séverin Lemaignan1, Francesco Ferro1, Helge Ritter2, Robert
Haschke2 This work was supported by the European Union Horizon 2020 Marie
Curie Actions under Grant 813713 NeuTouch.1PAL Robotics, Barcelona,
Spain2Neuroinformatics Group, Bielefeld University, Bielefeld, Germany

###### Abstract

We present a holistic grasping controller, combining free-space position
control and in-contact force-control for reliable grasping given uncertain
object pose estimates. Employing tactile fingertip sensors, undesired object
displacement during grasping is minimized by pausing the finger closing motion
for individual joints on first contact until force-closure is established.
While holding an object, the controller is compliant with external forces to
avoid high internal object forces and prevent object damage. Gravity as an
external force is explicitly considered and compensated for, thus preventing
gravity-induced object drift. We evaluate the controller in two experiments on
the TIAGo robot and its parallel-jaw gripper proving the effectiveness of the
approach for robust grasping and minimizing object displacement. In a series
of ablation studies, we demonstrate the utility of the individual controller
components.

##  I Introduction

One of the most important skills service robots need to master is that of
object manipulation. Since the majority of object manipulation approaches are
vision-based [1], they have a huge disadvantage compared to human grasping:
they lack environment feedback during physical object interaction and are
therefore prone to high error rates. This applies especially to mobile robots
since they have to deal with more sources of uncertainty than fixed setups
(e.g. unreliable hand-eye calibration). Robotic end-effectors equipped with
tactile sensors can provide the robot with data about physical object
interactions and thus enable more robust grasping approaches [2].

By measuring and controlling grasp forces, manipulation tasks can be modelled
and executed much more precisely. In situations where fragile objects are
being handled, the grasp force should be limited to prevent object damage and
yet ensure a firm grasp. Furthermore, undesired object movements during
grasping should be minimized to avoid undesired side-effects such as tipping
over the object. One key concept in grasping that formalizes these
characteristics is that of force closure. It requires the robot to only apply
forces on the object that result in a net-force of zero, hence avoiding object
displacement.

Many force controllers have been proposed for different robot setups and arm-
hand configurations that build upon force closure grasps. However, most of
them do not consider the effects of external forces on the controller
behavior. Mobile robots typically have to deliver objects to humans or place
them in another location after grasping. Thus, they also need to be able to
correctly handle external forces in the form of gravity or humans interacting
with the end-effector.

In this work, we propose a novel grasping controller with the following
characteristics:

  1. 1.

During the grasping process, the object is only minimally displaced,
exploiting tactile feedback to account for uncertainties in visual object pose
estimation.

  2. 2.

Holding the object, a desired grasp force is maintained, while also
compliantly reacting to external forces.

  3. 3.

Gravity as an external force is estimated and explicitly considered to avoid
undesired drift of the object.

  4. 4.

Small imbalanced force measurements are ignored, again avoiding undesired
object drift.

To the best of our knowledge, this is the first grasping controller, that
explicitly compensates external forces while minimizing object displacement
and accounting for gravity. We first derive the core concepts of our
controller from human grasping strategies and formulate them in a general
fashion. Subsequently we derive an instance specifically tailored for 2-DoF
grippers. Finally, we validate them on a sensorized parallel-jaw gripper of
the TIAGo service robot as shown in Fig. 1 and compare them to a standard
trajectory controller. We run experiments to show the effectiveness of the
proposed controller in terms of object displacement minimization and external
force compensation. In a set of ablation studies we show the impact of each
controller component on its behavior.

Figure 1: Grasping a soft object with TIAGo’s sensorized parallel-jaw gripper
using force control.

##  II Related Work

In a review of human grasping, Johansson and Flanagan [3] highlight the
importance of tactile sensations. They divide a grasping task into different
phases, where most of the phase transitions are determined by different
tactile events. When anticipating object contact, during the reach motion,
humans perform position control. Once object contact is detected, we switch to
the load phase where the grasp force is increased until a goal force is
reached. During subsequent phases, such as lifting, holding, and replacing, we
try to maintain that goal force. In the unload phase, we gradually decrease
the contact force until we release the fingers. These findings are supported
by experimental studies such as [4].

The human grasping approach presented in [3] has also been adopted by many
works in robotics. Romano et. al. [5] closely follow the phase distinctions
from [3] while modelling their force controller for the PR2 robot. They
concentrate on goal force estimation and slippage prevention using a small
sensor array in the 1-DoF PR2 gripper. One of their assumptions is the
equidistant placement of each finger around the object, thus implicitly
assuming force closure on first contact. In an earlier work, Hsiao et. al. [6]
also proposed a force controller for the PR2, but without the assumption of
equidistant finger-object placement. If one finger acquires object contact
first, they move the arm into the direction of that finger in order to avoid
object movements and guarantee force closure. Their work was integrated and
evaluated in a grasping pipeline for the PR2 in [7]. Since our work focuses on
controlling grippers with 2-DoF, we need to impose additional constraints on
the controller to avoid object drift. This allows us to be compliant with
external forces, which is not possible with a 1-DoF gripper.

Many works exist that consider grasp force control in fixed-arm setups with
2-DoF end-effectors. The authors in [8] designed a proximity, contact, and
force sensor using infrared sensors behind an elastomer. They leverage the
proximity measurements to center the gripper around the object and ensure a
safe grasp this way. Similar to [5], the authors process their sensed forces
to match human tactile perception channels, as described in [3].

The studies [9] and [10] use two fingers of a multi-fingered hand to perform
force control. In [9], the authors classify slippage using BioTac sensors.
Once object slip is detected, they increase the desired grasp force, thus
stabilizing the grasp. [10] concentrates on optimizing grasp stability during
the holding phase by updating fingertip positions based on force closure
constraints. In contrast to our work, these studies do not take the effect of
external object forces on the controllers into account.

A number of authors also aim for multi-fingered hands, which poses additional
challenges. [11] propose a control scheme for a three-fingered hand by
defining and controlling a virtual object frame that is based on finger
contacts. In [12], the authors propose a combined position and force
controller to ensure stable grasps for five-fingered hands that is based on
[11]. While both studies point out the need to consider external forces such
as gravity, the evaluations were conducted in simulation without taking
external forces into account.

More recent studies have evaluated their methods for multi-fingered hands also
on real hardware. [13] propose a grasp stabilization approach for the Allegro
hand, that controls a virtual object frame through force adaptation and finger
gaiting. In [14], Neural Networks and Gaussian Mixture Models are used to
detect contacts and estimate forces on the Shadow Hand. The results are
subsequently used by a grasp-stabilizing controller that adjusts finger
positions accordingly. In both works, the resilience to external disturbances
is evaluated, either by a human, shaking the arm ([14]) or by the arm
executing a shaking motion ([13]). However, these experiments do not
investigate the effect of constant and maintained forces on the fingers, such
as gravity or humans pushing or pulling on the fingers. Our controller is
modelled to explicitly handle both types of external forces, peaky
disturbances and slow continuous drift.

A growing subfield in the tactile manipulation community is that of
reinforcement learning. Some approaches aim to learn stable grasping policies
[15], while others focus on increasing grasp success using tactile feedback
[16]. One challenge of these approaches is the sim-to-real transfer [17],
since modelling tactile sensor responses is non-trivial in simulation [18].
External forces and their physical effects are also difficult to model. Hence,
most machine learning related works do not take external forces into account.
Our controller could be used as a component of a more complex machine learning
model to address these issues, provided that the output of the model are
desired forces.

##  III Grasp Controller

We propose a holistic control approach for grasping, covering all phases from
finger closing over establishing contacts to holding (cf. Fig. 2).
Consequently, the controller distinguishes these three phases and applies a
different control strategy in each of them, similar to human grasping as
described in [3]. During phase I, when the fingers are approaching the object,
the controller performs open-loop trajectory control by following a pre-
planned joint trajectory targeting a hand posture that would eventually
penetrate the object. Once a tactile sensor reliably detects contact, i.e.
perceives a contact force $f(t)$ just above the noise threshold $f_{\theta}$,
the corresponding finger transitions to phase II. As a result, all joints
along the kinematic chain from the sensor link up to the hand base stop moving
but just maintain object contact. This avoids undesired object movements
during grasping in situations where the object is not perfectly centered
between the fingers.

ptIclosingptIIestablishcontactptIIIholdingpt$f_{i}>f_{\theta}$ptforceclosure
Figure 2: State diagram illustrating the 3 phases of grasping.

Each time a new contact is established, the resulting grasp is evaluated for
force-closure [19] as described in sec. III-A. If the grasp is considered
force-closure, phase III is entered, which finally performs grasp force
control aiming to establish and maintain a desired grasp force as outlined in
sec. III-B. We assume desired goal forces to be provided by a higher-level
cognitive system component based on common-sense object knowledge, which is
analogous to how humans decide which grasping force to use [3]. Alternatively,
the grasp forces could be reactively adapted due to recognized incipient
slippage as suggested e.g. in [5, 13, 20, 21].

###  III-A Evaluating Force-Closure

Force-closure is a formally defined property of a grasp configuration stating
that the grasp can balance any (bounded) external disturbance wrench with
contact forces $\mathbf{f}_{i}$ that satisfy the friction cone constraints,
i.e. don’t induce local slippage. To determine the net wrench applied onto an
object through contact points $\mathbf{p}_{i}$, independent contact forces
$\mathbf{f}_{i}$ are mapped onto 6-dimensional contact wrenches
$\mathbf{F}_{C_{i}}=(\mathbf{f}_{i},\tau_{i})$ expressed in local contact
frames $C_{i}$, which are then transformed into the common object frame $O$
using the adjoint matrix $Ad^{\top}_{T_{oc_{i}}^{-1}}$, and finally summed up
to yield the net wrench $\mathbf{F}_{O}$ expressed in frame $O$ [19]:

$\displaystyle\mathbf{F}_{O}$
$\displaystyle=\sum_{i=1}^{n}Ad^{\top}_{T_{oc_{i}}^{-1}}\cdot\mathbf{F}_{C_{i}}=\sum_{i=1}^{n}Ad^{\top}_{T_{oc_{i}}^{-1}}B_{i}\cdot\mathbf{f}_{i}\equiv
G\cdot\mathbf{f}$ (1) $\displaystyle B^{\top}$
$\displaystyle=\left[\begin{smallmatrix}1&0&0&0&0&0\\\ 0&1&0&0&0&0\\\
0&0&1&0&0&0\\\ 0&0&0&0&0&1\\\ \end{smallmatrix}\right]\quad
Ad^{\top}_{T_{oc_{i}}^{-1}}=\begin{pmatrix}R_{oc_{i}}\\\
\hat{\mathbf{p}}_{oc_{i}}\cdot R_{oc_{i}}&&R_{oc_{i}}\end{pmatrix}$

Here, $\mathbf{f}=[\mathbf{f}_{1}^{\top},\dotsc,\mathbf{f}_{n}^{\top}]\top$
denotes the vector of concatenated contact forces $\mathbf{f}_{i}$, $G$
denotes the grasp matrix, and $B\equiv B_{i}$ is the wrench basis modelling
soft-finger contacts [19]. The latter is a point-contact model allowing for
_force_ transmission along the contact normal ($f_{i,z}$) and within the
tangential plane ($f_{i,x},f_{i,y}$) as well as for _torque_ transmission
along the normal axis of the contact ($f_{i,\tau}$). These independent force
components $\mathbf{f}_{i}\in\mathbb{R}^{4}$ need to satisfy friction cone
constraints $FC_{i}$ derived from Coulomb friction to avoid slippage:

$0\leq\sqrt{f_{i,x}^{2}+f_{i,y}^{2}}\leq\mu_{i}f_{i,z}\quad\text{and}\quad|f_{i,\tau}|\leq\mu_{i,\tau}f_{i,z}\,.$

A grasp defined by its grasp matrix $G$ and the collection of all friction
cone constraints $FC=FC_{1}\times\dotsc\times FC_{n}$ is force-closure, iff
$G$ is surjective and there exist strict internal forces, i.e. forces
$\mathbf{f}_{N}$ that strictly satisfy the friction cone constraints and that
don’t have a net effect onto the object ($G\cdot\mathbf{f}_{N}=0$). Both
conditions can be easily verified given the geometry of the grasp, i.e. its
contact locations $\mathbf{p}_{i}\equiv\mathbf{p}_{oc_{i}}$, contact normals
$\mathbf{n}_{i}$ (determining $R_{oc_{i}}$), and a conservative estimation of
the friction coefficients $\mu\equiv\mu_{i}$.

###  III-B Grasp Force Control

We assume that the tactile sensors only provide contact forces along the
normal direction at the contact point. This rather weak assumption allows
applying the grasping force controller to a large variety of tactile sensors
[22], but limits the capabilities of the method. Traditional grasp force
controllers like [11] strive for a globally optimal contact force distribution
ensuring grasp stability, i.e. ensuring that (i) all contact forces stay
within friction cone bounds, (ii) applied forces exactly balance external
forces (e.g. gravity), and (iii) local contact forces are minimized [19].
However, this approach is only meaningful if the full 3D contact force is
observable and thus controllable. Here, we assume that we cannot measure shear
forces nor determine the friction coefficients required to evaluate friction
cone constraints.

Consequently, we are not aiming at realizing a net wrench of zero applied onto
the object as proposed in traditional grasp force controllers. Rather, we
restrict control to the observable wrench component only, which in our case
are normal forces along the gripper’s closing direction $\mathbf{n}$. Hence,
instead of aiming for $G\cdot\mathbf{f}+\mathbf{F}^{\text{ext}}\\!=\\!0$, we
project onto the normal direction and yield:

$(f_{1}\mathbf{n}-f_{2}\mathbf{n}+\mathbf{f}^{\text{ext}})\cdot\mathbf{n}=f_{1}-f_{2}+f^{\text{ext}}\overset{!}{=}0\,,$
(2)

where $f_{i}$ denote scalar normal force components measured by the two
fingertip sensors and $f^{\text{ext}}$ is the scalar external force component
along the normal direction.

To maintain a desired grasp force $f_{\text{goal}}$, we consider the sum of
both contact forces, which essentially resembles the overall magnitude of
_internal_ forces applied to the object and thus should become equal to
$f_{\text{goal}}$:

$f^{\text{int}}\equiv f_{1}+f_{2}\overset{!}{=}f^{\text{goal}}$ (3)

By decoupling control of external and internal forces in this fashion, we
decouple object motion from object deformation. In contrast, the naive
approach controlling both contact forces $f_{1}$ and $f_{2}$ independently,
would face a common practical issue: in real-world settings, force
measurements are noisy and not perfectly calibrated. This means, that even
given balanced antipodal force _measurements_ , the _actual_ physical forces
might differ, thus resulting in a net force applied to the object. This
imprecision needs to be considered in order to avoid object drift when no
external forces are applied. Note that simple 1-DoF grippers, like in [5], do
not face this issue: their fingers move in sync and the object will always be
centered in the gripper.

When the measured imbalance of forces is larger, we expect the cause to be an
external force being exerted on the gripper. One type of external force are
those resulting from physical interactions with the environment, e.g. pushing
or pulling on the fingers or collisions with the environment. Since the
controller’s main objective is to maintain and not exceed $f^{\text{goal}}$,
it needs to be compliant with these forces. Another external force influencing
the sensors measurements is gravity. As we don’t want the fingers to drift due
to gravity, we need to explicitly consider gravity in eq. 2, resulting in:

$f_{1}-f_{2}+m\,\mathbf{g}\cdot\mathbf{n}=-f^{\text{ext}}\overset{!}{=}0$ (4)

where $m$ is the object’s (and sensor’s) mass in kilograms,
$\mathbf{g}=9.81\frac{m}{s^{2}}\,\mathbf{z}$ is the acceleration vector due to
gravity, and $f^{\text{ext}}$ is the remaining external force. This
formulation emphasizes the fact that we cannot explicitly counteract external
shear forces, but only normal forces.

In this work, we further assume that the end-effector is position-controlled.
Consequently, we transform force errors into Cartesian position errors
according to Hooke’s law:

$\Delta x(t)=\frac{1}{k}\Delta f(t)\,.$ (5)

Here, $k$ is the stiffness constant, i.e. in case of eq. 3 the object’s
stiffness and in case of eq. 4 the desired compliance when giving in to
external forces. Estimating the object’s stiffness accurately is difficult in
practice. Thus we utilize PI control to account for uncertainties in
estimating $k^{\text{int}}$:

$u^{\text{int}}(t)=\frac{K_{\text{P}}^{\text{int}}}{k^{\text{int}}}\cdot\Delta
f^{\text{int}}(t)+\frac{K_{\text{I}}^{\text{int}}}{k^{\text{int}}}\int_{\text{III}}\\!\\!\Delta
f^{\text{int}}(\tau)\,d\tau\,,$ (6)

where $K_{\text{P}}^{\text{int}}$ and $K_{\text{I}}^{\text{int}}$ denote the
coefficients of the proportional and integral terms. The integral is
calculated over all time steps since the transition into controller phase III.
Thus, for non-zero force errors $\Delta
f^{\text{int}}=f_{1}+f_{2}-f^{\text{goal}}$ the integral term will increase in
magnitude and thus force convergence of the error towards zero.

To react to external forces, expressed via eq. 4, we employ a simple
P-controller:

$u^{\text{ext}}(t)=\frac{K_{\text{P}}^{\text{ext}}}{k^{\text{ext}}}\cdot
f^{\text{ext}}\qquad\text{if }|f^{\text{ext}}|>f_{\phi}$ (7)

where $f_{\phi}$ is a small threshold to filter out sensor noise, similar to
$f_{\theta}$ for the force readings. To avoid object drift caused by sensor
noise or poor calibration, this compliant position controller is only applied
if $|f^{\text{ext}}|$ is larger than $f_{\phi}$. Inside this deadband, we
perform position control to maintain the object position $O(T)$ observed at
the last time $T$ when true external forces were measured:

$u^{\text{ext}}(t)=\Delta O(t)\equiv O(T)-O(t)\qquad\text{if
}|f^{\text{ext}}|\leq f_{\phi}$ (8)

In practice, we set $f_{\phi}=f_{\theta}$, however it is important for the
experiments to distinguish between $f_{\phi}$ and $f_{\theta}$. This
controller component can be thought of as a constraint on a virtual object
frame, an idea that has been pursued before [11, 13]. Our method differs from
previous approaches since we condition this object pose constraint on values
of $f^{\text{ext}}$ while taking gravity into account.

Finally, joint position deltas are computed via inverse velocity kinematics.
In our case, where increasing joint positions correspond to an opening of the
fingers, we yield:

$\displaystyle\Delta q_{1}$
$\displaystyle={\textstyle\frac{1}{2}}(-u^{\text{ext}}+u^{\text{int}})$ (9)
$\displaystyle\Delta q_{2}$
$\displaystyle={\textstyle\frac{1}{2}}(+u^{\text{ext}}+u^{\text{int}})\,,$
(10)

which distributes control signals evenly between the two fingers. The
commanded joint positions are determined by simple integration:

$\mathbf{q}^{\text{cmd}}(t)=\mathbf{q}(t)+\Delta\mathbf{q}(t)\,.$ (11)

###  III-C Implementation

We implemented the grasp controller using the robot operating system (ROS)
[23], the most commonly used robotics middleware nowadays. The default
controller library ros_control [24] offers a JointTrajectoryController (JTC),
which performs free position control for a set of joints given a trajectory.
Our controller inherits from the JTC class, keeping all its interfaces. Hence,
a grasp can be initiated by sending a JointTrajectoryActionGoal in the same
fashion a JTC task would have been triggered. Therefore, our controller serves
as a drop-in replacement for existing grasping pipelines. Parameters specific
to force control can be comfortably adjusted via the dynamic reconfigure
interface. The code is open-source and can be found on
GitHub111https://github.com/llach/ta11_sensor_tools.

The controller offers two different modes for phase III: an open-ended force
control mode and a mode finishing control upon reaching the desired force. For
the former one, the controller attempts to hold the desired force forever.
During this mode, the desired force can be adapted, e.g. reactively in
response to incipient slip detection or deliberately prior to planned
interactions with the environment. The latter mode signals action success once
$f^{\text{goal}}$ is reached.

##  IV Experiments

We evaluated the control approach on the TIAGo robot, a service robot
developed to serve in household settings. It has a 7 DoF arm with end-
effectors that can be easily exchanged or modified, making it an extensible
and adaptable manipulation platform. Since our work is designed for two DoF
end-effectors, we chose the parallel-jaw gripper for our experiments. With the
gripper, TIAGo has a maximum payload of $2.1kg$ if the arm is fully extended.
By default, TIAGo’s gripper has two plastic fingers that are _not_ sensorized.
For our experiments, we swapped the fingers for load cell sensors (TA11 from
Gicam) with 3D-printed, rigid plastic fingertips as shown in Fig. 1. Using the
sensors, we can read contact forces normal to the gripper’s closing direction
from each finger.

The raw sensor values are read using a LabJack U6 and then published in a
bias-corrected fashion to a ROS topic. To this end, the ROS node estimates the
sensor biases $b_{i}$ by averaging over $1000$ measurements in unloaded state
on each startup. Additionally, force measurements are also converted into
Newton. The conversion factors $\gamma_{i}$ were determined by aligning the
sensors parallel to the ground, recording de-biased sensor responses for three
objects with different weights, calculating their individual conversion
factors and then taking the mean of the three factors. This way, we determined
$\gamma_{1}=11.02$ and $\gamma_{2}=11.03$. Calibrated forces are then
calculated by:

$f^{\text{calibrated}}_{i}=\gamma_{i}\cdot(f^{\text{raw}}_{i}-b_{i})$

Furthermore, we determine the noise threshold $f_{\theta}=0.2N$ by observing
the maximum value during the bias correction process and approximately
doubling it. We choose it to be relatively conservative in order to avoid
false-positive contact detections. Choosing a well-suited value is crucial for
the controller’s performance. A too tight threshold can lead to premature
stopping of the finger-closing phase due to noise, while a too conservative
threshold degrades the controller’s sensitivity.

To assess the sensor’s first-touch sensitivity, we aligned them parallel to
the ground and placed small weights (different Euro coins) on their tips until
the measured sensor value was continuously above the threshold $f_{\theta}$
for a period of 30 seconds. In this fashion, we determined first-touch
sensitivities of $0.35N$ and $0.38N$ for the left and right sensors,
respectively.

Prior to the experiments, we determined the controller parameters in
preliminary grasping trials. The PI controller parameters as well as the
object stiffnesses were chosen such that objects were grasped without
overshooting the goal force. This can lead to a slightly slower convergence to
the goal force but ensures that the object is not damaged. Consequently, we
chose the PI coefficients $K_{\text{P}}^{\text{int}}=1.9$ and
$K_{\text{I}}^{\text{int}}=9.0$. We considered three different objects: a
styrofoam cylinder of 2$g$, a tape roll of 49$g$ and a wooden cuboid of
144$g$. For all experiments, $k$ was set to 1000. The estimation process for
$k$ has proven to be rather quick, requiring only about five trials per
object. We found that tuning it for soft objects is usually sufficient as the
controller’s integral part will still lead to stable grasps of stiffer
objects.

###  IV-A Object Displacement

In real-world scenarios, the end-effector is rarely centered perfectly around
the object when executing a grasp, resulting in object displacements if
initial object contact cannot be detected reliably. With experiment A, we
analyze our controller’s behavior in these situations and compare it to
standard position control.

For the experiment, we position the open gripper over a table and place an
object between its fingers. When placing the object, we do not center it
perfectly, but rather place it closer to one of the fingers. The gripper is
then closed, either with our force controller or with position control.
Finally, the distance the object was moved during the gripper closing is
recorded. This process is repeated for the three aforementioned objects at
five different object positions three times. The object offsets $o_{i}$ of
$2mm$, $5mm$, $8mm$, $11mm$ and $14mm$ were chosen such that all three objects
would fit into the gripper at each offset. In total, $5\cdot 3\cdot 3=45$
grasping trials were performed for each controller. The experimental setup is
shown in Fig. 3.

###  IV-B Grasp-force controller – ablation studies

With experiment B, we evaluate the utility of the individual controller
components introduced in sec. III-B. In particular, we perform ablation
studies where we evaluate grasping behavior while one of the controller
components is switched off. In this fashion, we evaluate the utility of the
decoupling of internal and external force control, the conditional external
force compensation, and the gravity compensation.

First, we study two different grasping scenarios with all components enabled.
In the first scenario, external forces are exerted on the gripper’s fingers by
a human while holding an object. Here, the controller is expected to be
compliant with the forces, but not to drift once they are removed. In the
second scenario, the gripper is being moved such that the influence of gravity
on the held object is changing. After evaluating the controller’s behavior
with all components enabled, we re-run these scenarios, but with individual
components switched off. As a result, we can qualitatively assess the utility
of a component.

##  V Experimental Results

Figure 3: Setup for experiment A: objects are placed with an offset relative to one finger (left) and then grasped (right). After the grasp, the object displacement is measured. TABLE I: Comparison of object displacements in $mm$ for standard position control (JTC) and our force controller. Each object was grasped $3$ times from $5$ different finger offsets to simulate imperfect gripper-object alignment. Displacements over all offsets were averaged. Object | Weight | JTC | Our Controller
---|---|---|---
Styrofoam | $2g$ | $19.0\pm 4.2$ | $17.0\pm 4.2$
Tape roll | $44g$ | $21.7\pm 4.5$ | $\textbf{3.5}\pm 0.5$
Wooden cuboid | $144g$ | $9.0\pm 4.2$ | $\textbf{1.0}\pm 0.3$

###  V-A Object Displacement

For experiment A, we used three objects with different weights. We measured
the object displacements by taking the difference of finger positions before
and after grasping minus the object offset:
$q^{\text{start}}-q^{\text{end}}-o_{i}$. Tab. I compares the displacements of
the objects for a series of 15 grasps for both, force and position control.
For the styrofoam cylinder, the displacement for both controllers is very
similar: this object was too lightweight to trigger the first-touch detection
on the force sensors. For the other two objects, the displacement was
significantly reduced by using our force controller. The experiments show that
for sufficiently heavy objects, our controller minimizes object displacement
to less than $1mm$. Furthermore, the results show that the object displacement
is equally minimized across different object offsets, as indicated by the low
standard deviation measured for the tape roll and the cuboid.

Figure 4: Controller behavior during object grasping (tape roll) and
application of external forces, first from the right and then from the left.
Figure 5: Object drift during gripper rotation while holding the tape roll,
with and without gravity compensation.

The results from experiment A are twofold: it shows, that using our force
controller, we can minimize undesired side effects of object manipulation
during the grasping stage. By dividing a grasp into different phases as humans
do, object displacements were minimized and force closure was guaranteed.
Furthermore, it shows that the performance of any force controller highly
depends on the sensors’ capability to detect first touch. That capability is
in turn dependent on the sensor sensitivity as well as its calibration. As
hardware continuously improves, we expect to see more sensitive tactile
sensors in the future. A higher sensitivity would decrease object
displacements even more and enable TIAGo to handle lighter objects.

###  V-B Grasp-force controller – ablation studies

For the first scenario of experiment B, we performed a grasp using our force
controller with the gripper being parallel to the floor and all components
enabled to establish a baseline. After initial grasping, we hold the object
without any external forces to verify that no object/gripper motion is
occurring. Subsequently, we applied external forces to one finger. In
response, the controller compliantly gave in as expected and started moving
away from the force. Once the external force was taken away, the finger
movement stopped promptly, and the object’s position was stable again. The
controller reacted the same when applying forces on the other finger. Fig. 4
shows the internal and external object forces as well as the object position
that were measured for this scenario. The different phases and events, such as
initial contacts, are annotated as well.

For the first ablation study, we disabled compliance to external forces
(setting $f_{\phi}=\infty$ in eq. 7). As a result, the gripper could not give
in to external forces. This caused the total force exerted on the object to
surpass $f^{\text{goal}}$ when external forces were applied. Obviously, this
is not a desired behavior as fragile objects might get damaged.

In the second ablation study, we set $f_{\phi}=0$, disabling object position
control within the deadband. While the controller was able to give in to
external forces, the gripper drifted away even after the force was removed.
This is likely caused by tiny differences in force measurements between both
sensors. Enabling the deadband $f_{\phi}$ results in the desired behavior of
stopping the motion promptly after taking away the external force.

In the second scenario, we rotated the wrist by approximately 180 degrees
after grasping an object. Consequently, the sensor’s normal vector changed
with respect to the gravity vector. During the first 90 degrees, the sensor
normals became aligned to gravity, hence maximizing its influence. For the
remaining rotation, the influence of gravity decreased again and became nearly
zero.

In the corresponding ablation study, we evaluated the controller behavior when
not explicitly compensating for gravity. Thus, $f^{\text{ext}}$ was _not_
corrected using $m\,\mathbf{g}\cdot\mathbf{n}$ and the controller confused
gravity for an external force. This caused the gripper’s finger to drift
towards the ground when the gripper was rotating. Fig. 5 compares the object
displacement with and without gravity compensation while performing the
described movement. Without gravity compensation, the object drifted as much
as 30 $mm$, while the object drift was negligible ($<1mm$) with gravity
compensation enabled.

In conclusion, these ablation studies have demonstrated the importance of each
controller component. If one was disabled, either undesired object drift was
occurring or $f^{\text{goal}}$ was surpassed. We also compare the controller’s
behavior with and without the individual components side-by-side in an
accompanying video, which illustrates these comparisons in an accessible
manner.

##  VI Conclusion And Future Work

In this work, we have presented a holistic grasp controller that combines free
position control during finger closing with force control during object
holding. We proposed a controller that minimizes undesired object displacement
while holding a pre-determined desired force. At the same time, the controller
is compliant with external forces in order not to surpass the goal force.
Furthermore, gravity as an external force is explicitly accounted for to avoid
undesired object drift.

We have validated our claims in two experiments: In experiment A, we have
shown our controller’s ability to minimize object displacement during grasping
and execute force closure grasps even under object pose uncertainty. The
ablation studies of experiment B have illustrated how each component is
contributing to the final behavior of the controller. Each component was shown
to be crucial to either minimize undesired object movements or to be compliant
with external forces only when they are actively applied to the fingers.

Although demonstrated for a simple parallel-jaw gripper only, we think the
approach will easily generalize to multi-fingered hands as well. To this end,
the general concept of decoupling control of external and internal forces
needs to be generalized, considering the full 6-dimensional wrench space and
the nullspace of the grasp matrix respectively. We plan to implement this
generalized version of the controller in our future work.

## References

  * [1] Guoguang Du, Kai Wang, Shiguo Lian and Kaiyong Zhao  “Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review”  In _Artificial Intelligence Review_ 54.3 Springer, 2021, pp. 1677–1734
  * [2] Chiara Bartolozzi, Lorenzo Natale, Francesco Nori and Giorgio Metta  “Robots with a sense of touch”  In _Nature Materials_ 15 Nature Publishing Group, 2016, pp. 921–925
  * [3] Roland S Johansson and J Randall Flanagan  “Coding and use of tactile signals from the fingertips in object manipulation tasks”  In _Nature Reviews Neuroscience_ 10.5 Nature Publishing Group, 2009, pp. 345–359
  * [4] Subramanian Sundaram et al.  “Learning the signatures of the human grasp using a scalable tactile glove”  In _Nature_ 569 Nature Publishing Group, 2019, pp. 698–702
  * [5] Joseph M Romano et al.  “Human-inspired robotic grasp control with tactile sensing”  In _IEEE Trans. on Robotics_ 27.6, 2011, pp. 1067–1079
  * [6] Kaijen Hsiao, Sachin Chitta, Matei Ciocarlie and E. Gil Jones  “Contact-Reactive Grasping of Objects with Partial Shape Information”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2010
  * [7] Matei Ciocarlie et al.  “Towards reliable grasping and manipulation in household environments”  In _Springer Tracts in Advanced Robotics_ 79 Springer Verlag, 2014, pp. 241–252
  * [8] Radhen Patel, Rebecca Cox and Nikolaus Correll  “Integrated proximity, contact and force sensing using elastomer-embedded commodity proximity sensors”  In _Autonomous Robots_ 42 Springer New York LLC, pp. 1443–1458
  * [9] Zhe Su et al.  “Force Estimation and Slip Detection/Classification for Grip Control using a Biomimetic Tactile Sensor”  In _2015 IEEE-RAS 15th Internatial Conference on Humanoid Robots (Humanoids)_ , 2015
  * [10] Andrés Montaño and Raúl Suárez  “Manipulation of unknown objects to improve the grasp quality using tactile information”  In _Sensors (Switzerland)_ 18 MDPI AG
  * [11] Kenji Tahara, Suguru Arimoto and Morio Yoshida  “Dynamic object manipulation using a virtual frame by a triple soft-fingered robotic hand”  In _Proc. ICRA_ , 2010
  * [12] Qiang Li, Robert Haschke, Helge Ritter and Bram Bolder  “Towards unknown objects manipulation”  In _IFAC Proceedings Volumes_ 45.22 Elsevier, 2012, pp. 289–294
  * [13] Kaiyu Hang et al.  “Hierarchical Fingertip Space: A Unified Framework for Grasp Planning and In-Hand Grasp Adaptation”  In _IEEE Transactions on Robotics_ 32 Institute of ElectricalElectronics Engineers Inc., 2016, pp. 960–972
  * [14] Zhen Deng, Yannick Jonetzko, Liwei Zhang and Jianwei Zhang  “Grasping force control of multi-fingered robotic hands through tactile sensing for object stabilization”  In _Sensors (Switzerland)_ 20 MDPI AG, 2020
  * [15] Hamza Merzic et al.  “Leveraging Contact Forces for Learning to Grasp”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019
  * [16] Bohan Wu, Iretiayo Akinola, Jacob Varley and Peter Allen  “MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning”  In _3rd Conference on Robot Learning (CoRL 2019),_ , 2019
  * [17] Alex Church, John Lloyd, hadsell and Nathan F. Lepora  “Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation”  In _5th Annual Conference on Robot Learning_ , 2021
  * [18] Ilge Akkaya et al.  “Solving rubik’s cube with a robot hand”, 2019
  * [19] Richard M Murray, Zexiang Li and S Shankar Sastry  “A mathematical introduction to robotic manipulation”  CRC, 1994
  * [20] Robert Haschke  “Grasping and Manipulation of Unknown Objects Based on Visual and Tactile Feedback”  In _Motion and Operation Planning of Robotic Systems_ 29 Springer, 2015, pp. 522
  * [21] Guillaume Walck, Robert Haschke, Martin Meier and Helge Ritter  “Robot Self-Protection by Virtual Actuator Fatigue: Application to Tendon-driven Dexterous Hands During Grasping”  In _Proc. IROS_ Vancouver: IEEE, 2017
  * [22] Zhanat Kappassov, Juan-Antonio Corrales and Véronique Perdereau  “Tactile sensing in dexterous robot hands”  In _Robotics and Autonomous Systems_ 74 Elsevier, 2015, pp. 195–220
  * [23] Morgan Quigley et al.  “ROS: an open-source Robot Operating System”  In _ICRA workshop on open source software_ , 2009  Kobe
  * [24] Sachin Chitta et al.  “ros_control: A generic and simple control framework for ROS”  In _Journal of Open Source Software_ 2, 2017, pp. 456