###### Abstract

The sense of touch plays a key role in enabling humans to understand and
interact with surrounding environments. For robots, tactile sensing is also
irreplaceable. While interacting with objects, tactile sensing provides useful
information for the robot to understand the object, such as distributed
pressure, temperature, vibrations and texture. During robot grasping, vision
is often occluded by its end-effectors, whereas tactile sensing can measure
areas that are not accessible by vision. In the past decades, a number of
tactile sensors have been developed for robots and used for different robotic
tasks. In this chapter, we focus on the use of tactile sensing for robotic
grasping and investigate the recent trends in tactile perception of object
properties. We first discuss works on tactile perception of three important
object properties in grasping, i.e., shape, pose and material properties. We
then review the recent development in grasping stability prediction with
tactile sensing. Among these works, we identify the requirement for
coordinating vision and tactile sensing in the robotic grasping. To
demonstrate the use of tactile sensing to improve the visual perception, our
recent development of vision-guided tactile perception for crack
reconstruction is presented. In the proposed framework, the large receptive
field of camera vision is first leveraged to achieve a quick search of
candidate regions containing cracks, a high-resolution optical tactile sensor
is then used to examine these candidate regions and reconstruct a refined
crack shape. The experiments show that our proposed method can achieve a
significant reduction of mean distance error from 0.82 mm to 0.24 mm for crack
reconstruction. Finally, we conclude this chapter with a discussion of open
issues and future directions for applying tactile sensing in robotic tasks.

Tactile sensing, robotic perception, robot grasping, material recognition,
shape recognition, pose estimation, grasping stability analysis.,

###### keywords:

##  Chapter 0 Robotic Perception of Object Properties using Tactile Sensing

###  1 Introduction

Humans explore the environment in their close vicinity using rich sensory
information such as the visual information obtained from their eyes and the
tactile feeling through physical interaction with cutaneous receptors. Among
these sensing modalities, the sense of touch is a key source of information
for predicting object properties. From touch sensing, diverse sensory
information can be obtained, such as pressure, vibration, pain and
temperature.

For robots, tactile sensing is also an irreplaceable modality. Compared to
vision and other sensing modalities, tactile sensing is superior at processing
material characteristics and detailed shapes of objects [1, 2, 3]. By
analyzing the data from tactile sensors, the properties of the object in
contact can be extracted such as distributed pressure, vibrations and surface
textures. Much of such information is inaccessible for remote sensors like
cameras due to the occlusion of the end-effectors, which makes tactile sensing
highly important in grasping and manipulation. Furthermore, tactile sensing is
not subject to factors like light conditions that the performance of vision
heavily depends on.

To enable robots to have the sense of touch as humans, in the past decades
researchers in the field of robotics have put many endeavours in the
development of tactile sensing solutions [4, 5, 6, 7, 8, 9, 10, 11] for the
perception of object properties. In this chapter, we review the recent
development in robotic perception of object properties with tactile sensing.
Three types of object properties that are important for robot grasping tasks
are investigated: the material, shape and pose of the object. The grasping
stability prediction using tactile sensing is also discussed, which is vital
in object grasping.

Through the survey, we find that many robotic tasks, especially dexterous
grasping and manipulation, can only be performed with vision and tactile
sensing together. To demonstrate the coordination of the visual and tactile
perception, our recent work in vision-guided tactile crack perception [12] is
introduced. In this work, camera vision is first used to quickly search the
candidate crack regions, a high-resolution optical tactile sensor is then
applied against these candidate regions. Finally, a refined crack shape is
reconstructed from the obtained tactile images. In this task, vision shows a
large field of view and is able to detect the potential crack areas. On the
other hand, tactile sensing can enhance the fineness of the reconstructed
crack shape thanks to the characteristics of being less susceptible to light
and noise. To evaluate the proposed method, we collected a dataset by
investigating 10 mock-up structures that were manufactured using 3D printing.
The extensive experiment results demonstrate that compared to only using
vision itself, our proposed approach achieves a significant reduction of mean
distance error from 0.82 mm to 0.24 mm for crack reconstruction. Furthermore,
the proposed method is more than 10 times faster than passive tactile
perception in terms of tactile data collection time.

The remainder of this chapter is organized as follows: the works on material
properties recognition using tactile sensing are first discussed in Section 2,
followed by discussions of object shape estimation and object pose estimation
via touch, and grasping stability prediction in Sections 3, Section 4 and
Section 5, respectively. The vision-guided tactile perception for crack
reconstruction is then introduced in Section 6. At the end of this chapter, a
conclusion is given along with the discussion on open issues and potential
future directions.

###  2 Material properties recognition using tactile sensing

To effectively grasp and manipulate objects, robots need to recognize the
materials of the objects that they are interacting with. The material
properties of an object include roughness, softness and hardness etc. By
recognizing the object’s material properties, the robot can infer a reasonable
grasping force and further achieve effective interaction with an object, e.g.,
grasping a fragile glass cup without damaging it. Furthermore, the
understanding of material properties enables robots to facilitate the task.
For example, in a task of laundry sorting, by recognizing the materials of
different garments with tactile sensing (and also vision), the robot will be
able to separate the items, select appropriate laundry options and prevent
potential damage to the clothes. In general, there are two types of approaches
for material recognition using tactile sensing: texture based and stiffness
based.

Texture based tactile material recognition. Surface texture information is
usually closely related to the friction coefficients, roughness, and micro-
structure patterns of objects. Among early studies, hand-crafted features such
as force variation [13, 14] were widely used. In [13], the vibrations detected
by the accelerometer were first transferred to the frequency-domain by Fast
Fourier Transform (FFT), then two machine learning algorithms, i.e., Support
Vector Machine (SVM) and k-Nearest Neighbors (k-NN) were used to infer the
surface texture class. In [14], the surface physical properties were
determined by the application of a dynamic friction model while a finger-
shaped force/torque sensor sliding along the object surface at varying speeds.
In [15], a microelectromechanical system (MEMS) based tactile sensor was
developed and used to discriminate the roughness of object surfaces based on
the estimated vibratory patterns during the sliding motion. In [16], a MEMS-
based tactile array sensor was employed to distinguish simple textures using
Maximum Likelihood (ML) estimation. In [17], a joint kernel sparse coding
method was developed, and this method can explicitly take the intrinsic
relations between tactile sensors of BarretHand into account.

Instead of using electronic tactile sensors that were used in the above works,
some studies classified surface textures with optical tactile sensors [18, 6].
In such sensors, cameras are used to capture the deformation of a silicone
layer that is placed on the top of the camera. Thanks to the use of a camera
in the optical tactile sensors, detailed textures of the object can be
captured. In [19], the camera-based GelSight sensor [20] was used to obtain
the tactile images by pressing the sensor against different objects. An
algorithm named Multi-scale Local Binary Pattern (MLBP) was proposed to
classify different surface textures. Similarly, another camera-based tactile
sensor TacTip [21] was used to analyze the object textures in [22].

Neural networks have recently been used as feature learners to infer material
properties. In [23], the raw 24,000-dimensional sensor signal was fed to a
convolutional neural network and a high classification accuracy of up to 97.3%
was achieved. In [24], convolutional and recurrent neural networks were
applied to learn feature representations from raw data acquired with the
hybrid touch and sliding movements. On the other hand, the use of deep neural
networks may result in high computational costs and make it challenging to
deploy the learned models in real-time operations.

To enhance the efficiency of the texture classification, a spiking neural
network was used to can achieve a high classification accuracy comparable to
Artificial Neural Network (ANN) with a faster inference in [25]. In [26], a
spatio-temporal attention model was proposed for the material recognition by
using the tactile texture sequence, where the spatial attention was used to
pay more attention to the dominant features for each tactile frame, and
temporal attention was used to model the relevance of different regions in the
whole contact event. As a result, redundant features can be suppressed, and
the salient features can be extracted effectively for the material
recognition.

Stiffness based tactile material recognition. Object stiffness is another cue
that can be used for material property recognition [27]. By using a BioTac
[28] sensor, the object compliance, i.e., the reciprocal of stiffness, can be
estimated either using the contact angle of the fingertip [29] or
investigating the BioTac electrode data [30]. In [31], a piezoresistive
tactile sensor was used, and the objects were classified by applying a KNN
classifier as well as the dynamic time warping algorithm [32] that can
calculate the distance between time series. The camera-based GelSight sensor
was also applied to object stiffness estimation [33]. In [33], the object
hardness was estimated based on a numerical model that compared the contact
area changes and contact surface geometry as well as the normal force value
during pressing the GelSight sensor.

The object materials can be recognized by using different tactile sensors and
different sensing cues. As summarized in Table 1, there are different motions
used in texture based and stiffness based material recognition methods: data
of acceleration, force vibration and acoustic data can be obtained via the
motions of sliding, tapping and scratching to reveal the surface textures,
whereas data of force variances can be acquired via the motions of squeezing,
knocking and pressing to reveal the stiffness of the object. When an optical
tactile sensor is used, cues of both object surface textures and stiffness can
be obtained by analyzing the tactile images. The motions employed and
resultant data types also result in pros and cons of each method: the use of
cameras in the optical tactile sensors can capture the detailed micro-
structure patterns of object textures in one high-resolution tactile image,
whereas it also comes along with high computational cost compared to the use
of other tactile sensors.

Table 1: A summary of material recognition methods with touch sensing Method Type | Texture based |  Stiffness based
---|---|---
Motions |  Sliding, tapping, scratching |  Pressing |  Squeezing, knocking, pressing
Data types |  Acceleration, force vibration, acoustic data |  Tactile images |  Force variances, tactile images
Advantages |  Low cost and limited computational expenses |  Micro-structure patterns of object textures can be captured in one tactile image |  Limited computational costs; complementary to other information
Disadvantages |  Interaction actions like scratching may damage objects |  Low resolution of tactile sensors may cause difficulties for processing |  Interaction actions like knocking may damage objects
References |  [13], [14] |  [19], [22] |  [31], [34]

###  3 Object shape estimation using tactile sensing

The ability to identify or reconstruct the shape of objects is crucial for
robots to perform multiple tasks such as grasping and in-hand manipulation. By
gathering the shape information of objects, robots can better plan and execute
grasping strategies and trajectories. There are two main approaches to achieve
shape estimation, i.e., shape recognition and shape exploration, and we will
discuss them below.

Tactile shape recognition. Due to the limitation of the low resolution, in
most early works tactile sensors were used to collect contact points, and
generate point clouds to represent object shapes [35]. The mosaics of tactile
measurements [36] were used to reconstruct object global geometry. Such
approaches are time-consuming, especially when a large object surface needs to
be explored.

Instead of collecting contact points or local measurements, the object shapes
can be represented by extracting features from tactile readings [37]. In [38],
a new tactile-SIFT descriptor was proposed to extract features in the tactile
image and 91.33% classification accuracy on shape recognition was achieved. In
[39, 40], a novel method named Iterative Closest Labeled Point (iCLAP) was
introduced that fundamentally linked kinesthetic cues (contact points) and
tactile features extracted from local patterns, and showed superior object
recognition performance compared to methods using either contact points or the
local tactile features only. When a tactile sensor interacts with an object,
two types of cues are obtained: local features of the object that indicate the
local shapes of the object and the distribution of the contact points that
gives a global view of the object in the 3D space.

In the above methods, either local contacts or local features are gathered to
train a model firstly, then the trained model is used to recognize the objects
in the test set by matching the new observations with the ones in the
training. To this end, a database of the object shapes needs to be built. Such
methods show a good performance in recognizing object shapes that are known in
the database. However, when the object shape is not available in the database,
tactile shape exploration is needed.

Tactile shape exploration. To reduce the dependence on prior knowledge such as
object shapes, researchers have proposed several approaches that reconstruct
object shape through shape exploration [41], particularly in the paradigm of
active shape exploration. In [42], a probabilistic approach based on Bayes’
rule was proposed to explore the object’s surface through rotating the tactile
sensor and moving the sensor tangentially over the object surface. To improve
the generalization performance of shape exploration, a deep convolutional
neural network was designed in [4] to replace the probabilistic model in [42].
Nevertheless, the performance of those methods heavily depends on the
localization error of end-effector and requires a long exploration time.
Hence, some other studies use Gaussian Process Implicit Surface (GPIS) [43] as
object shape representations [44, 45, 46, 47] to reduce the impact of
localization uncertainty and accelerate the exploration.

In [44, 45], active touch probing was performed at discrete query points. To
avoid inefficient touch-and-retract motions, an active learning framework was
proposed in [46] based on optimal query paths to address the efficiency issue
of tactile shape exploration. Furthermore, Driess et al. [47] extended the
previous work [46] and used multiple end-effectors at the same time to achieve
a more efficient exploration.

However, the objects used in the above works were fixed on the table to avoid
the movement of objects, which makes it hard to be applied in real grasping
tasks. To address the uncertainties of movable objects, some studies attempted
to recover a movable object’s shape by using a series of contacts [48, 49].
Inspired by the paradigm of Simultaneous Localization And Mapping (SLAM) in
mobile robotics, an approach that uses contact measurements and planar pushing
mechanics as constraints in batch optimization was proposed in [48]. Due to
the usage of ordered control points as object shape representations, the
method proposed in [48] is not suitable for exploring objects of arbitrary
shapes, and will fail easily when an incorrect data-association exists. To
address this issue, a recent work [49] expanded the method in [48] by
combining efficient GPIS regression with factor graph optimization over
geometric and physics-based constraints. The results showed a promising
localization and shape estimation performance in both simulated and real-world
settings. The localization and exploration of the contact on the object can
also be facilitated by matching the tactile features with a visual map of the
object. In [41], the visual-tactile localization problem is also treated as a
probabilistic estimation problem for the first time, i.e., a SLAM problem,
that was solved in a framework of recursive Bayesian filtering.

As summarized in Table 2, most works in tactile shape recognition focus on the
shape estimation of fixed objects, which simplifies the task by ignoring the
object’s dynamics models. Only a few works can estimate the shape of a movable
object, whereas they could only be applied in two-dimensional shape
estimation. As for the shape representations, there are three different ones:
shape class, ordered points and GPIS. The different object states and shape
representations also result in pros and cons of each method: those methods
using shape classes as the representations are only applicable to specific
scenarios such as assembly automation, whereas they are able to perform the
perception much faster than those methods using points and GPIS as shape
representations. Tactile object shape exploration methods are less dependent
on priors and therefore are more suitable for the estimation of unseen
objects, ones of arbitrary shapes and even movable objects.

Table 2: A summary of the methods for tactile object shape estimation Method Type |  Shape recognition | Shape exploration
---|---|---
Object State |  Fixed |  Fixed |  Fixed |  Movable
Shape Representation |  Shape class |  Ordered points |  GPIS |  Ordered points or GPIS
Advantages |  Low computational cost |  High accuracy |  Suitable for 3D exploration |  Suitable for movable objects
Disadvantages |  Need some prior shape information, not suitable for unseen objects |  Long exploration time and performance will drop when explorating 3D objects |  The computational effort is not suitable for real-time tasks |  Long exploration time and can only reconstruct 2D shapes
References |  [36], [38], [39] |  [42], [4] |  [46], [47] |  [48], [49]

###  4 Object pose estimation using tactile sensing

For dexterous manipulation, it is essential to estimate the pose of the object
in hand accurately. A small estimation error of object pose could result in
misplacing robotic fingers on the object and even damaging the object.
Recently, pose estimation is an important topic in the field of computer
vision. However, during object manipulation, the gripper or the arm of the
robot may partially or completely occlude the object from vision, which makes
the visual pose estimation not stable anymore. To address this problem,
tactile sensing can be used as a complementary modality for determining the
pose of an object in grasping and manipulation.

Particle filtering has been extensively used in vision-based robot
localization problems. In the past years, it has also found its applications
in tactile object pose estimation [50, 51, 52, 53]. In [50], particle filters
were applied to estimate a tube’s pose, in which some parts of the robot’s
fingers were not in contact with the tube. In [51], a particle filtering
approach using proprioceptive and tactile measurements was proposed to
localize the small objects, e.g., button or grommet, embedded in a flexible
material such as thin plastic. In [52] an approach named Memory Unscented
Particle Filter (MUPF) was proposed that localizes objects recursively in
real-time. In [53], a particle-filter based method was proposed to estimate
in-hand objects pose by bringing the object into contact with a surface.

In some other works, the pose of the in-hand object was estimated by matching
geometry with a pressure sensor [54] or a camera-based tactile sensor [55,
56]. In [54], the in-hand object pose was estimated by matching Principal
Component Analysis (PCA)-based features extracted from tactile pressure
readings to the object’s geometric features. In [55], a feature-based height
map registration method was proposed to localize the small objects, e.g., a
USB cable. Bauza et al. in [56] used CNNs to compute heightmaps and a coarse-
to-fine strategy for localizing objects. With this approach, each object has
to be explored individually to determine the tactile map of its global shape.

In summary, particle filtering and feature matching have been used to achieve
tactile object pose estimation. However, the pose estimation problems are
still not fully solved yet. For example, most particle filter based approaches
are sensitive to parameters of models and need a great number of iterations to
infer the estimated results. In contrast, feature-based matching methods for
pose estimation have to obtain a tactile map for each object and extensively
explore the object, which is time-consuming.

###  5 Grasping stability prediction using tactile sensing

Apart from the material, shape and pose of the object, it is also important to
predict the grasping stability with tactile sensing while they are being
grasped or manipulated. With the prediction of grasping stability, a robot can
find a stable grasping pose and prevent objects from falling. The prediction
methods can be summarised into two types: slip detection and grasping
stability regression, and we will discuss them as follows.

Tactile slip detection. Some researchers simplify the grasping stability
prediction as a slip detection task, when the object is being lifted. In [57],
a force/torque sensor and a matrix-based tactile sensor were used to detect
both translational and rotational slip with the prior of the contact surface’s
frictional coefficient. In [58], a fabric sensor of woven electroconductive
yarns was used to detect the slip by detecting the change of resistance that
depends on the yarn stretch. Although the above method showed good performance
in grasp stability prediction, the need for prior knowledge like the
frictional coefficient in [57] and the initial stretch of yarn in [58] pose a
restriction on their applications.

To predict the slip between the object and the hand without the priors, a
3-layer neural network was used in [59] to predict the slip, which achieves a
classification accuracy of 80%. However, as the data in [59] was squeezed into
a one-dimensional array, the spatial distribution inherent in the tactile
sensor was lost. To exploit the spatial relationships between different
electrodes on the BioTac sensor, in [60] the Graph Neural Networks (GNNs) were
used which yielded a validation accuracy of 92.7% on predicting the grasp
stability of novel objects.

Tactile images obtained from optical-based tactile sensors have also been used
to detect the slip in recent years. In [61], a two-fingered gripper composed
of two GelSight sensors was used to detect slip events by measuring the
relative displacement between object texture and markers. In [62, 63], two
other different optical tactile sensors, i.e., TacTip [64] and T-MO [65], were
employed for detecting slip. The results in [62, 63] showed that the slip is
detected effectively employing an SVM classifier with the movement of pins as
features.

Tactile grasping stability regression. Different from slip detection that is
used to prevent in-hand objects from dropping, grasping stability regression
is normally used for picking up objects with a robot arm. To assess the grasp
stability, multiple works [66, 67, 68] used the image moments of tactile
readings as features in their classifiers. In [66, 67], a kernel-logistic
regression model was presented. In the model, pose and touch are the
conditional factors for grasping probability. To reduce the modeling effort
and computational load, a wrench-based reasoning method was proposed in [68]
which achieved comparable performance with [66, 67]. In [69], CNNs were
combined with Long Short-Term Memory Networks (LSTMs) for predicting the grasp
stability.

Though promising results have been achieved, the above works can only estimate
the stability of an ongoing grasp with tactile readings and are not capable of
regrasping the object when a failure of grasp happens. The tactile readings
can also be used for computing the action level information [70]. Inspired by
the actions that humans can adjust hand posture to perform dexterous grasping
by using only tactile information, some researchers have investigated
assessing current grasp and selecting grasp adjustments to produce a stable
new grasp simultaneously. In [71], the grasp outcome estimated with a grasp
stability predictor was used as a reward signal that supervises and provides
feedback to the regrasping reinforcement learning algorithm. In [72], an
action-conditional model using tactile readings of GelSight sensor as input
was proposed to predict the outcome of a candidate grasp adjustment. In
contrast to [71], the action-conditional model presented in [72] learned the
actions entirely from raw inputs. Similar to [72], a re-grasping approach was
presented in [73] using a camera-based tactile sensor. It used CNNs to predict
grasp quality, and made local adjustments through simulating rigid-body
transformations of tactile readings.

As summarized in Table 3, slip detection and grasping stability regression are
used for different application scenarios. Slip detection methods are normally
used for preventing in-hand or lifted objects from dropping, whereas grasping
stability regression methods are usually used to select the best grasp
proposal for picking up the ungrasped objects. Moreover, tactile grasping
stability regression has a wider application prospect compared to tactile slip
detection. For example, the grasp outcome from tactile grasping stability
regression can be used to adjust grasp proposals and secure the object grasp.

Table 3: A summary of methods for tactile grasping stability prediction Method Type |  Slip detection |  Grasping stability regression
---|---|---
Object States |  In-hand or lifted |  Ungrasped
Application scenarios |  To prevent in-hand or lifted objects from dropping |  To pick up objects and prevent them from dropping
Advantages |  Low-cost computation, generalized to novel objects |  Can be used for ungrasped objects, and adjust hand posture to produce a stable new grasp based on ongoing grasp
Disadvantages |  Not suitable for ungrasped objects, and may need the prior knowledge of frictional coefficients |  Low resolution of tactile sensors may cause difficulties for processing, and may need the prior knowledge of object models
References |  [57], [59], [63] |  [67], [69], [72], [73]

###  6 Vision-guided tactile perception for crack reconstruction

By reviewing the above research on tactile object perception, it is found that
many robotic tasks can only be accomplished with vision and tactile sensing
together. To illustrate the coordination between visual and tactile
perception, we present our work on vision-guided tactile perception for crack
detection and reconstruction, with an overview of the framework illustrated in
Figure 1.

As for crack detection tasks, skilled inspectors typically first examine the
surface to search for areas that have color or shape characteristics similar
to cracks, and then use hammers or ultrasonic devices to double-check those
areas instead of traversing all regions. Inspired by those observations, in
this work camera vision is first used for a quick search of the candidate
crack regions; a high-resolution optical tactile sensor is then applied
against these candidate regions and a refined crack shape is reconstructed
from the obtained tactile images.

To evaluate our proposed method, we collected a dataset by investigating 10
mock-up structures that were manufactured using 3D printing. The extensive
experiment results demonstrate that compared to only using vision itself, our
proposed approach achieves a significant reduction of mean distance error from
0.82 mm to 0.24 mm for crack reconstruction. Furthermore, the proposed method
is more than 10 times faster than passive tactile perception in terms of
tactile data collection time.

####  1 Visual Guidance for Touch Sensing

A deep semantic segmentation network is first utilized to predict pixel-wise
masks of cracks. To guide the touch sensing, the contact points with cracks
skeletons are then generated. Details of these two steps are given below.

Figure 1: An overview of our vision-guided active tactile crack detection and
reconstruction method. Top row (from left to right): The Deeplabv3+ model is
used to segment the cracks in the visual image. Given the visual crack mask
and the depth image, a set of contact points are generated to guide the
collection of tactile images. Bottom row (from right to left): Another deep
convolutional network is used to segment the crack in the collected tactile
images. Given the detected tactile crack mask, the crack shape are
reconstructed based on the geometrical model of the GelSight sensor and the
coordinate transforming relation between the tactile sensor coordinate and the
world coordinate.

Visual Crack Segmentation. We treat visual crack segmentation as a semantic
segmentation problem that classifies pixels of an input image into one of two
categories: (a) background (b) cracks. In order to generate these
segmentations, we apply the DeepLabv3+ model [74] that is a state-of-the-art
deep learning model for semantic image segmentation. Considering that the
number of pixels in the background is much larger than that of the cracks, the
network may easily converge to the status that treats all the pixels as
background. To address this issue, we use the original images as input instead
of resizing it to a smaller size as done in the previous works [75] and use a
weighted cross-entropy loss with crack pixels weighted 10x more than
background pixels. In addition, we set the output stride value to 8 since
smaller values give finer details in the output mask. To train the model, we
started with a pretrained model for semantic segmentation based on the COCO
dataset [76] and use the following hyperparameters: SGD optimizer with a
constant learning rate of 1e-6, momentum 0.9 and weight decay 5e-4.

Contact Points Generation. Based on the predicted pixel-wise crack mask in the
color image, we can extract the crack mask skeleton with a pattern thinning
method [77]. To represent the topology of the crack pattern, two types of
keypoints (i.e., end points and branch points) and minimal edges are defined
as follows:

  * •

End points: if they have less than two neighbors;

  * •

Branch points: if they have more than two neighbors;

  * •

Minimal edge $E_{ij}$: if there is a continuous path between two keypoints
$p_{i}$ and $p_{j}$ and all points on the path are neither end points nor
branch points.

For every minimal edge $E_{ij}$ which consists of a number of ordered points,
the keypoint $p_{i}$ is initially selected as the current contact point
$p_{current}$. Then we iteratively choose the next contact point $p_{k}$ using
the following formula:

$\begin{array}[]{l}\max\limits_{k}D[p_{current},p_{k}]\\\ \text{ s.t.
}D[p_{current},p_{k}]<d\end{array}$ (1)

where $D[p_{current},p_{k}]$ is the distance between two points in the world
coordinate system. The hyper-parameter $d$ is the threshold of the distance
between two points that is related to the coverage and speed of the tactile
exploration and perception. A smaller $d$ will increase the coverage while
reducing the perception speed. In our case, $d$ is empirically set to four
fifths of the tactile sensor’s view length. As shown in Figure 1, the end-
point pixels and the generated contact points are tagged with red dots and
green dots, respectively. For each contact points $p_{i}$, the yaw angle of
the end-effector is parallel to the vector $<p_{i},p_{n}>$, where $p_{n}$ is
the nearest contact point to $p_{i}$, so that the end-effector can contact the
surface perpendicularly.

####  2 Guided Tactile Crack Perception

Tactile Crack Detection. To address the issue of false positives in the visual
crack detection caused by light changes and shadows. we apply tactile
information to refine the vision-based detection results and reconstruct crack
shapes in 3D space. At first, we control a robot arm with an embedded camera-
based Gelsight tactile sensor to collect tactile images autonomously at the
generated contact points in Section 1.

The GelSight sensor is a camera-based optical tactile sensor that can capture
fine details of the object surface. As shown in Figure 2, a webcam under an
elastomer captures the deformation of the elastomer as it interacts with the
object. The sensor has a flat surface and a view range of $14mm\times 10.5mm$
and can capture tactile images at a frequency of 30 Hz [78].

Figure 2: The geometrical model of the GelSight sensor. The webcam at the
bottom captures the deformations of the elastomer and the LEDs project light
to illuminate the space.

After collecting the tactile images, another Deeplabv3+ convolutional network
[74] with ResNet-101 [79] is then used to predict the crack masks in tactile
images. Since the number of background pixels is similar to that of the crack
pixels in tactile images, we use the vanilla cross-entropy loss instead of the
weighted cross-entropy loss in Section 1. Using the predicted masks in tactile
images, we can double-check the visual segmentation results. In case there is
more than one tactile image with a predicted crack area smaller than a
predetermined threshold, all minimal edges will be classified as false
positives and deleted. The threshold is empirically set as one-fiftieth of the
total number of pixels in the tactile image.

Tactile Crack Reconstruction. For crack perception, it is crucial to estimate
the shape and size of the crack when assessing its potential risk to the
building and infrastructure. Due to the limitation of the depth camera’s
accuracy, current vision-based reconstruction techniques cannot reconstruct
small cracks precisely. Therefore, we use the tactile images obtained from the
GelSight sensor for the reconstruction of cracks, whose spatial resolution is
approximately 20 to 30 microns.

Using the detected boundaries of pixel-wise masks in the tactile images, we
first predict the location of cracks on the surface of the GelSight sensor. To
simplify the problem, the webcam is modeled as a pinhole camera, and the
surface of the GelSight sensor is treated as a flat plane perpendicular to the
camera’s $z$ axis. Hence, the transformation between a contact point
$P=[X_{c},Y_{c},Z_{c}]^{T}$ in the tactile sensor coordinates (the tactile
sensor take the centre of the webcam as the origin) to the pixel
$P^{\prime}=[u,v]^{T}$ in the tactile image coordinates can be calculated:

$Z_{c}\left[\begin{array}[]{l}u\\\ v\\\
1\end{array}\right]=K\left[\begin{array}[]{c}X_{c}\\\ Y_{c}\\\ Z_{c}\\\
1\end{array}\right]$ (2)

where $Z_{c}$ is the distance from the optical center to the elastomer surface
of the tactile sensor. $K$ is the matrix of the intrinsic parameters of the
webcam and can be represented as:

$K=\left[\begin{array}[]{llll}\frac{f}{d_{x}}&0&u_{0}&0\\\
0&\frac{f}{d_{y}}&v_{0}&0\\\ 0&0&1&0\end{array}\right]$ (3)

where $f$ is the focal length of the webcam, $d_{x}$ and $d_{y}$ denote the
pixel size, and $(u_{0},v_{0})$ is the center point of the tactile image.

After obtaining the position $P$ in the tactile sensor coordinates, we can
calculate its position $P_{W}$ in the world coordinate system:

$P_{W}=T_{E}^{W}T_{C}^{E}P$ (4)

where $T_{C}^{E}$ and $T_{E}^{W}$ are the transformation matrix from the
tactile sensor coordinates to the end-effector coordinate system, and from the
end-effector coordinate system to the world coordinate system, respectively.

####  3 Experiment Setup

In this subsection, we introduce the robot setup used for data collection and
experiments with an overview shown in Figure 3.

Figure 3:  a) The UR5 robot setup with GelSight sensor and Realsense D435i; b)
GelSight sensor; c) Realsense D435i.

3D Printed Structures with Cracks. Following part of the data acquisition
protocol In [80], a set of 10 structures with cracks of different widths
(holes in the structures) are manufactured with PLA plastic. To test the
robustness of the proposed method, several structures are painted with fake
cracks on the surfaces. The samples are shown in Figure 4.

Figure 4: Sample structures used for collecting the visual and tactile
dataset. Top row: printed structures with real cracks (holes). Bottom row:
printed pad with fake cracks (painted black blocks).

Visual Data Collection. For training the visual model, we put those 3D-printed
structures on a table and took 10 images of each one containing only real
cracks. Then we take 3 images of each object separately for the test. All the
images are taken with a RealSense D435i camera.

Tactile Data Collection. The tactile data collection setup consists of two
parts: a 6-DOF Universal Robots UR5 collaborative robot arm and a GelSight
sensor mounted on a 3D-printed end effector. To collect the tactile data
autonomously and repeatedly, we build a data collection software based on
Robot Operating System (ROS). Using the software, a robot arm can be
controlled to move across a structure following the pre-determined initial
position, step length, and steps in $x$ and $y$ axes. In each position, we
rotate the sensor with different angles about the axis perpendicular to the
surface in order to generalize the dataset. When the pressure reaches a
threshold, the robot arm will stop moving and the GelSight will record a
tactile image. In this way, we can obtain high-quality tactile data and avoid
stopping the robot arm for unnecessarily protective reasons. In total, 544
valid tactile images were collected and split into training and test sets (370
and 174 for each, respectively), with some samples and their annotations shown
in Figure 5.

Figure 5: Visualization of the tactile images and their annotations for
different cracks.

####  4 Experiment Results

Detection Results. Our proposed methods are evaluated using standard
evaluation metrics of pixel accuracy (pixAcc) and Intersection over Union
(IoU) . The results of segmentation based solely on visual information and
based both on visual and tactile information are summarized in Table 4. Due to
the presence of fake painting cracks, both pixel accuracy and IoU of visual
semantic segmentation have dropped significantly from 0.899 to 0.866 and from
0.504 to 0.376, respectively. In addition, the results show that the crack
detection can be effectively improved after tactile information is used to
find the fake paintings and refine the visual results. Due to the fact that
cracks on the RGB images only take up fewer pixels compared to the background,
the pixel accuracy does not change dramatically as IoU.

Table 4: Crack Detection Accuracy.

Modalities | Fake Painting | pixAcc | IoU
---|---|---|---
vision | $\times$ | 0.899 | 0.504
vision | ✓ | 0.866 | 0.376
vision-tactile | ✓ | 0.909 | 0.636

Reconstruction Results. To evaluate the accuracy of our proposed crack
reconstruction method, we used the mean, maximum, and Standard Deviation (SD)
of the shortest distances between the actual crack shape and the reconstructed
crack location. There are four methods used for comparison. The vision method
uses point clouds obtained from visual detection and depth information to
represent cracks. Aligned vision reduces the impact of depth information
accuracy on reconstruction by projecting the point cloud to the table surface.
Passive tactile methods collect tactile images by traversing the entire
surface of a 3D printed object.

Table 5 shows that our approach significantly improves the mean distance error
from 0.55 mm to 0.24 mm for crack reconstruction compared to aligned vision
method. Two examples of reconstructed crack profiles are shown in Figure 6,
which shows that reconstructed crack profiles with tactile data are much
closer to the ground truth compared to the vision-based method. Furthermore,
our proposed vision-guided tactile perception is 10 times faster than passive
tactile perception in terms of the time it takes to collect tactile data
without affecting the accuracy of crack reconstruction.

Figure 6: Visual comparison of different reconstruction methods. Blue, red,
yellow, green curves represent the ground truth of the crack profile, aligned-
vision, passive-tactile and our method for crack reconstruction, respectively.
Table 5: Reconstruction Accuracy

Method | MeanD(mm) | SD(mm) | MaxD(mm) | time(s)
---|---|---|---|---
vision | 0.82 | 0.92 | 4.87 | 1
aligned-vision | 0.55 | 0.53 | 3.78 | 1
passive-tactile | 0.20 | 0.17 | 0.99 | 400
active-tactile[ours] | 0.24 | 0.16 | 0.82 | 35

###  7 Conclusion and Discussions

In this chapter, we first briefly illustrate the irreplaceability of tactile
sensing. Then we review the state-of-the-art tactile perception of object
properties from four perspectives: material properties, object shape, object
pose and grasping stability. A recent trend shows that deep learning,
especially CNNs, has been widely applied in robotic tactile perception and
optical tactile sensors play a greater role than traditional force tactile
sensors thanks to the quick development of both optical sensors and the
computer vision field. Moreover, we introduce a novel vision-guided tactile
perception for crack detection and reconstruction. The cooperation between
those two modalities addresses the false positives in visual detection
results. The experiment results show that our proposed method can reconstruct
crack shapes accurately and efficiently. It has the potential to enable robots
to inspect and repair the concrete infrastructure.

As for object property perception using tactile sensing, however, there still
exist multiple open issues expected to be investigated: 1) how to achieve
real-time perception with optical tactile sensors; 2) how to acquire the
costly data with labeling and ground truth; 3) how to close the perception-
action loop with tactile sensing. Those open issues and future directions are
discussed as follows:

Real time tactile perception. Due to the development of optical tactile sensor
technologies and the domination of deep learning in the computer vision field,
CNNs are investigated to achieve better performance at the expense of high
computational load. To this end, lighter networks obtained via methods like
network pruning can be also considered in those tasks that requires real-time
processing in the future.

Data acquirement and Labeling. It is challenging to obtain ground truth for
tactile perception tasks due to the high cost and difficulties in labeling or
measuring tactile properties of objects. Most of the recent literature still
uses a limited number of human annotators to label data which may bring bias
to the ground truths. To this end, unsupervised learning and sim-to-real
transfer learning [81, 82] will be of benefit in the future research of object
perception using tactile sensing.

Close the perception-action loop with tactile sensing. In the physical
interactions with an object, humans can not only infer object properties but
also adjust the hand posture to better feel features that are relevant to the
current task. However, most of the studies in the field still focus on
research on a dataset collected in advance, whereas the actions taken to
collect the data is not considered in the design of the algorithms. To close
the perception-action loop, it will be an interesting direction to generate
synergies between perception and actions when robots perform complex tasks
with tactile sensing [83].

## References

## References

  * [1] R. S. Dahiya, M. Valle, Robotic tactile sensing: technologies and system, Springer Science & Business Media, 2012.
  * [2] S. J. Lederman, R. L. Klatzky, Haptic perception: A tutorial, Attention, Perception, & Psychophysics 71 (7) (2009) 1439–1459.
  * [3] S. Luo, J. Bimbo, R. Dahiya, H. Liu, Robotic tactile perception of object properties: A review, Mechatronics 48 (2017) 54–67.
  * [4] N. F. Lepora, A. Church, C. De Kerckhove, R. Hadsell, J. Lloyd, From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor, IEEE Robotics and Automation Letters 4 (2) (2019) 2101–2107.
  * [5] S. Luo, L. Zhu, K. Althoefer, H. Liu, Knock-knock: acoustic object recognition by using stacked denoising autoencoders, Neurocomputing 267 (2017) 18–24.
  * [6] S. Luo, W. Yuan, E. Adelson, A. G. Cohn, R. Fuentes, Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 2722–2727.
  * [7] D. F. Gomes, Z. Lin, S. Luo, Geltip: A finger-shaped optical tactile sensor for robotic manipulation, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2020, pp. 9903–9909.
  * [8] D. F. Gomes, Z. Lin, S. Luo, Blocks world of touch: Exploiting the advantages of all-around finger sensing in robot grasping, Frontiers in Robotics and AI 7 (2020).
  * [9] G. Cao, J. Jiang, C. Lu, D. F. Gomes, S. Luo, Touchroller: A rolling optical tactile sensor for rapid assessment of large surfaces, arXiv preprint arXiv:2103.00595 (2021).
  * [10] H. Xie, H. Liu, S. Luo, L. D. Seneviratne, K. Althoefer, Fiber optics tactile array probe for tissue palpation during minimally invasive surgery, in: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2013, pp. 2539–2544.
  * [11] S. Luo, N. F. Lepora, U. Martinez-Hernandez, J. Bimbo, H. Liu, Vitac: Integrating vision and touch for multimodal and cross-modal perception, Frontiers in Robotics and AI 8 (2021).
  * [12] J. Jiang, G. Cao, D. F. Gomes, S. Luo, Vision-guided active tactile perception for crack detection and reconstruction, in: The 29th Mediterranean Conference on Control and Automation (MED 2021), 2021.
  * [13] J. Sinapov, V. Sukhoy, R. Sahai, A. Stoytchev, Vibrotactile recognition and categorization of surfaces by a humanoid robot, IEEE Transactions on Robotics 27 (3) (2011) 488–497.
  * [14] H. Liu, X. Song, J. Bimbo, L. Seneviratne, K. Althoefer, Surface material recognition through haptic exploration using an intelligent contact sensing finger, in: 2012 IEEE/RSJ international conference on intelligent robots and systems, IEEE, 2012, pp. 52–57.
  * [15] C. M. Oddo, M. Controzzi, L. Beccai, C. Cipriani, M. C. Carrozza, Roughness encoding for discrimination of surfaces in artificial active-touch, IEEE Transactions on Robotics 27 (3) (2011) 522–533.
  * [16] S.-H. Kim, J. Engel, C. Liu, D. L. Jones, Texture classification using a polymer-based mems tactile sensor, Journal of micromechanics and Microengineering 15 (5) (2005) 912.
  * [17] H. Liu, D. Guo, F. Sun, Object recognition using tactile measurements: Kernel sparse coding methods, IEEE Transactions on Instrumentation and Measurement 65 (3) (2016) 656–665.
  * [18] J.-T. Lee, D. Bollegala, S. Luo, “touching to see” and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 4276–4282.
  * [19] R. Li, E. H. Adelson, Sensing and recognizing surface textures using a gelsight sensor, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1241–1247.
  * [20] M. K. Johnson, F. Cole, A. Raj, E. H. Adelson, Microgeometry capture using an elastomeric sensor, ACM Transactions on Graphics (TOG) 30 (4) (2011) 1–8.
  * [21] C. Chorley, C. Melhuish, T. Pipe, J. Rossiter, Development of a tactile sensor based on biologically inspired edge encoding, in: 2009 International Conference on Advanced Robotics, 2009, pp. 1–6.
  * [22] B. Winstone, G. Griffiths, T. Pipe, C. Melhuish, J. Rossiter, Tactip-tactile fingertip device, texture analysis through optical tracking of skin features, in: Conference on Biomimetic and Biohybrid Systems, Springer, 2013, pp. 323–334.
  * [23] S. S. Baishya, B. Bäuml, Robust material classification with a tactile skin using deep learning, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016, pp. 8–15.
  * [24] T. Taunyazov, H. F. Koh, Y. Wu, C. Cai, H. Soh, Towards effective tactile identification of textures using a hybrid touch approach, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 4269–4275.
  * [25] T. Taunyazov, Y. Chua, R. Gao, H. Soh, Y. Wu, Fast texture classification using tactile neural coding and spiking neural network.
  * [26] G. Cao, Y. Zhou, D. Bollegala, S. Luo, Spatio-temporal attention model for tactile texture recognition, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 9896–9902.  doi:10.1109/IROS45743.2020.9341333.
  * [27] T. Nanayakkara, A. Jiang, M. d. R. A. Fernandez, H. Liu, K. Althoefer, J. Bimbo, Stable grip control on soft objects with time-varying stiffness, IEEE Transactions on Robotics 32 (3) (2016) 626–637.
  * [28] N. Wettels, V. J. Santos, R. S. Johansson, G. E. Loeb, Biomimetic tactile sensor array, Advanced Robotics 22 (8) (2008) 829–849.
  * [29] D. Xu, G. E. Loeb, J. A. Fishel, Tactile identification of objects using bayesian exploration, in: 2013 IEEE International Conference on Robotics and Automation, IEEE, 2013, pp. 3056–3061.
  * [30] Z. Su, J. A. Fishel, T. Yamamoto, G. E. Loeb, Use of tactile feedback to control exploratory movements to characterize object compliance, Frontiers in neurorobotics 6 (2012) 7.
  * [31] A. Drimus, G. Kootstra, A. Bilberg, D. Kragic, Design of a flexible tactile sensor for classification of rigid and deformable objects, Robotics and Autonomous Systems 62 (1) (2014) 3–15.
  * [32] H. Sakoe, S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE transactions on acoustics, speech, and signal processing 26 (1) (1978) 43–49.
  * [33] W. Yuan, M. A. Srinivasan, E. H. Adelson, Estimating object hardness with a gelsight touch sensor, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016, pp. 208–215.
  * [34] M. Kaboli, P. Mittendorfer, V. Hügel, G. Cheng, Humanoids learn object properties from robust tactile feature descriptors via multi-modal artificial skin, in: 2014 IEEE-RAS International Conference on Humanoid Robots, IEEE, 2014, pp. 187–192.
  * [35] P. K. Allen, K. S. Roberts, Haptic object recognition using a multi-fingered dextrous hand (1988).
  * [36] Z. Pezzementi, C. Reyda, G. D. Hager, Object mapping, recognition, and localization from tactile geometry, in: 2011 IEEE International Conference on Robotics and Automation, IEEE, 2011, pp. 5942–5948.
  * [37] S. Luo, X. Liu, K. Althoefer, H. Liu, Tactile object recognition with semi-supervised learning, in: International Conference on Intelligent Robotics and Applications, Springer, 2015, pp. 15–26.
  * [38] S. Luo, W. Mou, K. Althoefer, H. Liu, Novel Tactile-SIFT Descriptor for Object Shape Recognition, IEEE Sensors Journal 15 (9) (2015) 5001–5009.
  * [39] S. Luo, W. Mou, K. Althoefer, H. Liu, Iterative closest labeled point for tactile object shape recognition, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016, pp. 3137–3142.
  * [40] S. Luo, W. Mou, K. Althoefer, H. Liu, iCLAP: Shape recognition by combining proprioception and touch sensing, Autonomous Robots 43 (4) (2019) 993–1004.
  * [41] S. Luo, W. Mou, K. Althoefer, H. Liu, Localizing the object contact through matching tactile features with visual map, in: 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2015, pp. 3903–3908.
  * [42] N. F. Lepora, K. Aquilina, L. Cramphorn, Exploratory tactile servoing with active touch, IEEE Robotics and Automation Letters 2 (2) (2017) 1156–1163.
  * [43] O. Williams, A. Fitzgibbon, Gaussian process implicit surfaces, Gaussian Proc. in Practice (2007) 1–4.
  * [44] S. El-Khoury, M. Li, A. Billard, On the generation of a variety of grasps, Robotics and Autonomous Systems 61 (12) (2013) 1335–1349.
  * [45] N. Jamali, C. Ciliberto, L. Rosasco, L. Natale, Active perception: Building objects’ models using tactile exploration, in: 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), IEEE, 2016, pp. 179–185.
  * [46] D. Driess, P. Englert, M. Toussaint, Active learning with query paths for tactile object shape exploration, in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2017, pp. 65–72.
  * [47] D. Driess, D. Hennes, M. Toussaint, Active multi-contact continuous tactile exploration with gaussian process differential entropy, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 7844–7850.
  * [48] K.-T. Yu, J. Leonard, A. Rodriguez, Shape and pose recovery from planar pushing, in: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2015, pp. 1208–1215.
  * [49] S. Suresh, M. Bauza, K.-T. Yu, J. G. Mangelson, A. Rodriguez, M. Kaess, Tactile slam: Real-time inference of shape and pose from planar pushing, in: 2021 International Conference on Robotics and Automation (ICRA), IEEE, 2021.
  * [50] C. Corcoran, R. Platt, A measurement model for tracking hand-object state during dexterous manipulation, in: 2010 IEEE International Conference on Robotics and Automation, IEEE, 2010, pp. 4302–4308.
  * [51] R. Platt, F. Permenter, J. Pfeiffer, Using bayesian filtering to localize flexible materials during manipulation, IEEE Transactions on Robotics 27 (3) (2011) 586–598.
  * [52] G. Vezzani, U. Pattacini, G. Battistelli, L. Chisci, L. Natale, Memory unscented particle filter for 6-dof tactile localization, IEEE Transactions on Robotics 33 (5) (2017) 1139–1155.
  * [53] F. von Drigalski, S. Taniguchi, R. Lee, T. Matsubara, M. Hamaya, K. Tanaka, Y. Ijiri, Contact-based in-hand pose estimation using bayesian state estimation and particle filtering, in: 2020 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 7294–7299.
  * [54] J. Bimbo, S. Luo, K. Althoefer, H. Liu, In-hand object pose estimation using covariance-based tactile to geometry matching, IEEE Robotics and Automation Letters 1 (1) (2016) 570–577.
  * [55] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan, E. Adelson, Localization and manipulation of small parts using gelsight tactile sensing, in: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2014, pp. 3988–3993.
  * [56] M. Bauza, O. Canal, A. Rodriguez, Tactile mapping and localization from high-resolution tactile imprints, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 3811–3817.
  * [57] C. Melchiorri, Slip detection and control using tactile and force sensors, IEEE/ASME transactions on mechatronics 5 (3) (2000) 235–243.
  * [58] D. Kondo, S. Okada, T. Araki, E. Fujita, M. Makikawa, S. Hirai, et al., Development of a low-profile sensor using electro-conductive yarns in recognition of slippage, in: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2011, pp. 1946–1953.
  * [59] Z. Su, K. Hausman, Y. Chebotar, A. Molchanov, G. E. Loeb, G. S. Sukhatme, S. Schaal, Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor, in: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), IEEE, 2015, pp. 297–303.
  * [60] A. Garcia-Garcia, B. S. Zapata-Impata, S. Orts-Escolano, P. Gil, J. Garcia-Rodriguez, Tactilegcn: A graph convolutional network for predicting grasp stability with tactile sensors, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–8.
  * [61] S. Dong, W. Yuan, E. H. Adelson, Improved gelsight tactile sensor for measuring geometry and slip, in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2017, pp. 137–144.
  * [62] J. W. James, N. Pestell, N. F. Lepora, Slip detection with a biomimetic tactile sensor, IEEE Robotics and Automation Letters 3 (4) (2018) 3340–3346.
  * [63] J. W. James, N. F. Lepora, Slip detection for grasp stabilization with a multifingered tactile robot hand, IEEE Transactions on Robotics (2020).
  * [64] B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, N. F. Lepora, The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies, Soft robotics 5 (2) (2018) 216–227.
  * [65] J. W. James, A. Church, L. Cramphorn, N. F. Lepora, Tactile model o: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand, Soft Robotics (2020).
  * [66] Y. Bekiroglu, R. Detry, D. Kragic, Learning tactile characterizations of object-and pose-specific grasps, in: 2011 IEEE/RSJ international conference on Intelligent Robots and Systems, IEEE, 2011, pp. 1554–1560.
  * [67] Y. Bekiroglu, D. Song, L. Wang, D. Kragic, A probabilistic framework for task-oriented grasp stability assessment, in: 2013 IEEE International Conference on Robotics and Automation, IEEE, 2013, pp. 3040–3047.
  * [68] R. Krug, A. J. Lilienthal, D. Kragic, Y. Bekiroglu, Analytic grasp success prediction with tactile feedback, in: 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2016, pp. 165–171.
  * [69] J. Li, S. Dong, E. Adelson, Slip detection with combined tactile and visual information, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 7772–7777.
  * [70] Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, H. J. Ritter, A review of tactile information: Perception and action through touch, IEEE Transactions on Robotics 36 (6) (2020) 1619–1634.
  * [71] Y. Chebotar, K. Hausman, Z. Su, G. S. Sukhatme, S. Schaal, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2016, pp. 1960–1966.
  * [72] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, S. Levine, More than a feeling: Learning to grasp and regrasp using vision and touch, IEEE Robotics and Automation Letters 3 (4) (2018) 3300–3307.
  * [73] F. R. Hogan, M. Bauza, O. Canal, E. Donlon, A. Rodriguez, Tactile regrasp: Grasp adjustments via simulated tactile transformations, in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 2963–2970.
  * [74] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of European Conference on Computer Vision (ECCV), 2018.
  * [75] L. Pauly, S. Luo, D. Hogg, R. Fuentes, H. Peel, Deeper networks for pavement crack detection, in: Proceedings of the 34th ISARC, IAARC, 2017, pp. 479–485.
  * [76] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft COCO: Common Objects in Context, in: Proceedings of European Conference on Computer Vision (ECCV), Springer, 2014, pp. 740–755.
  * [77] T. Zhang, C. Y. Suen, A fast parallel algorithm for thinning digital patterns, Communications of the ACM 27 (3) (1984) 236–239.
  * [78] G. Cao, Y. Zhou, D. Bollegala, S. Luo, Spatio-temporal attention model for tactile texture recognition, in: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
  * [79] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [80] F. Palermo, J. Konstantinova, K. Althoefer, S. Poslad, I. Farkhatdinov, Implementing tactile and proximity sensing for crack detection, in: 2020 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 632–637.
  * [81] D. F. Gomes, P. Paoletti, S. Luo, Generation of gelsight tactile images for sim2real learning, IEEE Robotics and Automation Letters 6 (2) (2021) 4177–4184.
  * [82] D. F. Gomes, A. Wilson, S. Luo, Gelsight simulation for sim2real learning, in: ICRA ViTac Workshop, 2019.
  * [83] C. Lu, J. Wang, S. Luo, Surface following using deep reinforcement learning and a gelsighttactile sensor, arXiv preprint arXiv:1912.00745 (2019).


# A quantum analog of Huygen’s clock: noise-induced synchronization

Bhavay Tyagi  Department of Physics, University of Houston, Houston, Texas
77204, United States.  Hao Li  Department of Chemistry, University of Houston,
Houston, Texas 77204, United States.  Institut Courtois & Département de
physique, Université de Montréal, Caisse Postale 6128, Succursale centre-
ville, Montréal, Québec H3T 3J7, Canada  Eric R. Bittner
<EMAIL_ADDRESS>Department of Physics, University of Houston, Houston,
Texas 77204, United States.  Andrei Piryatinski  Theoretical Division, Los
Alamos National Laboratory, Los Alamos, New Mexico 87545, United States
Carlos Silva-Acuña<EMAIL_ADDRESS>Institut Courtois & Département
de physique, Université de Montréal, Caisse Postale 6128, Succursale centre-
ville, Montréal, Québec H3T 3J7, Canada

###### Abstract

We propose a quantum analogue of the Huygens clock, in which the phases of two
spins achieve synchronization through their interaction with a shared
environment. The environment functions analogously to the escapement mechanism
in a mechanical clock, regulating the gear train and permitting the
advancement of timing in discrete intervals. In our proposed model, the
relative phase of the two spins become synchronized through interaction with a
mutual, correlated, environment. We show that for a system of qubits, several
arguments can be made that significantly reduce the cardinality of the set of
allowed measurements and, hence, the complexity of the problem. We present a
numerically efficient method to calculate the degree of quantumness that
exists in the correlations of our final density matrix. This method also
provides a tight upper bound for when the system is described by rank-3 and
rank-4 density matrices.

##  I Introduction

At the close of the 17th century, Christiaan Huygens observed that two
identical pendulum clocks, when weakly coupled through a substantial beam,
rapidly synchronized with identical periods and amplitudes, yet with the
pendula oscillating in antiphasing directions. [1, 2, 3, 4, 5, 6, 7, 8] He
introduced his advances in pendulum clock technology to the Royal Society with
the intention of addressing the ”longitude problem,” which was pivotal for
ascertaining a ship’s precise position at sea. Initially, Huygens attributed
the synchronization of pendulum clocks to air currents. However, subsequent
experiments led him to surmise that the weak coupling via the beam instigated
this antiphase synchronization. Currently, oscillator synchronization is
recognized as a foundational concept in nonlinear science, encompassing both
the classical and quantum domains.[9, 10, 11, 12, 13, 14]

In this paper, we propose a quantum analogue of the Huygens clock, wherein the
phases of two spins achieve synchronization through their interaction with a
shared environment. In our proposed model, the environment functions
analogously to the escape mechanism in a mechanical clock, regulating the gear
train and permitting the advancement of time in discrete intervals. The
escapement also imparts a nominal amount of kinetic energy to the pendulum by
releasing potential energy stored in a coiled spring or a series of weights.
In our framework, escapement is represented by a stochastic term that induces
the transition from $|0\rangle$ to $|1\rangle$ at a specified rate.

Figure 1: Model for quantum “clocks” consisting of two weakly coupled qubits
coupled to local environments with known correlations.

Our results and analysis indicate that both synchronization and
antisynchronization can be achieved within this model, depending upon the
degree of correlation between the escapement mechanisms acting on the two
states. Our results have implications in terms of developing quantum
technologies that are robust against environmental effects such as
decoherence, as well as in understanding the underlying mechanisms for the
transport of quantum excitations in photosynthetic systems in which observed
long-lived exciton coherences are linked to vibronic correlation effects. The
enhanced excitonic coherence has a profound influence on the enhancement of
exciton transport and delocalization. [15, 16, 17, 18, 19, 20]

##  II Theoretical model

The scenario we envision is figuratively sketched in Fig. 1 whereby the two
qubits are directly coupled through an exchange term $J_{xy}$ and indirectly
through correlations within their local environments indicated by $\Xi$. A
suitable Hamiltonian corresponding to this scenario is given by

$\displaystyle H$
$\displaystyle=\frac{\Delta}{2}(\sigma^{z}_{1}+\sigma^{z}_{2})+\frac{\tau}{2}\left(\sigma^{x}_{1}+\sigma^{x}_{2}\right)+J_{xy}(\sigma^{+}_{1}\sigma^{-}_{2}+\sigma^{+}_{2}\sigma^{-}_{1})$
(1)

Similar models have been studied by us and others [21, 22, 23]. In this model,
each local site has a characteristic Rabi frequency $\hbar\tau$ that produces
a coherent oscillation between local states $|0_{i}\rangle$ and
$|1_{i}\rangle$. The qubits are coupled via $J_{xy}$, which allows the
transfer of a quantum from one qubit to the other. $\Delta$ sets the energy
gap between $|0_{i}\rangle$ and $|1_{i}\rangle$ for the decoupled spins.

Decoherence and dissipation are introduced into our model via the Lindblad
term. However, we also introduce correlation between the local environments by
first writing the system/bath interaction in the form.

$\displaystyle H_{s/b}=\sum_{i=1,2}\mathbf{\sigma}_{i}E_{i}(t)$ (2)

where each $\sigma_{i}$ is a local spin operator and $E_{i}(t)$ is a
stochastic term. For our purposes, we take the driving terms to be correlated
Ornstein-Uhlenbeck processes with

$\displaystyle d\mathbf{E}=-\mathbf{A}\cdot\mathbf{E}dt+\mathbf{B}\cdot
d\mathbf{W}$ (3)

with $\mathbf{\Xi}dt\delta(t-t^{\prime})=d\mathbf{W}\otimes d\mathbf{W}$ as
the correlation matrix,

$\displaystyle\Xi=\left(\begin{array}[]{cc}1&\xi\\\ \xi&1\end{array}\right).$
(6)

Working out the covariance and cross-covariance terms, we obtain the spectral
density for the noise as

$\displaystyle\mathbf{J}(\omega)=\frac{1}{2\pi}(\mathbf{A}+i\omega)^{-1}\cdot\mathbf{B}\cdot\mathbf{\Xi}\cdot\mathbf{B}^{T}\cdot(\mathbf{A}-i\omega)^{-1}$
(7)

which is a $2\times 2$ matrix. We bring this into diagonal form by defining a
transformation matrix $T$ that diagonalizes the $\Xi$ matrix (assuming that
both $\mathbf{A}$ and $\mathbf{B}$ are diagonal to begin with). This has the
effect of transforming the two correlated Wiener processes into a pair of
uncorrelated processes that satisfies the Ito identity
$d\tilde{W}_{\alpha}(t)d\tilde{W}_{\beta}(t^{\prime})=\delta(t-t^{\prime})dt$[23].

Following this, we can write the system/bath interaction as

$\displaystyle
H_{s/b}=\sum_{\alpha}\tilde{\sigma}_{\alpha}^{(n)}\tilde{E}_{\alpha}^{(n)}(t).$
(8)

where we transform the system operators as

$\displaystyle\tilde{\sigma}_{\alpha}^{(n)}=\sum_{i=1,2}T_{\alpha,i}\sigma_{i}^{(n)}$
(9)

The usual series of assumptions allow us to include the stochastic terms in
the form of a series of quantum jump processes described by Lindblad operators
of the form

$\displaystyle c^{(n)}_{S}$
$\displaystyle=\sqrt{\gamma(1+\xi)}\frac{1}{\sqrt{2}}(\sigma_{1}+\sigma_{2})$
(10) $\displaystyle c^{(n)}_{A}$
$\displaystyle=\sqrt{\gamma(1-\xi)}\frac{1}{\sqrt{2}}(\sigma_{1}-\sigma_{2})$
(11)

corresponding to symmetric and anti-symmetric system/bath interactions and
$\gamma$ gives the rate for this interaction. Note, that $\sigma_{1}$ and
$\sigma_{2}$ can be any operator acting on spin 1 or spin 2, respectively.

From this we write the Liouville-von Neumann equation for the system-density
matrix as follows.

$\displaystyle\partial_{t}\rho=\frac{1}{i\hbar}[H_{sys},\rho]+\left({\cal
D}_{S}(\rho)+{\cal D}_{A}(\rho)\right)$ (12)

where each ${\cal D}_{k}(\rho)$ is the Lindblad dissipator

$\displaystyle{\cal D}_{k}(\rho)=c_{k}\rho
c_{k}^{\dagger}-\frac{1}{2}\\{c_{k}^{\dagger}c_{k},\rho\\}.$ (13)

where $c_{k}$ are the Lindblad operators. The two dissipators in Eq. 12 can be
combined in terms of site-local terms and cross-terms

$\displaystyle\left({\cal D}_{S}(\rho)+{\cal D}_{A}(\rho)\right)={\cal
D}_{1}(\rho)+{\cal D}_{2}(\rho)+\xi{\cal D}_{12}(\rho)$ (14)

where

$\displaystyle{\cal D}_{12}(\rho)$
$\displaystyle=\gamma\left(\sigma_{1}\rho\sigma^{\dagger}_{2}+\sigma_{2}\rho\sigma^{\dagger}_{1})-\frac{1}{2}\\{\sigma^{\dagger}_{1}\sigma_{2}+\sigma^{\dagger}_{2}\sigma_{1},\rho\\}\right)$
(15)

The crucial parameter in this is $\xi$, which gives the correlation between
the two local environments. As we shall show, this cross-term has the effect
of creating a correlation between two spins even when the exchange coupling is
$J_{xy}=0$.

As mentioned above, a key feature of a pendulum clock is the escapement
mechanism, which provides a small “kick” to the pendulum. We introduce this
into our model using the $\sigma_{i}^{+}$ operators so that the escapement
provides a stochastic driving field on the pair of qubits inducing transitions
from the $|0_{i}\rangle\to|1_{i}\rangle$ state at a rate specified by
$\gamma$. As one of the qubits cycles through its $|0_{i}\rangle$ state over
the course of its normal unitary-time evolution, it can be kicked incoherently
to the $|1_{i}\rangle$ state by the interaction with the environment. If the
two qubits interact with uncorrelated baths ($\xi=0$), the two kick processes
are independent and no phase synchronisation is expected. However, the ${\cal
D}_{12}(\rho)$ dissipator introduces cross-terms such that the excitation of
one spin is contingent upon the state of the other spin. This induces
coherence synchronisation in $\rho$ by enhancing or suppressing the terms in-
phase or out-of-phase depending on whether the “kicks” are correlated or anti-
correlated.

Figure 2: (a-c) Expectation values of $\langle\sigma_{1}^{z}(t)\rangle$ (blue)
and $\langle\sigma_{2}^{z}(t)\rangle$(orange) for 2 qubit system subject to
various degrees of bath correlation as described by Eq. 1 with parameters:
$\Delta=1$, $\tau=1$, $J_{xy}=0.25$, and $\gamma=0.05$ in reduced units. (a)
Anti-correlated ($\xi=-1$); (b)Uncorrelated ($\xi=0$); (c) Correlated
($\xi=+1$). (d,e)Asymptotic phase shift and phase-locking value vs. $\xi$.

##  III Synchronization Dynamics

To study this effect, we performed numerical simulations of the system using
the QuTip (v5.0) quantum simulation package[24, 25], and the representative
results of our simulations are shown in Fig. 2. In the example given here, we
set $\Delta=1$, $\tau=1$, $J_{xy}=0.25$, and $\gamma=0.05$ in reduced units.
In each case, the initial state is set as
$|\psi_{0}\rangle=|1_{1}0_{2}\rangle$. In Fig. 2 we show the expectation value
of $\langle\sigma_{1}^{z}\rangle$ in blue and $\langle\sigma_{2}^{z}\rangle$
in orange when the environments are (a) anticorrelated with $\xi=-1$, (b)
uncorrelated with $\xi=0$, and (c) correlated with $\xi=+1$. At long time, the
two qubits become phase-synchronized depending upon the correlation.

We quantify this in Fig. 2(d) by comparing the asymptotic phase difference
between the two qubits as a function of the correlation parameter. For this,
we extracted the last 25% of the time series for
$\langle\sigma_{1}^{z}\rangle$ and $\langle\sigma_{1}^{z}\rangle$ subtracted
off the baseline and used the discrete Hilbert transform to generate the
analytic signal for the two qubits. Both the $\Delta\phi$ phase shift and the
phase locking value provide useful order parameters for quantifying the
synchronization between observables. In the Supplementary Information, we
include a series of movies showing the full dynamics projected onto a common
Bloch sphere. We conclude from the dynamics that synchronization or
antisynchronization can occur depending upon the correlation within the
driving forces acting on each qubit.

At long time, the system converges to a steady state that also depends upon
the degree of correlation within the environment since

$\displaystyle\frac{i}{i\hbar}[H_{sys},\rho_{ss}]+{\cal
D}_{1}(\rho_{ss})+{\cal D}_{2}(\rho_{ss})=-\xi{\cal D}_{12}(\rho_{ss}).$ (16)

Consequently, it is not obvious a priori whether $\rho_{ss}$ represents a
mixed state in which $\rho_{ss}=\rho_{A}\otimes\rho_{B}$ or represents an
entanglement between systems A and B. Furthermore, the question remains as to
whether or not the synchronization and entanglement between the two qubits is
of quantum or classical origin.

A useful way to quantify entanglement in the quantum state is the entanglement
entropy of the state. Given a bipartite system
$\mathcal{H}_{AB}=\mathcal{H}_{A}\otimes\mathcal{H}_{B}$, the reduced density
matrix describing subregion A of the system is given by tracing over subregion
B,

$\rho_{A}=tr_{B}\ket{\psi}\bra{\psi}$ (17)

where $\ket{\psi}\equiv\ket{\psi}_{AB}$ is the wavefunction of the whole
system. The entropy of entanglement of subregion A is defined as the von
Neumann entropy (vNE) of the reduced density matrix describing that subregion

$S(\rho_{A})=-tr\rho_{A}\ln\rho_{A}\;.$ (18)

This quantity is exactly zero when we know everything about the state and is
equal to $\log d$ when the state is maximally mixed, i.e.,
$\ket{\psi}=\mathbb{I}/d$. In order to distinguish between two quantum states
we look at the relative entropy between them,

$\displaystyle S(\rho||\sigma)=tr\left(\rho\ln\rho-\rho\ln\sigma\right)\geq
0.$ (19)

One can think of this as the amount of information about the distance between
the two states. The bound is saturated if and only if $\rho=\sigma$. In a
bipartite system, the density matrix representing an arbitrary preparation of
the entire system is $\rho_{AB}$. In general
$\rho_{AB}\neq\rho_{A}\otimes\rho_{B}$. We can then apply the above notion of
relative entropy and see how $\rho_{AB}$ and $\rho_{A}\otimes\rho_{B}$ differ
in terms of the information they contain. This relative entropy is defined as
the mutual information as given by

$\begin{split}\mathcal{I}_{\rho_{AB}}&(A:B)=S(\rho_{AB}||\rho_{A}\otimes\rho_{B})\\\
&=-S(\rho_{AB})-tr(\rho_{AB}\ln\rho_{A})-tr(\rho_{AB}\ln\rho_{B})\\\
&=S(\rho_{A})+S(\rho_{B})-S(\rho_{AB})\end{split}$ (20)

Figure 3: Mutual Entropy for $J_{xy}=0$ vs. correlation parameter and coupling
constant $\gamma$.

In Fig. 3, we show the mutual information for steady-state solutions. as a
function of the correlation parameter $\xi$ and coupling constant $\gamma$.
For $J_{xy}=0$ and $\xi=0$, the two spins are completely decoupled, and there
is no mutual information between the two subsystems. However, for $\xi\neq 0$,
correlations in the environment lead to increasing degrees of entanglement
even though there is no direct coupling between the two spins. Increasing
$J_{xy}$, leads to increased entanglement.

Mutual information also quantifies the total number of correlations in a given
state. These correlations can be classical or quantum. We know that if
$\rho_{AB}$ is a pure state, then the vNE $S(\rho_{AB})$ is zero and the
entropies of the subsystems are equal to each other. Therefore, mutual
information is

$\displaystyle\mathcal{I}_{\rho_{AB}}(A:B)=2S(\rho_{A})=2S(\rho_{B}).$ (21)

Figure 4: Mutual information at $J_{xy}=-1$ (a), $J_{xy}=0$ (b), and
$J_{xy}=+1$ (c) versus the coupling term $\gamma$ for correlated (blue), anti-
correlated (orange), and uncorrelated (green) environments. The dashed lines
represent the contribution due entirely to the quantum correlation effects,
$D(A:B)$ per equation 31. The shaded areas denote the difference between the
total mutual information and our lower-bound estimate of the “quantum-ness”
and represents the maximum amount of classical correlation.

For pure states, all existing correlations are truly quantum and represent
entanglement. When dealing with quantum systems, the type of measurement plays
an important role since it has a backaction on the system’s state. For a
bipartite system, we can learn something about subsystem A by making some
measurements on subsystem B. Consider a Positive Operator-Valued measurement
(POVM) $\Pi_{k}^{B}$ that satisfies
$\sum_{k}(\Pi_{k}^{B})^{\dagger}\Pi_{k}^{B}=\mathbb{I}$ and
$\bra{\psi}\Pi_{k}^{B}\ket{\psi}\geq 0$. We start our system in some arbitrary
$\rho_{AB}$ and perform this measurement in subsystem B. We know that the
probability of the k-th outcome is given by

$p_{k}^{B}=tr\left[(\mathbb{I}_{A}\otimes\Pi_{k}^{B})\rho_{AB}(\mathbb{I}\otimes\Pi_{k}^{B})^{\dagger}\right]$
(22)

and the post-measurement state is

$\rho_{AB|k}=\frac{(\mathbb{I}_{A}\otimes\Pi_{k}^{B})\rho_{AB}(\mathbb{I}\otimes\Pi_{k}^{B})^{\dagger}}{p_{k}^{B}}.$
(23)

This allows us to compute the conditional entropy

$S_{\Pi}(A|B)=\sum_{k}p_{k}^{B}S(\rho_{A|k})$ (24)

where $S(\rho_{A|k})$ is the reduced density matrix describing subsystem A,
given that we get the outcome $k$ when we perform a measurement on subsystem
B. In other words, this is the lack of information about state $\rho_{A}$ when
we know the outcomes of our measurement in subsystem B. Note that the
conditional entropy depends non-trivially on the choice of measurement. This
allows us to filter out the classical part of the total mutual information. It
is given by

$\begin{split}\mathcal{J}_{\Pi}(A|B)&=S(\rho_{A})-S_{\Pi}(A|B)\\\
&=S(\rho_{A})-\sum_{k}p_{k}^{B}S(\rho_{A|k}).\end{split}$ (25)

In essence, this quantity tells us how much we can learn about subsystem A by
performing some local measurements in subsystem B based on the classical
correlations that exist between the two subsystems[26]. The difference between
the total mutual information $\mathcal{I}(A:B)$ and this classical part is
known as the quantum discord [27]:

$\mathcal{D}_{\Pi}(A|B)=\mathcal{I}(A:B)-\mathcal{J}_{\Pi}(A|B).$ (26)

and it quantifies the amount of remaining shared information between
subsystems once we gain some new information by measuring subsystem B with our
specific choice of the set of measurements $\Pi_{k}^{B}$. This provides
information about the true quantum correlations that exist between subsystems.
Intuitively, this lack of agreement makes sense because measurements play a
“special” role in quantum mechanics and may not give a complete picture. Note
that this quantity is not symmetric i.e.,
$\mathcal{D}(A|B)\neq\mathcal{D}(B|A)$. To make this quantity independent of
the choice of measurements, we can try to maximize the classical correlations
over all possible POVMs

$\displaystyle\mathcal{D}(A|B)$
$\displaystyle=\mathcal{I}(A:B)-\max_{\\{\Pi_{k}^{B}\\}}\mathcal{J}_{\\{\Pi_{k}^{B}\\}}(A|B)$
$\displaystyle=\mathcal{I}(A:B)-S(\rho_{A})+\min_{\\{\Pi_{k}^{B}\\}}\sum_{k}p_{k}^{B}S(\rho_{A}|k)$
$\displaystyle=S(\rho_{B})-S(\rho_{AB})+\min_{\\{\Pi_{k}^{B}\\}}\sum_{k}p_{k}^{B}S(\rho_{A}|k)$
(27)

Therefore,

$\mathcal{D}(A|B)=\min_{\\{\Pi_{k}^{B}\\}}\mathcal{D}_{\Pi}(A|B)$ (28)

It is known that outcomes expected from local measurements do not completely
recover all correlations, and therefore, the discord is a non-zero quantity
for such post-measurement states. This minimising procedure is highly
nontrivial because infinitely many POVMs exist that differ by a simple change
of basis. In addition, finding a measurement that can be made that could
maximise $\mathcal{J}(A|B)$ is not straightforward or practical for anything
but a simple system. [22, 28, 29]

With this in mind, we define the classical mutual information by taking just
the diagonal elements of the density matrix.

$\displaystyle\tilde{\rho}_{k}={\rm diag}(\rho_{k})$ (29)

and computing the mutual information

$\displaystyle{\cal
I}(\tilde{A}:\tilde{B})=S(\tilde{\rho}_{A})+S(\tilde{\rho}_{B})-S(\tilde{\rho}_{AB})$
(30)

This definition is physically appealing since it reduces to the mutual
information for a purely mixed state and is independent of basis due to the
invariance of the trace under the unitary transformation of the density
matrix. Furthermore, ${\cal I}(\tilde{A}:\tilde{B})\leq{\cal I}(A:B)$,
implying that any difference between the two is entirely due to quantum nature
of correlations. With this, we define $D(A:B)$ as the degree of quantumness
between subsystems A and B,

$\displaystyle D(A:B)=\cal{I}(A:B)-\cal{I}(\tilde{A}:\tilde{B}).$ (31)

In testing the approach, this generally under-estimates the quantum
correlation and, hence, provides a lower bound on the quantum-ness. On the
other hand, $D(A:B)$ is strictly bound from above by ${\cal I}(A:B)$.

In Fig. 4, we present mutual information and the corresponding degree of
quantumness across various parametric regimes. Initially, in the scenario
where the two spins lack direct exchange coupling (Fig.4(b)), it is observed
that quantum contributions predominantly account for nearly all the
correlation between the two spins. This is unexpected, given that the
interaction between the two spins is mediated solely through the shared
environment. Evidently, in the context of uncorrelated environments, there is
an absence of shared information. Conversely, in correlated environments,
quantum correlations predominantly contribute to the mutual information,
whereas in the anti-correlated scenario, classical contributions assume a
dominant role.

##  IV Discussion

Synchronization has been a rich and diverse topic in both quantum and
classically described systems for well over 350 years. In most studies of this
sort, the two oscillators are linked in some direct way via mechanical
coupling as in the case of a classical pendula or via exchange of quanta.
Here, we show that indirect coupling mediated by contact with a common
stochastic environment can also lead to quantum entanglement and
synchronization even in the absence of direct coupling. While coupling to a
shared resource leads to correlation, that correlation may be quantum or
classical in origin. We propose that this can be quantified by subtracting the
maximal amount of classical information shared between the two components from
their total mutual entropy, ${\cal I}(A:B)$, and thus provides a lower bound
for the amount of quantum information shared between the two components. Our
analysis indicates that correlated noise leads to a high-degree of “quantum-
ness” in steady-state system, anti-correlated noise tends to result in more
classical-like correlation. Furthermore, when the direct coupling between the
spins is eliminated, nearly 100% of the correlation between the two qubits can
be attributed to quantum contributions.

###### Acknowledgements.

ERB acknowledges funding from the National Science Foundation (CHE-2404788)
and the Robert A. Welch Foundation (E-1337). HL and CSA acknowledge funding
from the Government of Canada (Canada Excellence Research Chair
CERC-2022-00055) and the Courtois Institute, Faculté des arts et des sciences,
Université de Montréal (Chaire de Recherche de l’Institut Courtois). The
authors also acknowledge Prof. Phillip Shushkov (Indiana University) for
discussions at a recent Telluride (TSRC) workshop that led to this
investigation.

## Data Availability Statement

Data files and source codes used in this work are available on GitHub at
https://github.com/BittnerTheoryGroup/Quantum_Synchro-131.

## Author Contribution Statement

ERB conceived of the idea. HL and ERB developed the correlated noise model.
ERB and BT developed the theory and performed the computations. ERB supervised
the findings of this work. All authors discussed the results and contributed
to the final manuscript.

## Competing Interests Statement

The authors have no competing interests in this work.

## Supplemental Information

Movies showing phase locking behavior for model qubit dimers are available on
GitHub (https://github.com/BittnerTheoryGroup/Quantum_Synchro-131).

## References

  * Huygens [1669] C. Huygens, “Instructions concerning the use of pendulum-watches, for finding the longitude at sea,” Philosophical Transactions of the Royal Society of London 4, 937–953 (1669).
  * Huygens [1673] C. Huygens,  _Horologium oscillatorium sive de motu pendulorum ad horologia aptato demonstrationes geometricae_ (Apud F. Muguet, 1673).
  * Ellicott [1739a] J. Ellicott, “Vi. Further observations and experiments concerning the two clocks above-mentioned, by the same,” Philosophical Transactions of the Royal Society of London 41, 128–135 (1739a), https://royalsocietypublishing.org/doi/pdf/10.1098/rstl.1739.0015 .
  * Ellicott [1739b] J. Ellicott, “V. An account of the influence which two pendulum clocks were observed to have upon each other,” Philosophical Transactions of the Royal Society of London 41, 126–128 (1739b), https://royalsocietypublishing.org/doi/pdf/10.1098/rstl.1739.0014 .
  * Airy [1830] S. G. B. Airy, “On the disturbances of pendulums and balances and on the theory of escapements,” (Cambridge Philosophical Society, 1830).
  * Newton [1833] I. Newton,  _Philosophiae naturalis principia mathematica_ , Vol. 1 (G. Brookman, 1833).
  * Korteweg [1906] D. Korteweg, “Les horloges sympathiques de huygens,” Archives Neerlandaises, Serie II 11, 273–295 (1906).
  * Huygens [1916] C. Huygens,  _Oeuvres complètes de Christiaan Huygens_ , Vol. 13 (M. Nijhoff, 1916).
  * Strogatz and Stewart [1993] S. H. Strogatz and I. Stewart, “Coupled oscillators and biological synchronization,” Scientific american 269, 102–109 (1993).
  * Pikovsky, Rosenblum, and Kurths [2001] A. Pikovsky, M. Rosenblum, and J. Kurths, “Synchronization,” Cambridge university press 12 (2001).
  * Bennett  _et al._ [2002] M. Bennett, M. F. Schatz, H. Rockwood, and K. Wiesenfeld, “Huygens’s clocks,” Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 458, 563–579 (2002), https://royalsocietypublishing.org/doi/pdf/10.1098/rspa.2001.0888 .
  * Goychuk  _et al._ [2006] I. Goychuk, J. Casado-Pascual, M. Morillo, J. Lehmann, and P. Hänggi, “Quantum stochastic synchronization,” Phys. Rev. Lett. 97, 210601 (2006).
  * Oliveira and Melo [2015] H. M. Oliveira and L. V. Melo, “Huygens synchronization of two clocks,” Scientific Reports 5, 11548 (2015).
  * Peña Ramirez  _et al._ [2016] J. Peña Ramirez, L. A. Olvera, H. Nijmeijer, and J. Alvarez, “The sympathy of two pendulum clocks: beyond huygens’ observations,” Scientific reports 6, 23580 (2016).
  * Engel  _et al._ [2007] G. S. Engel, T. R. Calhoun, E. L. Read, T.-K. Ahn, T. Mančal, Y.-C. Cheng, R. E. Blankenship, and G. R. Fleming, “Evidence for wavelike energy transfer through quantum coherence in photosynthetic systems,” Nature 446, 782–786 (2007).
  * Panitchayangkoon  _et al._ [2010] G. Panitchayangkoon, D. Hayes, K. A. Fransted, J. R. Caram, E. Harel, J. Wen, R. E. Blankenship, and G. S. Engel, “Long-lived quantum coherence in photosynthetic complexes at physiological temperature,” Proceedings of the National Academy of Sciences 107, 12766–12770 (2010).
  * Chenu and Scholes [2015] A. Chenu and G. D. Scholes, “Coherence in energy transfer and photosynthesis,” Annual review of physical chemistry 66, 69–96 (2015).
  * Scholes  _et al._ [2017] G. D. Scholes, G. R. Fleming, L. X. Chen, A. Aspuru-Guzik, A. Buchleitner, D. F. Coker, G. S. Engel, R. Van Grondelle, A. Ishizaki, D. M. Jonas, _et al._ , “Using coherence to enhance function in chemical and biophysical systems,” Nature 543, 647–656 (2017).
  * Zhu  _et al._ [2024] R. Zhu, W. Li, Z. Zhen, J. Zou, G. Liao, J. Wang, Z. Wang, H. Chen, S. Qin, and Y. Weng, “Quantum phase synchronization via exciton-vibrational energy dissipation sustains long-lived coherence in photosynthetic antennas,” Nature Communications 15, 3171 (2024).
  * Du  _et al._ [2021] M. Du, M. Qin, H. Cui, C. Wang, Y. Xu, X. Ma, and X. Yi, “Role of spatially correlated fluctuations in photosynthetic excitation energy transfer with an equilibrium and a nonequilibrium initial bath,” The Journal of Physical Chemistry B 125, 6417–6430 (2021).
  * Brox, Bergli, and Galperin [2012] H. Brox, J. Bergli, and Y. M. Galperin, “Bloch-sphere approach to correlated noise in coupled qubits,” Journal of Physics A: Mathematical and Theoretical 45, 455302 (2012).
  * Eneriz  _et al._ [2019] H. Eneriz, D. Z. Rossatto, F. A. Cárdenas-López, E. Solano, and M. Sanz, “Degree of quantumness in quantum synchronization,” Scientific Reports 9, 19933 (2019).
  * Bittner  _et al._ [2024] E. R. Bittner, H. Li, S. A. Shah, C. Silva-Acuña, and A. Piryatinski, “Correlated noise enhancement of coherence and fidelity in coupled qubits,” Philosophical Magazine 0, 1–17 (2024), https://doi.org/10.1080/14786435.2024.2341011 .
  * Johansson, Nation, and Nori [2012] J. Johansson, P. Nation, and F. Nori, “Qutip: An open-source python framework for the dynamics of open quantum systems,” Computer Physics Communications 183, 1760–1772 (2012).
  * Johansson, Nation, and Nori [2013] J. Johansson, P. Nation, and F. Nori, “Qutip 2: A python framework for the dynamics of open quantum systems,” Computer Physics Communications 184, 1234–1240 (2013).
  * Henderson and Vedral2 [2001] L. Henderson and V. Vedral2, “Classical, quantum and total correlations,” JOURNAL OF PHYSICS A: MATHEMATICAL AND GENERAL (2001).
  * Ollivier and Zurek [2001] H. Ollivier and W. H. Zurek, “Quantum discord: A measure of the quantumness of correlations,” Phys. Rev. Lett. 88, 017901 (2001).
  * Mohamed and Eleuch [2020] A.-B. A. Mohamed and H. Eleuch, “Quasi-probability information in a coupled two-qubit system interacting non-linearly with a coherent cavity under intrinsic decoherence,” Scientific Reports 10, 13240 (2020).
  * Foti  _et al._ [2021] C. Foti, A. Coppo, G. Barni, A. Cuccoli, and P. Verrucchi, “Time and classical equations of motion from quantum entanglement via the Page and Wootters mechanism with generalized coherent states,” Nature Communications 12, 1787 (2021).
  * Galve, Giorgi, and Zambrini [2011] F. Galve, G. L. Giorgi, and R. Zambrini, “Orthogonal measurements are almost sufficient for quantum discord of two qubits,” Europhysics Letters 96, 40005 (2011).
  * Hamieh, Kobes, and Zaraket [2004] S. Hamieh, R. Kobes, and H. Zaraket, “Positive-operator-valued measure optimization of classical correlations,” Phys. Rev. A 70, 052325 (2004).
  * D’Ariano, Presti, and Perinotti [2005] G. M. D’Ariano, P. L. Presti, and P. Perinotti, “Classical randomness in quantum measurements,” Journal of Physics A: Mathematical and General 38, 5979 (2005).

##  Appendix A Comments on Approximating the Discord.

Figure 5: The quantum discord for a thousand random rank-2 matrices (a) and
the tight upper bound on the discord for a thousand random density matrices
(including rank-2, 3 and 4)(b) for a two-qubit system can be exactly
calculated using the method described by Galve et. al[30]. The degree of
quantumness (c) calculated using the approximation used in Section III
underestimates the amount of quantum discord.

Despite having an exact expression to calculate the amount of true quantum
correlations that exist between the subsystems, this calculation is non-
trivial because it requires us to minimise over all possible sets of allowed
measurements on one of the subsystems. This optimization is challenging
because, given a general random unitary matrix, we can rotate this set of
measurements to a completely new and equally valid set of measurements.

For a system of qubits, several arguments can significantly reduce the
cardinality of the set of allowed measurements and the complexity of the
problem. The first result of Hamieh et al. [31] shows that the conditional
entropy $S(\rho_{A}|k)$ is a concave function over the convex set of POVMs;
hence, the set of allowed measurements or POVMs which correctly minimise the
classical part of the correlations are extremal, which means that they cannot
be constructed via a convex combination of other POVMs. Following this,
D’Ariano et al. [32] showed that these extremal POVMs for qubits are rank-1
and the allowed set can only have two, three or four elements. Finally, Galve
et al. [30] showed that for a system of two qubits, the two-element set of
POVMs corresponds to orthogonal measurements. This is based on the fact that a
rank-1 POVM of two elements must satisfy positivity, $\Pi_{1,2}>0$, and
normalization such that $\Pi_{1}+\Pi_{2}=\mathbb{I}$. They invented a method
to calculate the discord and provided numerical evidence that this calculation
is exact (with an error around $10^{-4}$) for a system of two qubits described
by a rank-2 density matrix. This method also provides a tight upper bound when
the system is described by rank-3 and rank-4 density matrices. We show
evidence for this in Fig. 5 by calculating the discord for random density
matrices describing a two-qubit system.

For rank-2 density matrices in Fig. 5(a), these matrices have higher purity
compared to general random matrices; hence, we see that the majority have
discord between 0.75 and 1. Note that the quantum discord for any maximally
entangled pure state is one and zero if the state is separable. For random
density matrices (including rank-2, 3, and 4) in Fig. 5(b), when a system
evolves under quantum dynamics and is also subject to some noisy environment,
the output density matrix will most likely be mixed. Therefore, we see a tight
upper bound as the quantum discord for maximally mixed density matrices is
zero. It is important to note here that these methods were developed for
systems in relatively small state spaces.

The search for the most general method for identifying the optimal allowed
measurement remains an active area of research. This motivated us to find an
approximate and numerically efficient method to calculate the degree of
quantumness that exists in the correlations of our final density matrix,
described in Section III. We argue that by calculating the difference between
the mutual information of our original density matrix $\rho_{k}$ and the
mutual information of the modified density matrix
$\tilde{\rho}_{k}=\mathrm{diag}(\rho_{k})$, we can estimate the degree of
quantum correlation in our quantum system. See Fig. 5(c). The intuition behind
this approximation is that the diagonal elements of a density matrix contain
all the classical information, i.e., the information that we get after local
measurements. So when we ignore the off-diagonal elements, we consider only
the classical part of the distribution and remove it from the total mutual
information. This approximation provides an upper bound on the amount of
classical correlations and a lower bound on the quantum correlations. One
advantage of this approximation is that it can be extended to higher-spin
systems and can be used as a quick estimation to extract the amount of quantum
correlations.