# Soft Biomimetic Optical Tactile Sensing
with the TacTip: A Review

Nathan F. Lepora This work was supported in part by Leverhulme Research
Leadership Award on ‘A biomimetic forebrain for robot touch’ (RL-2016-39). N.
Lepora is with the Department of Engineering Mathematics, Faculty of
Engineering, University of Bristol and Bristol Robotics Laboratory, Bristol,
UK. (e-mail: n.lepora@bristol.ac.uk).

###### Abstract

Reproducing the capabilities of the human sense of touch in machines is an
important step in enabling robot manipulation to have the ease of human
dexterity. A combination of robotic technologies will be needed, including
soft robotics, biomimetics and the high-resolution sensing offered by optical
tactile sensors. This combination is considered here as a SoftBOT (Soft
Biomimetic Optical Tactile) sensor. This article reviews the BRL TacTip as a
prototypical example of such a sensor. Topics include the relation between
artificial skin morphology and the transduction principles of human touch, the
nature and benefits of tactile shear sensing, 3D printing for fabrication and
integration into robot hands, the application of AI to tactile perception and
control, and the recent step-change in capabilities due to deep learning. This
review consolidates those advances from the past decade to indicate a path for
robots to reach human-like dexterity.

Force and tactile sensing, haptics, manipulation, robot dexterity, TacTip
sensor

##  1 Introduction

As humans, we take our sense of touch for granted as we mostly use it in a
subconscious way. Visual and auditory events grab our attention [1] while our
tactile sense continues unabated during everyday manual tasks and chores.
However, touch is arguably the sense that most differentiates humans from
other animals because the dexterous use of our hands relies on the intelligent
use of tactile perception. With our hands we have invented technology, the
hallmark of our species, spanning from the archaeological record of archaic
hominids to the modern era of advanced devices such as robots.

Reproducing the capabilities of the human tactile sense in machines is an
important step in enabling robotic hands to reach the dexterity of the human
hand. As argued compellingly by historian Yuval Noah Harrari in his book “Home
Deus: A Brief History of Tomorrow”, robot dexterity will have a profound
impact on human society as machines become commonplace for physical labour
[2]. This revolution in robot dexterity is beginning in the logistics
industry, but promises broader impacts in manufacturing, health, construction,
food production, recycling, conservation and renewable energy.

A range of robotic technologies will be needed to reach human-like dexterity
in machines. Soft robotics is needed for safe devices that adapt according to
the intelligence embodied in their materials and design. Biomimetics provides
the only known example of a general-purpose manipulation device: the human
hand and our manual intelligence to use it effectively. Intelligent
interaction requires a rich source of contact information, as offered by high-
resolution optical tactile sensors such as the MIT GelSight [3, 4] and BRL
TacTip [5, 6] with deep learning methods such as convolutional neural
networks.

Here, this combination of technologies is termed SoftBOT sensing, for Soft
Biomimetic Optical Tactile Sensing, encompassing SoftBOT sensors, hands and
robotic systems. The BRL TacTip uniquely combines all of these features as a
prototypical example of a SoftBOT sensor, and for this reason has led to a
large body of research on robot dexterity. Therefore, this article reviews
tactile sensing with the TacTip to consolidate the advances with this
technology over the past decade and so indicate a path to human-like robot
dexterity.

##  2 Soft, Biomimetic and Optical Tactile Sensing

To motivate this article, we define the terms ‘soft’, ‘biomimetic’ and
‘optical’ in the context of tactile sensing, bearing in mind that a tactile
sensor is a device that transduces deformation of a sensing surface into a
signal containing information about the physical contact. As emphasised in
past surveys [7, 8, 9, 10], this information can be used to infer properties
of the contacting object and its interaction with the tactile sensor.

Soft tactile sensors rely on soft materials to elicit information from
physical contact. While all tactile sensors use a deformable surface to sense,
they range in material construction from flexible printed e-skins [11] to
sensitive materials embedded within elastomeric substrates [12] to compliant
layers over electronic circuits that measure compression [13]. Softness can
have many benefits in the design and function of tactile sensors. For example,
it is important that e-skins are flexible so they can conform their sensing
surface to 3D objects and cover large areas of hard or soft actuated robots
[14].

Soft biomimetic tactile sensors are soft tactile sensors based on principles
distilled from the study of biological systems [15, 16]. There is a
distinction between ‘blind copying’, such as merely shaping a tactile sensor
like a human fingertip, and true biomimicry, such as transferring the
transduction principles of human skin into the design of an artificial sensor.
Soft robots are often inspired by soft-bodied animals [17], so it is expected
that biomimetic tactile sensors are usually soft. There are, however, many
ways in which biological principles can motivate soft designs. One example is
the inspiration for the Syntouch BioTac [18] from the multi-modality of human
touch to pressure, vibration and temperature. Another example is a biomimetic
e-skin that measures local shear and normal forces by reproducing the hill-
like structure of the dermal-epidermal boundary in human skin [19].

Soft optical tactile sensors are soft tactile sensors that use light to view
the deformation. Optical tactile sensors are often soft because they rely on
viewing physical changes to the sensing surface, usually from inside the
sensor with an internal light source. Optical touch sensing dates back to the
mid-1960s [20], with the earliest example relaying the view of a photoelastic
skin via optic fibres to a TV camera whose signal was viewed remotely to
teleoperate a robotic gripper [21].

For soft camera-based tactile sensors, there are many proposals for imaging
surface deformation, with categories including [20, 22]: (i) soft marker-based
optical tactile sensors that typically measure the (lateral) shear deformation
of the sensing surface, such as the GelForce (2004) with its markers embedded
in a supporting gel [23, 24]; and (ii) soft reflection-based optical tactile
sensors that typically measure the (normal) indentation of the sensing
surface, such as the GelSight (2009) which uses the surface shading from
multiple internal light sources to infer a depth map via photometric stereo
[3]. Combinations are also considered, such as by printing markers on the
GelSight skin [25] and by mixing coloured markers to indicate depth [26].
Recently, this area of research has been boosted by the remarkable advances in
image processing using convolutional neural networks [27], which has led to
many refinements of these designs [28, 29, 30, 31, 32].

Soft biomimetic optical tactile sensors combine optical imaging with
biological principles underlying the sense of touch in animals. A technology
that offers both optical and biomimetic tactile sensing is the BRL TacTip
(centre of figure in graphical abstract), proposed in 2009 as ‘a tactile
sensor based on biologically-inspired edge encoding’ [5]. This article will
review the biomimetic and optical principles of the TacTip as a prototypical
example of a SoftBOT sensor.

##  3 Soft Biomimetic Optical Tactile Sensing:
The TacTip

The TacTip is a soft biomimetic optical tactile sensor that mimics the
structure and function of the human fingertip (Figure LABEL:fig:1). Human skin
has an intricate morphology of layers, microstructures and sensory receptors
that underlie its many functions, from protecting the body to sensing surface
contact, temperature and injury [33, 34, 35]. Like most other mammals, our
skin is of two general types: hairy over much of our body and glabrous
(hairless) on the sensitive underside of our fingers and toes, palms, soles of
our feet, external genitalia, areolae, tongue, inner cheeks and lips.

The shallow layers of glabrous skin are structured to sense touch via the
deformation of its surface. These upper layers comprise an outer epidermis
over an inner papillar dermis, which interdigitate in a mesh of dermal
papillae and epidermal intermediate ridges (Figures LABEL:fig:1a,c). This 3D
structure transmits shear and normal deformation of the skin surface into the
displacement of sensory mechanoreceptors within the dermal-epidermal
interface. Thus, the dermal papillae and intermediate ridges are considered to
act as a mechanical amplifier of skin deformation into mechanoreceptor
activity [46].

The TacTip design is based on the shallow layers of glabrous skin [5, 6]. It
has an outer biomimetic epidermis made from a rubber-like material over a soft
inner biomimetic dermis made from an elastomer gel (Figures LABEL:fig:1b,d).
These two materials interdigitate in a mesh of biomimetic intermediate ridges
and dermal papillae, comprising stiff inner nodular pins that extend under the
biomimetic epidermis into the soft gel. This structure amplifies surface
deformation of the skin into lateral movement of visible markers on the pin
tips.

In human skin, two types of sensory mechanoreceptor are embedded within the
dermal-epidermal interface: Merkel cells on the intermediate ridges and
Meissner corpuscles within the dermal papillae (Figure LABEL:fig:1c). Merkel
cells respond to skin deformation, activating slowly-adapting (SA) neurons
that fire during sustained contact, such as when perceiving shapes or edges.
Meissner corpuscles respond to skin movement, activating rapidly-adapting (RA)
neurons that fire during changes of contact, such as when perceiving surface
slip and flutter. These sensory receptors work together to convey information
about the tangible aspects of touch [47].

The biomimetic counterparts to these sensory mechanoreceptors are the markers
on the pin tips, which can be imaged through a transparent gel forming the
dermis (Table LABEL:table:1). The marker displacements from rest are analogous
to the SA activity from the Merkel cells because they indicate sustained
deformation of the sensing surface [42] to enable accurate shape and edge
recognition [36, 48]. Likewise, the counterpart of RA mechanoreceptor activity
from the Meissner corpuscles are the marker velocities, because these comprise
a transient signal that indicates skin motion. Accordingly, the marker
velocities enable accurate slip detection [37, 38]. Together, these two
signals – the marker displacements and velocities – give information about the
sustained deformation and transient motion of the sensing surface.

The deeper layers of glabrous skin also have biomimetic counterparts in the
TacTip design (Table LABEL:table:1). Skin maintains its compliance from the
deep reticular dermis and subcutaneous fat (Figure LABEL:fig:1a), whose
biomimetic counterpart is the deeper elastomer gel held in place by a
transparent acrylic cap. Within the deep skin layers, a population of RA
mechanoreceptors called Pacinian corpuscles responds to high-frequency
vibration (peak sensitivity $\sim$250 Hz). A partial mimicry of their function
can be attained by using the TacTip with a high frame-rate (kHz) camera [39,
40]; however, questions remain about whether this approach to vibration
sensing is effective or even biomimetic, since it images fast pin movement
rather than vibration in the deeper gel. In our view, a biomimetic counterpart
of the vibration sense would be to embed a pressure sensor in the gel of the
TacTip, like the vibration modality of the BioTac [18]. Other tactile sensing
modalities can also be included, such as temperature sensing by using a
thermoactive smart material for the outer TacTip skin, which is imaged as a
background to the markers [41].

A consequence of the biomimetic design of the TacTip is that other properties
of human perception emerge. An important aspect of human tactile perception is
hyperacuity: a capacity to discriminate extended spatial features to a sub-
millimetre acuity that is finer than the millimetre-scale spacing between
mechanoreceptors [49]. The TacTip also exhibits tactile hyperacuity, with a
sub-millimetre capacity for spatial discrimination that is finer than its
millimetre-scale pin spacing [42]. Fundamentally, the hyperacuity arises
because both the biological and artificial tactile senses are comprised of
arrays of overlapping, broad but sensitive receptive fields. This structure
enables spatial interpolation over neighbouring receptors, which is analogous
to a well-known technique in optical imaging known as super-resolution [50].

Perhaps surprisingly, the role of the human fingerprint in the sense of touch
is still being investigated [51] after two centuries of study. A 3D-printed
fingerprint can be reproduced in the TacTip as raised bumps [39, 43] or
concentric raised rings over the papillae [52]. Benefits of a biomimetic
fingerprint include increased sensitivity to texture [39] and spatial
localisation [43]. A ringed biomimetic fingerprint can also induce incipient
slip [52], where a local region of skin slips in detectable way before the
grip slips [37, 38].

Finally, another direction to take biomimetic optical tactile sensing is to
adopt neuromorphic transduction using trains of ‘spike’ events [53, 54].
Biological nervous systems use event-based signalling to represent sensory
information in an efficient and temporally-precise way, which neuromorphic
engineering aims to reproduce artificially [55]. A benefit of optical tactile
sensing with the TacTip is that an event-based camera can be used to implement
a transduction mechanism that has both a biomimetic skin design and is
neuromorphic from the camera [44, 45]. Overall, neuromorphic computation
offers a new paradigm for robot touch that complements the capabilities of
soft biomimetic optical tactile sensors.

(a) Single tactile image processed into shear-strain field (coloured by
strain)
---
Tactile image & setup |  $x$-shears |  $y$-shears | Shear magnitude | Voronoi areas
|  |  |  |
|  |  |  |

(b) Biomimetic signal transduction
---
Skin indentation | Levering of pins |  Indentation and shear strain
|  |

(c) Time series of tactile images processed into shear strains (coloured by
marker)
---
Sensor position |  $x$-shear strains |  $y$-shear strains | Shear-strains | Voronoi areas
|  |  |  |
Sensor velocity |  $x$-shear-velocities |  $y$-shear-velocities | Shear-velocities | Voronoi area changes
|  |  |  |

Figure 1: Shape sensing with the BRL TacTip. (a) Processing of a tactile image
of an edge contact (left) into 2D $(x,y)$-shear strains, from which shear
magnitude and Voronoi area change are extracted (red/blue shows
positive/negative change; scale in panel b). (b) Biomimetic signal
transduction of skin indentation into marker displacement as a shear-strain
field. (c) Time series of quantities in panel a and their derivatives for a
contact onto then off an edged stimulus (colour labels marker). This
biomimetic representation using shear-strain is highly informative about the
contact.

##  4 Tactile Shear Sensing

###  4.1 Tactile sensing of normal strain and shear strain

A central aspect of tactile sensing with the TacTip is that it measures
contact via shear. Skin indentation tilts and moves the nodular pins, causing
lateral displacements of markers on the pin tips (Figure 1b, right). The
biomimetic structure of stiff nodular pins interdigitating with a soft
elastomeric gel causes mechanical transduction of normal-strain on the skin
surface into a measurable shear-strain underneath the skin.

This design also makes the TacTip highly sensitive to shear-strain of the skin
surface. The two sources of skin movement, normal and shear strain, produce
distinct patterns of marker displacement. Sliding or slip gives a more uniform
shear-strain field, whereas normal indentation can produce characteristic
dipole or multipole patterns (Figure 1a). In practise, both normal- and shear-
strain are usually present, which can introduce subtleties when inferring
contact shape information from the TacTip, as discussed later.

Therefore, the TacTip operates in a distinct manner from the common types of
taxel-based artificial tactile sensors that measure normal strain, such as the
BioTac [18] and the iCub fingertip [13]. The TacTip is also distinct from
other designs of optical tactile sensor. Reflection-based optical tactile
sensors such as the GelSight [3] reconstruct normal strain directly from light
shading. Some optical tactile sensors also use markers coated underneath the
skin [56] (including recent GelSight versions [25]), but those measure just
the shear strain of the skin conforming to a surface. In contrast, the TacTip
markers inform about both the normal and shear strain due to contact.

Several optical tactile sensors use markers floating in the elastomer
underneath the outer skin, such as the GelForce [23, 24] and ChromaTouch [26].
These tactile sensors differ from the TacTip by not having stiff nodular pins
connecting the deformation of the skin surface to the markers, but instead
rely on the forces transmitted through the soft elastomer. The GelForce uses
two layers of coloured spherical markers in a transparent elastomer to infer
force vectors within the gel [23, 24], and the ChromaTouch uses two layers of
partially-transparent coloured markers whose movement and colour-matching
inform about the sensor deformation [26].

All these designs have their pros and cons as artificial tactile sensors. The
overall point to emphasise is that the TacTip senses contact differently from
other tactile sensors and uses a biomimetic mechanism for this transduction.

###  4.2 The shear-sensing hypothesis

Platkiewicz, Lipson and Hayward have emphasised the importance of internal
shear strain for sensing external contact [57] with their shear-sensing
hypothesis: ‘we propose that shape-related tactile information is more
suitably recovered from shear strain than normal strain’.

Their reasoning is based on the contact mechanics of touch: ‘the pressure
distribution at the surface of a tactile sensor cannot be acquired directly
and must be inferred from the [strain] field induced by the touched object in
the sensor medium.’ In vision, it is well known that edge detection is
fundamental to processing shape, with the convolutions of deep neural networks
analogous to edge-detecting neurons in visual cortex. For touch, there are two
ways to detect edges: indirectly from gradients of the normal-strain field or
directly from zero crossings (sign changes) of the internal shear-strain field
(Figure 1b). Zero crossings are a more robust measure of shape from touch,
because they are affected less by distortion from the mechanics of skin and
require no computation of signal gradients [57], motivating the use of shear
strain.

This reasoning can be grounded in a biomechanical model of skin [57] that
approximates the shear-strain field $\gamma$ at depth $z$ and horizontal
position $x$ in a uniform elastomer (Young’s modulus $E$) due to a pressure
field $p(x)$ at the surface:

$\gamma(x,z)\approx\frac{z}{E/3}\frac{{\rm d}}{{\rm
d}x}p_{\epsilon}(x,z),\hskip
10.00002ptp_{\epsilon}(x,z)=p(x)*\phi_{\epsilon}(x,z),$ (1)

where $p_{\epsilon}(x,z)$ represents that the pressure field is mechanically
blurred by depth (modelled by convolving a Gaussian $\phi_{\epsilon}(x)$ of
width $\epsilon(z)$ that increases with depth). Overall, the shear-strain
field follows the gradient of the (blurred) pressure profile and increases
linearly with depth from zero at the surface – i.e. it has mechanically
calculated the signal gradient.

However, the skin of the TacTip is not a uniform elastic material, but has an
inner structure of stiff nodular pins interdigitating with soft elastomeric
gel. Hence, this model (1) is more suited to optical tactile sensors with
floating markers such as the GelForce and ChromaTouch [23, 24]

A biomechanical model of the TacTip and the dermal papillae should be based
instead on the levering of the nodular pins caused by the normal skin
indentation $\delta(x)$ due to surface pressure. For a pin along the normal to
the skin surface (Figure 1b), its tip moves horizontally with shear strain:

$\gamma(x,z)=z\sin\arctan\left(\frac{{\rm d}}{{\rm d}x}\delta(x)\right)\approx
z\frac{{\rm d}}{{\rm d}x}\delta(x),$ (2)

for pins of length $z$ forming the hypotenuse of a right-angled triangle with
shear strain $\gamma$ along the (horizontal) adjacent side. The gradient
$\frac{{\rm d}}{{\rm d}x}\delta(x)$ is tangential to the skin surface and
normal to the pins. The shear-strain field is linear in the surface gradient
when the small angle approximation $\frac{{\rm d}}{{\rm d}x}\delta(x)\ll 1$
holds; i.e. it also mechanically calculates the signal gradient.

It is interesting that these two biomechanical models lead to very similar
equations: in both, the shear-strain field $\gamma(x,z)$ is proportional to
depth $z$ and the gradient of a quantity (pressure or indentation) at the skin
surface. Both models amplify the transduction with depth $z$, but the pin
structure is beneficial in not being mechanically blurred $\epsilon(z)$
compared to the elastomer. Also, the shear strain from the stiff pins should
be larger than that of an elastic medium.

Overall, the reasoning behind the shear-sensing hypothesis appears consistent
with both the uniform elastomeric model considered originally (Equation 1) and
the pin model introduced here for the TacTip (Equation 2). The TacTip is
unique amongst related tactile sensors by sensing touch purely from shear via
a mechanism of stiff pins to transduce skin indentation into shear strain.
Therefore, the shear-sensing hypothesis appears to encapsulate the operation
of the TacTip as an embodiment of the theory.

##  5 Period I (2009-2014): Initial Development of the TacTip

Most of the research and development into soft biomimetic optical tactile
sensing has been within the Bristol Robotics Laboratory (BRL) in the U.K.
However, the focus has moved from soft to medical to tactile robotics, leading
to renewal of the underlying technology and research direction. Meanwhile, the
field of tactile robotics has risen in prominence since the TacTip was
introduced in 2009 as the aim of the field to enable artificial manipulation
has become closer to reality. Recently, the revolution in deep learning [27]
has raised the prominence of optical tactile sensors because of their
compatibility with convolutional neural networks, promising an entirely new
level of robot performance approaching human dexterity.

The first period of development (2009-2014) encompassed the original design
for an optical tactile sensor based on biologically-inspired encoding [5, 58].
The name TACTIP was coined [59, 39] as a contraction of ‘tactile fingertip’
after an EU-funded project that supported some of the research. The name has
stuck but is now written TacTip similarly to other well-known tactile sensors
(e.g. BioTac or GelSight).

The inspiration for the TacTip was the tactile contact lens [60, 61] that was
developed in a Toyota-funded research laboratory to help humans feel
manufacturing defects in automobile production. The tactile contact lens
magnifies the feeling of surface deformation via a slippery flexible base-
plate attached to an array of rigid pins held against a human fingertip
(Figure 2). As the base slides over a small bump or crack, the pins act as
levers to amplify the normal indentation in the base into a shear strain that
a human can feel [62]. Overall, the tactile contact lens magnifies
imperceptible surface features into larger shear displacements by about a
factor of four, so the contact feature can be discriminated more easily.

(a) Mechanics of the Tactile Contact Lens
---
|
(b) Mechanics of the Human Fingertip and TacTip
|
Figure 2: Analogous operation of the Tactile Contact Lens and TacTip. Both
devices magnify and transduce surface indentation into shear strain, using the
levering of an array of pins.

The TacTip uses this same mechanical principle to magnify indentations on a
flexible outer skin into lateral movement of markers on the pin tips [5]. Both
the TacTip and tactile contact lens are biomimetic, because this mechanism for
magnifying surface contact into shear strain seems to have evolved in human
tactile skin [46]: the human sense of touch relies on the mechanical structure
of dermal papillae and epidermal intermediate ridges, as explained in Section
3.

A central question when using any soft optical tactile sensor is how to
interpret the tactile image from the camera to infer aspects of the skin
deformation (Figure 1). Initial studies with the TacTip processed the images
into velocity vector fields of marker motion, taking inspiration from the
Meissner corpuscles embedded in the dermal papillae which are motion sensitive
[5, 58]. Surface edge features such as the contours of a coin were clearly
visible in the marker velocity field, motivating the initial research on
biologically-inspired edge encoding. Other early work used image filtering
(dilation/erosion) to transform a tactile image into a visualisation of marker
density showing the orientation of a contacted edge [63, 64]. Other simple
methods to infer contact features were also explored, such as binning the
tactile image into a discrete array, which was applied to lump detection and
localization in medical haptics [65, 66, 67].

Refinements of the original TacTip design were considered for an EU-funded
project that sought to address the need for tactile robot manipulators and
probes for industrial and healthcare applications. The tactile tip was
miniaturized to a 20 mm-diameter dome (from 40 mm) for integration as a
fingertip of an anthropomorphic robot hand [59] and a high frame rate camera
used to investigate texture perception [39]; however, in both cases the camera
was separate rather than integrated into the design. These early studies laid
the foundation for later advances that used 3D printing to improve the TacTip.

Sensor | Year | Design | Camera
---|---|---|---
TACTIP [5] | 2009 |  | cylindrical body (40 mm-dia. spacer)
---
molded hemispherical soft tip (40 mm dia., 532 pins)
LifeCam VX webcam, 480p 30fps, $f$$\approx$50 mm
Mini TACTIP [59] | 2012 |  | rigid 3DP mount on digit of Elumotion robotic hand
---
molded hemispherical soft tip (20 mm dia., 276 pins)
Not integrated
Open TacTip [68] | 2016 |  | rigid 3DP body (161 mm) modified for modular tip
---
molded hemispherical soft tip (40 mm dia., 532 pins)
LifeCam Cinema webcam, 720p 30fps, $f$$\approx$50 mm
TacTip [6] | 2016 |  | rigid 3DP body (85 mm) and modular tip
---
hemispherical 3DP soft tip (40 mm dia., 127 pins)
LifeCam Cinema board (disassembled camera)
TacTip-M2 [69] | 2016 |  | integrated as digit of 3DP M2 gripper
---
rectangular 3DP soft finger (32$\times$102$\times$44 mm; 80 pins)
LifeCam Cinema board
TacCylinder [70] | 2017 |  | 3DP body and soft skin
---
cylinder (63 mm dia., 82 mm length, 180 pins)
Catadioptric 360∘ lens; LifeCam Cinema HD
TacTip-GR2 [71] | 2017 |  | integrated as 2 fingertips of 3DP GR2 gripper
---
3DP fingertip (40 mm dia.$\times$ 44 mm depth, 127 pins)
Raspberry Pi Spycam, Fisheye lens, $f$$\approx$20 mm
TacTip-FP1 [43] | 2017 |  | modified tip with ‘fingerprint’ (raised bumps)
---
hemispherical 3DP soft tip (40 mm dia., 127 pins)
LifeCam Cinema board
TacTip-SYM [72] | 2017 |  | modified tip with 12-fold rotational symmetry
---
hemispherical 3DP soft tip (40 mm dia., 49 pins)
LifeCam Cinema board
TacTip-DM [40] | 2018 |  | customized TacTip for optical flow sensor
---
small hemispherical 3DP soft tip (28 mm; 19-61 pins)
ADNS-3080 dual mode, 30$\times$30 pix 3fps / 6.4 kHz
MultiTip [41] | 2018 |  | modified tip moulded from thermoactive material
---
body similar to 2009 TACTIP (40 mm dia., 127 pins)
LifeCam VX webcam, 480p 30fps, $f$$\approx$50 mm
TacWhisker [73] | 2018 |  | modified tip with 19 3DP whiskers fitting into pins
---
also a tendon-actuated version with 2$\times$5 whiskers
LifeCam Cinema board
TacTip [37] | 2018 | same as 2016 version | ELP camera module, 1080p 30fps - 360p 120fps
TacTip-SMG [74] | 2019 |  | integrated as 3 fingertips of Shadow Modular Grasper
---
custom 3DP soft fingertips ($\sim$40$\times$40$\times$44 mm, 100 pins)
ELP camera module; wide angle lens $f$$\approx$20 mm
TacTip-Mini [75] | 2020 |  | modified tip with small area and high pin density
---
small domed 3DP soft tip (25 mm dia., 331 pins)
ELP camera module
TacTip-FP2 [38] | 2020 |  | modified tip with fingerprint to induce incipient slip
---
ridged 3DP soft tip (40 mm dia., 44 pins)
ELP camera module
TacFoot [76] | 2020 |  | integrated as foot of Lynxmotion SQ3U quadruped
---
small hemispherical 3DP soft tip (28 mm; 37 pins)
Hydream USB endoscope, 480p 30fps f$\approx$40 mm
NeuroTac [44] | 2020 |  | neuromorphic TacTip with event-based output
---
hemispherical 3DP soft tip (40 mm dia, 61 pins)
Inivation DVS240, 240$\times$120pix 12Mevents/s
NeuroTac-SoftH [45] | 2020 |  | integrated as fingertip of Pisa/IIT SoftHand
---
small 3DP soft fingertip (20$\times$25$\times$30 mm; 38 pins)
Inivation mini-eDVS, 128$\times$128pix 0.6Mevents/s
TacTip-O [52] | 2020 |  | integrated as 3 fingertips of 3DP Model-O hand
---
small 3DP soft fingertips (20$\times$40$\times$35 mm, 30 pins)
| JeVois Vision module
---
1280$\times$1024pix 15fps to 276$\times$144pix 120fps
TacTip-SoftH [32] | 2021 |  | integrated as fingertip of Pisa/IIT SoftHand
---
small 3DP soft fingertip (12$\times$19$\times$17 mm; 35 pins)
Misumi Model SYD, 1080p 60fps f$\approx$10 mm
TacTip | 2021 |  | rigid 3DP body (45 mm) and modular tip
---
hemispherical 3DP soft tip (40 mm dia., 330 pins)
ELP camera module; wide angle lens $f$$\approx$10 mm

Table 1: TacTip family of soft biomimetic optical tactile sensors. Key:
3D-printed (3DP), 640$\times$480 pix (480p), 1280$\times$720 pix (720p),
1920$\times$1080 pix (1080p), frames per second (fps), focal length ($f$).
LifeCam CCD webcams by Microsoft, ELP (Ailipu Technology) USB CMOS camera
modules, Dynamic Vision Sensors (DVS) are event-based cameras.

Hand | DoA | Fingers | Examined capabilities
---|---|---|---
Tactile Model M2 (T-M2) | 1 | 2 (1 tactile) | precise rolling manipulation [69, 6]
Tactile Model GR2 (T-GR2) | 2 | 2 tactile | precise rolling manipulation [71, 6]
Tactile Model O (T-MO) | 4 | 3 tactile |  | retains grasping capabilities; tactile object recognition; grasp success prediction [52]
---
slip detection & correction; light grasping on first attempt [38]
Tactile Modular Grasper (T-MG) | 9 | 3 tactile |  | manipulation to desired grasp [74]
---
Tactile SoftHand (T-SoftH) | 1 | 5 (1 tactile) |  | sensorimotor control of touch [32]
---
tactile object recognition; grasp perturbation recognition [45]

Table 2: Integration of the TacTip into robot hands varying in degrees of
actuation (DoA) and numbers of fingertips. SoftBOT Sensors: Soft Biomimetic
Optical Tactile Sensors
---
SoftBOT Hands: Soft Biomimetic Optical Tactile Hands
SoftBOT Systems: Soft Biomimetic Optical Tactile Systems
Figure 3: SoftBOTS from the TacTip family. SoftBOT sensors (top row): original
2009 TACTIP [5], 3D-printed TacTip [6], TacTip-GR2 [71], TacTip-M2 [69],
TacCylinder [70], TacTip-O [52], TacTip-SoftHand [32], TacFoot [76] and
TacWhisker [73]. SoftBOT hands (middle row): tactile Model-M2 [69], tactile
Model-GR2 [71], tactile Model-O [52], tactile Shadow Modular Grasper [74] and
tactile SoftHand [32]. SoftBOT systems (bottom row): 3D-printed TacTip on ABB
robot arm for tactile servo control [48, 77], TacTip on UR5 arm for
reinforcement learning [75], TacFoot on walking robot [76], tactile Model-O on
UR5 robot arm for grasping and slip detection [52, 38] and tacWhisker mounted
on ABB robot arm [73].

##  6 Period II (2015-): The TacTip Family

The second period of the TacTip development (2015-present) introduced some
common approaches that allowed research on soft biomimetic optical tactile
sensing at BRL to build into a coherent body of work. Multi-material 3D
printing was adopted for fabrication alongside modular design principles [6],
which encouraged diversification into a family of tactile sensors, hands and
robotic systems (Figure 3). A common method based on tracking the markers was
used for tactile image processing [42, 36], so the data could be treated as
multi-dimensional time series for application of standard machine learning
methods. These developments enabled the TacTip to be integrated into complete
robotic systems that perceive, explore and manipulate their environments.

###  6.1 3D-printed TacTip and integration into robot hands

The adoption of multi-material 3D printing for TacTip manufacture [6] was key
to developing a family of rapidly-prototyped tactile probes, grippers and
manipulators (Figure 3). The soft biomimetic optical tactile sensor was
redesigned (2016/2018 TacTip; Table 2) to have a compact modular base housing
the circuit board from a web-camera; likewise, the tip was 3D-printed in one
piece combining a flexible skin joined to papillae tipped with rigid white
markers [6]. This redesigned TacTip became the standard device used in BRL for
tactile sensing research. The 3D-printed skin is robust to laboratory testing,
with the tip only needing replacing after human error in operating industrial
robot arms (even then, small tears have been repaired).

3D-printing also enabled the sensor to be customised for diverse applications,
from creating a tactile sensing foot for walking robots [76] to mimicking
rodent tactile whiskers [73] (Figure 3, bottom row). A more biomimetic version
of the TacTip skin with raised bumps over the pins (like a fingerprint) and
increased dermal-epidermal stiffness contrast (rigid cores to the pins)
improved the spatial acuity of the tactile sensor [43]. Further progression to
a ringed biomimetic fingerprint helped induce and detect incipient slip, by
encouraging the outer contact region to move before global slip occurs, giving
sufficient time to react before losing the grasp [78].

These advances in soft tactile sensors complement the rapid progress in
3D-printed robot hands, examplified by the Yale OpenHand Project: a library of
low-cost 3D-printed underactuated hand designs [79]. These tendon-driven, 2-4
fingered compliant hands have an underactuated adaptability that passively
conforms their grasping to a wide range of object geometries using only open-
loop control. The most well-known is the 3-fingered Model O, based on the iHY
(iRobot-Harvard-Yale) Hand [80] from the DARPA Robotic Manipulation-Hardware
(ARM-H) program, which has been commercialised as the Reflex Hand (RightHand
robotics). The hand combines an underlying capability for underactuated
grasping [81] with sufficient degrees-of-actuation for the manipulation tasks
set by the ARM-H program [80] (5 for the iHY Hand, reduced to 4 for the Model
O).

Integration of the TacTip with the OpenHands (Table 2) began with the Model M2
[82], a relatively simple gripper with just one movable finger and a large
immobile thumb that was replaced with the TacTip-M2 [69]: an elongated tactile
sensor with a rectangular sensing surface (Figure 3; middle row). Next, the
two-fingered GR2 gripper [83] was integrated [71], replacing each fingertip
with the TacTip-GR2: a compact version of the TacTip body using a fisheye lens
and small camera with a standard-sized tip. Recently, the OpenHand tactile
integration has culminated with the Tactile Model O (T-MO) [52]: the three
fingertips were each replaced with a miniaturized TacTip, offering a low-cost
3D-printed dexterous robot hand with multipurpose soft optical tactile
sensing.

A key question when considering the integration of tactile sensing into
robotic systems is: what new capabilities are given by the sense of touch? For
the 2-fingered tactile OpenHands (T-M2 and T-GR2), the integrated tactile
sense enabled precise in-hand manipulation for unknown held objects [69, 71].
The fingers could roll an object to a desired location or along a trajectory
over the tactile fingertip to millimetre accuracy [6]. The 3-fingered T-MO’s
capabilities are based around the hand’s high capability at grasping. Applying
supervised deep learning to the tactile images of grasped objects gave
accurate object classification (93%, 26 objects) and grasp success prediction
for lifting (95%, same objects) [52]. Furthermore, the application of slip-
detection methods enabled the hand to quickly re-grasp slipping objects (11
objects; 6 novel, 1 compliant) before being dropped [38] (see Section 6.2).
The tactile hand could also prevent an object being dropped when weight was
added, e.g. rice poured into a held tube, and lift an object on the first
attempt with minimal grasp force [38].

Two other state-of-the-art hands have also been integrated with TacTips (Table
2): the Shadow Smart Modular Grasper and the Pisa/IIT SoftHand. The Smart
Modular Grasper is a fully-actuated dexterous 3-fingered hand that was
combined with a compact TacTip using a wide-angle camera lens and customized
body and tip from the 2018 TacTip, which has been applied to stabilizing
grasps by controlling the fingertip contacts [74, 84]. The SoftHand is an
underactuated anthropomorphic robot hand designed around the principle of
adaptive synergies from human hand movements [85]. This 5-fingered hand has
been integrated with a miniaturized TacTip-SoftH of similar size to a human
fingertip [32], a key milestone in this family of tactile hands (Figure 3;
middle-row). A slightly-larger TacTip with an event-based camera (NeuroTac)
has also been integrated [44]. Both versions of the tactile SoftHand open new
possibilities for the sensorimotor control of anthropomorphic robot hands with
biomimetic touch.

(a) Pose-based Servo Control via | (b) Goal-based pushing via | (c) Tactile Deep RL:
---|---|---
Tactile Deep Learning [86, 87] |  tactile & proprioceptive feedback [88] |  Learning to type Braille [75]
|  |
(d) Object & grasp-success prediction | (e) In-hand tactile manipulation for | (f) Fine control of contact with
with an underactuated tactile hand [52] |  fully-actuated stable grasping [84] |  an anthropomorphic tactile hand [32]
|  |
Figure 4: Capabilities of deep learning with the BRL TacTip. The top row (a-c) shows examples of using a single tactile sensor mounted on a robot arm. The bottom row shows examples with tactile robot hands: (d) the Tactile Model O, (e) Tactile Modular Grasper and (f) the Pisa/IIT SoftHand. For the diverse examples shown in this figure, the use of convolutional neural networks was critical for the tactile capability to be reached. Capability | Year | Accuracy | Robot | Method
---|---|---|---|---
1D localization | 2015 | 0.7 mm | TacTip on ABB arm | probabilistic classifier on pin displacements [42, 36]
Rolling manipulation | 2016 |  $\sim$1 mm | TacTip on ABB arm | probabilistic classifier on pin displacements [89]
Gap measurement | 2016 | 0.5 mm | TacTip on ABB arm | probabilistic classifier on pin displacements [90]
Rolling manipulation | 2016 |  $\sim$1 mm | Tactile Model M2 hand | probabilistic classifier on pin displacements [69]
2D contour following (tapping) | 2017 |  $\sim$1 mm, $\sim$10 deg | TacTip on ABB arm | probabilistic classifier on pin displacements [48]
Rolling manipulation | 2017 |  $\sim$1 mm | Tactile Model GR2 hand | probabilistic classifier on pin displacements [71]
1D localization; curvature estimation | 2017 | 0.1 mm; $\sim$1 mm | TacTip-FP1 on ABB arm | probabilistic classifier on pin displacements [43]
Slip detection | 2018 |  $>$95% | TacTip on UR5 arm | SVM classifier on pin velocities [37]
2D contour following (sliding) | 2019 |  $\sim$1 mm, $\sim$10 deg | TacTip on ABB arm | PCA & GP regressor on pin displacements [91, 92]
2D contour following (sliding) | 2019 |  $\sim$1 mm, $\sim$10 deg | TacTip on ABB arm | ConvNet regressor on 128$\times$128 tactile image [77]
Grasp control (centre of contact) | 2019 |  $\sim$1 deg | Tactile Modular Grasper | polynomial regressor on pin displacements [74]
1D localization; Normal force | 2019 |  $\sim$0.5 mm, $\sim$0.2 N | TacTip on ABB arm | ConvNet autoencoder on ]32$\times$32 tactile image [93]
| Object recognition
---
Grasp success prediction
2020 |  |  $>$95%
---
$>$90%
Tactile Model O hand | ConvNet classifier on 60$\times 120$ tactile image [52]
Slip detection | 2020 |  $>$90% | Tactile Model O hand | SVM or logReg classifier on pin velocities [37]
Ultrasound detection | 2020 |  $\lesssim$10 micron | TacTip on ABB arm | GP regressor on pin displacements [94]
| 3D surface localization
---
(depth, roll/pitch)
2020 |  | 0.1 mm
---
0.3 deg
TacTip on ABB arm | ConvNet regressor on 128$\times$128 tactile image [86, 87]
| 3D edge localization
---
(horizontal, depth, roll/pitch, yaw)
2020 |  | 0.3 mm, 0.2 mm
---
1-2 deg, 4 deg
TacTip on ABB arm | ConvNet regressor on 128$\times$128 tactile image [86, 87]
Grasp control (stable pinch) | 2021 | 0.1 mm, 0.4 deg | Tactile Modular Grasper | servo control & ConvNet on 128$\times$128 tactile image [84]
| 3D surface following (sliding)
---
3D contour following (sliding)
2021 |  | 0.1-0.5 mm
---
1-5 deg
TacTip on ABB arm |  | pose-based tactile servo control &
---
ConvNet on 128$\times$128 tactile image [86, 87]
2D pushing (tapping motion) | 2021 |  $\sim$1 mm | TacTip on UR5 arm | servo control & ConvNet on 128$\times$128 tactile image [88]
Table 3: TacTip capabilities in order of development. Key: Support Vector
Machine (SVM), Principal Component Analysis (PCA), Gaussian Process (GP),
Convolutional Neural Network (ConvNet), Logistic Regression (LogReg). The ABB
arm is an IRB120 6-DoF industrial robot; the UR5 arm is a Universal Robotics
6-DoF arm. Tactile Hands are described in Table 2.

###  6.2 Progress in tactile capabilities

The purpose of an artificial sense of touch is to impart new dexterous
capabilities to robotic systems that physically interact with their
surroundings. These capabilities are built on sensorimotor perception and
control: perception to process the tactile sensations for inferring states of
the environment relative to the sensor, such as the location or shape of an
object feature; and control to change that state, such as to slide a fingertip
delicately over a feature or manipulate an object in a desired manner. A range
of tactile perception and control methods have been developed for the TacTip
family of soft biomimetic optical tactile sensors and hands (Table 3).

Over the period 2015-19, most of the methods used time series of marker
deflections from rest, as they gave an efficient representation of the tactile
image that has a biomimetic analogue with mechanoreceptor activity [42, 36,
43, 6] (see Section 3). The pin displacements are also easily visualised to
help interpret the tactile sensing (Figure 1).

Over the same period, a perception method based on a histogram likelihood
model over the marker displacements was used [42]. For control, the position
of an object feature was predicted, such as a cylinder position or edge
orientation [36], then applied to rolling manipulation [89, 69, 71] or tapping
around 2D contours [48, 72, 40]. However, those methods did not extend well
from discrete to continuous and dynamic environments. Consequently, techniques
interpolating over continuously-labelled data were tried, such as Gaussian
process regression to control sliding motion around 2D contours [91, 92] and
polynomial regression to smoothly control a multi-fingered grasp [74]. The
large amount of training data can be an issue in some circumstances, and so
online learning using Gaussian Process latent variable models has also been
explored [95, 76]. Methods for dynamic environments were also developed, such
as accurately detecting and correcting for object slippage [37, 38, 78] using
a support-vector machine.

The progress in machine learning methods has steadily improved the TacTip
accuracy. Initial performance of $\sim$1 mm for 1D localization [42, 36, 48]
or rolling manipulation [89, 69, 71] has improved by an order-of-magnitude to
0.1-0.5 mm [86]. Therefore, while the first results demonstrated hyperacuity
finer than the pin spacing [50, 42], the latest results demonstrate
sensitivity at the level of pixels on the tactile image. Thus far, the best
sensitivity with a TacTip has been to detect $\lesssim$10 microns indentation
from an ultrasound haptic display, using signal averaging over the pin
deflections from multiple tactile measurements to reduce noise [94].

##  7 Period III (2019-): Deep learning with the TacTip

Recently, the capabilities of the TacTip have undergone a step-change with the
adoption of deep learning over tactile images. This has enabled, for the first
time, tactile interaction in real time with complex objects in 3D (Figure 4).
Prior to deep learning, the capabilities were limited to basic demonstrations
such as rolling objects in 1D or exploring around flat shapes in 2D (Table 3).
The new dexterous capabilities are a step closer to those we possess as
humans.

The main benefit of deep learning is to predict quantities of interest
directly from the tactile images while being insensitive to unknown variations
that might otherwise interfere with the predictions. Consequently, the most
advanced robot dexterity with the TacTip (Table 3) has mainly used
convolutional neural networks (ConvNets), following their earlier success with
optical tactile sensors based on the GelSight [96, 97]. In principle, the
marker deflections (Section 6.2) could be used as a lower-dimensional input to
a neural network, although in practise it has been simpler to avoid the
additional image processing step by using the entire tactile image as input.

New tactile capabilities developed with the TacTip include: (a) pose-based
servo control, where a tactile fingertip mounted on a robot arm slides
delicately over unknown complex 3D objects (Figure 4a; [77, 86, 87]); (b)
pushing manipulation of unknown objects using only tactile sensing and
proprioceptive knowledge of where the sensor is positioned relative to a goal
location (Figure 4b; [88]); (c) acquiring the novel skill of single-fingered
typing on a braille keyboard, learning the identity of keys and how to
navigate the keyboard from touch (Figure 4c; [75]); (d) item recognition and
grasp-success prediction upon using the tactile sense of the three fingertips
of the Tactile Model-O (T-MO) hand (Figure 4d; [52]); (e) in-hand manipulation
of unknown objects to a stable grasp configuration with a fully-actuated
tactile Shadow Modular Grasper (Figure 4e; [84]); (f) fine control of contact
onto unknown objects placed in-hand using an anthropomorphic tactile hand
based on the Pisa/IIT SoftHand (Figure 4f; [32]). Humans can do all these
tasks with our sense of touch, which deep learning has enabled tactile robots
to likewise perform.

Why are ConvNets so useful when applied to tactile images from a soft
biomimetic optical tactile sensor? Considering all the tasks in Figure 4,
several reasons emerge:
(i) Ease of deployment – in all tasks, the trained neural network was applied
directly with only basic pre-processing steps such as cropping, thresholding
or concatenating tactile images. This simplifies the algorithmic pipeline and
leaves less room for software or other issues to emerge.
(ii) Robustness – once trained, the predictions were relatively unaffected by
uncontrolled variations that could cause issues, such as changes in ambient
and internal lighting, wear of the skin, visible dust inside the sensor, and
even after damage such as a (glued) tear in the TacTip skin after accidental
crushing.
(iii) Ease of scalability – the methods seem to extend straightforwardly from
simple test scenarios to more complex realistic situations; for example, 2D
servo control using a ConvNet with two pose variables extended relatively
easily to 3D control with more pose variables [77, 86].
(iv) Generalization – the predictions seemed to be accurate even after large
variations in the stimulus or task conditions; for example, ConvNets trained
on simple stimuli, such as the pose of a planar surface or straight edge,
maintained predictive performance on curved complex stimuli, enabling novel
objects to be explored or manipulated [87, 88].

Going forward, a key question for deep learning with optical tactile sensing
will be on the appropriate learning algorithm to acquire tactile skills. All
but one of the tasks in Figure 4 trained the neural network model with
supervised learning. However, a complementary approach is to use tactile deep
reinforcement learning (RL) with a reward signal that indicates the desired
interactions with a physical environment. The third task (Figure 4d)
successfully applied this approach to learning to type with a braille keyboard
[75], leading after many hours of training to a policy network that could
efficiently navigate the keyboard to press a desired key. In principle, all
other tasks could also be learnt using deep RL, but the long training times
are impractical on physical robots. Thus, our expectation is that most of the
learning should be within a simulated tactile environment [98]. This
expectation has just been confirmed on several of these tasks using the BRL
TacTip [99].

##  8 Conclusion

There are fundamental problems to be addressed in intelligent robotic
interaction with complex environments that once solved will open up many
application areas across engineering and robotics. One key problem is that
there is a huge gap between what is achievable in research laboratories
investigating robot manipulation and what is known about human dexterity and
our sense of touch. Research on the TacTip aims to bridge that gap as an
example of a SoftBOT sensor, combining Soft, Biomimetic, Optical and Tactile
sensing.

SoftBOT sensors offer the opportunity to artificially recreate key aspects of
the human sense of touch and our manual intelligence. This aim has two
interconnected goals: (1) to advance knowledge of how our sense of touch leads
to haptic intelligence from embodying those capabilities in robots; and (2) to
improve the intelligent dexterity of robots with accessible robot hardware and
software. Reaching human-like levels of dexterity has been the vision for
industrial robotics since its roots in the 1950s, and is captured in the
earlier origins of ‘robot’ from the Czech word robota for forced labour.
Likewise, using biomimetic touch to achieve that goal has driven developments
in robotic tactile sensing since the 1970s. A combination of advances in soft
robotics, biomimetic tactile sensing and AI could enable that vision to become
reality.

## Acknowledgments

I am fortunate to have the opportunity to work with many talented PhD students
and postdocs at Bristol Robotics Laboratory, including Noor Alakhawand, Kirsty
Aquilina, Sophie-Anne Baker, Thomas Cairnes, Alex Church, Pernilla Craig, Luke
Cramphorn, Elena Giannaccini, Kipp McAdam Freud, Michele Garibo, Thom
Griffith, Anupam Gupta, Fliss Inkpen, Jasper James, Haoran Li, Yijiong Lin,
John Lloyd, Fraser MacDonald, Dabal Pedamonti, Nick Pestell, Efi Psomopoulou,
Lizzie Stone, Emma Roscow and Ben Ward-Cherrier.

## References

  * [1] J. Kevin O’Regan.  Why Red Doesn’t Sound Like a Bell: Understanding the Feel of Consciousness. Oxford University Press, July 2011.
  * [2] Yuval Noah Harari.  Homo Deus: A Brief History of Tomorrow.  Random House, September 2016.
  * [3] Micah K. Johnson and Edward H. Adelson.  Retrographic sensing for the measurement of surface texture and shape.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1070–1077, June 2009.
  * [4] Wenzhen Yuan, Siyuan Dong, and Adelson Adelson, Edward H.  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force.  Sensors, 17(12):2762, 2017.
  * [5] Craig Chorley, Chris Melhuish, Tony Pipe, and Jonathan Rossiter.  Development of a tactile sensor based on biologically inspired edge encoding.  In 2009 International Conference on Advanced Robotics, pages 1–6, June 2009.
  * [6] Benjamin Ward-Cherrier, Nicholas Pestell, Luke Cramphorn, Benjamin Winstone, Maria Elena Giannaccini, Jonathan Rossiter, and Nathan F. Lepora.  The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies.  Soft Robotics, 5(2):216–227, April 2018.
  * [7] Shan Luo, Joao Bimbo, Ravinder Dahiya, and Hongbin Liu.  Robotic tactile perception of object properties: A review.  Mechatronics, 48:54–67, 2017.
  * [8] Zhanat Kappassov, Juan-Antonio Corrales, and Véronique Perdereau.  Tactile sensing in dexterous robot hands — Review.  Robotics and Autonomous Systems, 74:195–220, December 2015.
  * [9] Ravinder S. Dahiya, Giorgio Metta, Maurizio Valle, and Giulio Sandini.  Tactile Sensing—From Humans to Humanoids.  IEEE Transactions on Robotics, 26(1):1–20, February 2010.
  * [10] Mark R. Cutkosky, Robert D. Howe, and William R. Provancher.  Force and Tactile Sensors.  In Springer Handbook of Robotics, pages 455–476. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.
  * [11] Saleem Khan, Leandro Lorenzelli, and Ravinder S. Dahiya.  Technologies for Printing Sensors and Electronics Over Large Flexible Substrates: A Review.  IEEE Sensors Journal, 15(6):3164–3185, June 2015.
  * [12] Hongbo Wang, Massimo Totaro, and Lucia Beccai.  Toward Perceptive Soft Robots: Progress and Challenges.  Advanced Science, 5(9):1800541, September 2018.
  * [13] Alexander Schmitz, Perla Maiolino, Marco Maggiali, Lorenzo Natale, Giorgio Cannata, and Giorgio Metta.  Methods and Technologies for the Implementation of Large-Scale Robot Tactile Sensors.  IEEE Transactions on Robotics, 27(3):389–400, June 2011.
  * [14] Ravinder S. Dahiya.  E-Skin: From Humanoids to Humans [Point of View].  Proceedings of the IEEE, 107(2):247–252, February 2019.
  * [15] Nathan F Lepora, Paul Verschure, and Tony J Prescott.  The state of the art in biomimetics.  Bioinspiration & Biomimetics, 8(1):013001, January 2013.
  * [16] Rolf Pfeifer, Max Lungarella, and Fumiya Iida.  The challenges ahead for bio-inspired ’soft’ robotics.  Communications of the ACM, 55(11):76–87, November 2012.
  * [17] Sangbae Kim, Cecilia Laschi, and Barry Trimmer.  Soft robotics: A bioinspired evolution in robotics.  Trends in Biotechnology, 31(5):287–294, May 2013.
  * [18] Nicholas Wettels, Veronica J. Santos, Roland S. Johansson, and Gerald E. Loeb.  Biomimetic Tactile Sensor Array.  Advanced Robotics, 22(8):829–849, January 2008.
  * [19] Clementine Boutry, Marc Negre, Mikael Jorda, Orestis Vardoulis, Alex Chortos, Oussama Khatib, and Zhenan Bao.  A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics.  Science Robotics, 3(24), November 2018.
  * [20] Alexander C. Abad and Anuradha Ranasinghe.  Visuotactile Sensors With Emphasis on GelSight Sensor: A Review.  IEEE Sensors Journal, 20(14):7628–7638, July 2020.
  * [21] Thomas G. Strickler.  Design of an optical touch sensing system for a remote manipulator, 1966\.
  * [22] Kazuhiro Shimonomura.  Tactile Image Sensors Employing Camera: A Review.  Sensors, 19(18):3933, September 2019.
  * [23] Kazuto Kamiyama, Hiroyuki Kajimoto, Naoki Kawakami, and Susumu Tachi.  Evaluation of a vision-based tactile sensor.  In IEEE International Conference on Robotics and Automation (ICRA), pages 1542–1547 Vol.2, New Orleans, LA, USA, 2004\.
  * [24] Kazuto Kamiyama, Kevin Vlack, Terukazu Mizota, Hiroyuki Kajimoto, Naoki Kawakami, and Susumu Tachi.  Vision-based sensor for real-time measuring of surface traction fields.  IEEE Computer Graphics and Applications, 25(1):68–75, January 2005\.
  * [25] Wenzhen Yuan, Rui Li, Mandayam A. Srinivasan, and Edward H. Adelson.  Measurement of shear and slip with a GelSight tactile sensor.  In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 304–311, Seattle, USA, May 2015.
  * [26] Xi Lin and Michael Wiertlewski.  Sensing the Frictional State of a Robotic Skin via Subtractive Color Mixing.  IEEE Robotics and Automation Letters, 4(3):2386–2392, July 2019\.
  * [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, May 2017.
  * [28] Daniel F. Gomes, Zhonglin Lin, and Shan Luo.  GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9903–9909, Las Vegas, USA, October 2020.
  * [29] Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, and Roberto Calandra.  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation.  IEEE Robotics and Automation Letters, 5(3):3838–3845, July 2020\.
  * [30] Akhil Padmanabha, Frederik Ebert, Stephen Tian, Roberto Calandra, Chelsea Finn, and Sergey Levine.  OmniTact: A Multi-Directional High Resolution Touch Sensor.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 618–624, Paris, France, May 2020.
  * [31] Branden Romero, Filipe Veiga, and Edward H. Adelson.  Soft, Round, High Resolution Tactile Fingertip Sensors for Dexterous Robotic Manipulation.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4796–4802, Paris, France, May 2020.
  * [32] Nathan F. Lepora, Andrew Stinchcombe, Chris Ford, Alfred Brown, John Lloyd, Manuel G Catalano, Matteo Bianchi, and Benjamin Ward-Cherrier.  Towards integrated tactile sensorimotor control in anthropomorphic soft robotic hands.  In IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, May 2021.
  * [33] Roberta L. Klatzky and Susan J. Lederman.  Touch.  In Handbook of Psychology, pages 147–176. 2003.
  * [34] Victoria E. Abraira and David D. Ginty.  The Sensory Neurons of Touch.  Neuron, 79(4):618–639, August 2013.
  * [35] Amanda Zimmerman, Ling Bai, and David D. Ginty.  The gentle touch receptors of mammalian skin.  Science, 346(6212):950–954, November 2014.
  * [36] Nathan F. Lepora.  Biomimetic Active Touch with Fingertips and Whiskers.  IEEE Transactions on Haptics, 9(2):170–183, April 2016.
  * [37] Jasper Wollaston James, Nicholas Pestell, and Nathan F. Lepora.  Slip Detection With a Biomimetic Tactile Sensor.  IEEE Robotics and Automation Letters, 3(4):3340–3346, October 2018\.
  * [38] Jasper Wollaston James and Nathan F. Lepora.  Slip Detection for Grasp Stabilization With a Multifingered Tactile Robot Hand.  IEEE Transactions on Robotics, 37(2):506–519, 2020.
  * [39] Benjamin Winstone, Gareth Griffiths, Tony Pipe, Chris Melhuish, and Jonathon Rossiter.  TACTIP - Tactile Fingertip Device, Texture Analysis through Optical Tracking of Skin Features.  In Biomimetic and Biohybrid Systems, volume 8064, pages 323–334. Springer Berlin Heidelberg, 2013.
  * [40] Nicholas Pestell, John Lloyd, Jonathan Rossiter, and Nathan F. Lepora.  Dual-Modal Tactile Perception and Exploration.  IEEE Robotics and Automation Letters, 3(2):1033–1040, April 2018\.
  * [41] Gabor Soter, Andrew Conn, Helmut Hauser, Nathan F. Lepora, and Jonathan Rossiter.  MultiTip: A multimodal mechano-thermal soft fingertip.  In 2018 IEEE International Conference on Soft Robotics (RoboSoft), pages 239–244, Liverno, Italy, April 2018.
  * [42] Nathan F. Lepora and Benjamin Ward-Cherrier.  Superresolution with an optical tactile sensor.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2686–2691, Hamburg, Germany, September 2015.
  * [43] Luke Cramphorn, Benjamin Ward-Cherrier, and Nathan F. Lepora.  Addition of a Biomimetic Fingerprint on an Artificial Fingertip Enhances Tactile Spatial Acuity.  IEEE Robotics and Automation Letters, 2(3):1336–1343, July 2017\.
  * [44] Benjamin Ward-Cherrier, Nicholas Pestell, and Nathan F. Lepora.  NeuroTac: A Neuromorphic Optical Tactile Sensor applied to Texture Recognition.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2654–2660, Paris, France, May 2020.
  * [45] Benjamin Ward-Cherrier, Jorg Conradt, Manuel Giuseppe Catalano, Matteo Bianchi, and Nathan F. Lepora.  A Miniaturised Neuromorphic Tactile Sensor Integrated with an Anthropomorphic Robot Hand.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9883–9889, Las Vegas, USA, October 2020.
  * [46] Nikolajs Cauna.  Nature and functions of the papillary ridges of the digital skin.  The Anatomical Record, 119(4):449–468, August 1954.
  * [47] Hannes P. Saal and Sliman J. Bensmaia.  Touch is a team effort: Interplay of submodalities in cutaneous sensibility.  Trends in Neurosciences, 37(12):689–697, December 2014.
  * [48] Nathan F. Lepora, Kirsty Aquilina, and Luke Cramphorn.  Exploratory Tactile Servoing With Active Touch.  IEEE Robotics and Automation Letters, 2(2):1156–1163, April 2017\.
  * [49] Jack. M. Loomis.  An Investigation of Tactile Hyperacuity.  Sensory Processes, 3:289–302, 1980.
  * [50] Nathan F. Lepora, Uriel Martinez-Hernandez, Mathew Evans, Lorenzo Natale, Giorgio Metta, and Tony J. Prescott.  Tactile Superresolution and Biomimetic Hyperacuity.  IEEE Transactions on Robotics, 31(3):605–618, June 2015.
  * [51] Seoung-Mok Yum, In-Keun Baek, Dongpyo Hong, Juhan Kim, Kyunghoon Jung, Seontae Kim, Kihoon Eom, Jeongmin Jang, Seonmyeong Kim, Matlabjon Sattorov, Min-Geol Lee, Sungwan Kim, Michael J. Adams, and Gun-Sik Park.  Fingerprint ridges allow primates to regulate grip.  Proceedings of the National Academy of Sciences, 117(50):31665–31673, December 2020.
  * [52] Jasper W. James, Alex Church, Luke Cramphorn, and Nathan F. Lepora.  Tactile Model O: Fabrication and Testing of a 3D-Printed, Three-Fingered Tactile Robot Hand.  Soft Robotics, December 2020.
  * [53] Alex Chortos, Jia Liu, and Zhenan Bao.  Pursuing prosthetic electronic skin.  Nature Materials, 15(9):937–950, September 2016.
  * [54] Zhengkun Yi, Yilei Zhang, and Jan Peters.  Biomimetic tactile sensors and signal processing with spike trains: A review.  Sensors and Actuators A: Physical, 269:41–52, January 2018.
  * [55] Carver Mead.  Neuromorphic electronic systems.  Proceedings of the IEEE, 78(10):1629–1636, October 1990.
  * [56] Nicola J. Ferrier and Roger W. Brockett.  Reconstructing the Shape of a Deformable Membrane from Image Data.  The International Journal of Robotics Research, 19(9):795–816, September 2000.
  * [57] Jonathan Platkiewicz, Hod Lipson, and Vincent Hayward.  Haptic Edge Detection Through Shear.  Scientific Reports, 6(1):23551, September 2016.
  * [58] Craig Chorley, Chris Melhuish, Tony Pipe, and Jonathan Rossiter.  Tactile edge detection.  In 2010 IEEE Sensors, pages 2593–2598, Kona, Hawaii, November 2010.
  * [59] Benjamin Winstone, Gareth Griffiths, Chris Melhuish, Tony Pipe, and Jonathan Rossiter.  TACTIP — Tactile fingertip device, challenges in reduction of size to ready for robot hand integration.  In 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 160–166, Guangzhou, China, December 2012\.
  * [60] Ryo Kikuuwe, Akihito Sano, Hiromi Mochiyama, Naoyuki Takesue, K. Tsunekawa, S. Suzuki, and Hideo Fujimoto.  The tactile contact lens.  In Proceedings of IEEE Sensors, pages 535–538, Vienna, Austria, 2004.
  * [61] Ryo Kikuuwe, Akihito Sano, Hiromi Mochiyama, Naoyuki Takesue, and Hideo Fujimoto.  Enhancing haptic detection of surface undulation.  ACM Transactions on Applied Perception, 2(1):46–67, January 2005\.
  * [62] Mitsuhito Ando, Hiromi Mochiyama, Toshinobu Takei, and Hideo Fujimoto.  Effect of tactile contact lens on rubber artificial skin layer with a strain gauge.  In 2016 IEEE/SICE International Symposium on System Integration (SII), pages 397–402, Sapporo, Japan, December 2016.
  * [63] Tareq Assaf, Craig Chorley, Jonathan Rossiter, Tony Pipe, Cesare Stefanini, and Chris Melhuish.  Realtime processing of a biologically inspired tactile sensor for edge following and shape recognition.  In Towards Autonomous Robotic Systems, pages 13–19, Plymouth, UK, 2010.
  * [64] Tareq Assaf, Calum Roke, Jonathan Rossiter, Tony Pipe, and Chris Melhuish.  Seeing by Touch: Evaluation of a Soft Biologically-Inspired Artificial Fingertip in Real-Time Active Touch.  Sensors, 14(2):2561–2577, February 2014.
  * [65] Calum Roke, Chris Melhuish, Tony Pipe, David Drury, and Craig Chorley.  Deformation-Based Tactile Feedback Using a Biologically-Inspired Sensor and a Modified Display.  In Towards Autonomous Robotic Systems, Lecture Notes in Computer Science, pages 114–124. Springer, Berlin, Heidelberg, 2011.
  * [66] Calum Roke, Chris Melhuish, Tony Pipe, David Drury, and Craig Chorley.  Lump localisation through a deformation-based tactile feedback system using a biologically inspired finger sensor.  Robotics and Autonomous Systems, 60(11):1442–1448, November 2012\.
  * [67] Calum Roke, Adam Spiers, Tony Pipe, and Chris Melhuish.  The effects of laterotactile information on lump localization through a teletaction system.  In 2013 World Haptics Conference (WHC), pages 365–370, Daejeon, South Korea, April 2013.
  * [68] Benjamin Ward-Cherrier, Nicholas Pestell, Luke Cramphorn, Maria Elena Giannaccini, and Nathan F. Lepora.  TacTip.  In Soft Robotics Toolkit. https://softroboticstoolkit.com/tactip, 2016.
  * [69] Benjamin Ward-Cherrier, Luke Cramphorn, and Nathan F. Lepora.  Tactile Manipulation With a TacThumb Integrated on the Open-Hand M2 Gripper.  IEEE Robotics and Automation Letters, 1(1):169–175, January 2016\.
  * [70] Benjamin Winstone, Chris Melhuish, Tony Pipe, Mark Callaway, and Sanja Dogramadzi.  Toward Bio-Inspired Tactile Sensing Capsule Endoscopy for Detection of Submucosal Tumors.  IEEE Sensors Journal, 17(3):848–857, February 2017.
  * [71] Benjamin Ward-Cherrier, Nicolas Rojas, and Nathan F. Lepora.  Model-Free Precise in-Hand Manipulation with a 3D-Printed Tactile Gripper.  IEEE Robotics and Automation Letters, 2(4):2056–2063, October 2017\.
  * [72] Benjamin Ward-Cherrier, Luke Cramphorn, and Nathan F. Lepora.  Exploiting Sensor Symmetry for Generalized Tactile Perception in Biomimetic Touch.  IEEE Robotics and Automation Letters, 2(2):1218–1225, April 2017\.
  * [73] Nathan F. Lepora, Martin Pearson, and Luke Cramphorn.  TacWhiskers: Biomimetic Optical Tactile Whiskered Robots.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7628–7634, Madrid, Spain, October 2018.
  * [74] Nicholas Pestell, Luke Cramphorn, Fotios Papadopoulos, and Nathan F. Lepora.  A Sense of Touch for the Shadow Modular Grasper.  IEEE Robotics and Automation Letters, 4(2):2220–2226, April 2019\.
  * [75] Alex Church, John Lloyd, Raia Hadsell, and Nathan F. Lepora.  Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard.  IEEE Robotics and Automation Letters, 5(4):6145–6152, October 2020\.
  * [76] Elizabeth A. Stone, Nathan F. Lepora, and David A. W. Barton.  Walking on TacTip Toes: A Tactile Sensing Foot for Walking Robots.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9869–9875, Las Vegas, USA, October 2020.
  * [77] Nathan F. Lepora, Alex Church, Conrad de Kerckhove, Raia Hadsell, and John Lloyd.  From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and an Optical Biomimetic Tactile Sensor.  IEEE Robotics and Automation Letters, 4(2):2101–2107, April 2019\.
  * [78] Jasper Wollaston James, Stephen Redmond, and Nathan Lepora.  A Biomimetic Tactile Fingerprint Induces Incipient Slip.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9833–9839, Las Vegas, USA, October 2020.
  * [79] Raymond Ma and Aaron Dollar.  Yale OpenHand Project: Optimizing Open-Source Hand Designs for Ease of Fabrication and Adoption.  IEEE Robotics & Automation Magazine, 24(1):32–40, March 2017.
  * [80] Lael U. Odhner, Leif P. Jentoft, Mark R. Claffee, Nicholas Corson, Yaroslav Tenzer, Raymond R. Ma, Martin Buehler, Robert Kohout, Robert D. Howe, and Aaron M. Dollar.  A compliant, underactuated hand for robust manipulation.  The International Journal of Robotics Research, 33(5):736–752, April 2014.
  * [81] Aaron M. Dollar and Robert D. Howe.  The Highly Adaptive SDM Hand: Design and Performance Evaluation.  The International Journal of Robotics Research, 29(5):585–597, April 2010.
  * [82] Raymond R. Ma, Adam Spiers, and Aaron M. Dollar.  M2 Gripper: Extending the Dexterity of a Simple, Underactuated Gripper.  In Advances in Reconfigurable Mechanisms and Robots II, volume 36, pages 795–805. Springer International Publishing, 2016.
  * [83] Nicolas Rojas, Raymond R. Ma, and Aaron M. Dollar.  The GR2 Gripper: An Underactuated Hand for Open-Loop In-Hand Planar Manipulation.  IEEE Transactions on Robotics, 32(3):763–770, June 2016.
  * [84] Efi Psomopoulou, Nicholas Pestell, Fotios Papadopoulos, John Lloyd, Zoe Doulgeri, and Nathan F Lepora.  A robust controller for stable 3D pinching using tactile sensing.  IEEE Robotics and Automation Letters.
  * [85] Manuel G. Catalano, Giorgio Grioli, Edoardo Farnioli, Alessandro Serio, Cristina Piazza, and Antonio Bicchi.  Adaptive synergies for the design and control of the Pisa/IIT SoftHand.  The International Journal of Robotics Research, 33(5):768–782, 2014\.
  * [86] Nathan F. Lepora and John Lloyd.  Optimal Deep Learning for Robot Touch: Training Accurate Pose Models of 3D Surfaces and Edges.  IEEE Robotics and Automation Magazine, 27(2):66–77, June 2020.
  * [87] Nathan F. Lepora and John Lloyd.  Pose-based tactile servoing: Controlled soft touch using deep learning.  IEEE Robotics and Automation Magazine, 2021.
  * [88] John Lloyd and Nathan F Lepora.  Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback.  IEEE Transactions on Robotics, 2021.
  * [89] Luke Cramphorn, Benjamin Ward-Cherrier, and Nathan F. Lepora.  Tactile manipulation with biomimetic active touch.  In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 123–129, Stockholm, Sweden, May 2016.
  * [90] Nathan F. Lepora and Benjamin Ward-Cherrier.  Tactile Quality Control With Biomimetic Active Touch.  IEEE Robotics and Automation Letters, 1(2):646–652, July 2016.
  * [91] Kirsty Aquilina, David A. W. Barton, and Nathan F. Lepora.  Principal Components of Touch.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4071–4078, Brisbane, Australia, May 2018\.
  * [92] Kirsty Aquilina, David A. W. Barton, and Nathan F. Lepora.  Shear-invariant Sliding Contact Perception with a Soft Tactile Sensor.  In 2019 International Conference on Robotics and Automation (ICRA), pages 4283–4289, Montreal, Canada, May 2019.
  * [93] Marsela Polic, Ivona Krajacic, Nathan Lepora, and Matko Orsag.  Convolutional Autoencoder for Feature Extraction in Tactile Sensing.  IEEE Robotics and Automation Letters, 4(4):3671–3678, October 2019\.
  * [94] Noor Alakhawand, William Frier, Kipp McAdam Freud, Orestis Georgiou, and Nathan F. Lepora.  Sensing Ultrasonic Mid-Air Haptics with a Biomimetic Tactile Fingertip.  In Haptics: Science, Technology, Applications, volume 12272, pages 362–370. Springer International Publishing, 2020.
  * [95] Elizabeth A. Stone, Nathan F. Lepora, and David A. W. Barton.  Learning to Live Life on the Edge: Online Learning for Data-Efficient Tactile Contour Following.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9854–9860, Las Vegas, USA, October 2020.
  * [96] Wenzhen Yuan, Chenzhuo Zhu, Andrew Owens, Mandayam A. Srinivasan, and Edward H. Adelson.  Shape-independent hardness estimation using deep learning and a GelSight tactile sensor.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 951–958, Singapore, May 2017.
  * [97] Roberto Calandra, Andrew Owens, Manu Upadhyaya, Wenzhen Yuan, Justin Lin, Edward H. Adelson, and Sergey Levine.  The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?  In Conference on Robot Learning, pages 314–323. PMLR, October 2017.
  * [98] Zihan Ding, Nathan F. Lepora, and Edward Johns.  Sim-to-Real Transfer for Optical Tactile Sensing.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1639–1645, Paris, France, May 2020.
  * [99] Alex Church, John Lloyd, Raia Hadsell, and Nathan F. Lepora.  Optical tactile sim-to-real policy transfer via real-to-sim tactile image translation.  arXiv preprint https://arxiv.org/abs/2106.08796, 2021.