# Using Collocated Vision and Tactile Sensors for Visual Servoing and
Localization

Arkadeep Narayan Chaudhury1, Timothy Man1, Wenzhen Yuan1 and Christopher G.
Atkeson1 Manuscript received: 9-9-21; Revised: 12-9-21; Accepted: 1-4-22This
paper was recommended for publication by Associate Editor J. Falco and Editor
Dan Popa upon evaluation of the Reviewers’ comments. This work was supported
in part by the National Science Foundation through award 1717066 and the
Toyota Research Institute.1All authors are with the Robotics Institute,
Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, Pennsylvania, USA
ZIP: 15232. Email addresses: {arkadeepnc, yuanwz<EMAIL_ADDRESS>Object
Identifier (DOI): 10.1109/LRA.2022.3146565Note: This archival version of the
manuscript is significantly different in content from the reviewed and
published version. The published version can be accessed here:
https://ieeexplore.ieee.org/document/9699405.

# Using Collocated Vision and Tactile Sensors for Visual Servoing and
Localization

Arkadeep Narayan Chaudhury1, Timothy Man1, Wenzhen Yuan1 and Christopher G.
Atkeson1 Manuscript received: 9-9-21; Revised: 12-9-21; Accepted: 1-4-22This
paper was recommended for publication by Associate Editor J. Falco and Editor
Dan Popa upon evaluation of the Reviewers’ comments. This work was supported
in part by the National Science Foundation through award 1717066 and the
Toyota Research Institute.1All authors are with the Robotics Institute,
Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, Pennsylvania, USA
ZIP: 15232. Email addresses: {arkadeepnc, yuanwz<EMAIL_ADDRESS>Object
Identifier (DOI): 10.1109/LRA.2022.3146565Note: This archival version of the
manuscript is significantly different in content from the reviewed and
published version. The published version can be accessed here:
https://ieeexplore.ieee.org/document/9699405.

###### Abstract

Coordinating proximity and tactile imaging by collocating cameras with tactile
sensors can 1) provide useful information before contact such as object pose
estimates and visually servo a robot to a target with reduced occlusion and
higher resolution compared to head-mounted or external depth cameras, 2)
simplify the contact point and pose estimation problems and help tactile
sensing avoid erroneous matches when a surface does not have significant
texture or has repetitive texture with many possible matches, and 3) use
tactile imaging to further refine contact point and object pose estimation. We
demonstrate our results with objects that have more surface texture than most
objects in standard manipulation datasets. We learn that optic flow needs to
be integrated over a substantial amount of camera travel to be useful in
predicting movement direction. Most importantly, we also learn that state of
the art vision algorithms do not do a good job localizing tactile images on
object models, unless a reasonable prior can be provided from collocated
cameras.

###### Index Terms:

tactile sensing, sensor fusion, localization.

##  I INTRODUCTION

(a)

(b)

(c)

(d)

(e)

(f)

Figure 1:  We demonstrate an approach to integrate sensors with different
fields of view to visually servo the robot arm to a predetermined contact
point and estimate the pose of a fixed object relative to the sensors at
contact. Figures 1(a) and 1(b) show our sensor platform. We use 2 cameras with
$70^{\circ}$ and $100^{\circ}$ fields of view, which we collocate with a
modified GelSight sensor in the middle (fig. 1(b)). The cameras are used to
visually servo the robot (fig. 1(c)) and generate a preliminary pose estimate
(fig. 1(d)) while the robot is moving towards the target. At contact, the
GelSight data is observed (fig. 1(e)) and the preliminary pose is then refined
to generate the object pose at contact. Figure 1(f) shows the camera pose
superimposed on the mesh model of the object.

This work is motivated by FingerVision [1] where the same camera was used for
tactile sensing and to view nearby objects through a transparent elastomer.
Although FingerVision convinced us of the importance of proximity imaging, it
did not provide high resolution images of contact surface texture, and the
transparent elastomer blurred proximity imaging, attracted dust, and got
scratched and worn so the view of nearby objects was often not as good as we
would like. Separating tactile and proximity imaging enables us to get the
high resolution of GelSight [2] tactile sensors that produce images of the
surface texture (tactile imaging), and better proximity imaging with rigid
lenses that don’t attract dust as much, are easier to clean and don’t scratch
or get worn as easily. This paper explores an alternative to using the same
camera for tactile and proximity imaging, where a tactile sensor is collocated
with a camera for proximity sensing (fig. 1). Since the tactile sensor we use,
a GelSight variant, is also based on a camera, we are actually collocating
multiple cameras to provide tactile and proximity imaging.

Cameras that move with a robot hand can have less occlusion and more
resolution since they are closer to manipulated objects. Direct measurement of
the direction or bearing to an object and its pose relative to the hand can be
used to guide the hand to a particular contact location and center the hand
with respect to an object. External and head-mounted cameras are often
occluded by the robot itself as well as manipulated objects, and need to use
stereo, multi-view, or other forms of depth measurement to locate the hand
relative to the approach axis and object, which involves subtracting two noisy
estimates (typically large numbers) to estimate a smaller quantity, which is
usually less accurate than directly measuring the smaller quantity. We have
found depth measurements from stereo or time of flight (TOF) cameras usually
have low spatial resolution, so getting the camera close to the hand is useful
to improve depth resolution.

We divide the problem of contact pose estimation into two parts – The initial
phase before contact, when cameras can be used for vision-based servoing to a
contact point target as well as estimating a prior for the tactile sensor
contact point and object pose estimation, and the contact phase which refines
the prior pose estimates. In this paper we assume that 1) the object is fixed,
even during contact, 2) the object is a single rigid body with no
articulations, and 3) we have a prior 3D model of the object (potentially
provided by our vision of the current object) so we can express the pose of
the object with respect to this model. For this paper we put aside the gross
object localization and recognition problems in order to focus on fine
localization, so we assume a vision system has already located the object,
created a bounding box, and recognized the object by creating or selecting an
appropriate 3D model that we want to register the actual object to (e.g. [3]).
Our experimental pipeline involves selecting a goal and then visually servoing
to that goal, recording color and depth data from the vision sensors,
generating and maintaining pose estimates of the object, and using the
estimates along with tactile information received at contact to localize the
contact point on the object. Through this work we show that:

  * •

The optic flow, as observed by the hand mounted cameras, can be used to
predict the heading direction of the hand using image-based techniques rather
than 3D geometry. In our setup frame-to-frame optic flow was dominated by
small changes in orientation of the hand and thus the cameras. Optic flow had
to be integrated across about 10cm of hand travel to be useful.

  * •

A few “mid-course” corrections can correct almost all the error in
trajectories.

  * •

Pose errors, when measured only with the cameras are about $\pm 1.5$ cm and
$\pm 2^{\circ}$ in translation and rotation respectively about a vertical
axis.

  * •

Given these priors, tactile estimation based on a GelSight sensor further
improved the pose estimates to an uncertainty of $\pm 1.5$mm and $\pm
0.5^{\circ}$ in translation and rotation respectively in cases distinct
tactile signals were available.

  * •

Collocated vision is particularly useful when an object does not have
distinctive tactile surface texture, or has repetitive surface texture. We
show that using tactile sensing collocated with vision can help disambiguate
tactile signals when used for localization.

We provide additional details of our work, tables of results and reference
implementations of our algorithms described in the paper here:
https://arkadeepnc.github.io/projects/collocated_vision_touch/index.html

##  II Related work

In this section we provide a survey of related work on visual servoing and
pose estimation using hand-mounted cameras and tactile sensing. Recent work on
addressing these issues has used hand-mounted cameras to demonstrate superior
performance in classical manipulation tasks such as grasping and bin
picking[4]. With the availability of a visual perspective complementary to
external (or head mounted) cameras, researchers have diversified the moving
cameras to serve as tactile devices ([1, 2]) and have implemented delicate
manipulation behaviors (see e.g. [5, 6]). Recent research has also developed
tactile sensors and algorithms for estimating contact pose and inferring
object from contacts [7, 8], tracking object motion by fusing externally
mounted cameras and tactile sensors[9], and transferring information between
external cameras and hand-mounted cameras (see e.g. [10]). A closely related
work by [11] discusses integration of a visual and tactile measurement through
a Bayesian filter.

Vision-based localization and contact prediction: Camera-on-hand or more
generally camera-on-mobile-agent arrangements have been investigated by
several researchers to pursue diverse goals such as visual servoing to a
workspace goal (e.g. [12]), collision avoidance systems on miniature aerial
vehicles (e.g. [13]), and how flying insects, birds, and rapidly moving
animals perceive motion[14]. Literature on quantitative analysis of
looming111Looming or visual looming is defined as the phenomena of an object
getting bigger in the visual field as the relative distance between the
observer and the object decreases. (e.g. [15], [16]) is of particular interest
to us as we try to identify an area in the image space corresponding to the
direction of heading of the robot at any particular moment. Recent research on
optical expansion (e.g. [17]) is focussed on supervised learning, instead of
hand crafted functions (e.g. [15, 16]) to compute dense scene flow from optic
flow to identify relative motion of objects and the agents, and has been
demonstrated to exhibit state of the art performance in identifying objects
heading towards the agent. In the current work we build upon research on optic
flow for scene understanding to identify an area in the robot’s visual field
corresponding to the physical point in the workspace where the robot is
currently headed.

(a)

(b)

(c)

(d)

Figure 2: Figure 2(a) shows our sensor platform which is attached to the robot
manipulator. The sensors, from left to right, are the Intel RealSense L515
LiDAR camera, our modified version of the GelSight and an RGB camera. The RGB
camera and the RealSense are at a distance of 6 cm each from the camera to the
left and right respectively. Figure 2(b) shows the schematic of our modified
version of the GelSight, fig. 2(c) is an instance of the raw data collected by
our GelSight when pressed against an 18mm long 6mm diameter bolt with 1mm
pitch. Through the image, we note the physical scope of the sensor is almost
entirely covered by the 18mm long bolt. We process the raw GelSight data to
yield metric depth and normal maps. The depth of the reconstructed surface is
rendered as a shaded point cloud in 2 views in fig. 2(d).

Tactile localization and contact estimation: Vision sensors can have a wide
“field of view” and are good for making large scale models. A tactile sensor
has a much smaller measurement area (or field of view) and can potentially
capture minute details of the surface it interacts with. Early research on
tactile sensing leveraged this capability, even with a seemingly low
resolution tactile sensor (Weiss Robotics DSA9205), to demonstrate object
recognition using image feature descriptors[18] and, recognize and localize an
articulated object through a sequence of touches (see e.g. [19]). With the
introduction of camera-based higher resolution tactile sensors, most notably
the GelSight (see [2]) and its derivative GelSlim[20], investigations on
tactile object recognition and localization have made significant progress in
tactile sensing driven perception. [21] described tactile localization using
the GelSight sensor using conventional feature based image alignment. [22]
integrated the GelSlim with a gripper and interfaced it with a model based
controller to successfully perform re-grasps of a cable. More recently, [23]
demonstrated tactile localization and shape reconstruction using a GelSlim
sensor, where the authors trained neural networks to generate height maps with
ground truth data obtained from robot experiments with the sensor and known
objects. The trained network was then used to generate height maps of the
surface of an object and the height maps were registered to reconstruct the
object surface. This work was extended by [24] where the authors used a
renderer to generate and cache a large number of possible tactile signals of
objects from a data set touching a tactile sensor (GelSlim in this case) in
different orientations. An actual tactile signal corresponding to a particular
object at a particular pose was then compared with the cache to retrieve
candidate object and pose pairs and the network demonstrated in [23] was then
used to generate a height map which was registered with the candidate object
at the candidate pose to localize contact. In [25] the authors describe a
method to integrate a learned representation of a tactile signals (from a
GelSight in this case) as a factor in a task of estimating object pose with a
stream of subsequent measurements. [26] introduced a new time of flight based
tactile sensor, the soft-bubble, and demonstrate object identification using
learned embeddings and object localization by registering the tactile signal
(obtained as a point cloud by their sensor) with the retrieved geometric model
of the recognized object. Recently, and perhaps closest to the current work,
[27] introduced a method to reconstruct the shape of an object by refining a
coarse prior shape obtained by an RGBD sensor using tactile measurements from
a GelSight attached to a robot. Also related to the current work, Dikhale et
al.[28] demonstrated a method to fuse off-the-shelf learned pose estimators
(PoseCNN [29] in this case) from external RGBD sensors and in-hand tactile
sensor outputs to estimate the poses of grasped objects. In the current work
we build upon literature on contact localization using high resolution tactile
sensors, visual localization and hand mounted cameras to demonstrate a suite
of collocated vision based sensors that can be used to visually servo a robot
to touch and localize objects in the workspace.

##  III Methods

In this section we describe our sensor platform (A) and algorithms to estimate
where the robot hand will go (B1), visually servo the robot to a target
contact point (B2), estimate the pose of the target object (C), and combine
vision and tactile information to estimate both the contact point and refine
the object pose estimate (D).

###  III-A Sensor platform

In this work, we collocate cameras with a camera-based tactile sensor by
putting the cameras and the tactile sensor in close physical proximity while
operating them independently. The sensor platform consists of a GelSight
tactile sensor in the middle (co-incident with the robot wrist’s axis) (figs.
1 and 2), with a camera on either side. We modify the GelSight as described by
[30] in the physical sensor form factor introduced by [2]. We use a LIDAR-
based RGBD sensor (Intel RealSense L515) with a $70^{\circ}$ field of view to
provide depth (figs. 1(a) and 2(a) left) and color images of the workspace,
and a USB camera (a Sony IMX 291 sensor) with a wider $100^{\circ}$ field of
view lens (figs. 1(a) and 2(a) right). We chose these cameras partly because
they were of comparable size to our GelSight sensor, thus ruling out most
other popular RGBD cameras. Also, the USB camera yielded high resolution
images which were better than the color channel of our 3D sensor and helped us
localize small objects. Further details on the sensors can be found in fig. 2.

(a)

(b)

Figure 3: Figure 3(a) shows the surface of the squared magnitude of the
optical flow between a consecutive frame pair in 2 views. We note that this
surface assumes a parabolic shape. The red dot is the minima of the optical
flow surface as identified by our algorithm. Figure 3(b) demonstrates the
usage of our algorithm to correct trajectory errors using both cameras as
shown in fig. 2(a). The X, Y and Z axes are marked in red, green and blue in
fig. 3(b) on the bottom right of the figure

###  III-B Visual servoing with hand mounted cameras

In this section we describe a method to estimate the robot hand’s heading
direction in the workspace and then we describe how to servo to a goal using
that estimate.

####  III-B1 Optical point of expansion from in hand cameras

We process the scene to identify the location of a 3D point corresponding to
the heading direction of the robot in the image space. To achieve this, we
calculate the optical flow between the consecutive frames obtained by the hand
mounted cameras and identify the region in the image from which the optic flow
seems to be emerging (i.e., we look for a portion of the scene which has zero
translation) as the camera moves towards the scene. Assuming that the world
scene is relatively flat (object depth $\ll$ projection depth), the square of
the magnitude of the optic flow at each pixel is roughly distributed as a
parabolic surface (see fig. 3(a)). We calculate the motion field (per pixel
optic flow magnitude and direction) between two consecutive frames using the
OpenCV implementation[31] of the Lucas-Kanade dense optical flow, which solves
for per pixel motions (along horizontal and vertical directions) over the full
image. We also tested the Farenbäck optical flow [32] and the Brox optical
flow [33], and found that the dense Lucas-Kanade optical flow performs
slightly better in computation speed and produces smoother optic flow fields.
The optical point of expansion (POE) is obtained as the minima of the surface
representing the square of the magnitude of the optic flow. We use a robust
algorithm described in algorithm 1 to detect the POE.

Algorithm 1 Calculating the point of expansion

1:procedure calculatePOE (image stream $\\{I\\}$)

2: Make pointers list $\\{L\\}$ for overlapping image tiles

3: for $(I_{t},~{}I_{t-1})\in I$ do

4: $\\{d_{x},d_{y}\\}\longleftarrow$ denseLKOpticalFlow($I_{t},~{}I_{t-1}$)

5: $d_{t}\longleftarrow{d_{x}^{2}+d_{y}^{2}}$,
$\theta_{t}\longleftarrow\arctan(\dfrac{d_{y}}{d_{x}})$

6: $d_{t}\longleftarrow$ normalizeAndHistogramEqualize($d_{t}$)

7: for tile $\in L$ do $\triangleright$ exec. with n workers

8: if Histogram($\theta_{t}$[tile]) is uniform then

9: tile$\rightarrow L_{ang}$

10: end if

11: add one random tile to $L_{ang}$ $\triangleright$ if tracking bad

12: end for

13: for tile $\in L_{ang}$ do $\triangleright$ exec. with n workers

14: fit $\leftarrow$ Least sq. fit paraboloid to $d_{t}$[tile]

15: end for

16: choose $k$ best fits from fit $\rightarrow$ best fits

17: find pixel $P$ of minima of the fit $\in$ best fits

18: POE $\leftarrow$ $P$ with the most uniform Histogram around
$\theta_{t}$[P])

19: end for

20: return list of POEs per frame pair

21:end procedure

The heading direction estimated by the optical flow between consecutive frames
was too noisy to yield meaningful heading estimates. To address this, we
looked at the trajectory correction estimates for each of the on hand cameras
and found their prediction to be very closely correlated (almost equally
incorrect or equally correct), which led us to rule out camera noise and
incorrect robot kinematics including the camera mounts as the cause of the
noise. We concluded that the errors were being caused by small unmeasured
rotations of the robot wrists (play or backlash). Numerical modeling showed
that the expected amount of play in orientation led to optic flow values
comparable to the “noisy” shifts in the POE. We also noted that the noise in
the predicted trajectory errors was centered about zero. We averaged POE
shifts over a robot travel of at least about 10 cm to get usable POE
estimates. For our robot setup. where a maximum of 1m downward travel was
possible, empirically we observed that about 10cm non-overlapping intervals
provided useful trajectory corrections and provided the opportunity for
several corrections as the robot moved to the target. We describe the
experiment that led us to this conclusion in fig. 4.

(a) Int.: 1cm; St.dev.: (284,98)

(b) Int.: 5cm; St.dev.: (146,60)

(c) Int.: 7.5cm; St.dev.: (62,55)

(d) Int.: 10cm; St.dev.: (19,27)

(e) Int.: 15cm; St.dev.: (21,20)

(f) Int.: 20cm; St.dev.: (17,15)

Figure 4: In this experiment, we move the robot vertically down by 65 cm to a
goal location slightly below the yellow cross mark on the handle of the glue
gun. There are no errors in the goal location being tracked for this case.
This experiment is repeated 10 times. The red dots are the predictions of
potential point of contact (calculated as the instantaneous POE). We note that
the predictions are centered about the actual point of contact – a point
slightly below the yellow cross mark on the glue gun. We report 6 cases where
we predict the potential point of contact by looking at various intervals of
the trajectory. We report the interval lengths (in cm) and the standard
deviation in predicting the point of contact (in pixels) as the labels of the
figures. We note that as we increase the length of the interval, the standard
deviation of the prediction decreases (as seen through “clumping” of the
predicted potential points of contact), but the number of possible predictions
decreases (as seen through fewer number of red dots with increasing interval
sizes). This leads us to conclude that the averages across larger temporal
(and spatial) windows produce smoother and more stable error signals (or
correction signals in the case of visual servoing). For our use case,
averaging across 10 cm intervals provided us with “enough” number of
correction signals while having reasonably low variance.

####  III-B2 Correcting trajectory errors using pixel space errors

In the previous section we described a method to identify the image
coordinates of the point in the workspace to which the robot is headed. In
this section we address the problem of correcting trajectory errors using
those predictions. This can be useful when the robot’s trajectory needs
correction and the only information about the updated goal is available in
pixel space – possibly from a hand mounted RGB camera which detected a
movement of the target. We use the following intuition for visual servoing: to
reach a pre-planned goal in the robot’s workspace, the robot, almost always,
needs to move towards that goal, hence the image of the goal point and the
observed POE should be very close. Errors in the pixel space between these
two, indicates an error in the trajectory being followed by the robot, which
we aim to correct. In more precise terms, let us denote the true workspace
goal as $\mathbf{X}_{G}^{W}$ and it’s estimate $\widehat{\mathbf{X}_{G}^{W}}$
in homogeneous 3 space. The robot is initially planned to move to
$\widehat{\mathbf{X}_{G}^{W}}$ and let the POEs obtained for the subsequent
frames be $\mathbf{p}$, in the pixel space. As the camera is registered to the
robot, a given point in time, we can calculate the world to camera projection
matrix as
$\mathbf{P}=\mathbf{K}[\mathbf{R}_{world}^{cam}|\mathbf{t}_{world}^{cam}]$ and
can project $\mathbf{X}_{G}^{W}$ and it’s estimate
$\widehat{\mathbf{X}_{G}^{W}}$ to $\mathbf{x}_{G}$ and
$\widehat{\mathbf{x}_{G}}$ in homogeneous 2 space. If the motion between two
frames captured by the camera-in-hand is mostly perpendicular to the imaging
plane, we can approximate $\widehat{\mathbf{x}_{G}}$ with the point of
expansion $\mathbf{p}$ for each subsequent frames. We also note that, as
$\mathbf{p}$ is in the pixel space, with this approximation, we discard the
“z-buffer” of the projective transformations associated with $\mathbf{x}_{G}$
and as a consequence, cannot correct for a error in the camera’s projective
axis (camera’s z axis). However, this is not a problem for us as the location
of the table is known, and we combine the correction signals from the 2 in
hand cameras to mitigate possible errors due to rejecting the “z-buffer”.
We use the camera projection Jacobian and calculate the task-space error as
eq. 5

$\displaystyle\Delta=\left[\begin{array}[]{ccc}\frac{f_{x}}{Z_{W}}&0&-\frac{X^{W}_{G}f_{x}+Z^{W}_{G}c_{x}}{{Z^{W}_{G}}^{2}}+\frac{c_{x}}{Z^{W}_{G}}\\\
0&\frac{f_{y}}{Z^{W}_{G}}&-\frac{Y^{W}_{G}f_{y}+Z^{W}_{G}c_{y}}{{Z^{W}_{G}}^{2}}+\frac{y}{Z^{W}_{G}}\end{array}\right]^{+}\left[\begin{array}[]{c}p^{u}-\mathbf{x}_{G}^{u}\\\
p^{v}-\mathbf{x}_{G}^{v}\end{array}\right]$ (5)

In eq. 5, $[f_{x},f_{y},c_{x},c_{y}]$ are the camera intrinsics, and we denote
$\mathbf{X}_{G}^{W}=[X^{W}_{G},Y^{W}_{G},Z^{W}_{G}]$ and the $[\cdot]^{+}$
denotes the Moore-Penrose pseudo-inverse. As an example use case, we use the
following procedure to correct trajectory goals as the robot moves
approximately 1m towards the true workspace goal $\mathbf{X}_{G}^{W}$ with a
erroneous initial estimates of upto $\pm 10$cm in each of X and Y directions.

  * •

steps 1–20: Plan a path with 100 waypoints till goal, and execute the first 20
steps.

  * •

steps 21–40: For the next 20 steps, calculate the POE and then use the camera
intrinsics and pose of the camera with respect to the robot base to obtain the
mean error in trajectory space. Correct the trajectory error and plan for the
remaining 60 steps.

  * •

steps 41–60: Execute the current path re-planned at step 41, do not track the
point of expansion or generate new trajectory error estimates.

  * •

steps 61–80: Repeat steps 21–40 while moving through the corrected trajectory,
at the end correct trajectory again and re-plan if necessary.

  * •

steps 81–100: Continue with the last planned trajectory, and stop if goal is
reached to given tolerance.

In fig. 3(b), on the two sides of the robot, we show 2 trajectories where the
robot travels about 1m (vertically) from the start to the final contact
position, and each of the trajectories need a correction of 10cm errors in the
X and Y directions in the robot workspace. We show the initial portions of the
trajectories in yellow, the planned portions in red, and the corrected
portions in green. The final accuracy achieved was within 5mm of the target.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Our pipeline to estimate pose through vision and touch. Figure 5(a)
is the color channel of the RGB-D sensor stream captured at a particular point
in the trajectory. Figure 5(b) is the pose estimate obtained after solving eq.
6, which is refined by solving eq. 8 and we obtain a camera pose estimate
$\hat{\zeta}$ (shown in fig. 5(c)) which is mostly correct in the camera
projection depth and camera yaw. We transfer $\hat{\zeta}$ to the reference
frame of the GelSight and obtain fig. 5(d) showing the relative pose of the
object and the GelSight. This corresponds to the pose $\hat{\zeta}_{GS}$. This
estimate is further refined by minimizing eq. 9 using the GelSight data (fig.
5(e)) obtained at contact, and we obtain the final pose shown in fig. 5(f)

###  III-C Pose estimation through vision

In this section we discuss visual localization of an object with known 3D
geometry. At least in the early stages of approach, the hand-mounted cameras
can see all or large portions of the target object, and standard image based
registration methods ([34, 35]) can be used to estimate the object’s pose
relative to the robot hand. This is quite different from the situation with
tactile sensing where only a small part of the object is visualized. As is
common in the whole-object-visible camera-based pose estimation literature
(see e.g. [34, 35] and [36]), we decompose pose estimation into two parts –
coarse pose estimation by aligning the centroids and edge moments, and finer
alignment using a distance-based cost applied densely.
As a prerequisite for this part we need the edge pixels of the object in the
image and for this we use either the depth edges on the object if available or
else use the image gradient edges of the object – we use a Canny edge finder
for this purpose. Let us denote this binary edge image as $\mathbf{I}_{S}$.
Next, we formulate an optimization problem to identify an object pose that
produces the most similar edge distribution to $\mathbf{I}_{S}$.
To do this, we generate an initial guess for $\theta$ (the angle of rotation
about the camera projection axis) for the object from the principal components
of the camera image and if an aligned depth map is available, we use its mean
to initialize the $z$ component (distance between the camera and object along
the camera projection axis), or else we initialize the $z$ depth arbitrarily.
The $x$ and $y$ directions along the image plane are initialized by back-
projecting the centroid of the edge pixels using the camera intrinsics and the
initial value of $z$. With the initial guess $\omega=[x,y,z,0,0,\theta]$, we
use a differentiable renderer (we use a modified version of the DIRT renderer
from [37]) to render the mesh model of the object, extract the corresponding
edge image $\mathbf{I}_{R}(\omega)$ and solve the following minimization to
obtain a rough pose estimate from the camera image. For each $\mathbf{I}_{R}$,
we identify the image edge pixels $\mathbf{p}_{R}^{i}$ and $\mathbf{p}_{S}$
for $\mathbf{I}_{R}$ and $\mathbf{I}_{S}$ respectively and minimize the
following sparse edge matching cost $\mathbb{E}_{cs}$ in eq. 6

$\displaystyle\mathbb{E}_{cs}(\omega)$
$\displaystyle=\gamma\left[||\bar{\mathbf{p}_{R}}(\omega)-\bar{\mathbf{p}_{S}}||_{2}\right]+(1-\gamma)$
(6)
$\displaystyle\langle\mathcal{V}\left(\mathbf{p}_{R}(\omega)-\bar{\mathbf{p}_{R}}\right)\cdot\mathcal{V}\left(\mathbf{p}_{S}-\bar{\mathbf{p}_{S}}\right)\rangle$

where, $\mathcal{V}(\mathbf{p})$ is the direction of largest variance of the
mean centered point set $\mathbf{p}\in\mathbb{R}^{2}$, given by the
eigenvector corresponding to the maximum eigenvalue,
$\langle\cdotp,\cdotp\rangle$ is the dot product between vectors and $\gamma$
is a weighting factor. Minimizing eq. 6 aligns the centroid and approximately
recovers the angle of rotation along the camera projection axis.
The expression for $\mathbb{E}_{cs}$ does not admit automatic gradient
calculation due to the non-differentiable selection of pixel indices to obtain
$\mathbf{p}_{R}$ from $\mathbf{I}_{R}$, therefore, we obtain a finite
difference gradient using central differences. We minimize
$\mathbb{E}_{cs}\forall\theta_{i}$ and obtain the candidate pose parameters
$\hat{\omega}=\\{X_{cs},Y_{cs},Z_{cs},\theta_{cs}\\}$ in the camera coordinate
frame, corresponding to the minimum $E_{cs}$. From fig. 5(b) we note that
minimization of $\mathbb{E}_{cs}$ is not expected to solve for the projection
depth, as it only recovers the orientation of the camera and not the
projection depth. In the next part, we solve the projection depth by
minimizing a modified version of the dense differentiable cost using the
directional chamfer matching energy as discussed in [34] and [35] (eq. 7).

$\displaystyle\mathbb{E}_{cm}(\mathbf{I}_{S},\mathbf{I}_{R}^{\zeta})$
$\displaystyle=\sum_{{}^{\xi}\mathbf{p}_{s}^{i}\in^{\xi}\mathbf{I}_{S}}\left[\min_{{}^{\xi}\mathbf{p}_{R}^{j}\in^{\xi}\mathbf{I}_{R}(\zeta)}||^{\xi}\mathbf{p}_{S}^{i}-^{\xi}\mathbf{p}_{R}^{j}(\zeta)||\right]$
(7)

In eq. 7, the outer sum
$\sum_{{}^{\xi}\mathbf{p}_{s}^{i}\in^{\xi}\mathbf{I}_{S}}(\cdot)$ implements
the edge awareness by binning the edges according to their orientation (as
quantized by $\xi\in[-\pi,\pi]$), and implicitly assigning edge pixel
correspondences. We observe that the inner minimization problem
$\min_{{}^{\xi}\mathbf{p}_{R}^{j}\in^{\xi}\mathbf{I}_{R}(\zeta)}||^{\xi}\mathbf{p}_{S}^{i}-^{\xi}\mathbf{p}_{R}^{j}(\zeta)||$
for each ${}^{\xi}\mathbb{E}_{cm}$ can be solved by the Euclidean distance
transform. As the ${}^{\xi}\mathbb{E}_{cm}$ cost is cumulative over the
${}^{\xi}\mathbf{I}_{S}$ image, the cost boils down to the pixel-wise sum of
absolute differences between the Euclidean distance transforms (EDT) of images
${}^{\xi}\mathbf{I}_{S}$ and ${}^{\xi}\mathbf{I}_{R}(\zeta)$. So using the
definition of Euclidean distance transform from [38] in eq. 7, we simplify our
dense edge matching energy as

$\displaystyle\mathbb{E}_{cm}(\zeta)$
$\displaystyle=\sum_{\xi}\left[\sum_{\mathcal{G}}\left[|\rm{EDT}(^{\xi}\mathbf{I}_{S})-\rm{EDT}(^{\xi}\mathbf{I}_{R}(\zeta))|\right]\right]$
(8)

We minimize this function with gradient descent to obtain a coarse pose
estimate $\hat{\zeta}$ from the camera image and transfer the pose estimate to
the GelSight camera frame as $\hat{\zeta}_{GS}$. In contrast to [35, 34], we
implemented modified and differentiable versions of the matching costs and
thus, our gradient steps are about 80% faster than the reference
implementations of [35, 34] for the same size of the image. In practice, to
keep the computational cost low, we maintain a coarse pose estimate using eq.
6 throughout the major part of the trajectory and switch to eq. 8 at a point
beyond which the objects are de-focussed. For objects in figs. 1, 5 and 6 this
distance was around $15$cm and for objects in fig. 7 it was $10$cm.

###  III-D Precise contact pose estimation through touch

We found that our vision-based localization had errors on the order of a
centimeter, due to our algorithms imperfect camera calibration, local minimums
in matching, and sometimes a lack of visual features to match or track. In
addition, as the hand comes near the object, the depth and image sensor
measurements become unusable – the LiDAR based sensor provides reliable depth
estimates at distances greater than 25cm, and the camera image measurements
were usable at distances greater than 12-15cm to the object, after which the
items of interest often go out of view and the image becomes too blurry to
extract high quality edges needed by our 3D pose estimation algorithms. We
find that using the tactile image on contact can improve contact point and
pose estimation. In fig. 2 we noted that we could generate metrically correct
depth and normal maps of the deformed GelSight surface at contact. In this
section, we use that information, along with the pose estimated from the
previous section to localize contact, given that we know the geometry of the
object. To achieve this, we use multi-scale dense depth and normal map
alignment to obtain the pose of object with respect to the tactile sensor.
We capture the tactile image in our GelSight’s native camera resolution of
$(640\times 480)$ pixels and obtain depth and normal maps. We then decompose
each of the normal and depth maps into 4 lower pyramid levels. We denote these
5 normal and depth maps of the source image as $N_{S}$ and $D_{S}$
respectively. Next, we render the object (using [37]) through the GelSight’s
viewport using the pose estimated in the previous section and obtain normal
and depth maps corresponding to the frame sizes of $N_{S}$ and $D_{S}$
respectively.
We note here that this is not an exact simulation of the GelSight sensor
through our renderer – the ideal GelSight should only measure objects touching
it i.e. $\sim$25mm from the camera. Anything beyond or closer than that is
either not touching the sensor or interfering with it. We relax this
requirement, and also neglect the effects of the soft body contact between the
gel and the object to get a better basin of convergence in the optimization
problem. We denote these sets of rendered depth maps by $N_{R}(\hat{\zeta})$
and $D_{R}(\zeta)$. Our alignment cost function $\mathbb{E}_{gs}(\zeta)$ for
the GelSight data is

$\displaystyle\mathbb{E}_{gs}(\zeta)=\sum_{i=1}^{5}\left[\left|D_{S}^{i}-D_{R}^{i}(\zeta)\right|+\left[1-\langle
N_{S}^{i}\cdot N_{R}^{i}(\zeta)\rangle\right]\right]$ (9)

where $\langle N_{S}^{i}\cdot N_{R}^{i}\rangle$ denotes the pixel-wise dot
products of the normal maps. We solve this with $\hat{\zeta}_{GS}$ as the
starting guess using gradient descent. Figure 5 describes the steps discussed
above.

##  IV Results

In this section we present the results of localizing contacts using the sensor
setup and the algorithms discussed in this work through 6 scenarios. For the
experiments presented below, we use a black background to simplify the object
segmentation and edge detection. Except for the glue gun, all the other
objects had reflective parts and were painted matte white to remove
specularities.

####  IV-1 Localization with color and depth edges

Our RGBD sensor captures the depth of the scene reliably from 30 cm away and
can produce aligned depth and color images. As the depth images are unchanged
by changes in lighting and texture, extracting edges from depth images and
maintaining coarse pose estimates as the robot moves closer to the object is a
viable strategy to generate initial pose estimates which are later refined to
estimate the pose at contact. For the objects described in figs. 1, 5 and 6,
when the depth sensor output degraded (at 20 cm) we transferred the pose
estimates obtained (and maintained) using the depth image, to the color image.
refined the pose estimates from the final color image using the tactile image
to obtain the pose at contact.

(a)

(b)

(c)

Figure 6: Figure 6(a) is the pose estimated using the color channel image.
This pose estimate was initialized with coarse pose estimates obtained using
depth images. Figure 6(b) is the GelSight data obtained at contact and fig.
6(c) is the camera pose at contact. Figures 1 and 5 contain similar results.

####  IV-2 Localizing small, flat and thin objects

Depth cameras don’t capture much in situations where an object has little
depth variation. The limited field of view of most depth cameras means they
lose sight of where the GelSight sensor will land as the hand approaches an
object. For these reasons we collocated a high resolution large field of view
RGB camera with the GelSight sensor, so the system would also work well with
small thinner objects with less depth variation. In fig. 7 we demonstrate the
localization of a slender metallic pin and a 4cm square circuit board.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7: Figures 7(a), 7(b) and 7(c) are the steps in localizing a slender
metallic object, figs. 7(d), 7(e) and 7(f) are the steps in localizing a
miniature circuit board. L-R we generate a coarse pose estimate with just the
color image, receive the GelSight image on contact and finally obtain the
camera pose on contact. The object in fig. 7(a) is 10 cm long and 8.5 mm in
diameter at the thickest part. The circuit board in fig. 7(d) is 38mm $\times$
38mm square.

####  IV-3 Disambiguating tactile measurements with vision

With collocated vision, we can disambiguate between repetitive surface
features, which would not have been possible with just tactile sensing. To
show this, we set up an experiment (fig. 8) where the robot approaches the box
cutter from above and touches around the middle of the slider. The tactile
image recorded is shown in fig. 8(b). The pose estimates obtained from the
camera when transferred to the GelSight yield useful initial guesses which can
then be refined with the tactile data to localize the contact.

(a)

(b)

(c)

Figure 8: Collocated vision can help us disambiguate between repeated tactile
features. In this experiment we touch the middle of the set of teeth and could
localize the contact using our pipeline.

####  IV-4 Estimating object pose with vision and touch

Object (exp. name) | $\Delta T_{x}^{v}$ | $\Delta T_{y}^{v}$ | $\Delta T_{z}^{v}$ | $\Delta R_{x}^{v}$ | $\Delta R_{y}^{v}$ | $\Delta R_{x}^{v}$ | $\Delta T_{x}^{t}$ | $\Delta T_{y}^{t}$ | $\Delta T_{z}^{t}$ | $\Delta R_{x}^{t}$ | $\Delta R_{y}^{t}$ | $\Delta R_{x}^{t}$
---|---|---|---|---|---|---|---|---|---|---|---|---
Box cutter slider (5) | 2.5672 | 1.7059 | 5.4776 | 0.5199 | 0.3862 | 0.1761 | 0.7174 | 0.839 | 0.6012 | 0.5031 | 0.3692 | 0.1842
Box cutter teeth (8) | 0.6949 | 0.5968 | 3.7251 | 0.131 | 0.1059 | 0.3534 | 6.7531 | 1.9073 | 0.927 | 0.1433 | 0.0667 | 0.3014
Folding knife (6) | 2.3934 | 3.4957 | 8.4574 | 0.3459 | 0.2366 | 0.2701 | 0.8343 | 1.4932 | 2.4594 | 0.3543 | 0.1062 | 0.2392
Glue Gun (1(f)) | 1.0771 | 1.0904 | 8.3263 | 0.1114 | 0.1547 | 0.2233 | 2.1286 | 2.2714 | 1.8284 | 0.081 | 0.1175 | 0.1628
Camera circuit (7(f)) | 0.2987 | 2.2541 | 2.2718 | 0.393 | 0.2401 | 2.1186 | 0.8923 | 0.3378 | 1.0733 | 0.3518 | 0.2725 | 0.5157
Metallic object (7(c)) | 0.6863 | 0.76 | 0.6261 | 0.0019 | 0.0012 | 0.0473 | 0.3077 | 1.4643 | 0.0132 | 0.002 | 0.0057 | 0.0452
TABLE I: Relative uncertainties in localization using only vision and tactile
sensing augmented by vision, in case good tactile signals are guaranteed.
$\Delta T_{x}$ is the standard deviation in localization along the $x$ axis of
the robot and $\Delta R_{x}$ is the standard deviation in orientation along
the $x$ axis of the robot. Same naming convention applies to the $y$ and $z$
axes. The translation uncertainties are in mm. and the rotation uncertainties
are in degrees. Superscript $[\cdot]^{v}$ indicates localization using vision
only and $[\cdot]^{t}$ indicates localization using vision and touch. Lower
values are better in all cases. The position uncertainties are in mm and the
angle uncertainties are in degrees.

In this section we present our results on localizing objects with vision and
touch, and evaluate the performance of our localization pipeline. We focus on
the case where the contact point has a unique arrangement of useful tactile
features. This is the best case for a combined approach. In practice, if the
tactile sensor contacts a featureless area, we can detect and ignore the
output of the tactile sensor. Most work in this area specializes for a set of
objects used in a training set for a learning approach. We did not find work
that combined a camera with a high resolution tactile imaging sensor for
localizing objects upon contact, so there isn’t an obvious method to compare
to. This along with our choice of smaller and highly textured objects preempts
easy comparison with recent learned localization approaches (e.g. [23, 24, 29]
etc.). Instead, we present overall results of accuracy experiments below.
For each of the 6 objects used in this work (figs. 5, 8, 7(c), 6, 1(f) and
7(f)), we fix the object to the robot table. Assuming that there is zero error
in position control of our robot (we use a Universal Robots UR5E manipulator
fixed to a vention.io table of recommended design for the same robot), we
register the object with respect to the robot base and treat this pose as our
ground truth. Next, we move the robot vertically 1 m above the object and move
the robot down to touch the object at a chosen point that will yield good
tactile information, and localize the object with respect to the robot. We
repeat this 3 times for the same object positions and repeat this experiment
for 2 more positions of the object with respect to the robot – i.e. localizing
each object 9 times with respect to the robot. Fixing the objects is a
restrictive assumption in the context of localizing objects especially with
touch, however, to ensure repeatability of the experiments reported in the
section, we had to fix the objects to a rigid base. Following [34, 35] we
report the repeatibility of our pose estimation pipeline as the measure of its
performance.
Using tactile sensing the localization errors were brought down to $\pm 1.5$mm
in translation and $\pm 0.5^{\circ}$ in rotation from about 1.5cm in
translation and $2^{\circ}$ in rotation using only vision. However, for cases
where the tactile features were not unique, e.g. the box cutter teeth (fig. 8)
and the metallic object (fig. 7(c)), the tactile sensing actually increased
the localization errors in the horizontal directions. The order of these
errors were equivalent to the scale of the repeated features – 5mm for the
experiment described in fig. 8(c) (the box-cutter teeth are about 3mm wide
placed in intervals of 5mm) and about 2mm for the experiment described in fig.
7(c) (the embossed features are very similar at intervals of 3.5 mm). This
observation is consistent with the fact that the final gradient descent step
(eq. 9) to refine the camera based pose estimates will converge to the wrong
local minima if the tactile measurements are not distinctive enough. However,
for objects with rich tactile features, using tactile sensing assisted with
vision provides better localization than exclusively using either as we show
in section IV-6. We repeated a subset of the experiments reported above where
we first corrected the robot trajectory using the procedure described in
section III-B2 – and observed similar localization performance. We present the
results of the experiment reported in this section in table I.

####  IV-5 Effect of points of contact on localization accuracy

In this section we present the effect of randomly selected contacts for
localization. For this set of experiments, we fix each of the objects and the
black background plate used in section IV-4 to a graduated compound slide
capable of in plane translation and rotation. We then moved the robot
vertically down to make contact at the same location on the object used for
the experiments reported in section IV-4 to generate a starting pose. Next, we
generated 5 random configurations per object in translation and orientation on
the plane of the table and moved the robot vertically down to touch the object
and attempted to recover the randomly generated pose perturbations we
introduced. For each of the objects, as expected, we observed similar errors
in localization using only vision as reported in section IV-4. For the box
cutter (fig. 5) most of the contacts yielded useful tactile signals so the
errors in recovering the perturbations in pose were in the range reported in
section IV-4 – i.e. $\sim 8$mm in translation and $\sim 1.5^{\circ}$ in
rotation. This observation was also consistent for the smaller textured
objects3 (fig. 7). However, tactile sensing was not always helpful in
localizing the objects – for the glue gun (fig. 1) and the folding knife (fig.
6, significant parts of the object were featureless and the tactile signals
obtained when touched at these parts were unusable in localizing the objects
as the final gradient descent step (eq. 9 in section III-D) re-introduced
localization errors of about $3$-$4$ cm and $15^{\circ}$ by converging to
incorrect poses. We report the results of the experiments described in this
section in table II.

Object (exp. name) | Ground Truth (x,y) | Vision only (x,y) | Vision + Touch (x,y) | Comment
---|---|---|---|---
Box cutter best case | 41 (0, 41) | 50 (11, 49) | 45 (5, 45) | –
Box cutter worst case | 78 (0, 78) | 89 (40, 79) | 55 (41, 35) | poor tactile signal due to interference
Metallic pin cutter best case | 50 (0, 50) | 40 (5, 40) | 53 (6, 52) | –
Metallic pin cutter worst case | 61 (0, 60) | 65 (8, 64) | 89 (10, 88) | partial contact between object and table
Glue gun best case | 24 (24, 0) | 15 (12, 8) | 26 (25, 8) | –
Glue gun worst case | 26 (24, 10) | 21 (20, 6) | 63 (60, 23) | tactile sensor touched flat part
Monkey best case | 13 (6, 12) | 9 (7, 6) | 15 (8, 13) | –
Monkey worst case | 14 (14, 4) | 18 (17, 6) | 36 (10, 35) | tactile sensor touched edge of arm and part of table
TABLE II: Best and worst case localization errors when the contacts are
selected at random. The ground truth measurements are calculated from the
starting pose by reading the changes in X and Y translations on the graduated
compound slide, the third column reports the magnitude of the translation
recovered using vision only, and the fourth column reports the same using
vision and touch. The X and Y components of the motion are indicated inside
the brackets. We note that, even in the worst case, including the tactile
measurements did not predict the contact in the wrong direction. In the last
column, we provide possible explanations for the worst case errors as
comments. All measurements have been rounded to mm. The errors in orientation
are within $2^{\circ}-3^{\circ}$ for the cases when the pose estimation was
successful and have not been reported. Some of the successful localization
results can be viewed on the paper website.

####  IV-6 Localizing contact using only vision

We explored two approaches for predicting contact with only vision – an image-
based approach that tracks the POE similar to visual servoing and a 3D
geometric approach that can estimate contact based on a single image captured
before the object gets out of focus or gets mostly out of the frame.
In the first case, we identify the POE as described in section III-B1. Based
on the known offset between the camera and the Gelsight, we can identify the
predicted GelSight contact point in any camera image. This approach works best
when the hand is moving straight to the contact point (no trajectory
corrections or going around obstacles), there are a lot of visual features to
track, and the hand starts high enough so there is little offset of the object
and the background caused by the different distances to the object and the
background. We tested performance with the data from the previous section for
objects in figs. 5, 8 and 1(f). We could predict the location of contact with
1cm accuracy.

In the second case, we took the final usable image in these trajectories and
used the procedure described in section III-C to obtain the pose of the object
with respect to the RGB camera. Next, we transformed the pose of the object to
the frame of the GelSight camera and simulated moving blindly until the
vertical distance between the object and the GelSight is 3cm (the gel surface
is 2.5 cm away from the GelGight camera), and reported the location of
contact. We compared these estimates of the point of contact for all the
objects we used. We found that for experiments with larger objects (as shown
in figs. 5, 8, 6 and 1(f)) the average error between the estimated true
location of contact and the location estimated by blindly moving downwards in
the frame of the GelSight camera was 3.5 cm, whereas the same metric for
experiments with smaller objects (as shown in fig. 7) were about 1.5 cm. The
experiments with the smaller objects have lower errors because the distance
between the GelSight and the object at the point the pose of the object was
measured with only vision was much lower (8-10 cm) than the experiments with
the larger object where the distance was about 25-30 cm, so the length of the
blind descent was smaller for the smaller objects and hence the accumulated
error in predicting the point of contact was smaller.

To localize contact with only single instances of touch, To localize contact
with only a single tactile image, we obtained a normal map and point cloud
from the tactile image. We rendered a mesh model of the object (e.g. fig.
8(c)) in a stable configuration, with the surface being touched facing the
camera. The mesh is colored according to the normals with respect to a
simulated camera, which is looking at the object vertically. Then we captured
a high resolution image of the colored mesh and used ORB and SIFT features
with FLANN based matching (ratio = 90%) to find correspondences between our
high resolution image of the rendered mesh and the normal map from the image.
We also tried matching the raw images captured by the sensor (e.g. fig. 8(b))
with the corresponding image of the geometry (e.g. fig. 8(c)). Finally, we
rendered the mesh as a point cloud and used point cloud feature matching
algorithms (B-SHOT[39] and FPFH[40]) to generate correspondences between the
template object and the tactile measurement processed as a point cloud, and
identify the point of contact on the object. None of the above experiments
were able to generate reliable correspondences between the object model and
the tactile image. We observe that in the absence of good initial pose
estimates from an external sensor or learned embeddings (e.g. [24]), the
portion of the object observed by the GelSight is often not large enough for
conventional feature matching algorithms to work. Additionally, the GelSight
elastomer mechanically operates as a nonlinear low pass spatial filter and
does not preserve the true shape of depth discontinuities. Cliffs become
gradual slopes, and the inverted surface is filtered quite differently. As a
result, point cloud feature matching often fails to match the sharp surface
features from the object model to the data captured by the tactile sensor. The
GelSight sensor elastomer mechanics need to be simulated to provide a better
template to match. To localize contact with only a single tactile image, we
attempted to match the processed tactile image obtained from the modified
GelSight (fig. 2(b)) to a representation of the object model. First, we
treated the tactile signal as an image (fig. 2(c)) and extracted image
features (SIFT and ORB) and attempted to match those features with a canonical
image of the object mesh model. Next, we processed the raw GelSight data into
a point cloud (fig. 2(d)) and used local point cloud feature matching
algorithms (B-SHOT[39] and FPFH[40]) between the processed tactile signal and
the object mesh (e.g. fig. 5(f)). None of the above experiments were able to
generate reliable correspondences between the object model and the tactile
image which leads us to conclude that in the absence of good initial pose
estimates from an external sensor or learned embeddings (e.g. [24]), the
portion of the object observed by the GelSight is often not large enough for
conventional feature matching algorithms to work.

##  V Discussion and future work

We focused on objects with significant surface tactile texture that our small
tactile sensor could image. This led us to not use existing sets of objects
for manipulation research (YCB[41], McMaster[42]). This different choice is
due to the absence of a dataset to benchmark tactile imaging and vision
working together. It is unclear what objects should be in the dataset, how
approaches trained on such a dataset would generalize to novel objects and how
much effort is needed to introduce new objects into the dataset and pipelines
based on them. In our approach, we used an off the shelf 3D scanner (EinScan-
SE[43]) to generate 3D models of objects and reference poses for our initial
estimates (see section III-C). We do expect that versions of the current
state-of-the-art edge based localization pipelines (what we use) to be
inherently slower than learned pipelines for generating initial pose estimates
(e.g. [24] or [29]), but we believe that our localization pipeline would
perform well for objects we have not tested here. Also, in the current
literature, we did not find explicit or implicit feature transforms invariant
across tactile images (or processed tactile data) and visual images. Although
there exists work on learning features (see [23, 24, 10]), using them for
explicit correspondences to match tactile data to visual data is relatively
unexplored and we consider promising future work.
While integrating visual sensors of different capabilities, we noted that
having optical sensors with almost co-incident optical axes would trivially
solve some of the issues we faced while localizing small objects and
disambiguating possible localizations. Collocating an ensemble of cameras of
different capabilities (similar to smart phone multi-camera systems) creating
a synthetic vision system is a natural next step of this work. Such a system
could create a virtual fovea focused on where the tactile sensor could contact
the object and bring down the overall footprint of the sensor setup so that it
can be efficiently collocated with a standard gripper. Using a gripper with
the ensemble of sensors is also future work. The GelSight data can be
inexpensively processed to obtain surface normals of objects with respect to a
fixed frame. This is valuable for pose estimation and should be emphasized in
future object shape representations and manipulation algorithms. Naturally,
computing object surface normals by controlling illumination at the scale of
the robot workspace is also future work.

##  VI Conclusion

In this work we show that collocating cameras with tactile sensors on a robot
hand in many cases enables tactile localization to work where vision only or
tactile sensing only localization may perform badly or fail. We demonstrated
this with common objects without substantial pre-computation and without using
a known and fixed set of object models (learning). Vision can also help
disambiguate tactile image matching, especially in cases of limited,
repetitive, or otherwise ambiguous tactile features. Tactile sensing almost
always improves visual localization as well. We also found that using optic
flow from hand mounted cameras had to be integrated across about 10cm of
camera travel to provide useful heading estimates, and could be used to
correct trajectories.

## References

  * [1] A. Yamaguchi and C. G. Atkeson, “Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables,” in _2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)_. IEEE, 2016, pp. 1045–1051.
  * [2] W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” _Sensors_ , vol. 17, no. 12, p. 2762, 2017.
  * [3] “Detectron: FAIR’s platform for Object Detection Research.”
  * [4] S. Song, A. Zeng, J. Lee, and T. Funkhouser, “Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations,” _IEEE Robotics and Automation Letters 2020_ , vol. 5, no. 3.
  * [5] A. Yamaguchi and C. G. Atkeson, “Implementing tactile behaviors using fingervision,” in _2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids)_. IEEE, 2017, pp. 241–248.
  * [6] W. Yuan, M. A. Srinivasan, and E. H. Adelson, “Estimating object hardness with a gelsight touch sensor,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 208–15.
  * [7] S. Wang, J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson, “3D shape perception from monocular vision, touch, and shape priors,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 1606–13.
  * [8] E. J. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal, “3D shape reconstruction from vision and touch,” _arXiv preprint arXiv:2007.03778_ , 2020.
  * [9] G. Izatt, G. Mirano, E. Adelson, and R. Tedrake, “Tracking objects with point clouds from vision and touch,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017.
  * [10] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba, “Connecting touch and vision via cross-modal prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019_.
  * [11] S. Luo, W. Mou, K. Althoefer, and H. Liu, “Localizing the object contact through matching tactile features with visual map,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_.
  * [12] R. Kelly, R. Carelli, O. Nasisi, B. Kuchen, and F. Reyes, “Stable visual servoing of camera-in-hand robotic systems,” _IEEE/ASME Transactions on Mechatronics_ , vol. 5, no. 1, pp. 39–48, 2000.
  * [13] T. Mori and S. Scherer, “First results in detecting and avoiding frontal obstacles from a monocular camera for micro unmanned aerial vehicles,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 1750–1757.
  * [14] J. R. Serres and F. Ruffier, “Optic flow-based collision-free strategies: From insects to robots,” _Arthropod structure & development_, vol. 46, no. 5, pp. 703–717, 2017\.
  * [15] D. Raviv, _A quantitative approach to looming_. US Department of Commerce, National Institute of Standards and Technology, 1992.
  * [16] D. Raviv and K. Joarder, “The visual looming navigation cue: a unified approach,” _Computer Vision and Image Understanding_ , 2000.
  * [17] G. Yang and D. Ramanan, “Upgrading optical flow to 3d scene flow through optical expansion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [18] S. Luo, W. Mou, K. Althoefer, and H. Liu, “Novel tactile-sift descriptor for object shape recognition,” _IEEE Sensors Journal_ , vol. 15, no. 9, pp. 5001–5009, 2015.
  * [19] ——, “Iterative closest labeled point for tactile object shape recognition,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 3137–3142.
  * [20] E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez, “Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 1927–1934.
  * [21] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson, “Localization and manipulation of small parts using gelsight tactile sensing,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 3988–3993.
  * [22] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson, “Cable manipulation with a tactile-reactive gripper,” 2019.
  * [23] M. Bauza, O. Canal, and A. Rodriguez, “Tactile mapping and localization from high-resolution tactile imprints,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019.
  * [24] M. Bauza, E. Valls, B. Lim, T. Sechopoulos, and A. Rodriguez, “Tactile object pose estimation from the first touch with geometric contact rendering,” _arXiv preprint arXiv:2012.05205_ , 2020.
  * [25] P. Sodhi, M. Kaess, M. Mukadam, and S. Anderson, “Learning Tactile Models for Factor Graph-based State Estimation,” _arXiv preprint arXiv:2012.03768_ , 2020.
  * [26] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake, “Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation,” in _2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)_. IEEE, 2019, pp. 597–604.
  * [27] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess, “Efficient shape mapping through dense touch and vision,” _arXiv preprint arXiv:2109.09884_ , 2021.
  * [28] S. Dikhale, K. Patel, D. Dhingra, I. Naramura, A. Hayashi, S. Iba, and N. Jamali, “VisuoTactile 6D Pose Estimation of an In-Hand Object using Vision and Tactile Sensor Data,” _IEEE Robotics and Automation Letters_ , 2022.
  * [29] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes,” _arXiv preprint arXiv:1711.00199_ , 2017.
  * [30] M. K. Johnson, F. Cole, A. Raj, and E. H. Adelson, “Microgeometry capture using an elastomeric sensor,” _ACM Transactions on Graphics (TOG)_ , vol. 30, no. 4, pp. 1–8, 2011.
  * [31] OpenCV, “Open Source Computer Vision Library,” 2021.
  * [32] G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in _Scand. conf. on Image anal._ Springer, 2003.
  * [33] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in _European conference on computer vision_. Springer, 2004, pp. 25–36.
  * [34] M.-Y. Liu, O. Tuzel, A. Veeraraghavan, Y. Taguchi, T. K. Marks, and R. Chellappa, “Fast object localization and pose estimation in heavy clutter for robotic bin picking,” _The International Journal of Robotics Research_ , vol. 31, no. 8, pp. 951–973, 2012.
  * [35] M. Imperoli and A. Pretto, “$\mathrm{D^{2}CO}$: Fast and Robust Registration of 3D Textureless Objects Using the Directional Chamfer Distance,” in _International conference on computer vision systems_. Springer, 2015.
  * [36] C. Choi and H. I. Christensen, “3d textureless object detection and tracking: An edge-based approach,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012.
  * [37] P. Henderson and V. Ferrari, “Learning single-image 3d reconstruction by generative modelling of shape, pose and shading,” _International Journal of Computer Vision_ , pp. 1–20, 2019.
  * [38] P. F. Felzenszwalb and D. P. Huttenlocher, “Distance transforms of sampled functions,” _Theory of computing_ , pp. 415–428, 2012\.
  * [39] S. M. Prakhya, B. Liu, W. Lin, V. Jakhetiya, and S. C. Guntuku, “B-SHOT: a binary 3D feature descriptor for fast Keypoint matching on 3D point clouds,” _Autonomous Robots_ , vol. 41, no. 7, 2017.
  * [40] R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (FPFH) for 3D registration,” in _2009 IEEE international conference on robotics and automation_. IEEE, 2009, pp. 3212–3217.
  * [41] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Yale-CMU-Berkeley dataset for robotic manipulation research,” _The International Journal of Robotics Research_ , vol. 36, no. 3, pp. 261–268, 2017.
  * [42] E. Corona, K. Kundu, and S. Fidler, “Pose estimation for objects with rotational symmetry,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 7215–7222.
  * [43] “Einscan-SE.” [Online]. Available: https://www.einscan.com/desktop-3d-scanners/einscan-se/


# SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing
Mixture of Experts

###### Abstract

Recently, Mixture of Experts (MoE) based Transformer has shown promising
results in many domains. This is largely due to the following advantages of
this architecture: firstly, MoE based Transformer can increase model capacity
without computational cost increasing both at training and inference time.
Besides, MoE based Transformer is a dynamic network which can adapt to the
varying complexity of input instances in real-world applications. In this
work, we explore the MoE based model for speech recognition, named SpeechMoE.
To further control the sparsity of router activation and improve the diversity
of gate values, we propose a sparsity $L1$ loss and a mean importance loss
respectively. In addition, a new router architecture is used in SpeechMoE
which can simultaneously utilize the information from a shared embedding
network and the hierarchical representation of different MoE layers.
Experimental results show that SpeechMoE can achieve lower character error
rate (CER) with comparable computation cost than traditional static networks,
providing 7.0%$\sim$ 23.0% relative CER improvements on four evaluation
datasets.

Index Terms: mixture of experts, dynamic routing, acoustic model, speech
recognition

11footnotetext: Equal contribution.

##  1 Introduction

Owing to powerful representation, Deep Neural Networks (DNN) have gained great
success in speech recognition [1, 2]. Various types of neural network
architectures have been employed in ASR systems, such as convolutional neural
networks (CNNs) [3, 4], long short-term memory (LSTM) [5], gated recurrent
unit[6], time-delayed neural network [7], feedforward sequential memory
networks (FSMN) [8], etc. Recently, more powerful deep models such as
Transformer[9], Emformer[10] and Conformer[11] have proved their efficacy to
further improve the speech recognition performance.

Increasing model and training data size has been shown an effective way to
improve the system performance, which is especially demonstrated in the field
of language modeling [12, 13]. Recently, deep mixture of experts (MoE) based
approaches [14, 15] have been intensively investigated and applied in
different tasks such as language modeling [16, 17] and image
classification[18, 19, 20, 21]. The benefits mainly come from two aspects:
First, MoE is an effective way to increase model capacity. Second, with
introduction of the sparsely-gated mixture-of-experts layer [22], an
attractive property of MoE models is the sparsely dynamic routing, which
enables us to satisfy training and inference efficiency by having a sub-
network activated on a per-example basis.

In real-world applications, speech recognition systems need to be robust with
different input conditions such as speakers, recording channels and acoustic
environments. Larger models are appealing while the increase of training and
inference cost can not be afforded. The major problem is that the computation
cost of a static model is fixed and can not be adaptive to the varying
complexity of input instances. Therefore, developing mixture of expert models
for speech recognition with dynamic routing mechanism is a promising
exploration.

In this study, we explore mixture of experts approach for speech recognition.
We propose a novel dynamic routing mixture of experts architecture, similar to
[17], which comprises of a set of experts and a router network. The router
takes output of the previous layer as input and routes it to the best
determined expert network. We find that the balance loss proposed in [17]
achieves balanced routing but the sparsity of router activation can not always
be guaranteed. Here, we propose a sparsity $L1$ loss to encourage the router
activation to be sparse for each example. Besides, we use a mean importance
loss to further improve the balance of expert utilization. Furthermore, a
shared embedding network is used in our architecture to improve the route
decisions, whose output will be combined with the output of previous layers as
the input of routers.

The rest of the paper is organized as follows. Section 2 reviews the related
works of MoE and Section 3 represents our proposed method SpeechMoE. The
experimental setup is described in Section 4 and the experimental results are
reported in Section 5. Finally, we conclude this paper in Section 6.

##  2 Related works

In this section, we mainly describe two different architectures of MoE.

Figure 1:  (a), (b) and (c) represent the architecture of DeepMoE, Switch
Transformer and SpeechMoE respectively. Similar to Switch Transformer, only
one expert with the largest router probability in each MoE layer is used in
the SpeechMoE, which is different from DeepMoE. Besides, the SpeechMoE
utilizes a shared embedding and output of the previous layer as the input of
each router.

###  2.1 DeepMoE

The DeepMoE architecture proposed in [20] can achieve lower computation cost
and higher prediction accuracy than standard convolutional networks. The
architecture designs a sparse gating network which can dynamically select and
re-weight the channels in each layer of the base convolutional network.
Fig.1(a) shows the detailed architecture of DeepMoE. The DeepMoE consists of a
base convolutinal network, a shared embedding network and a multi-headed
sparse gating network. The gating network transforms the output of the shared
embedding network into sparse mixture weights:

$g^{l}(e)=f({W^{l}_{g}}\cdot e)$ (1)

where $g^{l}(e)$ is the sparse mixture weights of $l$-th convolutional layer,
e is the output of the shared embedding network, and $f$ is the activation
operation(i.e., Relu). Then, the output of $l$-th convolutional layer can be
formulated as:

$y^{l}=\sum_{i=1}^{n}g^{l}_{i}E^{l}_{i}$ (2)

where $n$ is the input channels number of $l$-th convolutional layer and
$E^{l}_{i}$ is the $i$-th channel of $l$-th convolutional layer, treated as
the $i$-th expert in $l$-th layer.

The loss function for training DeepMoE is defined as:

$L(x;y)=L_{b}(x;y)+\alpha L_{g}(x;y)+\beta L_{e}(x;y)$ (3)

where $x$ and $y$ are the input image feature and target label, respectively.
$L_{b}$ is the classification loss, $L_{g}$ is the L1 regularization term
which controls sparsity of the gating network and $L_{e}$ is the additional
classification loss which encourages the diversity of shared embedding
network.

###  2.2 Switch Transformer

Fedus et al. proposed the Switch Transformer [17] for language modeling, which
further reduces computation and communication costs by simplifying the MoE
routing algorithm. The architecture of Switch Transformer is described in
Fig.1(b), where experts refer to feed-forward networks and the non-expert
layers refer to the self-attention layers. Each MoE layer consists of $n$
experts and a router layer. It takes output for the previous layer as input
and routes it to the top-1 expert with the largest router probability. Let
$W^{l}_{r}$ and $o^{l-1}$ be the router weights of the $l$-th layer an the
output of the previous layer, then the router probability can be defined as
follows:

$r^{l}=W^{l}_{r}\cdot o^{l-1}$ (4)

${p^{l}_{i}}=\frac{exp^{r^{l}_{i}}}{\sum_{j=1}^{n}exp^{r^{l}_{j}}}$ (5)

Then, the selected expert’s output is also gated by router probability to get
output of the MoE layer,

$y^{l}=p^{l}_{i}E^{l}_{i}$ (6)

Since only one expert is active in each layer, the Switch Transformer can keep
the computation cost constant while scaling to a very large model. To
encourage a balance load across experts, the balancing loss [17] is added into
the loss function and defined as:

$L_{b}=n\cdot\sum_{i=1}^{n}s_{i}\cdot P_{i}$ (7)

where $s_{i}$ is the fraction of samples dispatched to expert i, $P_{i}$ is
the fraction of router probability allocated for expert $i$.

##  3 SpeechMoE

###  3.1 Model architecture

Fig.1(c) shows an overview of the architecture of our proposed SpeechMoE. For
speech recognition, its input is speech features (e.g. fbanks) and the input
frames will be dispatched to experts in each layer. Similar to the Switch
Transformer, SpeechMoE only selects one expert in each layer to reduce the
computation cost. Compared with Switch Transformer and DeepMoE, the SpeechMoE
concatenates the shared embedding with output of the previous layer as the
input of routers, which can be defined as:

$r^{l}=W_{r}^{l}\cdot Concat(e;o^{l-1})$ (8)

This router mechanism comes from two considerations: (1) All gating values in
DeepMoE are controlled by the shared embedding, which may decay to similar
gating results in each layer. Utilizing the hierarchical representation from
output of each layer may lead to diverse routing results for SpeechMoE. (2)
The shared embedding relative to the goal task may be helpful to get a better
routing strategy, providing a high-level distinctive representation and making
the experts specialized to process distinct input frames.

###  3.2 Training objective

####  3.2.1 sparsity $L1$ loss

In our study, we find that the router probability distribution tends to be
uniform when we only use the balancing loss proposed in [17], resulting in a
bad performance. In order to encourage the sparsity of router activation, we
propose a sparsity $L1$ loss, defined as follows:

$L_{s}=\frac{1}{m}\sum_{i=1}^{m}\parallel{\hat{f}_{i}}\parallel_{1}$ (9)

where $\hat{f}_{i}=\frac{f_{i}}{{\parallel}{{f}_{i}}\parallel_{2}}$, stands
for the unit normalized router probability distribution of sample $i$, and $m$
is the number of samples in this mini-batch. Due to the unit normalization,
minimizing the $L1$ norm will force the distribution close to space axes and
attain sparsity.

####  3.2.2 Mean importance loss

We have also observed that model isn’t balanced enough when increasing the
number of experts. To solve this problem, we use a modified importance
loss[22] to replace the balancing loss, defined as follows:

$Imp=\frac{1}{m}\sum_{i=1}^{m}p_{i}$ (10)
$\bar{L}_{m}=n\sum_{j=1}^{n}{{Imp}_{j}}^{2}$ (11)

The mean importance is defined as the mean activation of experts on batch of
samples and the loss is defined as the squared sum of mean importance of each
expert. It’s clear that when mean importance of each expert is averaged
$\frac{1}{n}$, the loss reaches the minimum. Compared with the balancing loss
in which $s_{i}$ is not differentiable, the mean importance loss is more
smooth, leading to a more balanced routing strategy.

####  3.2.3 Loss function

Given the input $x$ and the target $y$, the full loss function of our method
is defined as

$\begin{split}L(x;y)=L_{r}(x;y)+\alpha L_{s}(x)+\beta\bar{L}_{m}(x)+\gamma
L_{e}(x;y)\end{split}$ (12)

Among these items, $L_{r}$ is the CTC loss[23] for speech recognition, $L_{s}$
and $\bar{L}_{m}$ are the mentioned sparsity $L1$ loss and mean importance
loss, used to encourage sparsity and diversity of the SpeechMoE model. Similar
to [20], we introduce an additional embedding loss $L_{e}$, which is also the
CTC loss. It shares the same goal with our SpeechMoE model and provides
reliable embeddings for the routers. $\alpha$, $\beta$, and $\gamma$ are the
scale for $L_{s}$, $\bar{L}_{m}$ and $L_{e}$ respectively.

##  4 Experimental Setup

###  4.1 Training setup

The speech features used in all the experiments are 40-dimensional log-Mel
filterbank features appended with the first-order and the second-order
derivatives. Log-mel filterbank features are computed with a 25ms window and
shifted every 10ms. We stack 8 consecutive frames and subsample the input
frames with 3. A global mean and variance normalization is applied for each
frame. All the experiments are based on the CTC learning framework. We use the
CI-syllable-based acoustic modeling method [24] for CTC learning. The target
labels of CTC learning are defined to include 1394 Mandarin syllables, 39
English phones, and a blank. Character error rate results are measured on the
test sets and the floating point operations (FLOPs) for a one-second example
is used to evaluate the inference computation cost. We use a pruned, first
pass, 5-gram language model. All the systems use a vocabulary that consists of
millions of words. Decoding is performed with a beam search algorithm by using
the weighted finite-state transducers (WFSTs).

###  4.2 Datasets

Our training corpus is mixed data sets collected from several different
application domains, all in Mandarin. In order to improve system robustness, a
set of simulated room impulse responses (RIRs) are created with different
rectangular room sizes, speaker positions, and microphone positions, as
proposed in [25]. Totally, It comes to a 10k hours training corpus.

To evaluate the performance of our proposed method, we report performance on 3
types of test sets which consist of hand-transcribed anonymized utterances
extracted from reading speech (1001 utterances), conversation speech (1665
utterances) and spontaneous speech (2952 utterances). We refer them as Read,
Chat, and Spon respectively. In addition, to provide a public benchmark, we
also use AISHELL-2 development set (2500 utterances) recorded by high fidelity
microphone as the test set.

###  4.3 Acoustic Model

Our acoustic models consist of four components: MoE layer, sequential memory
layer [26], self-attention layer [27] and the output softmax layer. Each MoE
layer includes a router and a set of experts which is a feed forward network
with one hidden layer of size 1024 activated by ReLU and an projection layer
of size 512. For the sequential memory layer, the look-back order and look-
ahead order of each memory block is 5 and 1 respectively, and the strides are
2 and 1 respectively. For the self-attention layer, we set the model dimension
$d=512$ and the number of heads $h=8$. For every layer excluding the output
softmax layer, the residual connection is applied.

The backbone of our model consists of 30 MoE layers, 30 sequential memory
layers and 3 self-attention layers. Each MoE layer is followed by one
sequential memory layer, and a self-attention layer is inserted after each 10
consecutive MoE and sequential memory layers. In our experiments, we vary the
number of experts of MoE layers to be 2, 4 and 8, which are marked as MoE-2e,
MoE-4e and MoE-8e respectively. The shared embedding network is a static model
without MoE layers but a similar structure to the backbone.

In our study, we built two baseline systems for evaluating the performance of
our proposed method:

  * -

Baseline 1 (B1): The static model without MoE layers but a similar structure
to the backbone of SpeechMoE models, which can also be treated as MoE-1e.
Since the proposed method uses an extra embedding network, B1 model is
designed to have 60 layers to be FLOP-matched with our MoE models.

  * -

Baseline 2 (B2): The model with 4 experts in each MoE layer, which does not
have the shared embedding network and is trained with only the auxiliary
balancing loss proposed in Switch Transformer.

For all experiments on MoE models, we set the hyper-parameters $\alpha=0.1$,
$\beta=0.1$ and $\gamma=0.01$.

##  5 Experimental Results

###  5.1 Adding sparsity $L1$ loss

In this section, we investigate the performance of adding the sparsity $L1$
loss in training. We have trained two baseline systems for this evaluation.
The first baseline system(B1) is the static model trained based on $L_{r}$
loss and The other one(B2) is trained based on $L_{r}$ and $L_{b}$ loss
mentioned above. Our result of adding sparsity $L1$ loss relative to B2 is
marked as MoE-$L1$.

Table 1: Results of adding sparseness $L1$ loss.

Model | Params | FLOPs | Test set
---|---|---|---
Read | Chat | Spon | AISHELL
B1 | 71M | 2.3B | 2.0 | 22.92 | 24.95 | 4.52
B2 | 134M | 2.3B | 1.81 | 22.49 | 24.90 | 4.50
MoE-$L1$ | 134M | 2.3B | 1.69 | 22.47 | 24.70 | 4.25

As shown in table 1, B2 performs a little better than B1 with more parameters
and comparable computation cost. It is as expected that the MoE-$L1$ which
uses both balancing loss and sparsity $L1$ loss achieves the best performance
compared with two baseline systems. This indicates that the additional
sparsity $L1$ loss brings about more sparsity to router probability
distribution. The routers become more distinctive and specialized for varying
input frames so that the model get a better performance.

###  5.2 Augmenting shared embedding network

In this section, we evaluate the performance of the new router architecture
which concatenates the shared embedding with output of the previous layer as
the input of the router. As can be observed in table 2, the proposed router
architecture achieves lower character error rate comparing with MoE-$L1$
model.

It is worthy to note that only using output of previous layer as input does
not work very well, which contradict with the method used in [17]. A
reasonable explanation is that for language modeling, the word input as high-
level representation already has good distinction, while for speech
recognition the spectrum input is low-level feature which can not provide
enough distinction information for routers, so the shared embedding network
which converts low-level features to high-level embedding, is necessary to
help router attain better selecting effect.

Table 2: Results of augmenting shared embedding network and utilizing mean
importance loss.

Model | Params | FLOPs | Test set
---|---|---|---
Read | Chat | Spon | AISHELL
MoE-$L1$ | 134M | 2.3B | 1.69 | 22.47 | 24.70 | 4.25
+emb | 170M | 2.3B | 1.63 | 22.15 | 24.15 | 4.16
+imp loss | 170M | 2.3B | 1.58 | 21.57 | 23.31 | 4.00

###  5.3 Utilizing mean importance loss

The last line of table 2 presents the effects of the mean importance loss in
place of the balancing loss. We observe that the proposed loss can further
achieve lower character error rate than MoE-$L1$ model with embedding network
on the four test sets. Since the mean importance loss encourages all experts
to have equal importance, it will help the routers dispatch input frames to
experts in a balanced way, avoiding the situation that some experts get no
samples for training. Thus, the experts will be more diverse and result in a
better performance.

###  5.4 Increasing the number of experts

Figure 2: Validation CTC loss for increasing expert number

In this section, we investigate the effect of increasing the number of
experts. Table 3 shows the performance comparison on different number of
experts with SpeechMoE. Line 2 presents the results of the baseline system
(B1). The following three lines present results of 3 different number of
experts which are marked as MoE-2e, MoE-4e and MoE-8e respectively. The
results clearly show that performance get better as the number of experts
increases. Specifically, MoE-8e achieves up to 23.0% relative CER improvement
over the baseline model on the Read test set, and the gain is between
7.0%$\sim$11.9% for other more realistic test sets.

Figure 2 shows the validation CTC loss of MoE with different number of experts
and the baseline model. As shown, the MoE-8e model produces the lowest CTC
loss compared with both the baseline model and the other SpeechMoE models.
Moreover, we observe that having more experts speeds up training. This
suggests that increasing the number of expert leads to more powerful models.

Table 3: Results of increasing the number of experts.

Model | Params | FLOPs | Test set
---|---|---|---
Read | Chat | Spon | AISHELL
B1 | 71M | 2.3B | 2.0 | 22.92 | 24.95 | 4.52
MoE-2e | 105M | 2.3B | 1.62 | 21.82 | 23.52 | 4.08
MoE-4e | 170M | 2.3B | 1.58 | 21.57 | 23.31 | 4.00
MoE-8e | 297M | 2.3B |  | 1.54
---
(-23.0%)
| 21.31
---
(-7.0%)
| 22.97
---
(-7.9%)
| 3.98
---
(-11.9%)

##  6 Conclusions and future work

In this paper, we explore a mixture of experts approach for speech
recognition. We propose a novel dynamic routing acoustic model architecture,
the router module is enhanced by combining the previous layer’s output and
embedding from an isolated embedding network. We also improve the training
loss that can both achieve better sparsity and balancing among different
experts. Thorough experiments are conducted on training with different loss
and varied number of experts. Future work includes both extending training
data scale and number of experts, increasing by one or two orders of
magnitudes, and exploring the proposed SpeechMoE model with other end-to-end
training framework such as transformer transducers.

## References

  * [1] G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” in _IEEE Transactions on audio, speech, and language processing_ , vol. 20. IEEE, 2012, p. 30–42.
  * [2] D. Yu and J. Li, “Recent progresses in deep learning based acoustic models,” in _IEEE/CAA Journal of Automatica Sinica_ , vol. 4. IEEE, 2017, p. 396–409.
  * [3] T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for lvcsr,” in _2013 IEEE international conference on acoustics, speech and signal processing_. IEEE, 2013, pp. 8614–8618.
  * [4] Y. Qian and P. C. Woodland, “Very deep convolutional neural networks for robust speech recognition,” in _2016 IEEE Spoken Language Technology Workshop (SLT)_ , 2016, pp. 481–488.
  * [5] A. Graves, N. Jaitly, and A.-r. Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in _2013 IEEE workshop on automatic speech recognition and understanding_. IEEE, 2013, pp. 273–278.
  * [6] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, “Light gated recurrent units for speech recognition,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 2, no. 2, pp. 92–102, 2018.
  * [7] V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [8] S. Zhang, M. Lei, Z. Yan, and L. Dai, “Deep-fsmn for large vocabulary continuous speech recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5869–5873.
  * [9] L. Dong, S. Xu, and B. Xu, “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5884–5888.
  * [10] Y. Shi, Y. Wang, C. Wu, C.-F. Yeh, J. Chan, F. Zhang, D. Le, and M. Seltzer, “Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition,” _arXiv e-prints_ , p. arXiv:2010.10759, Oct. 2020.
  * [11] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [12] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” _arXiv preprint arXiv:1909.08053_ , 2019.
  * [13] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _arXiv preprint arXiv:2005.14165_ , 2020.
  * [14] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” _Neural computation_ , vol. 3, no. 1, pp. 79–87, 1991\.
  * [15] M. I. Jordan and R. A. Jacobs, “Hierarchical mixtures of experts and the em algorithm,” _Neural computation_ , vol. 6, no. 2, pp. 181–214, 1994.
  * [16] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” _arXiv preprint arXiv:2006.16668_ , 2020.
  * [17] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” _arXiv preprint arXiv:2101.03961_ , 2021.
  * [18] S. Gross, M. Ranzato, and A. Szlam, “Hard mixtures of experts for large scale weakly supervised vision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6865–6873.
  * [19] K. Ahmed, M. H. Baig, and L. Torresani, “Network of experts for large-scale image categorization,” in _European Conference on Computer Vision_. Springer, 2016, pp. 516–532.
  * [20] X. Wang, F. Yu, L. Dunlap, Y.-A. Ma, R. Wang, A. Mirhoseini, T. Darrell, and J. E. Gonzalez, “Deep mixture of experts via shallow embedding,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 552–562.
  * [21] S. Cai, Y. Shu, and W. Wang, “Dynamic routing networks,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 3588–3597.
  * [22] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” _arXiv preprint arXiv:1701.06538_ , 2017.
  * [23] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [24] Z. Qu, P. Haghani, E. Weinstein, and P. Moreno, “Syllable-based acoustic modeling with ctc-smbr-lstm,” in _Automatic Speech Recognition and Understanding Workshop (ASRU), 2017 IEEE_. IEEE, 2017, pp. 173–177.
  * [25] I. Himawan, P. Motlicek, D. Imseng, B. Potard, N. Kim, and J. Lee, “Learning feature mapping using deep neural network bottleneck features for distant large vocabulary speech recognition,” in _International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [26] S. Zhang, M. Lei, Z. Yan, and L. Dai, “Deep-fsmn for large vocabulary continuous speech recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5869–5873.
  * [27] Z. You, D. Su, J. Chen, C. Weng, and D. Yu, “Dfsmn-san with persistent memory model for automatic speech recognition,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7704–7708.