A globally convergent method to accelerate large-scale optimization using on-the-fly model hyperreduction: application to shape optimization

rvt1]Tianshu Wenfn1

rvt1]Matthew J. Zahrfn2cor1

[rvt1]Department of Aerospace and Mechanical Engineering, University
of Notre Dame, Notre Dame, IN 46556, United States
[cor1]Corresponding author

[fn1]Graduate Student, Department of Aerospace and Mechanical
Engineering, University of Notre Dame
[fn2]Assistant Professor, Department of Aerospace and Mechanical
Engineering, University of Notre Dame

PDE-constrained optimization, reduced-order model, hyperreduction, shape optimization, reduced mesh motion, trust-region method, on-the-fly sampling

We present a numerical method to efficiently solve optimization problems governed by large-scale nonlinear systems of
equations, including discretized partial differential equations, using projection-based reduced-order models accelerated
with hyperreduction (empirical quadrature) and embedded in a trust-region framework that guarantees global convergence.
The proposed framework constructs a hyperreduced model on-the-fly during the solution of the optimization problem,
which completely avoids an offline training phase. This ensures all snapshot information is collected along the optimization
trajectory, which avoids wasting samples in remote regions of the parameters space that are never visited, and inherently avoids
the curse of dimensionality of sampling in a high-dimensional parameter space. At each iteration of the proposed algorithm, a
reduced basis and empirical quadrature weights are constructed precisely to ensure the global convergence criteria of the
trust-region method are satisfied, ensuring global convergence to a local minimum of the original (unreduced) problem.
Numerical experiments are performed on two fluid shape optimization problems to verify the global convergence of the method
and demonstrate its computational efficiency; speedups over $18\times$ (accounting for all computational cost, even cost that is
traditionally considered “offline” such as snapshot collection and data compression) relative to standard optimization
approaches that do not leverage model reduction are shown.

§ ACKNOWLEDGMENTS

This material is based upon work supported by the Air Force Office of
Scientific Research (AFOSR) under award numbers FA9550-20-1-0236
and FA9550-22-1-0004. The content of this publication does not necessarily
reflect the position or policy of any of these supporters, and no official endorsement
should be inferred.

[1]
Anshul Agarwal and Lorenz T Biegler.
A trust-region framework for constrained optimization using reduced
order modeling.
Optimization and Engineering, 14(1):3–35, 2013.

[2]
Natalia M. Alexandrov, John E. Dennis, Robert M. Lewis, and Virginia Torczon.
A trust-region framework for managing the use of approximation models
in optimization.
Structural Optimization, 15(1):16–23, February 1998.

[3]
Natalia M. Alexandrov, Robert Michael Lewis, Clyde R. Gumbert, Lawrence L.
Green, and Perry A. Newman.
Approximation and Model Management in Aerodynamic
Optimization with Variable-Fidelity Models.
Journal of Aircraft, 38(6):1093–1101, November 2001.

[4]
David Amsallem, Matthew Zahr, Youngsoo Choi, and Charbel Farhat.
Design optimization using hyper-reduced-order models.
Structural and Multidisciplinary Optimization, 51(4):919–940,
April 2015.

[5]
Eyal Arian, Marco Fahl, and Ekkehard W. Sachs.
Trust-region proper orthogonal decomposition for flow control.

[6]
Stefan Banholzer, Luca Mechelli, and Stefan Volkwein.
A trust region reduced basis Pascoletti-Serafini algorithm for
multi-objective PDE-constrained parameter optimization.
arXiv preprint arXiv:2201.07744, 2022.

[7]
Maxime Barrault, Yvon Maday, Ngoc Cuong Nguyen, and Anthony T. Patera.
An `empirical interpolation' method: application to efficient
reduced-basis discretization of partial differential equations.
Comptes Rendus Mathematique, 339(9):667–672, November 2004.

[8]
Gabriele Boncoraglio and Charbel Farhat.
Active Manifold and Model-Order Reduction to Accelerate
Multidisciplinary Analysis and Optimization.
AIAA Journal, 59(11):4739–4753, November 2021.

[9]
Matthew Brand.
Fast low-rank modifications of the thin singular value decomposition.

[10]
Richard Carter.
On the global convergence of trust region algorithms using inexact
gradient information.
SIAM Journal on Numerical Analysis, 28(1):251–265, 1991.

[11]
Saifon Chaturantabut and Danny C. Sorensen.
Discrete Empirical Interpolation for nonlinear model reduction.
In Proceedings of the 48h IEEE Conference on Decision and
Control (CDC) held jointly with 2009 28th Chinese Control
Conference, pages 4316–4321, Shanghai, China, December 2009. IEEE.

[12]
Youngsoo Choi, Gabriele Boncoraglio, Spenser Anderson, David Amsallem, and
Charbel Farhat.
Gradient-based constrained optimization using a database of linear
reduced-order models.
Journal of Computational Physics, 423:109787, December 2020.

[13]
Andrew R. Conn, Nicholas I. M. Gould, and Philippe L. Toint.
Trust-region methods.
MPS-SIAM series on optimization. Society for Industrial and
Applied Mathematics, Philadelphia, PA, 2000.

[14]
Markus A Dihlmann and Bernard Haasdonk.
Certified PDE-constrained parameter optimization using reduced
basis surrogate models for evolution problems.
Computational Optimization and Applications, 60(3):753–787,

[15]
Eugene Du, Michael Sleeman, and Masayuki Yano.
Adaptive Discontinuous-Galerkin Reduced-Basis
Reduced-Quadrature Method for Many-Query CFD Problems.
In AIAA Aviation 2021 Forum, Virtual event, August 2021.
American Institute of Aeronautics and Astronautics.

[16]
Eugene Dua and Masayuki Yanoa.
Efficient hyperreduction of high-order discontinuous Galerkin
methods: element-wise and point-wise reduced quadrature formulations.

[17]
Mohammad Esmaeili, Mohammad Ahmadi, and Alireza Kazemi.
A generalized DEIM technique for model order reduction of porous
media simulations in reservoir optimizations.
Journal of Computational Physics, 422:109769, December 2020.

[18]
Charbel Farhat, Philip Avery, Todd Chapman, and Julien Cortial.
Dimensional reduction of nonlinear finite element dynamic models with
finite rotations and energy-based mesh sampling and weighting for
computational efficiency.
International Journal for Numerical Methods in Engineering,
98(9):625–662, 2014.

[19]
Charbel Farhat, Christoph Degand, Bruno Koobus, and Michel Lesoinne.
Torsional springs for two-dimensional dynamic unstructured fluid
Computer Methods in Applied Mechanics and Engineering,
163(1-4):231–245, 1998.

[20]
Alexander I.J Forrester, Neil W Bressloff, and Andy J Keane.
Optimization using surrogate models and partially converged
computational fluid dynamics simulations.
Proceedings of the Royal Society A: Mathematical, Physical and
Engineering Sciences, 462(2071):2177–2204, July 2006.

[21]
Alexander I.J. Forrester and Andy J. Keane.
Recent advances in surrogate-based optimization.

[22]
Matthias Heinkenschloss and Caleb Magruder.
Reduced order model hessian approximations in newton methods for
optimal control.
In Christopher Beattie, Peter Benner, Mark Embree, Serkan Gugercin,
and Sanda Lefteriu, editors, Realization and Model Reduction of
Dynamical Systems: A Festschrift in Honor of the 70th Birthday of Thanos
Antoulas, pages 335–351. Springer International Publishing.

[23]
Tim Keil, Luca Mechelli, Mario Ohlberger, Felix Schindler, and Stefan Volkwein.
A non-conforming dual approach for adaptive Trust-Region reduced
basis approximation of PDE-constrained parameter optimization.
ESAIM: Mathematical Modelling and Numerical Analysis,
55(3):1239–1269, 2021.

[24]
Tim Keil and Mario Ohlberger.
A relaxed localized trust-region reduced basis approach for
optimization of multiscale problems.
arXiv preprint arXiv:2203.09964, 2022.

[25]
Drew P. Kouri, Matthias Heinkenschloss, Denis Ridzal, and Bart G. van
Bloemen Waanders.
A Trust-Region Algorithm with Adaptive Stochastic
Collocation for PDE Optimization under Uncertainty.
SIAM Journal on Scientific Computing, 35(4):A1847–A1879,
January 2013.

[26]
Patrick LeGresley and Juan Alonso.
Airfoil design optimization using reduced order models based on
proper orthogonal decomposition.
In Fluids 2000 Conference and Exhibit, Denver,CO,U.S.A.,
June 2000. American Institute of Aeronautics and Astronautics.

[27]
Michel Lesoinne, Marcus Sarkis, Ulrich Hetmaniuk, and Charbel Farhat.
A linearized method for the frequency analysis of three-dimensional
fluid/structure interaction problems in all flow regimes.
Comput. Methods Appl. Mech. Engrg., page 26, 2001.

[28]
Kjetil O. Lye, Siddhartha Mishra, Deep Ray, and Praveen Chandrashekar.
Iterative surrogate model optimization (ISMO): An active learning
algorithm for PDE constrained optimization with deep neural networks.
Computer Methods in Applied Mechanics and Engineering,
374:113575, February 2021.

[29]
Andrea Manzoni, Alfio Quarteroni, and Gianluigi Rozza.
Shape optimization for viscous flows by reduced basis methods and
free-form deformation.
International Journal for Numerical Methods in Fluids,
70(5):646–670, 2012.

[30]
Simao P. Marques, Lucas Kob, Trevor T. Robinson, Weigang Yao, and Liang Sun.
Non-intrusive aerodynamic shape optimisation with a discrete
empirical interpolation method.
In AIAA Scitech 2021 Forum, VIRTUAL EVENT, January 2021.
American Institute of Aeronautics and Astronautics.

[31]
Per-Olof Persson and Gilbert Strang.
A Simple Mesh Generator in MATLAB.
SIAM Review, 46(2):329–345, January 2004.

[32]
Elizabeth Qian, Martin Grepl, Karen Veroy, and Karen Willcox.
A Certified Trust Region Reduced Basis Approach to
PDE-Constrained Optimization.
SIAM Journal on Scientific Computing, 39(5):S434–S460, January

[33]
S. Ashwin Renganathan.
Koopman-Based Approach to Nonintrusive Reduced Order
Modeling: Application to Aerodynamic Shape Optimization and
Uncertainty Propagation.
AIAA Journal, 58(5):2221–2235, May 2020.

[34]
Matteo Ripepi, Mark. J. Verveld, N. W. Karcher, Tobias Franz, Mohammad
Abu-Zurayk, Stefan Görtz, and Thiemo M. Kier.
Reduced-order models for aerodynamic applications, loads and MDO.
CEAS Aeronautical Journal, 9(1):171–193, March 2018.

[35]
Gianluigi Rozza.
On optimization, control and shape design of an arterial bypass.
International Journal for Numerical Methods in Fluids,
47(10-11):1411–1419, 2005.

[36]
Daniel Scheffold, Christopher Bach, Fabian Duddeck, Gerhard Müller, and
Martin Buchschmid.
Vibration Frequency Optimization of Jointed Structures with
Contact Nonlinearities using Hyper-Reduction.
IFAC-PapersOnLine, 51(2):843–848, 2018.

[37]
Nadav Sorek, Hardikkumar Zalavadia, and Eduardo Gildin.
Model Order Reduction and Control Polynomial Approximation
for Well-Control Production Optimization.
In Day 2 Tue, February 21, 2017, Montgomery, Texas, USA,
February 2017. SPE.

[38]
Răzvan Ştefănescu, Adrian Sandu, and Ionel M. Navon.
POD/DEIM reduced-order strategies for efficient four dimensional
variational data assimilation.
Journal of Computational Physics, 295:569–595, August 2015.

[39]
Eka Suwartadi, Stein Krogstad, and Bjarne Foss.
Adjoint-based surrogate optimization of oil reservoir water flooding.
Optimization and Engineering, 16(2):441–481, June 2015.

[40]
Jun Tao and Gang Sun.
Application of deep learning based multi-fidelity surrogate model to
robust aerodynamic design optimization.
Aerospace Science and Technology, 92:722–737, September 2019.

[41]
Sumeet Trehan and Louis J. Durlofsky.
Trajectory piecewise quadratic reduced-order model for subsurface
flow, with application to PDE-constrained optimization.
Journal of Computational Physics, 326:446–473, December 2016.

[42]
Huanhuan Yang and Alessandro Veneziani.
Efficient estimation of cardiac conductivities via POD-DEIM model
order reduction.
Applied Numerical Mathematics, 115:180–199, May 2017.

[43]
Masayuki Yano.
Discontinuous Galerkin reduced basis empirical quadrature procedure
for model reduction of parametrized nonlinear conservation laws.
Advances in Computational Mathematics, 45(5-6):2287–2320,
December 2019.

[44]
Masayuki Yano.
Goal-oriented model reduction of parametrized nonlinear partial
differential equations: Application to aerodynamics.
International Journal for Numerical Methods in Engineering,
121(23):5200–5226, 2020.

[45]
Masayuki Yano, Tianci Huang, and Matthew J. Zahr.
A globally convergent method to accelerate topology optimization
using on-the-fly model reduction.
Computer Methods in Applied Mechanics and Engineering,
375:113635, March 2021.

[46]
Masayuki Yano and Anthony T. Patera.
An LP empirical quadrature procedure for reduced basis treatment of
parametrized nonlinear PDEs.
Computer Methods in Applied Mechanics and Engineering,
344:1104–1123, February 2019.

[47]
Yao Yue and Karl Meerbergen.
Accelerating optimization of parametric linear systems by model order
SIAM Journal on Optimization, 23(2):1344–1370, 2013.

[48]
Matthew J. Zahr.
Adaptive model reduction to accelerate optimization problems
governed by partial differential equations.
PhD Thesis, Stanford University, 2016.

[49]
Matthew J. Zahr, Kevin T. Carlberg, and Drew P. Kouri.
An Efficient, Globally Convergent Method for Optimization
Under Uncertainty Using Adaptive Model Reduction and Sparse
SIAM/ASA Journal on Uncertainty Quantification, 7(3):877–912,
January 2019.

[50]
Matthew J. Zahr and Charbel Farhat.
Progressive construction of a parametric reduced‐order model for
PDE‐constrained optimization.
International Journal for Numerical Methods in Engineering,
102(5):1111–1135, May 2015.

[51]
Matthew J. Zahr, Andrew Shi, and Per-Olof Persson.
Implicit shock tracking using an optimization-based high-order
discontinuous Galerkin method.
Journal of Computational Physics, 410:109385, June 2020.

[52]
J. Carsten Ziems and Stefan Ulbrich.
Adaptive Multilevel Inexact SQP Methods for
PDE-Constrained Optimization.
SIAM Journal on Optimization, 21(1):1–40, January 2011.


# Visuo-Tactile Transformers for Manipulation

Yizhou Chen Andrea Sipos Mark Van der Merwe Nima Fazeli
Department of Robotics, University of Michigan
Ann Arbor, MI 48109, United States
{yizhouch, asipos, markvdm<EMAIL_ADDRESS>
https://www.mmintlab.com/vtt

###### Abstract

Learning representations in the joint domain of vision and touch can improve
manipulation dexterity, robustness, and sample-complexity by exploiting mutual
information and complementary cues. Here, we present Visuo-Tactile
Transformers (VTTs), a novel multimodal representation learning approach
suited for model-based reinforcement learning and planning. Our approach
extends the Visual Transformer [1] to handle visuo-tactile feedback.
Specifically, VTT uses tactile feedback together with self and cross-modal
attention to build latent heatmap representations that focus attention on
important task features in the visual domain. We demonstrate the efficacy of
VTT for representation learning with a comparative evaluation against
baselines on four simulated robot tasks and one real world block pushing task.
We conduct an ablation study over the components of VTT to highlight the
importance of cross-modality in representation learning.

> Keywords: Multimodal Learning, Reinforcement Learning, Manipulation

##  1 Introduction

Deep reinforcement learning (RL) algorithms have solved numerous challenging
tasks from raw observations (e.g., images) [2, 3, 4, 5]. Despite these
successes, learning directly from such high-dimensional observation spaces can
be numerically difficult/unstable, sensitive to hyper-parameters, and sample
inefficient [4]. These challenges have limited the utility of these methods
for robotics applications because data collection is time intensive, costly,
and task variance is high.

To address this, recent research in representation learning aims to build
compact “latent” representations of the underlying states [4, 6, 7, 8, 9, 10].
These compact representations are typically trained together with a policy
using prediction and reconstruction losses along with task-dependant rewards.
The policy then learns from the low-dimensional latent representation rather
than the high-dimensional raw sensory observation. This approach offers
stability and sample-efficiency for policy learning owing to the information-
dense latent representations generated from rich prediction and reconstruction
supervisory signals. As such, latent representations are a promising tool for
real-world robotic planning and RL.

Most research in robot latent representation learning has focused on images
and proprioception [2, 5, 11, 12, 13, 14, 15]. The sense of touch, together
with vision, plays an important role in robotic manipulation; however, visuo-
tactile representation learning is still in its relative nascency. Recent
works [10, 16, 17, 18, 19, 20, 21, 22] have proposed visuo-tactile methods;
however, the latent representations used are typically vectors or clusters in
$\mathbb{R}^{n}$ and do not exploit attention architectures [23]. These
representations can smooth over fine details and struggle with locality in
images. These issues are particularly troublesome for manipulation as the
difference between in-contact and out-of-contact can be small in pixel space
or the robot may need to move significantly to contact objects.

In this paper, we propose a novel latent representation learning method that
addresses these challenges. Our approach extends the popular Visual
Transformer (ViT) [1] to integrate multimodal feedback from vision and touch –
the Visuo-Tactile Transformer (VTT). Our approach focuses spatially-aware
attention on important visual task features using tactile feedback. We show
how this cross-modal attention creates a rich spatially distributed latent
space that facilitates more efficient task execution. We evaluate the efficacy
of our approach on four simulated and one real-world contact-rich
interactions, benchmarking against state-of-the-art techniques.

##  2 Related Work

The learning community has made significant progress in latent space dynamics
learning, demonstrating promise in relatively low-dimensional [24, 25, 26, 27]
and high-dimensional [4, 6, 7, 8, 9] spaces. The key idea in these approaches
is to learn a latent dynamics model that compactly represents the high-
dimensional observation space that is then used for planning or model-based
RL. Many of these approaches consider continuous control tasks that may be
applicable to the robotics domain [28, 29, 30, 31, 32, 33]. Existing
approaches focus on learning latent dynamics from visual feedback. Here, our
work explicitly addresses multimodal learning in the visuo-tactile domain.

The robot learning community has also made progress in learning latent space
dynamics. In particular, cross-modal learning has recently gained traction
[10, 16, 17, 18, 19, 20, 21, 22] where the current trend is in visuo-tactile
representations. Our work is most similar to [10, 16] and we use their
encoders as our baselines. The key difference between these works and ours is
the latent representation. Rather than using a vector in $\mathbb{R}^{n}$, VTT
learns a multimodal latent heatmap where attention is distributed spatially.
We hypothesize that VTT’s explicitly spatially-aware cross-attention is more
effective for representation learning in terms of sample efficiency and
expected reward than either of these approaches. We evaluate this claim by
benchmarking against existing approaches.

Our proposed approach is an extension of the Transformer [23] and ViT [1] to
the visio-tactile domain. Recent research has demonstrated the efficacy of
attention mechanisms for sequence-to-sequence and spatially-distributed
inputs. A tighter integration of encoders developed by the computer vision
community [34, 35, 36] with robotic applications can lead to significant
advances in representation learning that enable efficient planning and RL.

##  3 Visuo-Tactile Transformers

Figure 1: Architecture: Schematic of our proposed Visuo-Tactile Transformer.
Parts of the diagram with gray backgrounds are novel contributions and
differentiate our structure from ViT.

At a high level, VTT produces compact latent representations by fusing visuo-
tactile inputs with self and cross-modal attention mechanisms. This enables
robotic agents to leverage the complementary nature of these sensing
modalities for policy learning. Additionally, VTT supplements image and
tactile input patches with learned contact, alignment, and position/modality
embeddings to further improve multi-modal reasoning. The overall VTT
architecture is shown in Fig. 1. We present the details of the architecture
and its components in the following sections.

###  3.1 Modality Patches

Before sending inputs to the transformer encoder to compute self and cross-
modal attention, both the image and tactile inputs need to be patched. RGB
image inputs are patched and embedded with one 2-D convolution layer as in ViT
[1]. Tactile inputs from a wrist-mounted force-torque sensor are patched into
reaction forces and torques, then a linear projection is applied to embed
them. Image and tactile modalities are notated $I$ and $T$ respectively. We
denote the embedded modality patch as $X_{M}\in\mathbb{R}^{(P_{I}+P_{T})\times
d}$, where $P$ is the number of patches in each modality and $d$ is the number
of features in each modality, which we assume to be identical. This embedded
modality patch $X_{M}$ can be written $[X_{I};X_{T}]$ and is used as input to
the first attention layer, as shown in Fig. 1.

###  3.2 Self and Cross-Modal Attention

We implement self and cross-modal attention mechanisms by adapting the multi-
head attention method from Transformer architecture [23]. This adaptation is
shown on the right side of Fig. 1. The transformer encoder is composed of a
stack of $N$ identical attention layers. Each attention layer has two sub-
layers: a multi-head attention sub-layer and a feed-forward sub-layer. When
$1\leq n+1\leq N$, the output $X_{n}$ of the $n^{\text{th}}$ attention layer
$A_{n}$ will be the input for the $(n+1)^{\text{th}}$ attention layer in the
transformer encoder. To show how self and cross-modal attention are computed
through $N$ attention fusion layers, we first analyze $A_{n=1}$ and then
generalize to $n>1$ layers.

To calculate the self and cross-modal attention at layer $n=1$, we first
project the post layer normalization ($LN$) of embedded modality patch $X_{M}$
into the Query ($Q^{i}_{n=1}$), Key ($K^{i}_{n=1}$), and Value ($V^{i}_{n=1})$
shown in Eq. 1, where $h$ is the number of attention heads and $1\leq i\leq
h$.

$\displaystyle\begin{split}Q^{i}_{n=1}=LN\begin{bmatrix}X_{I}\\\
X_{T}\end{bmatrix}W^{i}_{Q},\>\>K^{i}_{n=1}=LN\begin{bmatrix}X_{I}\\\
X_{T}\end{bmatrix}W^{i}_{K},\>\>V^{i}_{n=1}=LN\begin{bmatrix}X_{I}\\\
X_{T}\end{bmatrix}W^{i}_{V}\end{split}$ (1)

In this formulation, $W^{i}_{Q}\in\mathbb{R}^{d\times d_{K}}$,
$W^{i}_{K}\in\mathbb{R}^{d\times d_{K}}$, and
$W^{i}_{V}\in\mathbb{R}^{d_{K}\times\frac{d}{h}}$ are weights for Query, Key
and Value, where $d$ is defined in Sec. 3.1 and $d_{K}$ is the dimension of
features in the Key. Following Eq. 1, we note that the Query, Key, and Value
can be written as

$\displaystyle\begin{split}Q^{i}_{n=1}=\begin{bmatrix}Q_{I}\\\
Q_{T}\end{bmatrix}^{i}_{n=1},\>\>K^{i}_{n=1}=\begin{bmatrix}K_{I}\\\
K_{T}\end{bmatrix}^{i}_{n=1},\>\>V^{i}_{n=1}=\begin{bmatrix}V_{I}\\\
V_{T}\end{bmatrix}^{i}_{n=1}\end{split}$ (2)

After projecting the embedded modality patch $X_{M}$ into $Q_{n=1}$,
$K_{n=1}$, and $V_{n=1}$, we use Eq. 3 to calculate the attention at each head
$i$ using a scaled dot product with regularization factor $a=\sqrt{d}$.

$\displaystyle\begin{split}A^{i}_{n=1}=Atten(X_{n-1}=X_{M})^{i}=softmax(\frac{Q_{n=1}^{i}(K_{n=1}^{i})^{T}}{a})V_{n=1}^{i}\end{split}$
(3)

For clarity and brevity, we expand Eq. 3 in terms of $Q$, $K$, and $V$ to
isolate self and cross-modal attention.

$\displaystyle\begin{split}A^{i}_{n=1}\sim
QK^{T}V=\underbrace{\begin{bmatrix}Q_{I}K_{I}&Q_{I}K_{T}\\\
Q_{T}K_{I}&Q_{T}K_{T}\end{bmatrix}}_{\text{Attention
Heatmap}}\begin{bmatrix}V_{I}\\\
V_{T}\end{bmatrix}={\underbrace{\begin{bmatrix}Q_{I}K_{I}V_{I}\\\
Q_{T}K_{T}V_{T}\end{bmatrix}}_{\text{Self
Attention}}}^{i}+{\underbrace{\begin{bmatrix}Q_{I}K_{T}V_{T}\\\
Q_{T}K_{I}V_{I}\end{bmatrix}}_{\text{Cross Attention}}}^{i}\end{split}$ (4)

From Eq. 4 we can see that the attention heatmap is constructed with image and
tactile self attentions $Q_{I}K_{I}$ and $Q_{T}K_{T}$ on the diagonal and
cross-modal attentions $Q_{I}K_{T}$ and $Q_{T}K_{I}$ in the upper and lower
triangles. The attention output of Eq. 4 is the sum of self and cross-modal
attention. Finally, the multi-head attention $A_{n=1}$ is formed by
concatenating each attention head as shown in Eq. 5. Multi-head attention
encourages the model to explore the attention subspace.

$\displaystyle\begin{split}A_{n=1}=[A^{1}_{n=1},A^{2}_{n=1},...,A^{h}_{n=1}]\in\mathbb{R}^{(P_{I}+P_{T})\times
d}\end{split}$ (5)

To connect the sub-layers in each attention layer, we implement the residual
connection method from [37]. Finally, the output $X_{n=1}$ of the attention
layer $A_{n=1}$ can be written

$\displaystyle\begin{split}X_{n=1}=f_{n=1}(A_{n=1},X_{n-1}=X_{M})\end{split}$
(6)

where $f_{n=1}$ is a learned nonlinear function induced by nonlinear
activation functions and operations.

To generalize to $n>1$ attention fusion layers, $X_{n-1}$ is the input for
attention layer $A_{n}$ and is projected into $Q_{n}$, $K_{n}$ and $V_{n}$
using Eq. 1. This proposed attention mixture allows the agent to iteratively
leverage weights of self and cross-modal attention.

###  3.3 Learned Embeddings

To improve feature learning for RL policies, we propose three learnable
embeddings to bolster the modality patches and attention mechanisms we
introduced in Sec. 3.1 and Sec. 3.2. We evaluate effects of these embeddings
on model performance in the ablation study presented in Sec. 4.2.
Contact Embedding: Intuitively, contact embeddings are used to pull latent
codes for alike contact states together while pushing latent codes for
dissimilar contact states apart. After $N$ layers, the contact embedding
$X_{C}\in\mathbb{R}^{1\times d}$ becomes the contact head
$C_{head}\in\mathbb{R}^{1\times d}$ following the same method outlined in Sec.
3.2. We form a contact loss in Eq. 7 by using $C_{head}$ to predict the
contact state of the system then comparing that prediction to the ground truth
$C_{gt}$.
Alignment Embedding: Although contact embeddings introduce binary contact
recognition, temporal alignment of tactile signatures to images is non-trivial
[10, 16]. Alignment embeddings shape the latent space to seek similarities in
temporally aligned modalities and differences in temporally misaligned
modalities [10, 38]. Therefore, alignment embedding
$X_{Al}\in\mathbb{R}^{1\times d}$ is introduced and becomes the alignment head
$Al_{head}\in\mathbb{R}^{1\times d}$ after $N$ layers. As in the previous
section, we form a loss in Eq. 7 by using $Al_{head}$ to predict whether the
tactile and image data are aligned then comparing that prediction to the
ground truth $Al_{gt}$.

The contact and alignment losses are defined by a binary cross entropy with
logits loss $BCE_{logits}(\cdot)$.

$\displaystyle\begin{split}\ell_{VTT}=BCE_{logits}(MLP(Al_{head})),Al_{gt})+BCE_{logits}(MLP(C_{head}),C_{gt})\end{split}$
(7)

Position/Modality Embedding: The final type of learnable embedding we
implement is a position/modality embedding. We found that
$[X_{C};X_{Al};X_{M}]$ do not contain the position or modality of information,
so it is necessary to include position/modality embedding
$X_{P}\in\mathbb{R}^{1\times(2+P_{I}+P_{T})}$. Inspired by [1, 39], we add
this embedding to our structure to form $[X_{C};X_{Al};X_{M}]+X_{P}$.

###  3.4 Discussion of Compressed Representation Head

The output $X_{N}$ of the transformer encoder is a series of fused heads
originating from the visuo-tactile embedded patches $X_{M}$, contact
embeddings $X_{C}$, alignment embeddings $X_{Al}$, and position/modality
embeddings $X_{P}$. In order to use our learned representation from VTT for
RL, we need to think about the dimensionality of the latent vector z. If we
use $X_{N}\in\mathbb{R}^{(2+P_{I}+P_{T})\times d}$ directly, z will be too
high-dimensional. Consequently, we propose compressing all fused heads from
$\mathbb{R}^{(2+P_{I}+P_{T})\times d}$ to
$\mathbb{R}^{(2+P_{I}+P_{T})\times\frac{d}{c}}$ with constant $c>4$ by using
an MLP. This compression results in
$\textbf{z}\in\mathbb{R}^{1\times\frac{d}{c}(2+P_{I}+P_{T})}$. Our proposed
method will allow image and tactile attention heads from VTT to be used for
latent dynamics learning while introducing minimum inductive bias.

###  3.5 Combining with Reinforcement Learning

Due to the frequency of visual occlusions and high levels of tactile
perception uncertainty in our manipulation benchmarks, we formulate our
manipulation tasks as partially observable Markov decision processes (POMDPs)
[40]. There have been a number of advances in model-based RL for POMDPs [41,
4, 42, 2]. From these, we chose the Stochastic Latent Actor Critic (SLAC) [4]
algorithm because of its stability and robustness. These characteristics
accommodate high stochasticity and provide an effective baseline method to
compare visuo-tactile fusion mechanisms.

SLAC is composed of a model-learning component and a policy-learning
component. The model-learning piece is built with a factorized sequential
variational autoencoder (VAE) $f_{\theta}$ [43]. $f_{\theta}$ uses a Gaussian
distribution to form a prior model $p$ and a posterior model $q$. The prior
model $p(\textbf{z}^{d}_{t}|\textbf{z}^{d}_{t-1},a_{t-1})$ propagates latent
dynamics from an action $a$ and the posterior model
$q(\textbf{z}^{d}_{t}|\textbf{z}^{d}_{t-1},\textbf{z}_{t-1},a_{t-1})$
integrates observations. In this notation, $\textbf{z}_{t}^{d}\sim
f_{\theta}(\textbf{z}_{0:t},\textbf{z}_{0:t-1}^{d},a_{0:t-1})$. To close the
distribution gap between the prior and posterior models, KL divergence is
introduced. Finally, $\textbf{z}^{d}_{t}$ is decoded to reconstruct the visual
input $O$ and infer a reward $r$. VTT losses from Eq. 7 are composed with SLAC
losses to form the total model-learning losses in Eq. 8.

$\displaystyle\begin{split}\ell_{model}=\ell(O_{t}|\textbf{z}_{t}^{d},a_{t-1})+\ell(r_{t}|\textbf{z}_{t}^{d},a_{t-1})+\ell_{KL}(q||p)+\ell_{VTT}\end{split}$
(8)

The policy-learning component is implemented using the soft actor critic (SAC)
algorithm [44]. SAC is an off-policy actor-critic deep RL algorithm that
maximizes future return and entropy for exploration. For further details and
derivation on the model and policy learning, see [44, 4]. It is important to
note that the latent representation z from VTT is utilized by both components.
Additionally, the critic’s value function loss is backpropagated through VTT
to the attention mechanism.

##  4 Experiments and Results

Figure 2: Baseline Structure: We compare VTT to concatenation and product of
experts (PoE) baselines. These methods are inserted at the ‘fusion module’.

In this section, we benchmark VTT’s performance against two popular multi-
modality fusion techniques on four simulated robot manipulation tasks. Next,
we present an ablation study that evaluates the importance of our contact loss
and alignment loss on simulation tasks. We then evaluate VTT on a real-world
pushing task. Finally, we visualize the attention maps produced by VTT on
simulation and real-world tasks to illustrate the effects and intuition of
cross-modal attention in the latent space.

###  4.1 Simulated Experiments

Baseline Structures: To evaluate the performance of VTT, we benchmark against
two popular multi-modality fusion methods: concatenation and product of
experts (PoE) [10, 16]. Fig. 2 shows the architecture of these baselines,
where the fusion model is the key distinction between the methods. Both
baselines follow the same training pipeline as VTT to allow for fair
comparisons. We denote the inputs of the fusion module as
$E_{I}\in\mathbb{R}^{1\times d_{I}}$ and $E_{T}\in\mathbb{R}^{1\times d_{T}}$.

In the concatenation fusion module, the latent vector z becomes
$\textbf{z}\in\mathbb{R}^{1\times(d_{I}+d_{T})}=[E_{I};E_{T}]$. However, the
PoE fusion module treats $E_{I}$ and $E_{T}$ independently and applies a VAE-
like structure to each latent code individually. $E_{I}$ and $E_{T}$ pass
through separate MLPs to generate separate means $(\mu_{E_{I}},\mu_{E_{T}})$
and variances $(\sigma_{E_{I}},\sigma_{E_{T}})$ that are mapped to a normal
distribution using KL divergence, then fused using PoE as shown in Eq. 9 for
$i$ modalities and $j$ features. The latent space in this method is also
shaped by contact and alignment classification. Further details about each of
these baselines can be found in their published works.

$\displaystyle\begin{split}\sigma^{2}_{j}=\Bigg{(}\sum_{i=1}^{2}\sigma^{2}_{ij}\Bigg{)}^{-1},\mu_{j}=\Bigg{(}\sum_{i=1}^{2}\frac{\mu_{ij}}{\sigma_{ij}^{2}}\Bigg{)}\Bigg{(}\sum_{i=1}^{2}\sigma^{2}_{ij}\Bigg{)}^{-1}\end{split}$
(9)

Simulation Experiment Setup: We conduct simulated experiments of four
manipulation tasks in Pybullet. The selected tasks are Pushing, Door-Open,
Picking, and Peg-Insertion. Based on the continuous control manipulation
benchmark developed by ElementAI [45], we construct a dense reward setting for
these tasks. The agent receives an $84\times 84\times 3$ RGB image and a
$1\times 6$ wrist reaction wrench. Task rewards and parameters are outlined
below.

1\. Pushing Push a white block to a grey target pose. The agent receives +25
reward for finishing the task. In this task, we vary the initial pose and mass
of the block as well as the target pose.

2\. Door-Open Open a door from $\theta_{door}=0$ to $\theta_{goal}$. When
$\theta_{door}<\theta_{goal}$, we penalize the agent
$-\lambda(\theta_{goal}-\theta_{door})$. When $\theta_{goal}$ =
$\theta_{goal}$, the agent receives +25 reward. We incorporate curriculum
learning [46] into this task by randomly initializing $\theta_{door}>0$ during
training only. In this task, we vary the color of the door and frame.

3\. Picking Pick up a white block. We force the gripper to touch the block
with a location-based -$\lambda(|Loc_{gripper}-Loc_{block}|)$ penalty. The
agent receives +1 reward for grasping when the wrist feels significant
downward force. The agent receives +25 reward when the block is lifted to a
certain height. In this task, we vary the initial pose, mass, and size of the
block.

4\. Peg-Insertion Insert a peg into a hole. We force peg-hole alignment with a
location-based $-\lambda(|Loc_{peg}-Loc_{hole}|)$ penalty. To encourage the
agent to lower the peg into the hole, the agent receives
+$\lambda(|z_{max}-z_{peg})|)$ reward after alignment. The agent receives +25
reward when the peg in fully inserted. In this task, we vary the color of the
peg and the color of the hole.

Figure 3: Manipulation Tasks: We evaluate VTT on four tasks in Pybullet. We
vary visual and physical parameters in each task.

Training Details: We follow the SLAC training schedule to pretrain the
dynamics model $f_{\theta}$ and fusion modules VTT, concatenation, and PoE.
After the pretraining, SLAC and VTT are trained together. For the model we use
$lr=0.0001$ and batch size 32. For the policy we use $lr=0.0003$ and batch
size 256. We train using the Adam optimizer [47]. Further implementation
details are included in the supplementary material.

Figure 4: Simulation Results: We compare the performance of our method to
baselines across 8 trials and find that VTT outperforms the baselines in all
but one task. The variance of VTT’s learning curve is significantly smaller
than either baseline, suggesting better numerical stability of VTT.

Baseline Comparison: We compared VTT with the concatenation and PoE baselines
over 8 trials. The results for each task with each fusion module are shown in
Fig. 4. Compared to the baselines, our proposed approach achieves higher
sample efficiency on all tasks and higher success rate in all but the Peg-
Insertion task. We speculate that this task may not require as much spatial
reasoning as the other tasks because the robot is always grasping the peg and
only has to make contact with the table then insert. The remaining tasks all
require the robot to first reach for and grasp/push an object, then perform
the remainder of the task. Additionally, our approach shows lower variance
with respect to the baselines. This suggests that the latent representation is
numerically easier and more stable to learn.

###  4.2 Ablation Studies

In this section, we conduct three ablation studies to verify the fidelity of
our method. Firstly, we ablate the contact and alignment losses in the VTT
loss function. Secondly, we ablate the compression of latent vector z.
Finally, we ablate model sizes by adjusting parameter counts (Appendix Sec.
6.4).

####  4.2.1 Loss Function

To show the importance of contact and alignment losses in VTT, we conduct an
ablation study by sequentially removing the contact and alignment embeddings
and losses and performing the Door-Open and Peg-Insertion tasks. Overall, our
results in Fig. 5a indicate that each of these methods is critical for
achieving high sampling efficiency and success rate. The lack of contact and
alignment losses can cause miscorrelation of images and tactile feedback. The
representation induced by such miscorrelations may mislead the agent’s state
value estimation and decision making, leading to worse overall performance.
Additionally, we notice that the agent performs worse without the contact loss
than without the alignment loss. This behavior is likely due to the contact-
rich tasks we use to evaluate our method. One potentially useful application
of this misalignment may be as a metric for evaluating just how out of
distribution a novel observation is.

Figure 5: a) Ablation Results: We sequentially remove the contact and
alignment losses from VTT to show performance in two simulated tasks without
them. We find that the full method outperforms each partial method. b) Real-
World Demonstration: We find that VTT also outperformed the baselines in the
real-world Pushing task in terms of sample efficiency and success rate.

####  4.2.2 Latent Vector Compression

To investigate the effect of compression and the amount of compression, we
conducted an ablation study. We find that our chosen compression value of
$c=12$ yielded the best results in the Door-Open and Peg-Insertion tasks. This
level of compression allowed VTT to learn faster and yield generally better
performance than other compression values, as shown in Fig. 6.

Figure 6: Latent Vector Compression: We examine how compression of the latent
vector z impacts the performance of VTT and find that a value of $c=12$ yields
the best results.

###  4.3 Real-World Demonstration

Real-World Demonstration Setup: We conduct the Pushing task in the real world
using a Franka Emika Panda robot arm with a wrist-mounted ATI Gamma F/T sensor
and custom end-effector. The block is a 2.25-in steel cube with Apriltags on
its visible sides. These tags are tracked by two Intel RealSense D435 cameras
located in front of the robot (Camera 1) and to the back right of the robot
(Camera 2). We use the RGB image from Camera 1 as input $E_{I}$ and utilize
Camera 2 to track the block in case of occlusion. We use the reaction force
and torque from the F/T sensor as input $E_{T}$. This setup is shown in Fig.
7. The agent will receive +25 reward for finishing the task and
$-10|Loc_{block}-Loc_{goal}|$ penalty when the task is not finished. Each
trial has a 100 step maximum.

Real-World Baseline Comparison: Similarly to the simulated tasks, we evaluated
VTT against the concatenation and PoE baselines. As in the simulation results,
our approach outperforms the other fusion modules in the real world as shown
in Fig. 5.

Figure 7: Real-World Demonstration Setup: In the left pane we show two
successful block pushing tasks. In the right pane we note the location of our
cameras and F/T sensor, block, and goal.

###  4.4 Attention Map Visualization

We illustrate the output of VTT’s attention mechanism in Fig. 8 for simulated
Pushing and Door-Open tasks. These heatmaps are the average of the multi-head
attention from Eq. 4. We empirically observe 2 distinct phases: contact-free
and in-contact. In contact-free steps, we observe the attention heatmap
highlight the robot and the object (Pushing $t=1$, Door-Open $t=2$). This is
likely due to correlation between the goal and the change in the tactile
signature that arises from contact between end-effector and object. Once
contact is present, the attention focuses mostly on the contact interaction
and the goal. Again, this is likely due to the correlation between the reward
signal and the tactile signature. We note that the object and the goal can be
far from the robot in the visual scene, yet our attention mechanism is able to
correctly identify these key features.

Figure 8: Attention Heatmap: We ovelay the attention heatmaps from Eq. 4 onto
RGB inputs. Further visualizations are available in Sec. 6.3 along with plots
of visual and tactile attention at the timesteps visualized. Additional
examples are shown in Figs. 9 and 10 (Sim and Real-world).

##  5 Discussion and Limitations

We expect that VTT will enable important downstream applications such as
visuo-tactile curiosity. Curiosity [48] makes exploration more effective by
maximizing a proxy reward. This reward is often formulated as a series of
actions that increase representation entropy. Similarly to [45], VTT can
measure potential mismatch in vision and touch and use this score to explore.
VTT advantageously incorporates spatial attention to localize regions of
interest in an image, which sets it apart from other multi-modal
representation methods. Despite its promise, the current iteration of VTT has
a few limitations. Firstly, our implementation uses 6-dimensional reaction
wrenches from F/T sensors as tactile feedback. This low-dimensional signal can
often alias important contact events and is generally not as informative as
collocated tactile sensing like [49, 50]. To address this, the tactile
patching and embedding need to be adapted to handle high-dimensional tactile
signatures. Secondly, we only evaluated VTT on rigid-body interactions. This
choice was motivated by the availability of simulation platforms that
incorporate both tactile feedback and deformables and that are fast enough for
RL applications. It is not entirely clear how VTT will handle deformables;
though we suspect that attention will be distributed over both the deformation
and the contact region.

## References

  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929, 2020.
  * Hafner et al. [2019a] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi.  Dream to control: Learning behaviors by latent imagination.  _CoRR_ , abs/1912.01603, 2019a.
  * Hafner et al. [2019b] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International Conference on Machine Learning_ , pages 2555–2565, 2019b.
  * Lee et al. [2020] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 741–752. Curran Associates, Inc., 2020.
  * Yarats et al. [2019] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus.  Improving sample efficiency in model-free reinforcement learning from images, 10 2019.
  * Ha and Schmidhuber [2018] D. Ha and J. Schmidhuber.  World models.  _CoRR_ , abs/1803.10122, 2018.
  * De Bruin et al. [2018] T. De Bruin, J. Kober, K. Tuyls, and R. Babuška.  Integrating state representation learning into deep reinforcement learning.  _IEEE Robotics and Automation Letters_ , 3(3):1394–1401, 2018.
  * Hafner et al. [2018] D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  _CoRR, abs/1811.04551_ , 2018.
  * Stooke et al. [2021] A. Stooke, K. Lee, P. Abbeel, and M. Laskin.  Decoupling representation learning from reinforcement learning.  In _International Conference on Machine Learning_ , pages 9870–9879. PMLR, 2021.
  * Lee et al. [2018] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg.  Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks.  _CoRR_ , abs/1810.10191, 2018.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine.  Soft actor-critic algorithms and applications.  _CoRR_ , abs/1812.05905, 2018.
  * Zhang et al. [2018] M. Zhang, S. Vikram, L. M. Smith, P. Abbeel, M. J. Johnson, and S. Levine.  SOLAR: deep structured latent representations for model-based reinforcement learning.  _CoRR_ , abs/1808.09105, 2018.  URL http://arxiv.org/abs/1808.09105.
  * Fu et al. [2021] Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak.  Coupling vision and proprioception for navigation of legged robots.  _CoRR_ , abs/2112.02094, 2021.
  * Torabi et al. [2019] F. Torabi, G. Warnell, and P. Stone.  Imitation learning from video by leveraging proprioception.  _CoRR_ , abs/1905.09335, 2019.
  * Cong et al. [2022] L. Cong, H. Liang, P. Ruppel, Y. Shi, M. Görner, N. Hendrich, and J. Zhang.  Reinforcement learning with vision-proprioception model for robot planar pushing.  _Frontiers in Neurorobotics_ , 16:829437, 03 2022.  doi:10.3389/fnbot.2022.829437.
  * Lee et al. [2020] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg.  Making sense of vision and touch: Learning multimodal representations for contact-rich tasks.  _IEEE Transactions on Robotics_ , 36(3):582–596, 2020.
  * Li et al. [2019] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba.  Connecting touch and vision via cross-modal prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10609–10618, 2019.
  * Fazeli et al. [2019] N. Fazeli, M. Oller, J. Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez.  See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion.  _Science Robotics_ , 4(26):eaav3123, 2019.
  * Narang et al. [2021] Y. Narang, B. Sundaralingam, M. Macklin, A. Mousavian, and D. Fox.  Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6444–6451. IEEE, 2021.
  * Zheng et al. [2020] W. Zheng, H. Liu, and F. Sun.  Lifelong visual-tactile cross-modal learning for robotic material perception.  _IEEE transactions on neural networks and learning systems_ , 32(3):1192–1203, 2020.
  * Driess et al. [2022] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint.  Learning multi-object dynamics with compositional neural radiance fields.  _arXiv preprint arXiv:2202.11855_ , 2022.
  * Wi et al. [2022] Y. Wi, P. Florence, A. Zeng, and N. Fazeli.  Virdo: Visio-tactile implicit representations of deformable objects.  _arXiv preprint arXiv:2202.00868_ , 2022.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need, 2017.
  * Deisenroth and Rasmussen [2011] M. Deisenroth and C. E. Rasmussen.  Pilco: A model-based and data-efficient approach to policy search.  In _Proceedings of the 28th International Conference on machine learning (ICML-11)_ , pages 465–472. Citeseer, 2011.
  * Gal et al. [2016] Y. Gal, R. McAllister, and C. E. Rasmussen.  Improving pilco with bayesian neural network dynamics models.  In _Data-Efficient Machine Learning workshop, ICML_ , volume 4, page 25, 2016.
  * Henaff et al. [2017] M. Henaff, W. F. Whitney, and Y. LeCun.  Model-based planning with discrete and continuous actions.  _arXiv preprint arXiv:1705.07177_ , 2017.
  * Amos et al. [2018] B. Amos, L. Dinh, S. Cabi, T. Rothörl, S. G. Colmenarejo, A. Muldal, T. Erez, Y. Tassa, N. de Freitas, and M. Denil.  Learning awareness models.  _arXiv preprint arXiv:1804.06318_ , 2018.
  * Rafailov et al. [2021] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn.  Offline reinforcement learning from images with latent space models.  In _Learning for Dynamics and Control_ , pages 1154–1168. PMLR, 2021\.
  * Rybkin et al. [2021] O. Rybkin, C. Zhu, A. Nagabandi, K. Daniilidis, I. Mordatch, and S. Levine.  Model-based reinforcement learning via latent-space collocation.  In _International Conference on Machine Learning_ , pages 9190–9201. PMLR, 2021.
  * Allshire et al. [2021] A. Allshire, R. Martín-Martín, C. Lin, S. Manuel, S. Savarese, and A. Garg.  Laser: Learning a latent action space for efficient reinforcement learning.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6650–6656. IEEE, 2021.
  * Zhang et al. [2020] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine.  Learning invariant representations for reinforcement learning without reconstruction.  _arXiv preprint arXiv:2006.10742_ , 2020.
  * Gelada et al. [2019] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare.  Deepmdp: Learning continuous latent space models for representation learning.  In _International Conference on Machine Learning_ , pages 2170–2179. PMLR, 2019.
  * Okada and Taniguchi [2021] M. Okada and T. Taniguchi.  Dreaming: Model-based reinforcement learning by latent imagination without reconstruction.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4209–4215. IEEE, 2021.
  * Tsai et al. [2019] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Florence, Italy, 7 2019\. Association for Computational Linguistics.
  * Wei et al. [2020] X. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu.  Multi-modality cross attention network for image and sentence matching.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10938–10947, 2020.  doi:10.1109/CVPR42600.2020.01095.
  * Gao et al. [2019] P. Gao, H. You, Z. Zhang, X. Wang, and H. Li.  Multi-modality latent interaction network for visual question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep Residual Learning for Image Recognition.  _CoRR_ , abs/1512.03385, 2015.
  * Baltrusaitis et al. [2017] T. Baltrusaitis, C. Ahuja, and L. Morency.  Multimodal machine learning: A survey and taxonomy.  _CoRR_ , abs/1705.09406, 2017.
  * Ryoo et al. [2021] M. S. Ryoo, A. J. Piergiovanni, A. Arnab, M. Dehghani, and A. Angelova.  Tokenlearner: What can 8 learned tokens do for images and videos?  _CoRR_ , abs/2106.11297, 2021.
  * Kurniawati [2021] H. Kurniawati.  Partially Observable Markov Decision Processes (POMDPs) and Robotics.  _CoRR_ , abs/2107.07599, 2021.
  * Igl et al. [2018] M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson.  Deep variational reinforcement learning for POMDPs.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2117–2126. PMLR, 10–15 Jul 2018.
  * Hafner et al. [2018] D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  _CoRR_ , abs/1811.04551, 2018.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  _CoRR_ , abs/1312.6114, 2013.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1861–1870. PMLR, 10–15 Jul 2018.
  * Rajeswar et al. [2021] S. Rajeswar, C. Ibrahim, N. Surya, F. Golemo, D. Vázquez, A. C. Courville, and P. O. Pinheiro.  Touch-based curiosity for sparse-reward tasks.  _CoRR_ , abs/2104.00442, 2021.
  * Narvekar et al. [2020] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone.  Curriculum learning for reinforcement learning domains: A framework and survey.  _CoRR_ , abs/2003.04960, 2020.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _ICML_ , 2017.
  * Taylor et al. [2021] I. Taylor, S. Dong, and A. Rodriguez.  Gelslim3. 0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger.  _arXiv preprint arXiv:2103.12269_ , 2021.
  * Alspach et al. [2019] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake.  Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation.  In _2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)_ , pages 597–604. IEEE, 2019.

##  6 Appendix

###  6.1 Implementation Details of VTT

With reference to the notation in Sec. 3 and Fig. 1, VTT uses one 2-D
convolution layer to patch an $84\times 84\times 3$ image into
$X_{I}\in\mathbb{R}^{36\times 3}$. Each patches goes through one linear
projection layer to make $X_{I}\in\mathbb{R}^{36\times 384}$. In the real-
world experiment, we reduce all the channel sizes to 128. Prior to patching,
VTT separates the $1\times 6$ wrist tactile feedback into $1\times 3$ reaction
force and $1\times 3$ torque. Features are extracted through one linear
projection layer to form $X_{T}\in\mathbb{R}^{2\times 384}$. Contact embedding
$X_{C}\in\mathbb{R}^{1\times 384}$, alignment embedding
$X_{A}\in\mathbb{R}^{1\times 384}$, and position embedding
$X_{P}\in\mathbb{R}^{40\times 384}$ are initialized with weights based on a
truncated normal distribution by following the class embedding initialization
in [1]. We note here that $d_{I}=d_{T}=d_{A}=d_{C}$ is required for patch-wise
concatenation.

$W_{Q}$, $W_{K}$, $W_{V}$ are formed with 2-layer MLPs. Each MLP is
constructed with two $384\times 384$ linear layers and has GELUs as activation
functions after each linear layer. The feed forward block is also constructed
with the same MLP. We chose the attention head number $h=8$ to effectively
spread attention distribution and $N=6$ attention fusion iterations.

The contact and alignment heads from the transformer encoder’s output, denoted
$C_{head}$ and $Al_{head}$, are passed through a 1-layer MLP. The MLP is
constructed with one $384\times 1$ linear layer followed by sigmoid activation
function for alignment/contact binary classification. 1 is assigned for
positive pairing/contact, 0 is assigned for negative pairing/non-contact. When
feeding negative pairs, RL’s parameters are frozen to prevent misleading
gradients for policy learning. As in [10, 16], the alignment embedding is used
alongside a training scheme that samples both aligned data and temporally
misaligned data.

The fused heads that are output from the transformer encoder
$F_{head}\in\mathbb{R}^{40\times 384}$ have dimensions that are too large for
model-based RL. Therefore, each fused head is compressed by a 2-layer MLP
constructed with $384\times 40\times 32$ linear layers followed with RELU
activation functions amid each linear layer. The resulting
$A_{N}\in\mathbb{R}^{40\times 32}$ are flattened and passed through the
2-layer MLP to form $\textbf{z}\in\mathbb{R}^{1\times 288}$.

###  6.2 Further Details of VTT with SLAC

With reference to notation in Sec. 3, the main objective of SAC is to maximize
entropy of policy $\mathcal{H}(\pi_{\phi}(\cdot|a_{t-1},\textbf{z}_{t}))$
while also maximize following POMDP based actor-critic objective functions:

$\displaystyle\begin{split}&J_{Q}(\beta)=\displaystyle\mathop{\mathbb{E}}_{\textbf{z}_{1:\tau+1}^{d}\sim
q}[\frac{1}{2}(Q_{\beta}(\textbf{z}_{\tau}^{d},a_{\tau})-(r_{\tau}+\gamma
V_{\bar{\beta}}(\textbf{z}_{\tau+1}^{d})))^{2}]\\\ &J_{\pi}(\zeta)=\
\displaystyle\mathop{\mathbb{E}}_{\textbf{z}_{1:\tau+1}^{d}\sim
q}[\displaystyle\mathop{\mathbb{E}}_{a_{\tau}\sim\pi_{\zeta}}[\alpha
log(\pi_{\zeta}(a_{\tau+1}|\textbf{z}_{1:\tau+1},a_{1:\tau}))-Q_{\beta}(\textbf{z}_{\tau+1}^{d},a_{\tau+1})]]\end{split}$
(10)

Where $\beta$ is the parameter of soft Q-function, and ${\bar{\beta}}$ is
delay of $\beta$ for state value V-function. $\zeta$ is parameter of policy.
Unlike typical model-based RL, instead of using belief representation, SLAC’s
policy learning is only conditioned on the past and current $\tau+1$ steps.
The hyperparameters of SLAC follow its published code.

###  6.3 Further Attention Visualization

In addition to visualizing the attention on the RGB inputs, we plot the
proportion of attention between visual and tactile inputs over time during the
simulated Door-Open task and the real-world Pushing task. We find that at the
beginning of the task the attention is dominated by vision in both simulation
and the real world. Upon instances of contact, we find that the attention
heatmap contracts to the pixel-space of the RGB input where contact is
located. At the same time, the proportion of tactile attention increases. In
some cases, such as in Row 4 of Fig. 10 (green/green), the modalities meet in
the middle near a 50/50 split of attention. We find that in the real-world
task, the attention is slightly more balanced at the start of the task and
again becomes evenly split upon contact. In some real-world examples, the
tactile attention spikes above the visual attention immediately after contact
then becomes balanced around 50/50 (Row 1, Row 2). We find this result to be
unsurprising because it is in line with our intuition about the nature of
multimodality in contact-rich settings such as robotic manipulation. In
general, we think of vision as being a global sensing modality and touch as
being a local sensing modality. Thus, it stands to reason that until contact
is detected, vision outweighs touch.

Figure 9: Attention Visualization for Simulated Tasks: On the left side, we
overlay the attention heatmaps from Eq. 4 onto RGB inputs for 5 instances of
the simulated Door-Open task. On the right side, we plot the visual and
tactile attention during the task and note the time steps that correspond to
the RGB visualizations on the left. Figure 10: Attention Visualization for
Real-World Tasks: On the left side, we overlay the attention heatmaps from Eq.
4 onto RGB inputs for 4 instances of the real-world Pushing task. On the right
side, we plot the visual and tactile attention during the task and note the
time steps that correspond to the RGB visualizations on the left.

###  6.4 Parameter Count

We present the exact parameter counts for each model ablated in Sec. 6.4. This
ablation is motivated by VTT having roughly 5-6x parameters as our
implementation of the concatenation and PoE baselines. After adjusting each of
these baselines to have comparable parameters to VTT, we found that their
performance did not improve.

Table 1: Parameter counts for all methods. Method | Parameters
---|---
Concatenation |  $2.228$E$5$
PoE |  $2.889$E$5$
Concatenation w/ Adjustment |  $1.110$E$6$
PoE w/ Adjustment |  $1.201$E$6$
MulT |  $1.118$E$6$
VTT |  $1.193$E$6$

To ensure that VTT’s outperformance of the baselines is not due to the
expressiveness of the models, we conduct an ablation study over parameter
count. We increase the size of the concatenation and PoE baselines to be
comparable to the size of VTT. As we show in Fig. 11, we find that increasing
the sizes of concatenation and PoE worsen their performance. Parameter counts
are reported in Table 1.

Figure 11: Parameter Adjustment: We examine how the size of each model impacts
performance and find that larger concatenation and PoE models do not yield
better results.