# Atomic Norm Minimization-based Low-Overhead Channel Estimation for RIS-aided
MIMO Systems ††thanks: This research was supported by the MSIP (Ministry of
Science, ICT and Future Planning), Korea, under the ITRC (Information
Technology Research Center) support program (IITP-2021-2017-0-01637)
supervised by the IITP (Institute for Information & communications Technology
Promotion).

Hyeonjin Chung and Sunwoo Kim
Department of Electronics and Computer Engineering, Hanyang University, Seoul,
South Korea
E-mail<EMAIL_ADDRESS>
###### Abstract

Large beam training overhead has been considered as one of main issues in the
channel estimation for reconfigurable intelligent surface (RIS)-aided systems.
In this paper, we propose an atomic norm minimization (ANM)-based low-overhead
channel estimation for RIS-aided multiple-input-multiple-output (MIMO)
systems. When the number of beam training is reduced, some multipath signals
may not be received during beam training, and this causes channel estimation
failure. To solve this issue, the width of beams created by RIS is widened to
capture all multipath signals. Pilot signals received during beam training are
compiled into one matrix to define the atomic norm of the channel for RIS-
aided MIMO systems. Simulation results show that the proposed algorithm
outperforms other channel estimation algorithms.

###### Index Terms:

Reconfigurable intelligent surface, atomic norm minimization, channel
estimation, low-overhead, MIMO

##  I Introduction

Reconfigurable intelligent surface (RIS) has been received considerable
attention owing to the property that it can programmably change the
propagation characteristic of the signal [1]. When it comes to millimeter-wave
communications, RIS can make the signal bypass the blockage by reflecting the
signal from the base station (BS) to the user equipment (UE) [2]. The advent
of RIS has generated various new research, and one of them is the channel
estimation for RIS-aided systems. One of major concerns in the channel
estimation for RIS-aided systems is that the RIS has been added to the link
between BS and UE, and the addition of RIS causes the excessive beam training
overhead for channel estimation.

To address the issue on large overhead, the channel estimation algorithms
based on compressive sensing (CS) [3, 4, 5] and atomic norm minimization (ANM)
[6] have been introduced, assuming that signal paths are sparse. Channel
estimation algorithms for RIS-aided systems in [3, 4, 6] commonly estimate
angle-of-departures (AoDs) and angle-of-arrivals (AoAs), then estimate channel
gains to construct the channel. Algorithms in [3, 4] exploit CS for AoD/AoA
estimation, but in this case, a grid-mismatch limits the estimation accuracy
[7]. In [6], the grid-mismatch is handled by ANM, however, the channel
estimation becomes inaccurate when AoAs and AoDs are closely separated. The
channel estimation for RIS-aided multi-user systems is proposed in [5], where
the algorithm in [5] jointly estimates multiple channels by using CS-based
multi-user joint channel estimator.

Although there have been various studies on the low-overhead channel
estimation, the problem induced by the shortage of training beams and how it
affects the channel estimation have not been discussed properly. In this
paper, we propose an ANM-based low-overhead channel estimation for RIS-aided
multiple-input-multiple-output (MIMO) systems. When the beam training is
reduced, an erroneous multipath reception may occur and induces the channel
estimation failure. To tackle this issue, a training beamwidth adaptation is
proposed to widen the beamwidth when there is less beam training. Then, the
atomic norm of the channel for RIS-aided MIMO systems is defined, where
defining the atomic norm is feasible when pilot signals received during beam
training are compiled in a specified manner. A detailed explanation on the
proposed algorithm is presented in following sections.

Notations: We use lower-case and upper-case bold characters to respectively
represent vectors and matrices throughout this paper. $(\cdot)^{T}$,
$(\cdot)^{H}$, and $(\cdot)^{*}$ respectively denote transpose, conjugate
transpose, and complex conjugation. $\textrm{Tr}(\cdot)$ denotes the trace of
a matrix, and $\textrm{diag}(\cdot)$ denotes the diagonal matrix whose
diagonal entries equal to entries of given vector. $\textrm{vec}(\cdot)$
denotes vectorization of given matrix. $\lVert\cdot\rVert_{\textrm{F}}$
denotes Frobenius norm. The curled inequality symbol $\succeq$ denotes matrix
inequality. If $\mathbf{A}\succeq\mathbf{B}$, a matrix $\mathbf{A}-\mathbf{B}$
is positive semidefinite. $\otimes$ and $\diamond$ respectively denote
Kronecker product and Khatri-Rao product. $\mathbf{I}_{N}$ denotes a $N\times
N$ identity matrix.

##  II Channel and Signal Model

We consider a downlink RIS-aided MIMO system, which means that a base station
(BS) transmits the signal to a RIS and the RIS bounces back the signal to an
user equipment (UE). The BS, the RIS, and the UE are equipped with
$M_{\textrm{B}}$, $M_{\textrm{R}}$, and $M_{\textrm{U}}$ antennas
respectively. Here, antenna arrays that BS, RIS, and UE use are uniform linear
arrays (ULAs) with half-lambda spacing. In this paper, the BS and the UE
employ full-complexity hybrid beamforming structure [8], where the BS and the
UE are respectively equipped with $N_{\textrm{B}}$ and $N_{\textrm{U}}$ RF
chains.

The steering vector of the ULA with half-lambda spacing, $\mathbf{a}(\theta)$
is

$\mathbf{a}(\theta)=[1,e^{j\pi\cos\theta},\ldots,e^{j\pi(M-1)\cos\theta}]^{T}\in\mathbb{C}^{M\times
1},$ (1)

where $\theta$ denotes the steering direction, and $M$ denotes the number of
antennas. A scheme of RIS-aided MIMO system considered in this paper is given
in Fig. 1. Assuming all signal paths between BS and UE are blocked, a channel
for RIS-aided MIMO system can be represented as a cascade of two separate
channels: BS-to-RIS channel and RIS-to-UE channel. The BS-to-RIS channel
$\mathbf{H}_{\textrm{BR}}$ can be given by

$\begin{split}\mathbf{H}_{\textrm{BR}}&=\sum_{l=1}^{L_{\textrm{BR}}}\alpha_{\textrm{BR}}^{l}\mathbf{a}(\phi_{\textrm{BR}}^{l})\mathbf{a}(\theta_{\textrm{BR}}^{l})^{H}\\\
&=\mathbf{A}(\bm{\phi}_{\textrm{BR}})\textrm{diag}(\bm{\rho}_{\textrm{BR}})\mathbf{A}(\bm{\theta}_{\textrm{BR}})^{H}\in\mathbb{C}^{M_{\textrm{R}}\times
M_{\textrm{B}}},\end{split}$ (2)

where $L_{\textrm{BR}}$ denotes the number of signal paths between BS and RIS.
$\alpha_{\textrm{BR}}^{l}$, $\phi_{\textrm{BR}}^{l}$, and
$\theta_{\textrm{BR}}^{l}$ respectively denote the channel gain, the BS-to-RIS
AoA, and the BS-to-RIS AoD of the $l$-th signal path.
$\bm{\phi}_{\textrm{BR}}=\\{{\phi}^{1}_{\textrm{BR}},\ldots,{\phi}^{L_{\textrm{BR}}}_{\textrm{BR}}\\}$
and
$\bm{\theta}_{\textrm{BR}}=\\{{\theta}^{1}_{\textrm{BR}},\ldots,{\theta}^{L_{\textrm{BR}}}_{\textrm{BR}}\\}$.
$\mathbf{A}(\bm{\phi}_{\textrm{BR}})=[\mathbf{a}(\phi_{\textrm{BR}}^{1}),\ldots,\mathbf{a}(\phi_{\textrm{BR}}^{L_{\textrm{BR}}})]\in\mathbb{C}^{M_{\textrm{R}}\times
L_{\textrm{BR}}}$,
$\mathbf{A}(\bm{\theta}_{\textrm{BR}})=[\mathbf{a}(\theta_{\textrm{BR}}^{1}),\ldots,\mathbf{a}(\theta_{\textrm{BR}}^{L_{\textrm{BR}}})]\in\mathbb{C}^{M_{\textrm{B}}\times
L_{\textrm{BR}}}$, and
$\bm{\rho}_{\textrm{BR}}=[\alpha_{\textrm{BR}}^{1},\ldots,\alpha_{\textrm{BR}}^{L_{\textrm{BR}}}]^{T}$.
The RIS-to-UE channel $\mathbf{H}_{\textrm{RU}}$ can be given by

$\begin{split}\mathbf{H}_{\textrm{RU}}&=\sum_{l=1}^{L_{\textrm{RU}}}\alpha_{\textrm{RU}}^{l}\mathbf{a}(\phi_{\textrm{RU}}^{l})\mathbf{a}(\theta_{\textrm{RU}}^{l})^{H}\\\
&=\mathbf{A}(\bm{\phi}_{\textrm{RU}})\textrm{diag}(\bm{\rho}_{\textrm{RU}})\mathbf{A}(\bm{\theta}_{\textrm{RU}})^{H}\in\mathbb{C}^{M_{\textrm{U}}\times
M_{\textrm{R}}},\end{split}$ (3)

where $L_{\textrm{RU}}$ denotes the number of signal paths between RIS and UE.
$\alpha_{\textrm{RU}}^{l}$, $\phi_{\textrm{RU}}^{l}$, and
$\theta_{\textrm{RU}}^{l}$ respectively denote the channel gain, the RIS-to-UE
AoA, and the RIS-to-UE AoD of the $l$-th signal path.
$\bm{\phi}_{\textrm{RU}}=\\{{\phi}^{1}_{\textrm{RU}},\ldots,{\phi}^{L_{\textrm{RU}}}_{\textrm{RU}}\\}$
and
$\bm{\theta}_{\textrm{RU}}=\\{{\theta}^{1}_{\textrm{RU}},\ldots,{\theta}^{L_{\textrm{RU}}}_{\textrm{RU}}\\}$.
$\mathbf{A}(\bm{\phi}_{\textrm{RU}})=[\mathbf{a}(\phi_{\textrm{RU}}^{1}),\ldots,\mathbf{a}(\phi_{\textrm{RU}}^{L_{\textrm{RU}}})]\in\mathbb{C}^{M_{\textrm{U}}\times
L_{\textrm{RU}}}$,
$\mathbf{A}(\bm{\theta}_{\textrm{RU}})=[\mathbf{a}(\theta_{\textrm{RU}}^{1}),\ldots,\mathbf{a}(\theta_{\textrm{RU}}^{L_{\textrm{RU}}})]\in\mathbb{C}^{M_{\textrm{R}}\times
L_{\textrm{RU}}}$, and
$\bm{\rho}_{\textrm{RU}}=[\alpha_{\textrm{RU}}^{1},\ldots,\alpha_{\textrm{RU}}^{L_{\textrm{RU}}}]^{T}$.

Figure 1: A scheme of RIS-aided MIMO system. The signal path between BS and UE
is blocked.

A RIS control matrix $\bm{\Omega}$ can be given by

$\bm{\Omega}=\begin{bmatrix}\beta_{1}e^{j\vartheta_{1}}&0&\ldots&0\\\
0&\beta_{2}e^{j\vartheta_{2}}&\ldots&0\\\ \vdots&\vdots&\ddots&\vdots\\\
0&0&\ldots&\beta_{M_{\textrm{R}}}e^{j\vartheta_{M_{\textrm{R}}}}\end{bmatrix}\in\mathbb{C}^{M_{\textrm{R}}\times
M_{\textrm{R}}},$ (4)

where $\beta_{m}$ and $\vartheta_{m}$ respectively denote a reflection
coefficient and a phase shift of the $m$-th antenna in RIS.
$\vartheta_{m}\in[0,2\pi)$ and $\beta_{m}$ can be either $0$ or $1$, where $0$
and $1$ respectively denotes the deactivation and the activation of the $m$-th
antenna in the RIS. For a convenient representation of $\bm{\Omega}$, a RIS
control vector $\bm{\omega}$ is defined as

$\bm{\omega}=\left[\beta_{1}e^{j\vartheta_{1}},\beta_{2}e^{j\vartheta_{2}},\ldots,\beta_{M_{\textrm{R}}}e^{j\vartheta_{M_{\textrm{R}}}}\right]^{T}\in\mathbb{C}^{M_{\textrm{R}}\times
1}.$ (5)

Note that $\bm{\Omega}=\textrm{diag}(\bm{\omega})$. The cascaded channel for
RIS-aided MIMO system, $\mathbf{H}$ can be given by

$\mathbf{H}=\mathbf{H}_{\textrm{RU}}\bm{\Omega}\mathbf{H}_{\textrm{BR}}\in\mathbb{C}^{M_{\textrm{U}}\times
M_{\textrm{B}}}.$ (6)

A frame structure for beam training procedure in RIS-aided MIMO systems is
depicted in Fig. 2. To simplify notations, $M_{\textrm{B}}/N_{\textrm{B}}$ and
$M_{\textrm{U}}/N_{\textrm{U}}$ are respectively defined as $P_{\textrm{B}}$
and $P_{\textrm{U}}$. There are $P_{\textrm{B}}$ precoding matrices and
$P_{\textrm{U}}$ combining matrices, and
$\mathbf{F}_{i}\in\mathbb{C}^{M_{\textrm{B}}\times N_{\textrm{B}}}$ and
$\mathbf{C}_{j}\in\mathbb{C}^{M_{\textrm{U}}\times N_{\textrm{U}}}$
respectively denote the $i$-th precoding matrix and the $j$-th combining
matrix. The RIS control matrix changes frame by frame, and the BS and the UE
perform a total $P_{\textrm{B}}P_{\textrm{U}}$ beam training at each frame.
Letting $B$ denotes the number of frames, the total number of beam training
$P$ equals to $BP_{\textrm{B}}P_{\textrm{U}}$. $\mathbf{X}^{i,j}_{b}$, a
received pilot signal at the $b$-th frame which uses the $i$-th precoding
matrix and the $j$-th combining matrix, can be given by

$\mathbf{X}^{i,j}_{b}=\mathbf{C}_{j}^{H}\mathbf{H}_{\textrm{RU}}\bm{\Omega}_{b}\mathbf{H}_{\textrm{BR}}\mathbf{F}_{i}\mathbf{S}+\mathbf{N}^{i,j}_{b}\in\mathbb{C}^{N_{\textrm{U}}\times
D},$ (7)

where $\bm{\Omega}_{b}$ denotes the RIS control matrix at the $b$-th frame,
and $D$ denotes the number of signal samples per one beam training.
$\mathbf{S}=\left[\mathbf{s}_{1},\ldots,\mathbf{s}_{N_{\textrm{B}}}\right]^{T}\in\mathbb{C}^{N_{\textrm{B}}\times
D}$, where $\mathbf{s}_{n}$ is the $n$-th unit-energy pilot signal that
satisfies $\mathbf{s}^{H}_{n}\mathbf{s}_{n}/D=1$. Note that pilot signals are
orthogonal to each other so that
$\mathbf{S}\mathbf{S}^{H}/D=\mathbf{I}_{N_{\textrm{B}}}$.
$\mathbf{N}^{i,j}_{b}\in\mathbb{C}^{N_{\textrm{U}}\times D}$ is a noise matrix
whose entries follow a circularly-symmetric complex Gaussian distribution.
Here, the mean and the covariance are $0$ and $\sigma^{2}$ respectively.

Figure 2: A frame structure for beam training procedure in RIS-aided MIMO
communication.

(a) Radiation patterns without training beamwidth adaptation

(b) Radiation patterns with training beamwidth adaptation

Figure 3: Radiation patterns of beams created by RIS when $L_{\textrm{BR}}=1$,
$L_{\textrm{RU}}=3$, $M_{\textrm{R}}=16$, and $B=10$. For successful multipath
signal reception, every RIS-to-UE AoD has to be captured within one of beams.

After receiving $\mathbf{X}^{i,j}_{b}$, $\mathbf{S}^{H}$ are multiplied to
$\mathbf{X}^{i,j}_{b}$ to filter the noise. We define this filtered signal
$\mathbf{Y}^{i,j}_{b}$ as

$\mathbf{Y}^{i,j}_{b}=\frac{\mathbf{X}^{i,j}_{b}\mathbf{S}^{H}}{D}=\mathbf{C}_{j}^{H}\mathbf{H}_{\textrm{RU}}\bm{\Omega}_{b}\mathbf{H}_{\textrm{BR}}\mathbf{F}_{i}+\frac{\mathbf{N}^{i,j}_{b}\mathbf{S}^{H}}{D}\in\mathbb{C}^{N_{\textrm{U}}\times
N_{\textrm{B}}}.$ (8)

A total $P_{\textrm{B}}P_{\textrm{U}}$ filtered signals received at the $b$-th
frame is organized as

$\mathbf{Y}_{b}=\begin{bmatrix}\mathbf{Y}^{1,1}_{b}&\mathbf{Y}^{2,1}_{b}&\cdots&\mathbf{Y}^{P_{\textrm{B}},1}_{b}\\\
\mathbf{Y}^{1,2}_{b}&\mathbf{Y}^{2,2}_{b}&\cdots&\mathbf{Y}^{P_{\textrm{B}},2}_{b}\\\
\vdots&\vdots&\cdots&\vdots\\\
\mathbf{Y}^{1,P_{\textrm{U}}}_{b}&\mathbf{Y}^{2,P_{\textrm{U}}}_{b}&\cdots&\mathbf{Y}^{P_{\textrm{B}},P_{\textrm{U}}}_{b}\end{bmatrix}\in\mathbb{C}^{M_{\textrm{U}}\times
M_{\textrm{B}}},$ (9)

where $\mathbf{Y}_{b}$ is a compilation of all filtered signals at the $b$-th
frame. $\mathbf{Y}_{b}$ can also be represented as

$\mathbf{Y}_{b}=\mathbf{C}^{H}\mathbf{H}_{\textrm{RU}}\bm{\Omega}_{b}\mathbf{H}_{\textrm{BR}}\mathbf{F}+\mathbf{V}_{b},$
(10)

where $\mathbf{F}\in\mathbb{C}^{M_{\textrm{B}}\times M_{\textrm{B}}}$ and
$\mathbf{C}\in\mathbb{C}^{M_{\textrm{U}}\times M_{\textrm{U}}}$ respectively
denote a full-rank precoding matrix and a full-rank combining matrix.
$\mathbf{V}_{b}\in\mathbb{C}^{M_{\textrm{U}}\times M_{\textrm{B}}}$ is a
matrix that represents the remaining noise.

##  III The Proposed Low-Overhead Channel Estimation for RIS-aided MIMO
Systems

###  III-A Compilation of Filtered Signals and Its Representation via
Kronecker Product and Khatri-Rao Product

To define the atomic norm of the channel, we compile filtered signals and
present an organized representation of compiled filtered signals. For the
compilation, two following properties are employed.

  * •

Property 1:
$\textrm{vec}\left(\mathbf{A\textrm{diag}(\mathbf{b})C}\right)=(\mathbf{C}^{T}\diamond\mathbf{A})\mathbf{b}.$

  * •

Property 2:
$(\mathbf{AB}\diamond\mathbf{CD})=(\mathbf{A}\otimes\mathbf{C})(\mathbf{B}\diamond\mathbf{D}).$

Definitions and properties of Kronecker product and Khatri-Rao product are
well-explained in [9]. Letting a lengthy column vector $\mathbf{y}_{b}$ equals
$\textrm{vec}(\mathbf{Y}_{b})$, $\mathbf{y}_{b}$ can be represented as follows
by using Property 1.

$\mathbf{y}_{b}=\textrm{vec}(\mathbf{Y}_{b})=\left(\mathbf{F}^{T}\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{C}^{H}\mathbf{H}_{\textrm{RU}}\right)\bm{\omega}_{b}+\mathbf{v}_{b}\in\mathbb{C}^{M_{\textrm{B}}M_{\textrm{U}}\times
1},$ (11)

where $\bm{\omega}_{b}$ denotes the RIS control vector at the $b$-th frame,
and $\mathbf{v}_{b}=\textrm{vec}(\mathbf{V}_{b})$.

Then, we form a matrix $\bm{\mathcal{Y}}$ which is constructed by stacking
$\mathbf{y}_{b}$ for $b=1,\ldots,B$ as follows.

$\begin{split}\bm{\mathcal{Y}}&=\left[\mathbf{y}_{1},\ldots,\mathbf{y}_{B}\right]\\\
&=\left(\mathbf{F}^{T}\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{C}^{H}\mathbf{H}_{\textrm{RU}}\right)\mathbf{W}+\bm{\mathcal{V}}\in\mathbb{C}^{M_{\textrm{B}}M_{\textrm{U}}\times
B},\end{split}$ (12)

where
$\mathbf{W}=[\bm{\omega}_{1},\ldots,\bm{\omega}_{B}]\in\mathbb{C}^{M_{\textrm{R}}\times
B}$ and
$\bm{\mathcal{V}}=[\mathbf{v}_{1},\ldots,\mathbf{v}_{B}]\in\mathbb{C}^{M_{\textrm{B}}M_{\textrm{U}}\times
B}$. By using Property 2, $\bm{\mathcal{Y}}$ can be also represented as

$\bm{\mathcal{Y}}=\left(\mathbf{F}^{T}\otimes\mathbf{C}^{H}\right)\left(\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}\right)\mathbf{W}+\bm{\mathcal{V}}.$
(13)

Here,
$\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}\in\mathbb{C}^{M_{\textrm{B}}M_{\textrm{U}}\times
M_{\textrm{R}}}$ contains a channel information that is independent of
$\mathbf{F}$, $\mathbf{C}$, and $\mathbf{W}$. Once
$\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}$ is successfully
estimated, the optimal RIS control matrix that maximizes SNR can be derived by
conducting singular value decomposition (SVD) to
$\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}$ [6]. Throughout
this paper, we define
$\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}$ as an effective
channel $\mathbf{H}_{\textrm{eff}}$, which is a goal of channel estimation for
RIS-aided MIMO systems.

With (2), (3), and Property 2, $\bm{\mathcal{Y}}$ can be fully unfolded as

$\begin{split}\bm{\mathcal{Y}}&=\left(\mathbf{F}^{T}\otimes\mathbf{C}^{H}\right)\left(\mathbf{A}(\bm{\theta}_{\textrm{BR}})^{*}\otimes\mathbf{A}(\bm{\phi}_{\textrm{RU}})\right)\\\
&\left(\textrm{diag}(\bm{\rho}_{\textrm{BR}})\otimes\textrm{diag}(\bm{\rho}_{\textrm{RU}})\right)\left(\mathbf{A}(\bm{\phi}_{\textrm{BR}})^{T}\diamond\mathbf{A}(\bm{\theta}_{\textrm{RU}})^{H}\right)\mathbf{W}+\bm{\mathcal{V}}.\end{split}$
(14)

To simplify
$\mathbf{A}(\bm{\phi}_{\textrm{BR}})^{T}\diamond\mathbf{A}(\bm{\theta}_{\textrm{RU}})^{H}\in\mathbb{C}^{L_{\textrm{BR}}L_{\textrm{RU}}\times
M_{\textrm{R}}}$, $\bm{\varphi}$ is defined as

$\begin{split}\bm{\varphi}=\\{\varphi_{i,j}&:\cos^{-1}(\cos\theta^{j}_{\textrm{RU}}-\cos\phi^{i}_{\textrm{BR}}),\\\
&i=1,\ldots,L_{\textrm{BR}},j=1,\ldots,L_{\textrm{RU}}\\}.\end{split}$ (15)

Then,
$\mathbf{A}(\bm{\phi}_{\textrm{BR}})^{T}\diamond\mathbf{A}(\bm{\theta}_{\textrm{RU}})^{H}$
can be rewritten as

$\begin{split}&\mathbf{A}(\bm{\phi}_{\textrm{BR}})^{T}\diamond\mathbf{A}(\bm{\theta}_{\textrm{RU}})^{H}=\mathbf{A}(\bm{\varphi})^{H}\\\
&=\left[\mathbf{a}(\varphi_{1,1}),\ldots,\mathbf{a}(\varphi_{1,L_{\textrm{RU}}}),\ldots,\mathbf{a}(\varphi_{L_{\textrm{BR}},1}),\ldots,\mathbf{a}(\varphi_{L_{\textrm{BR}},L_{\textrm{RU}}})\right]^{H}.\end{split}$
(16)

###  III-B Robust Multipath Signal Reception When Using Fewer Beam Training
via Training Beamwidth Adaptation

One of general ways to control RIS is to use a discrete Fourier transform
(DFT) matrix [10]. $N\times N$ DFT matrix $\bm{\Psi}_{N}$ can be given by

$\bm{\Psi}_{N}=\begin{bmatrix}1&1&\cdots&1\\\
1&e^{j\frac{2\pi}{N}}&\cdots&e^{j\frac{2\pi(N-1)}{N}}\\\
1&e^{j\frac{4\pi}{N}}&\cdots&e^{j\frac{4\pi(N-1)}{N}}\\\
\vdots&\vdots&\cdots&\vdots\\\
1&e^{j2\pi}&\cdots&e^{j2\pi(N-1)}\end{bmatrix}\in\mathbb{C}^{N\times N}.$ (17)

For mainlobes of beams to cover the entire angular domain, $\mathbf{W}$ should
be equal to $\bm{\Psi}_{M_{\textrm{R}}}$ so that $B=M_{\textrm{R}}$. However,
considering $P=BP_{\textrm{B}}P_{\textrm{U}}$, $B$ should be reduced in order
to prevent the beam training overhead from getting excessively large. In this
subsection, we discuss on the erroneous multipath signal reception that occurs
when $B<M_{\textrm{R}}$ and how to resolve it.

The simplest way of determining RIS control vectors when $B<M_{\textrm{R}}$ is
to select $B$ columns from $\bm{\Psi}_{M_{\textrm{R}}}$ as in [4], but lack of
beams can cause erroneous multipath signal reception as in Fig. 3(a). Fig.
3(a) shows the case of the erroneous multipath signal reception when one of
RIS-to-UE signal paths does not fall onto mainlobes of $B$ beams. In this
case, the channel estimation fails since signals from all paths are required
for perfect channel estimation.

To address the issue on the erroneous multipath signal reception, a training
beamwidth adaptation is proposed to make multipath signal reception robust
when $B<M_{\textrm{R}}$. The beamwidth of RIS can be widened by deactivating
the part of RIS, and $\mathbf{W}$ that contains $B$ widened beams can be given
by

$\mathbf{W}=\begin{bmatrix}\bm{\Psi}_{B}\\\
\mathbf{O}_{M_{\textrm{R}}-B,B}\end{bmatrix}\in\mathbb{C}^{M_{\textrm{R}}\times
B},$ (18)

where $\mathbf{O}_{M,N}$ denotes a $M\times N$ zero matrix. Fig. 3(b) shows
beams that are widened by training beamwidth adaptation when
$M_{\textrm{R}}=16$ and $B=10$. In Fig. 3(b), mainlobes of $B$ beams cover the
entire angular domain so that every RIS-to-UE signal path is captured within
one of $B$ beams, although the beam gain decreases.

###  III-C Atomic Norm Minimization-based Channel Estimation for RIS-aided
MIMO Systems

The simplest approach to estimate the effective channel is to use least square
(LS) estimator. If $\mathbf{F}^{T}\otimes\mathbf{C}^{H}$ and $\mathbf{W}$ are
both full-rank matrices, the effective channel can be estimated via LS
estimator as follows.

$\hat{\mathbf{H}}_{\textrm{eff}}^{\textrm{LS}}=\left(\mathbf{F}^{T}\otimes\mathbf{C}^{H}\right)^{-1}\bm{\mathcal{Y}}\mathbf{W}^{-1},$
(19)

where $\hat{\mathbf{H}}_{\textrm{eff}}^{\textrm{LS}}$ is the effective channel
estimated by LS estimator. The LS estimator requires at least
$M_{\textrm{R}}P_{\textrm{B}}P_{\textrm{U}}$ beam training since $B$ should be
larger than $M_{\textrm{R}}$ to make $\mathbf{W}$ a full-rank matrix.
$\mathbf{F}^{T}\otimes\mathbf{C}^{H}$ is full-rank since $\mathbf{F}$ and
$\mathbf{C}$ are both full-rank matrices. It is worth noting that the beam
training overhead of LS estimator is generally large considering
$M_{\textrm{R}}$ can be more than hundreds in practice [11]. On the other
hand, the effective channel can be estimated by ANM even when
$B<M_{\textrm{R}}$ if signal paths are sparse and all multipath signals are
received properly. In this subsection, an atomic norm of the effective channel
is defined, where defining proper atomic norm leads to accurate channel
estimation.

Figure 4: A summary of the proposed ANM-based low-overhead channel estimation
for RIS-aided MIMO systems.

We assume the signal paths are sparse so that
$B>L_{\textrm{BR}}L_{\textrm{RU}}$. For simplification of equations and
notations, we define $\mathbf{G}$ and $\mathbf{Z}$ as follows.

$\begin{split}\mathbf{G}=\left\\{\left(\mathbf{F}^{T}\otimes\mathbf{C}^{H}\right)^{-1}\bm{\mathcal{Y}}\right\\}^{H}=\mathbf{W}^{H}\mathbf{Z}+\mathbf{E}\in\mathbf{C}^{B\times
M_{\textrm{B}}M_{\textrm{U}}},\end{split}$ (20)
$\begin{split}\mathbf{Z}&=\mathbf{H}_{\textrm{eff}}^{H}=\left(\mathbf{H}^{T}_{\textrm{BR}}\diamond\mathbf{H}_{\textrm{RU}}\right)^{H}\\\
&=\mathbf{A}(\bm{\varphi})\left(\textrm{diag}(\bm{\rho}_{\textrm{BR}})\otimes\textrm{diag}(\bm{\rho}_{\textrm{RU}})\right)^{H}\left(\mathbf{A}(\bm{\theta}_{\textrm{BR}})^{*}\otimes\mathbf{A}(\bm{\phi}_{\textrm{RU}})\right)^{H}.\end{split}$
(21)

Here,
$\mathbf{E}=\\{\left(\mathbf{F}^{T}\otimes\mathbf{C}^{H}\right)^{-1}\bm{\mathcal{V}}\\}^{H}$.
For the estimation of $\mathbf{Z}$, an atomic set $\mathcal{A}$ is defined as
follows.

$\mathcal{A}=\left\\{\mathbf{a}(\theta)\mathbf{b}^{T}\in\mathbb{C}^{M_{\textrm{R}}\times
M_{\textrm{B}}M_{\textrm{U}}}:0^{\circ}<\theta<180^{\circ},\lVert\mathbf{b}\rVert_{2}=1\right\\},$
(22)

where $\mathbf{a}(\theta)\mathbf{b}^{T}$ is defined as an atom, and
$\mathbf{Z}$ can be represented as a linear combination of atoms. Properties
of the atom defined in (22) and its atomic norm have been studied in seminal
works of the ANM [12, 13, 14]. The atomic norm of $\mathbf{Z}$,
$\lVert\mathbf{Z}\rVert_{\mathcal{A}}$ can be represented by following SDP
[12]:

$\begin{split}\lVert\mathbf{Z}\rVert_{\mathcal{A}}=&\min_{\mathbf{u},\mathbf{T}}\;\frac{1}{2M_{\textrm{R}}}\textrm{Tr}(\textrm{Toep}(\mathbf{u}))+\frac{1}{2}\textrm{Tr}(\mathbf{T})\\\
&\;\textrm{s.t.}\begin{bmatrix}\textrm{Toep}(\mathbf{u})&\mathbf{Z}\\\
\mathbf{Z}^{H}&\mathbf{T}\end{bmatrix}\succeq 0,\end{split}$ (23)

where $\textrm{Toep}(\mathbf{u})$ denotes a Hermitian Toeplitz matrix whose
first column equals $\mathbf{u}$. With the ANM denoising theorem studied in
[14], an equation that estimates $\mathbf{Z}$ from $\mathbf{G}$ can be given
by

$\hat{\mathbf{Z}}=\operatorname*{argmin}_{\bar{\mathbf{Z}}}\;\tau\lVert\bar{\mathbf{Z}}\rVert_{\mathcal{A}}+\frac{1}{2}\lVert\mathbf{G}-\mathbf{W}^{H}\bar{\mathbf{Z}}\rVert_{\textrm{F}}^{2},$
(24)

where $\hat{\mathbf{Z}}$ and $\bar{\mathbf{Z}}$ respectively denote the
estimate of $\mathbf{Z}$ and the variable for estimation of $\mathbf{Z}$. A
regularization parameter $\tau$ is set as in [14]:

$\begin{split}\tau=\frac{\sigma}{\sqrt{D}}&\Big{(}1+\frac{1}{\log
M_{\textrm{R}}}\Big{)}^{\frac{1}{2}}\Big{(}M_{\textrm{B}}M_{\textrm{U}}+\log(\alpha
M_{\textrm{B}}M_{\textrm{U}})+\\\
&\sqrt{2M_{\textrm{B}}M_{\textrm{U}}\log(\alpha
M_{\textrm{B}}M_{\textrm{U}})}+\sqrt{\frac{\pi
M_{\textrm{B}}M_{\textrm{U}}}{2}}+1\Big{)}^{\frac{1}{2}},\end{split}$ (25)

where $\alpha=8\pi M_{\textrm{R}}\log M_{\textrm{R}}$. (24) can be fully
unfolded as

$\begin{split}&\\{\hat{\mathbf{u}},\hat{\mathbf{T}},\hat{\mathbf{Z}}\\}=\\\
&\operatorname*{argmin}_{\mathbf{u},\mathbf{T},\bar{\mathbf{Z}}}\;\frac{\tau}{2M_{\textrm{R}}}\textrm{Tr}(\textrm{Toep}(\mathbf{u}))+\frac{\tau}{2}\textrm{Tr}(\mathbf{T})+\frac{1}{2}\lVert\mathbf{G}-\mathbf{W}^{H}\bar{\mathbf{Z}}\rVert_{\textrm{F}}^{2}\\\
&\;\textrm{s.t.}\begin{bmatrix}\textrm{Toep}(\mathbf{u})&\bar{\mathbf{Z}}\\\
\bar{\mathbf{Z}}^{H}&\mathbf{T}\end{bmatrix}\succeq 0.\end{split}$ (26)

Finally, $\mathbf{H}_{\textrm{eff}}$ can be approximated as
$\hat{\mathbf{Z}}^{H}$. The proposed ANM-based low-overhead channel estimation
for RIS-aided MIMO systems can be summarized as Fig. 4.

##  IV Simulation Results and Discussions

In this section, the proposed algorithm is compared with [4] and [6], and LS
is added as a benchmark if $B=M_{\textrm{R}}$. Note that [4] and [6] also
employ the training beamwidth adaptation in this simulation. The algorithm in
[3] is excluded from the comparison since the computation time becomes
excessively high when reforming the algorithm to work under RIS-aided MIMO
systems. For simulation, $M_{\textrm{B}}$, $M_{\textrm{R}}$, and
$M_{\textrm{U}}$ are respectively set to 8, 32, and 4, and $N_{\textrm{B}}$
and $N_{\textrm{U}}$ are set to 4 and 2 so that there are 4 beam training per
frame. $\alpha^{l}_{\textrm{BR}}\sim\mathcal{CN}(0,1)$ for
$l=1,\ldots,L_{\textrm{BR}}$ and
$\alpha^{l}_{\textrm{RU}}\sim\mathcal{CN}(0,1)$ for
$l=1,\ldots,L_{\textrm{RU}}$. All AoAs and AoDs such as
$\phi_{\textrm{BR}}^{l}$, $\theta_{\textrm{BR}}^{l}$,
$\phi_{\textrm{RU}}^{l}$, and $\theta_{\textrm{RU}}^{l}$ are chosen randomly
between $\left[30^{\circ},150^{\circ}\right]$. The SNR is defined as

$\textrm{SNR}=10\log_{10}\frac{\left|\left(\sum_{l=1}^{L_{\textrm{RU}}}\alpha_{\textrm{RU}}^{l}\right)\left(\sum_{l=1}^{L_{\textrm{BR}}}\alpha_{\textrm{BR}}^{l}\right)\right|^{2}}{\sigma^{2}}(\textrm{dB}).$
(27)

The SNR defined in (27) is a ratio of signal power and noise power when
$M_{\textrm{B}}=M_{\textrm{R}}=M_{\textrm{U}}=1$. Note that the defined SNR
does not depend on precoding matrix, combining matrix, and RIS control matrix.
The NMSE is defined as

$\textrm{NMSE}=\frac{1}{Q}\sum_{q=1}^{Q}\frac{\lVert\hat{\mathbf{H}}^{q}_{\textrm{eff}}-\mathbf{H}^{q}_{\textrm{eff}}\rVert^{2}_{\textrm{F}}}{\lVert\mathbf{H}^{q}_{\textrm{eff}}\rVert^{2}_{\textrm{F}}},$
(28)

where $Q$ is the number of Monte Carlo trials for NMSE calculation and is set
to 300. $\hat{\mathbf{H}}^{q}_{\textrm{eff}}$ and
$\mathbf{H}^{q}_{\textrm{eff}}$ respectively denote the estimated effective
channel and the actual effective channel on the $q$-th Monte Carlo trial.

Figure 5: NMSE versus interval between two RIS-to-UE AoAs. The SNR is set to 0
dB. $L_{\textrm{BR}}=1$, $L_{\textrm{RU}}=2$, and $B=16$.

Fig. 5 shows the NMSE versus interval between two RIS-to-UE AoAs when
$L_{\textrm{BR}}=1$, $L_{\textrm{RU}}=2$, and $B=16$. The SNR is set to 0 dB.
Considering that AoDs and AoAs are uniformly distributed in all angles [15],
there is a possibility that AoDs and AoAs of signal paths are closely
separated. In [4] and [6], AoD/AoA estimation can be inaccurate when AoDs and
AoAs are closely separated, where inaccurate AoD/AoA estimation leads to
channel estimation failure. To show the correlation between the channel
estimation accuracy and the separation between AoAs or AoDs, we analyze NMSE
with respect to the interval between two RIS-to-UE AoAs. In Fig. 5, the NMSE
of [4] and [6] are high when the two RIS-to-UE AoAs are closely separated as
the estimation on RIS-to-UE AoAs fails. On the other hand, the NMSE of the
proposed algorithm remains relatively low even when the interval between two
RIS-to-UE AoAs is close. When the interval between two RIS-to-UE AoAs is
$4^{\circ}$, the NMSE of proposed algorithm, [4], and [6] are $0.014$, $0.29$,
and $0.66$ respectively.

Figure 6: NMSE versus number of frames. The SNR is set to 0 dB.
$L_{\textrm{BR}}=2$ and $L_{\textrm{RU}}=2$. The number of frames equals to
the number of beams created by RIS during beam training.

Fig. 6 shows the NMSE versus number of beam training when $L_{\textrm{BR}}=2$
and $L_{\textrm{RU}}=2$. The SNR is set to 0 dB. To show that the training
beamwidth adaptation ensures robust channel estimation when there is less beam
training, the NMSE of the proposed algorithm without training beamwidth
adaptation is also presented in Fig. 6. Since the deficiency of beam training
may cause the erroneous multipath signal reception, the proposed algorithm
without training beamwidth adaptation shows high NMSE when $B<M_{\textrm{R}}$.
On the other hand, the NMSE of the proposed algorithm is lower than those of
other algorithms at every $B$. When $B$ reaches $M_{\textrm{R}}$, there is no
need for broadening beamwidth so that the NMSE of the proposed algorithm
without training beamwidth adaptation becomes equivalent to the NMSE of the
proposed algorithm. Since the channel estimation is inaccurate when AoDs or
AoAs are closely separated, the NMSE of [4] and [6] remains high even when
sufficient beam training is performed.

Figure 7: NMSE versus SNR. $L_{\textrm{BR}}=1$, $L_{\textrm{RU}}=2$, and
$B=16$.

Fig. 7 shows the NMSE versus SNR when $L_{\textrm{BR}}=2$,
$L_{\textrm{RU}}=2$, and $B=16$. The NMSE of the proposed algorithm is lower
than those of other algorithms at every SNR. On the other hand, the NMSE of
[4] and [6] do not improve with the increase of SNR and remains relatively
high. For the same reason as previous results, this is because the estimation
of [4] and [6] fails when AoDs or AoAs are closely separated, and this failure
is independent of the SNR.

From Fig. 6 and Fig. 7, we can conclude that the proposed algorithm shows the
best channel estimation accuracy when the identical number of beam training
and SNR are given. As shown in Fig. 5, the superiority of the proposed
algorithm is attributed to the robustness against close separation between
AoDs and AoAs. Also, Fig. 6 supports that the training beamwidth control makes
channel estimation accurate when there is less beam training.

However, the channel estimation becomes inaccurate as the number of active RIS
antennas decreases, where this inaccuracy is induced by the decrease of beam
gain and the decrease of the number of detectable signal paths. Thus, the idea
to maintain high channel estimation accuracy while extremely lowering beam
training overhead needs to be discussed in further work, as well as the
analysis on the relationship between channel estimation accuracy and the
number of active RIS antennas.

##  V Conclusions

In this paper, we propose a low-overhead channel estimation algorithm for RIS-
aided MIMO systems. When there is less beam training, some multipath signals
may not be received, and this causes the channel estimation failure. To
address this issue, the beamwidth of RIS is adaptively widened so that the
beamwidth is inversely proportional to the number of beams created by RIS. The
atomic norm of the effective channel for RIS-aided MIMO systems is defined,
where defining the atomic norm requires the compilation of pilot signals
received during beam training. The effective channel is estimated by solving
the SDP that represents the atomic norm. Simulation results show that the
proposed channel estimation algorithm has the lowest NMSE among other
algorithms for RIS-aided MIMO systems when the identical number of beam
training and SNR are given.

## References

  * [1] M. Di Renzo et al., “Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead,” IEEE J. Sel. Areas Commun., vol. 38, pp. 2450–2525, Jul. 2020\.
  * [2] M. Di Renzo et al., “Reconfigurable intelligent surfaces vs. relaying: Differences, similarities, and performance comparison,” IEEE Open J. Commun. Society, vol. 1, pp. 798–807, Jun. 2020.
  * [3] P. Wang, J. Fang, H. Duan, and H. Li, “Compressed channel estimation for intelligent reflecting surface-assisted millimeter wave systems,” IEEE Signal Process. Lett., vol. 27, pp. 905–909, May 2020.
  * [4] K. Ardah, S. Gherekhloo, A. Andre L. F. de, and M. Haardt, “TRICE: A channel estimation framework for RIS-aided millimeter-wave MIMO systems,” IEEE Signal Process. Lett., pp. 1–1, Feb. 2021.
  * [5] J. Chen, Y.-C. Liang, H. Victor Cheng, and W. Yu, “Channel estimation for reconfigurable intelligent surface aided multi-user MIMO systems,” Arxiv, Dec. 2019.
  * [6] J. He, H. Wymeersch, and M. Juntti, “Channel estimation for RIS-aided mmWave MIMO systems via atomic norm minimization,” Arxiv, Jan. 2021.
  * [7] Y. Chi, L. L. Scharf, A. Pezeshki, and A. R. Calderbank, “Sensitivity to basis mismatch in compressed sensing,” IEEE Trans. Signal Process., vol. 59, pp. 2182–2195, May 2011.
  * [8] A. F. Molisch et al., “Hybrid beamforming for massive MIMO: A survey,” IEEE Commun. Mag., vol. 55, pp. 134–141, Sep. 2017.
  * [9] C. R. Rao and M. B. Rao, Matrix algebra and its applications to statistics and econometrics.  Singapore: World Scientific, 1998.
  * [10] A. Sayeed and N. Behdad, “Continuous aperture phased MIMO: Basic theory and applications,” in 2010 48th Annu. Allerton Conf. Commun., Control, Comput. (Allerton), pp. 1196–1203, Oct. 2010.
  * [11] M. A. ElMossallamy et al., “Reconfigurable intelligent surfaces for wireless communications: Principles, challenges, and opportunities,” IEEE Trans. Cogn. Commun. Netw., vol. 6, pp. 990–1002, Sep. 2020.
  * [12] Y. Chi and M. Ferreira Da Costa, “Harnessing sparsity over the continuum: Atomic norm minimization for superresolution,” IEEE Signal Process. Mag., vol. 37, pp. 39–57, Mar. 2020.
  * [13] Z. Yang and L. Xie, “Exact joint sparse frequency recovery via optimization methods,” IEEE Trans. Signal Process., vol. 64, pp. 5145–5157, Oct. 2016.
  * [14] Y. Li and Y. Chi, “Off-the-grid line spectrum denoising and estimation with multiple measurement vectors,” IEEE Trans. Signal Process., vol. 64, pp. 1257–1269, Mar. 2016.
  * [15] M. K. Samimi and T. S. Rappaport, “3-D millimeter-wave statistical channel model for 5G wireless system design,” IEEE Trans. Microw. Theory Tech., vol. 64, pp. 2207–2225, Jun. 2016.


# Multimodal Visual-Tactile Representation Learning through Self-Supervised
Contrastive Pre-Training

Vedant Dave*, Fotios Lygerakis*, Elmar Rueckert
Cyber-Physical-Systems Lab at Montanuniversität Leoben, Austria *These authors
contributed equally to this work.Corresponding Author:
<EMAIL_ADDRESS>
###### Abstract

The rapidly evolving field of robotics necessitates methods that can
facilitate the fusion of multiple modalities. Specifically, when it comes to
interacting with tangible objects, effectively combining visual and tactile
sensory data is key to understanding and navigating the complex dynamics of
the physical world, enabling a more nuanced and adaptable response to changing
environments. Nevertheless, much of the earlier work in merging these two
sensory modalities has relied on supervised methods utilizing datasets labeled
by humans. This paper introduces MViTac, a novel methodology that leverages
contrastive learning to integrate vision and touch sensations in a self-
supervised fashion. By availing both sensory inputs, MViTac leverages intra
and inter-modality losses for learning representations, resulting in enhanced
material property classification and more adept grasping prediction. Through a
series of experiments, we showcase the effectiveness of our method and its
superiority over existing state-of-the-art self-supervised and supervised
techniques. In evaluating our methodology, we focus on two distinct tasks:
material classification and grasping success prediction. Our results indicate
that MViTac facilitates the development of improved modality encoders,
yielding more robust representations as evidenced by linear probing
assessments. https://sites.google.com/view/mvitac/home

##  I INTRODUCTION

In the realm of robotics, visual perception has traditionally served as a
central modality extensively leveraged for acquiring nuanced environmental
representations, a role emphasized in a range of studies [1, 2]. However, this
approach harbors intrinsic limitations in fully encapsulating the dynamic and
intricate state of the surrounding environment [3]. Conversely, tactile
sensing excels in delineating fine-grained attributes that are beyond the
grasp of visual modalities, effectively capturing the subtleties that evade
visual systems.

Thus, a synergic integration of both visual and tactile modalities offers a
more robust and comprehensive world representation, wherein the visual systems
predominantly decipher global features while tactile sensing augments this
with enriched local feature representations [4, 5]. However, the fusion of
these modalities is far from trivial, given the disparate information density
each conveys over time. Especially in manipulation task settings, the reliance
on vision remains high until the robot engages physically with an object or
surface; following this interaction, the tactile modality often becomes the
principal source of nuanced data, particularly in scenarios involving
occlusion by the robotic arm.

Recent endeavors are increasingly focusing on the efficient fusion of vision
and tactile representations to navigate the challenges delineated above [6, 7,
8, 9, 10]. This is further propelled by the significant advancements in self-
supervised learning (SSL) approaches, catalyzed by the lack of labeled tactile
and visuotactile datasets [10, 11, 12, 9]. In contrast to visual data, where
simulations and virtual environments can often provide substantial and rich
datasets, tactile data collection requires physical interaction with a wide
array of materials and objects to encapsulate a rich diversity of tactile
experiences. This not only escalates the complexity but also significantly
extends the time needed for data collection. Recognizing these challenges,
researchers are turning towards the SSL strategy to leverage unlabeled data,
which is easier and quicker to collect [10, 11, 12]. SSL facilitates the
learning of useful representations from this unlabeled data, potentially
accelerating the development of sophisticated visuotactile systems by reducing
the dependency on labor-intensive labeled datasets, thus presenting a
promising avenue to mitigate the hurdles in tactile data collection. In recent
years, contrastive learning (CL) has risen as a prominent subfield of self-
supervised learning (SSL), establishing itself as the predominant methodology
for pretraining visual encoders [13, 14, 15, 16, 17, 18].

In this study, we propose a Multi-modal Visual-Tactile (MViTac) representation
learning algorithm for effectively fusing the two modalities. Our methodology
facilitates the learning of both intra and inter-modal representations,
leveraging the richness of visual and tactile observations. We use two sets of
encoders to compute the InfoNCE loss for the two ranges of losses. We employ
the within-modality (intra) loss to maximize the agreement of the
representations of similar modality instances. Similarly, we employ the
across-modality (inter) loss to maximize the similarity of representations of
different modalities across the same sample. Utilizing the Touch-and-Go [11]
and Calandra [4] datasets for training the encoders, we undertake an
exhaustive evaluation across diverse downstream tasks, including material
property identifications and robot grasp prediction. We thereby substantiate
the efficacy of our approach in transcending the performance benchmarks set by
previous state-of-the-art frameworks.

##  II RELATED WORK

The integration of visual and tactile modalities in robotic systems is a
growing research area, with numerous studies delving into its complexities.
This section highlights key contributions in this field.

###  II-A Tactile Sensing in Robotics

Humans can identify physical properties (hardness, roughness, texture) of
objects exclusively through tactile interactions [19] and exhibit significant
reliance on tactile feedback for grasping and manipulation tasks. In recent
years, various methods have been developed for the extraction of tactile
information, proving instrumental for robotic applications [20, 21]. Works
such as [22, 23, 24] have utilised tactile signals to estimate the object
grasp stability, while Veiga et al. [25] extended this idea to independent
fingers for detecting local slips. Significant robotics research focuses on
manipulation and grasping that integrates object properties and geometry,
gripper configurations, and environmental conditions [26, 27]. Various
methodologies like Gaussian Processes [28, 29], Movement Primitives [30, 31],
Reinforcement Learning [32, 33] are incorporated to obtain the correlation
between the manipulation skills and tactile sensations. In recent years,
vision-based optical tactile sensors such as Gelsight [34], Digit [35], and
XELA uSkin [36] have gained significant traction due to their ability to
provide rich information concerning object geometry, forces, and shear.

###  II-B Vision and Touch in Robotic Manipulation

Several advancements have been made in the integration of tactile sensing for
enhanced grasping and manipulation. Calandra et al. [37] demonstrated that
integrating tactile sensing significantly enhances grasping results. Their
approach involved training vision-only, tactile-only, and visual-tactile grasp
success predictors and selecting the one with the highest success probability.
In another study [38], a tactile encoder was trained in simulation across
three distinct tasks and managed to achieve zero-shot sim-to-real transfer
using a generative adversarial network. Delving deeper into tactile
representations, Guzey et al. [12] found that utilizing a self-supervised
method to learn these representations led to superior results in manipulation
tasks. Their approach gathered a dataset of teleoperated, contact-rich, albeit
arbitrary, interactions with the environment, which they then used to derive
tactile representations. For subsequent tasks, they employed a nearest
neighbors approach to recall actions. Meanwhile, the research outlined in [39]
introduced Visuotactile-RL, a methodology combining visual and tactile
feedback for manipulation. They harnessed tactile data sourced from optical
sensors and explored two image encoder architectures, namely MultiPath (MP)
and SinglePath (SP), complemented by tactics such as tactile gating.

###  II-C Visual-Tactile Joint Representation learning

In recent studies, the fusion of visual and tactile data has emerged as a
focal point in advancing object recognition and manipulation tasks. Li et al.
in [40] presented a cross-modal prediction system, which addresses the
significant scale gap between visual and tactile signals using conditional
adversarial networks. The system synthesizes temporal tactile signals from
visual inputs and identifies touched object parts from tactile inputs,
enhancing the interaction between vision and touch. The authors in [40] employ
a data rebalancing strategy to prevent mode collapse during GAN training and
the inclusion of touch scale and location data in the model. H. Liu et al. [6]
developed a visual-tactile fusion framework using a joint group kernel sparse
coding method to address the weak pairing issue in visual-tactile data
samples. Further contribution by H. Li et al. leveraged three distinct
attention mechanisms including multi-head self-attentions across different
modalities and timelines, proving beneficial in mastering dense packing and
pouring tasks [7]. S. Luo et al. [8] aimed at amplifying cloth texture
recognition precision by focusing on shared features across different
modalities. They tested their system in robotic tactile exploration tasks for
cloth material identification. The studies extended into robot-assisted
dressing with a notable focus on garment unfolding, integrating visual-tactile
prediction models with reinforcement learning to guide robotic movements
effectively during the unfolding process [41]. Chen et al. [9] introduced the
Visuo-Tactile Transformer (VTT) which utilized spatial attention for merging
visual and tactile data, demonstrating enhanced efficiency in correlating
tactile events with visual cues. Kerr et al. [10] build a self-supervised
learning strategy that leverages intra-modal contrastive loss for learning
representations in tasks such as garment feature tracking and manipulation.
These endeavors collectively underscore the promising trajectory of
integrating visual and tactile feedback for refined robotic functionalities in
recognition and manipulation tasks.

##  III MULTIMODAL SELF-SUPERVISED LEARNING

We introduce a cross-modal self-supervised learning approach for learning
representations between visual and tactile data, which can subsequently be
employed for downstream applications such as material classification and
robotic manipulation tasks.

###  III-A Problem Statement

Given a visual observation $\mathcal{O}_{V}$ and a tactile observation
$\mathcal{O}_{T}$ both corresponding to the same object, our objective is to
find a function $f$ that encapsulates the projections
$z_{V}=f(\mathcal{O}_{V}$) and $z_{T}=f(\mathcal{O}_{T})$, which lie close to
each other in a lower-dimensional latent space $\mathcal{Z}$,
$z_{V},z_{T}\in\mathcal{Z}$. The observation spaces $\mathcal{O}_{V}$ and
$\mathcal{O}_{T}$ are RGB images in $\mathbb{R}^{H\times W\times 3}$, where
$H$ and $W$ are the height and the width of the images.

Figure 1: Left: Architecture of MViTaC that consists of two training
mechanisms, the inter-modal (green and pink) and intra-modal contrastive
learning (blue and cyan), which collectively contribute to the self-supervised
learning loss. Right: The SSL pre-trained model is subsequently evaluated on
downstream tasks via linear probing. The task we consider are material
property classification (top) and grasp success prediction (bottom).

###  III-B Multimodal Learning

Our approach is built upon Multimodal Contrastive Training (MCT) [42], a
method originally developed for learning relationships between visual and
textual data. The overall adapted architecture of the proposed approach is
illustrated in Figure 1. Within this framework, we utilize dual encoders: one
for processing visual data and another for handling tactile information. The
training is comprised of two distinct contrastive training paradigms: intra-
modal and inter-modal learning. Intra-modal refers to the learning that occurs
exclusively within the same data modality, either between tactile-to-tactile
or visual-to-visual representations. This loss is designed to maintain the
similarity among augmented images while capturing their inherent structures,
thereby enhancing the model’s robustness to varying perspectives (Details
regarding the augmentations are in the Appendix). Inter-modal refers to the
learning that incorporates multiple data modalities, facilitating the
understanding of relationships between different types of data, such as
tactile and visual representations.

###  III-C Architecture

Our dataset, comprised of $N$ samples, is formally defined as
$\mathcal{D}=\\{(v_{i},t_{i})\\}_{i=1}^{N}$, where the tuple $(v_{i},t_{i})$
corresponds to an image-tactile pair. Referring to the architecture
illustrated in Figure 1, the image encoder and its momentum-based counterpart
are convolutional neural networks, parameterized by $\theta$ and $\theta_{k}$
respectively. These encoders are built upon a ResNet-18 backbone [43], pre-
trained on ImageNet [44]. We additionally use a 2-layer MLP Projection head,
which serves to map the representations into an embedding space for
contrastive estimation. For the image encoder and its momentum counterpart,
two distinct MLPs are utilized to generate latent representations tailored for
intra-modal and inter-modal learning tasks. The momentum encoder as well as
the MLP projection heads $(\phi_{v}^{k},\phi_{t}^{k})$ are updated with the
momentum update [15]:

$\displaystyle\theta_{k}$ $\displaystyle\leftarrow m\theta_{k}+(1-m)\theta,$
(1) $\displaystyle\phi_{v}^{k}$ $\displaystyle\leftarrow
m\phi_{v}^{k}+(1-m)\phi_{v}^{q},$ (2) $\displaystyle\phi_{t}^{k}$
$\displaystyle\leftarrow m\phi_{t}^{k}+(1-m)\phi_{t}^{q}.$ (3)

Regarding the tactile models, the encoder and its momentum counterpart are
instantiated with identical configurations and are utilized to ensure
consistency in the encoding process.

###  III-D Intra-modal learning

Intra-modal learning aims at optimizing the feature representations within the
same sensory modality, such as visual-to-visual or tactile-to-tactile, to
enhance the discriminative power of the extracted features and the
generalization capabilities of the model for downstream tasks. One common
practice in contrastive learning is to randomly augment a single image to
obtain different representations [14], which are then passed to the encoder to
obtain a representation called “query” and its momentum counterpart to obtain
a“key” [15]. We derive two encodings for each modality from their respective
inputs: for visual images $\\{o_{v}^{q},o_{v}^{k}\\}$ from $o_{v}$ and for
tactile image $\\{o_{t}^{q},o_{t}^{k}\\}$ from $o_{t}$. The encodings for
visual data can be written as follows

$\displaystyle z_{vv}^{q}$
$\displaystyle=g(f(o_{v}^{q};\theta);\phi_{v}^{q})_{\textstyle,}$ (4)
$\displaystyle z_{vv}^{k}$
$\displaystyle=g(f(o_{v}^{k};\theta_{k});\phi_{v}^{k})_{\textstyle,}$ (5)

where $\phi_{v}^{q}$ and $\phi_{v}^{k}$ are the query and key projection heads
for visual data respectively and $z_{vv}^{q}$ and $z_{vv}^{k}$ are their
corresponding visual embeddings generated by their respective query and key
encoders (represented as $f(\cdot)$) in conjunction with their associated
projection heads (represented as $g(\cdot)$).
Similarly, we obtain encodings for the tactile images as

$\displaystyle z_{tt}^{q}$
$\displaystyle=g(f(o_{t}^{q};\psi);\Phi_{t}^{q})_{\textstyle,}$ (6)
$\displaystyle z_{tt}^{k}$
$\displaystyle=g(f(o_{t}^{k};\psi_{k});\Phi_{t}^{k})_{\textstyle.}$ (7)

We adopt InfoNCE [13] loss to optimize the contrastive learning objective.
Given a query $z_{vv}^{q}$ with a positive sample $z_{vv}^{k+}$ and K negative
samples denoted by $z_{vv}^{k_{i}}$ with $k_{i}=\\{1...K\\}$, the InfoNCE loss
can be defined as

$\mathcal{L}_{vv}=-\log\frac{\exp(z_{vv}^{q}\cdot
z_{vv}^{k^{+}}/\tau)}{\sum_{i=0}^{K}\exp(z_{vv}^{q}\cdot
z_{vv}^{k_{i}}/\tau)}.$ (8)

Here $\tau$ is the temperature hyper-parameter [45] for $\ell_{2}$-normalized
q and k. In contrast to MoCo [15] and MCT [42], which employs a memory queue
to maintain negative samples, our approach aligns with previous works [14, 46,
47], utilizing the other keys present in the same batch as negative samples.
In our model, the intra-modal learning consists of two parts, the visual-to-
visual $\mathcal{L}_{vv}$ as Eq. (8) and the tactile-to-tactile contrastive
learning, formalised as $\mathcal{L}_{tt}$ loss and in a similar manner as Eq.
$(\ref{eq5:infonce_vv})$.

###  III-E Inter-modal learning

Inter-modal learning aims at aligning the feature spaces across the visual and
touch sensory modalities, in order to develop a unified multimodal latent
representation that captures the underlying relationships and shared
attributes between these distinct data sources.

####  III-E1 Image-to-Tactile representation learning

For a visual image $o_{v}$ and its corresponding tactile image $o_{t}$, we
utilize the image encoder to produce the query feature and the momentum
tactile encoder for the generation of the key feature. They are then projected
to latent space by their respective projection heads

$\displaystyle z_{vt}^{q}$
$\displaystyle=g(f(o_{v}^{q};\theta);\phi_{t}^{q})_{\textstyle,}$ (9)
$\displaystyle z_{vt}^{k}$
$\displaystyle=g(f(o_{t}^{k};\psi_{k});\Phi_{v}^{k})_{\textstyle.}$ (10)

Contrary to the approach employed by MCT [42], which utilizes dot products
between learned representations and introduces a ‘margin” hyper-parameter, our
method adopts the InfoNCE loss as specified in Eq. (8). This deviation arises
as we operate in the image space for both modalities, a distinction from the
scenario in [42] that handles textual data instead. The corresponding loss
$\mathcal{L}_{vt}$ is also defined as InfoNCE loss and is formulated in the
same manner as in Eq. (8)

$\mathcal{L}_{vt}=-\log\frac{\exp(z_{vt}^{q}\cdot
z_{vt}^{k^{+}}/\tau)}{\sum_{i=0}^{K}\exp(z_{vt}^{q}\cdot
z_{vt}^{k_{i}}/\tau)}$ (11)

####  III-E2 Tactile-to-Image representation learning

Similar to Image-to-Tactile learning, given a visual-tactile image pair
${o_{v},o_{t}}$, we generate the query feature from the tactile encoder and
the key feature from the momentum image encoder, which are subsequently
projected to the latent space by their respective projection heads as

$\displaystyle z_{tv}^{q}$ $\displaystyle=g(f(o_{t}^{q};\psi);\Phi_{v}^{q})$
(12) $\displaystyle z_{tv}^{k}$
$\displaystyle=g(f(o_{v}^{k};\theta_{k});\phi_{t}^{k})$ (13)

The loss $\mathcal{L}_{tv}$ is also formalised as InfoNCE loss and is
formulated in the same manner as in Eq. (11).

###  III-F Combined Loss

The overall multimodal contrastive loss used in MViTac is formulated as a
combination of both, inter-modal and intra-modal contrastive loss, weighted by
a term $\lambda_{inter}$ that is responsible for the trade-off between between
the optimization objectives for inter-modal and intra-modal representations.
The overall loss $\mathcal{L}_{mm}$ is defined as

$\displaystyle\mathcal{L}_{mm}=\mathcal{L}_{vv}+\mathcal{L}_{tt}+\lambda_{inter}(\mathcal{L}_{vt}+\mathcal{L}_{tv})$
(14)

##  IV EXPERIMENTS

In this section, we discuss the experiments aimed at validating the
performance of our proposed MViTac learning framework. We elaborate on the two
datasets used and evaluate the learned representations on four downstream
tasks. Our model is benchmarked against state-of-the-art self-supervised
methods in visuotactile learning and supervised learning method.

TABLE I: Comparison on different material property identification downstream tasks Dataset |  Method |  Modality |  Category Accuracy% |  Hard/Soft Accuracy (%) |  Rough/Smooth Accuracy (%)
---|---|---|---|---|---
Chance | - | Tactile | 18.6 | 66.1 | 56.3
ResNet18 [43] | Supervised Learning | Tactile | 57.4 |  89.1 | 79.3
|  | Tactile + Visual | 48.0 | 85.9 | 80.0
TAG [11] | Contrastive Multiview Coding | Tactile | 54.7 | 77.3 | 79.4
|  | Tactile + Visual | 68.6 | 87.1 | 82.4
SSVTP [10] | InfoNCE | Tactile | 46.1 | 79.7 | 75.8
|  | Tactile + Visual | 70.7 | 88.6 | 83.6
MViTac (Ours) | Multimodal Contrastive Training | Tactile |  57.6 | 86.2 | 82.1
|  | Tactile + Visual |  74.9 |  91.8 | 84.1
We report the evaluation of Top-1% accuracy across various downstream tasks.
We evaluate on both, tactile-only and visual-tactile data. The blue are the
best results in tactile-only modality and red shows the best result in
tactile+visual modality.

###  IV-A Experimental Setup

####  IV-A1 Material property identification

We evaluate on the Touch-and-Go (TAG) [11] dataset to solve the task of
material property identification. This dataset encompasses a diverse range of
tactile features that are instrumental in bifurcating various material
properties. We consider three downstream tasks: 1) categorization of
materials, 2) distinction between hard and soft surfaces, and 3) distinction
between smooth and textured surfaces. We adhere to the dataset splits
prescribed by the authors of [11] to maintain experimental consistency. This
partitioning ensures that our evaluations are directly comparable to prior
work and their baselines. For the classification task, the dataset comprises a
collection of 20 distinct objects, whereas Hard/Soft and Rough/Smooth are
binary classification task.

####  IV-A2 Robot Grasping Prediction

The Calandra dataset [4] provides the data from a pair of tactile sensors
attached to a jaw gripper (left and right) alongside the RGB images. A triplet
of samples was captured ’before’, ’during’, and ’after’ grasping a plethora of
objects. The objective is to determine the success or the failure of the grasp
attempt. Unlike in TAG [11], the Calandra dataset does not provide a
predefined train/test split, leading us to create our own randomized split. We
train our model on a subset of 40 unique objects from the total 106 of the
original dataset, keeping only the demonstrations with the most grasping
attempts. In contrast to TAG, which only utilizes tactile-tactile encoding in
their experiments, we use visual-tactile representations to predict the grasp
success. The only difference with the setup in the previous section is that we
stack the tactile images from the two sensors across the channel dimension
before passing them through the respective encoder to obtain their common
representation. Finally, we evaluate the learned representations by using the
tactile pair and the RGB image from the ’during’ samples to train a linear
classifier.

###  IV-B Evaluation and Results

We evaluate our MViTac model against established models on tactile-only and
visual-tactile data. For reproducibility and further research, we provide our
test/train split of the Calandra dataset, along with the models and code, on
our project website111 https://sites.google.com/view/mvitac/home .

####  IV-B1 Material property identification

We first evaluate MViTac on only the tactile data for all the three material
property identification tasks. For a comparative assessment, we employed
multiple methodologies within the same problem domain. These encompass a
supervised learning framework that relies on ResNet-18 architecture [43],
where the model is trained on labeled data. Furthermore, we incorporate
comparative metrics from TAG [11], a model that utilizes the Contrastive
Multiview Coding (CMC) approach [48] to learn cross-modal representations. The
baseline results for TAG are sourced from the original paper as the
experimental conditions, including data splits, and were maintained in strict
accordance with the original study. In addition, we also consider SSVTP [10],
a recent approach that employs InfoNCE [13] loss in its pre-training phase for
self-supervised learning.
In accordance with common practice, we evaluate the quality of the learned
representations through linear probing. Following the self-supervised pre-
training phase, we detach the projection heads and freeze the encoder. On
these static representations, we then train a linear classifier in a
supervised fashion.
As observed in Table I, the tactile-only model surpasses the performance of
both the cross-modal representation learning methods, TAG and SSTVP. Notably,
our method exceeds the performance of not just self-supervised approaches like
TAG and SSTVP, but also the supervised learning model, which commonly
surpasses self-supervised methods in general efficacy. Nonetheless, the
supervised learning approach exhibits a marginal advantage in tasks related to
the classification of hard and soft surfaces. Remarkably, the integration of
visual and tactile data results in substantial performance gains. Across all
methodologies, incorporating visual information appears to uniformly elevate
prediction accuracy—an outcome that, while expected, underscores its
significance.
Within this framework, our method consistently outperforms all alternative
approaches, signifying the effectiveness of our multimodal strategy. Although
both TAG and SSVTP leverage contrastive learning frameworks and operate on
closely related loss functions, their performance lags behind our model in
terms of prediction accuracy. Even though TAG and SSVTP are contrastive
learning models, and work on almost similar loss functions, they achieve less
prediction accuracy than our model. Our model not only distinguishes
similarities and differences between visual and tactile data but also places
substantial emphasis on learning within the same sensory modality.

TABLE II: Comparison of predicting the success of grasping Dataset | Method |  Grasping Pred. Accuracy%
---|---|---
TAG [11] | Contrastive Multiview Coding | 56.3
MViTac (Ours) | Multimodal Contrastive Training | 60.3
Calandra et. al. [4] | Supervised Learning (Baseline) | 73.1
Top 1% Accuracy. The best result is in bold.

####  IV-B2 Robot Grasping Prediction

We compare the robustness of the learned representations using again the
common linear probing method and train a linear classifier to predict robot
grasping outcomes. We compare our approach against the CMC method [11] and the
supervised method proposed in [4] for solving this classification task. We
report the comparison results in Table I. The superiority of the learned
embeddings using MViTac is apparent, where we outperform CMC by almost $4\%$
on predicting the grasping of unseen objects. We must note here, that the
supervised method of [4] greatly outperforms both self-supervised modalities.
This discrepancy in performance is expected due to the small size of the
training dataset which is around 18000 samples and its imbalanced distribution
of samples across objects. CL techniques necessitate bigger and more diverse
datasets to perform comparably with supervised methods.

##  V DISCUSSION AND LIMITATIONS

We have demonstrated that our model exhibits better generalizability across
real world datasets. For the material property classification, we report
considerable performance improvements when visual and tactile data are
combined. The results suggest that while tactile information is adept at
capturing fine-grained material properties, it provides insufficient
information for the accurate classification of complex surfaces, such as those
encountered in uncontrolled environments. While we acknowledge that self-
supervised approaches may not surpass supervised methods in scenarios with
limited data, as evidenced in our grasp prediction experiment, efforts are
underway to narrow this performance gap. In the grasp prediction task, we
acknowledge that self-supervised methods do not yet outperform supervised
methods as data is limited; however, efforts are underway to minimize this
performance disparity. Our evaluations leverage datasets that, albeit
comprehensive, might not fully encapsulate the complexity and variability seen
in real-world scenarios. Extending the evaluation to real robotic systems and
understanding how well the model performs in real-time tasks is an essential
next step. Expanding the scope of evaluation to include a broader spectrum of
tasks, including more sophisticated manipulation tasks, could provide a more
comprehensive understanding of the model’s capabilities and limitations.
Nonetheless, it is important to note that our experiments are based on data
collected from real-world tasks, thus affirming the relevance and value of our
findings.

##  VI CONCLUSIONS

In this paper, we presented MViTac, a novel approach for incorporating both
visual and tactile sensory for various tasks. More specifically, our
methodology learns inter-modal and intra-modal representations via self-
supervised learning which leads to more efficient representations. As a
consequence, MViTac consistently surpasses existing self-supervised
methodologies across all the benchmarks and outperforms supervised learning
approaches on material property recognition. Furthermore, it outperforms self-
supervised state-of-the-art method on grasping success prediction with linear
probing. Nonetheless, it necessitates further validations on real robotic
platforms to ensure its efficacy in real-world applications. Our future works
will aim at refining the learning architecture and venturing into real robot
experiments to foster advancements in multimodal robotic tasks.

## APPENDIX

###  VI-A Implementation Details

For image and tactile modality, we use ResNet-18 [43] as the backbone. In
order to obtain representations, we apply average pooling on the last layer of
the backbone. For all the encoders i.e. for inter-modal and intra-modal
contrastive learning, we use a 2-layer MLP as projection heads that converts
the 512-dimensional output from the backbone into a 128-dimensional final
representation. We use ReLU [49] as the activation function for the first
layer and no activation function for the final layer. All the representation
vectors are normalised before calculating the contrastive loss. We set the
temperature $\tau$ as $\\{0.07,0.2,0.5,1\\}$. The best results were obtained
when $\tau$ was 0.07.
All the networks are trained jointly using ADAM [50] optimizer with parameters
$\beta_{1}=0.9,\beta_{2}=0.999,\epsilon_{ADAM}=10^{-7}$. During the pre-
training phase, a learning rate of 0.03 is employed, while for the subsequent
downstream tasks, a reduced learning rate of $10^{-4}$ is used. To mitigate
overfitting, a dropout layer is integrated into the classifier, featuring a
dropout probability of 0.2, serving as regularization. We use the batch size
of 256 training on a single 4090 GPU for 240 epochs for pre-training and 60
epochs for the downstream tasks. For the downstream tasks, the projection
heads [14], are bypassed; instead, the encoder’s output is directly fed into
the input layer of the classifier.

###  VI-B Augmentations

We follow the standard practice of resizing the images into 256x256 pixels and
subsequently normalized using predefined mean and standard deviation metrics.
Then the images are subjected to a randomly resized crop with dimensions of
224x224 pixels, falling within a scale range of 0.2 to 1.0. Additional
stochastic operations include the application of horizontal flipping with a
50% probability and the conversion to grayscale with a 20% probability. It
should be noted that the grayscale transformation is excluded from the
grasping task, as it offers little utility in such a controlled experimental
setting.

###  VI-C Baseline Methods

In the case of supervised learning, we employ a pair of ResNet-18 encoders
[43] that independently generate tactile and visual representations, which are
subsequently concatenated and directly subjected to the classification
process. This strategy diverges from our other baselines, where we employ a
linear classifier post-encoding for task-specific adaptations. In our study,
we adopt a feature dimension of 128 and a batch size of 128, diverging from
SSVTP’s settings of 8 and 512 respectively, due to dataset size and
computational constraints [10]. In our study, the TAG model [11] was executed
as per the original specifications, leveraging the code made publicly
available by the authors. Encoders in all the baselines are built upon a
ResNet-18 backbone [43], pre-trained on ImageNet [44], with a modification in
the final layer. For the grasping experiment [4], we employed the
implementation available in Tacto [51].

## ACKNOWLEDGMENT

This project has received funding from the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) - No #430054590 (TRAIN).

## References

  * [1] Chen Hua  “A Review of Visual Perception for Mobile Robot Navigation: Methods, Limitation, and Applications”  In _2022 2nd International Conference on Algorithms, High Performance Computing and Artificial Intelligence (AHPCAI)_ , 2022, pp. 729–737  DOI: 10.1109/AHPCAI57455.2022.10087821
  * [2] Md Tanzil Shahria et al.  “A Comprehensive Review of Vision-Based Robotic Applications: Current State, Components, Approaches, Barriers, and Potential Solutions”  In _Robotics_ 11.6, 2022  DOI: 10.3390/robotics11060139
  * [3] Chiara Bozzacchi, Robert Volcic and Fulvio Domini  “Effect of visual and haptic feedback on grasping movements” PMID: 25231616  In _Journal of Neurophysiology_ 112.12, 2014, pp. 3189–3196  DOI: 10.1152/jn.00439.2014
  * [4] Roberto Calandra et al.  “More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch”  In _IEEE Robotics and Automation Letters_ 3, 2018, pp. 3300–3307  URL: https://api.semanticscholar.org/CorpusID:38553603
  * [5] Benoit P. Delhaye, Katie H. Long and Sliman J. Bensmaia  “Neural Basis of Touch and Proprioception in Primate Cortex”  In _Comprehensive Physiology_ John Wiley & Sons, Ltd, 2018, pp. 1575–1602  DOI: https://doi.org/10.1002/cphy.c170033
  * [6] Huaping Liu, Yuanlong Yu, Fuchun Sun and Jason Gu  “Visual–Tactile Fusion for Object Recognition”  In _IEEE Transactions on Automation Science and Engineering_ 14.2, 2017, pp. 996–1008  DOI: 10.1109/TASE.2016.2549552
  * [7] Hao Li et al.  “See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation”  In _CoRL_ , 2022
  * [8] Shan Luo et al.  “ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 2722–2727  DOI: 10.1109/ICRA.2018.8460494
  * [9] Yizhou Chen, Mark Van Merwe, Andrea Sipos and Nima Fazeli  “Visuo-Tactile Transformers for Manipulation”  In _6th Annual Conference on Robot Learning_ , 2022  URL: https://openreview.net/forum?id=JqqSTgdQ85F
  * [10] Justin Kerr et al.  “Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features”, 2023  arXiv:2209.13042 [cs.RO]
  * [11] Fengyu Yang et al.  “Touch and Go: Learning from Human-Collected Vision and Touch”  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022
  * [12] Irmak Guzey, Ben Evans, Soumith Chintala and Lerrel Pinto  “Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play”, 2023  arXiv:2303.12076 [cs.RO]
  * [13] Aäron Oord, Yazhe Li and Oriol Vinyals  “Representation Learning with Contrastive Predictive Coding”  In _CoRR_ abs/1807.03748, 2018  arXiv: http://arxiv.org/abs/1807.03748
  * [14] Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton  “A Simple Framework for Contrastive Learning of Visual Representations”  In _Proceedings of the 37th International Conference on Machine Learning_ 119, Proceedings of Machine Learning Research  PMLR, 2020, pp. 1597–1607  URL: https://proceedings.mlr.press/v119/chen20j.html
  * [15] Kaiming He et al.  “Momentum Contrast for Unsupervised Visual Representation Learning”  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9726–9735  DOI: 10.1109/CVPR42600.2020.00975
  * [16] Jean-Bastien Grill et al.  “Bootstrap Your Own Latent a New Approach to Self-Supervised Learning”  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20  Vancouver, BC, Canada: Curran Associates Inc., 2020
  * [17] Mathilde Caron et al.  “Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20  Vancouver, BC, Canada: Curran Associates Inc., 2020
  * [18] Adrien Bardes, Jean Ponce and Yann LeCun  “VICReg: Variance-Invariance-Covariance Regularization For Self-Supervised Learning”  In _ICLR_ , 2022
  * [19] Roberta L Klatzky, Susan J Lederman and Victoria A Metzger  “Identifying objects by touch: An expert system”  In _Perception & psychophysics_ 37.4 Springer, 1985, pp. 299–302
  * [20] Ravinder Dahiya, Giorgio Metta, Maurizio Valle and Giulio Sandini  “Tactile Sensing—From Humans to Humanoids”  In _IEEE Transactions on Robotics_ 26, 2010, pp. 1–20
  * [21] Akihiko Yamaguchi and Christopher G. Atkeson  “Recent progress in tactile sensing and sensors for robotic manipulation: can we turn tactile sensing into vision?”  In _Advanced Robotics_ 33.14 Taylor & Francis, 2019, pp. 661–673
  * [22] Zhen Deng, Yannick Jonetzko, Liwei Zhang and Jianwei Zhang  “Grasping force control of multi-fingered robotic hands through tactile sensing for object stabilization”  In _Sensors_ 20.4 MDPI, 2020, pp. 1050
  * [23] J. Schill et al.  “Learning continuous grasp stability for a humanoid robot hand based on tactile sensing”  In _2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob)_, 2012, pp. 1901–1906  DOI: 10.1109/BioRob.2012.6290749
  * [24] Zilin Si et al.  “Grasp stability prediction with sim-to-real transfer from tactile sensing”  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 7809–7816  IEEE
  * [25] Filipe Veiga, Benoni Edin and Jan Peters  “Grip stabilization through independent finger tactile feedback control”  In _Sensors_ 20.6 Mdpi, 2020, pp. 1748
  * [26] Akihiko Yamaguchi and Christopher G. Atkeson  “Recent progress in tactile sensing and sensors for robotic manipulation: can we turn tactile sensing into vision?”  In _Advanced Robotics_ 33.14 Taylor & Francis, 2019, pp. 661–673  DOI: 10.1080/01691864.2019.1632222
  * [27] Qiang Li et al.  “A review of tactile information: Perception and action through touch”  In _IEEE Transactions on Robotics_ 36.6 IEEE, 2020, pp. 1619–1634
  * [28] Roberto Calandra et al.  “Learning inverse dynamics models with contacts”  In _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 3186–3191  IEEE
  * [29] Filipe Veiga, Dominik Notz, Thomas Hesse and Jan Peters  “Tactile based forward modeling for contact location control”  In _RSS Workshop on Tactile Sensing for Manipulation_ , 2017
  * [30] Francois R Hogan, Jose Ballester, Siyuan Dong and Alberto Rodriguez  “Tactile dexterity: Manipulation primitives with tactile feedback”  In _2020 IEEE international conference on robotics and automation (ICRA)_ , 2020, pp. 8863–8869  IEEE
  * [31] Vedant Dave and Elmar Rueckert  “Predicting full-arm grasping motions from anticipated tactile responses”  In _2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids)_ , 2022, pp. 464–471  DOI: 10.1109/Humanoids53995.2022.9999743
  * [32] Yevgen Chebotar et al.  “Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning”  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2016, pp. 1960–1966  DOI: 10.1109/IROS.2016.7759309
  * [33] Yi Zheng, Filipe Fernandes Veiga, Jan Peters and Veronica J. Santos  “Autonomous Learning of Page Flipping Movements via Tactile Feedback”  In _IEEE Transactions on Robotics_ 38.5, 2022, pp. 2734–2749  DOI: 10.1109/TRO.2022.3168731
  * [34] Wenzhen Yuan, Siyuan Dong and Edward H. Adelson  “GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force”  In _Sensors_ 17.12, 2017  DOI: 10.3390/s17122762
  * [35] Mike Lambeta et al.  “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation”  In _IEEE Robotics and Automation Letters_ 5.3 IEEE, 2020, pp. 3838–3845
  * [36] Tito Pradhono Tomo et al.  “A New Silicone Structure for uSkin—A Soft, Distributed, Digital 3-Axis Skin Sensor and Its Integration on the Humanoid Robot iCub”  In _IEEE Robotics and Automation Letters_ 3.3, 2018, pp. 2584–2591  DOI: 10.1109/LRA.2018.2812915
  * [37] Roberto Calandra et al.  “The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?”  In _CoRR_ abs/1710.05512, 2017  arXiv: http://arxiv.org/abs/1710.05512
  * [38] Yijiong Lin, John Lloyd, Alex Church and Nathan F. Lepora  “Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch”  In _IEEE Robotics and Automation Letters_ 7.4, Proceedings of Machine Learning Research  IEEE, 2022, pp. 10754–10761  DOI: 10.1109/LRA.2022.3195195
  * [39] Johanna Hansen et al.  “Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 8298–8304  DOI: 10.1109/ICRA46639.2022.9812019
  * [40] Yunzhu Li, Jun-Yan Zhu, Russ Tedrake and Antonio Torralba  “Connecting touch and vision via cross-modal prediction”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10609–10618
  * [41] Fan Zhang and Yiannis Demiris  “Visual-Tactile Learning of Garment Unfolding for Robot-Assisted Dressing”  In _IEEE Robotics and Automation Letters_ 8.9, 2023, pp. 5512–5519  DOI: 10.1109/LRA.2023.3296371
  * [42] Xin Yuan et al.  “Multimodal contrastive training for visual representation learning”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6995–7004
  * [43] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [44] Jia Deng et al.  “ImageNet: A large-scale hierarchical image database”  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255  DOI: 10.1109/CVPR.2009.5206848
  * [45] Zhirong Wu, Yuanjun Xiong, Stella X Yu and Dahua Lin  “Unsupervised feature learning via non-parametric instance discrimination”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3733–3742
  * [46] Mang Ye, Xu Zhang, Pong C Yuen and Shih-Fu Chang  “Unsupervised embedding learning via invariant and spreading instance feature”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6210–6219
  * [47] Xinlei Chen, Saining Xie and Kaiming He  “An Empirical Study of Training Self-Supervised Vision Transformers”, 2021  arXiv:2104.02057 [cs.CV]
  * [48] Yonglong Tian, Dilip Krishnan and Phillip Isola  “Contrastive multiview coding”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , 2020, pp. 776–794  Springer
  * [49] Vinod Nair and Geoffrey E Hinton  “Rectified linear units improve restricted boltzmann machines”  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , 2010, pp. 807–814
  * [50] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [51] Shaoxiong Wang, Mike Lambeta, Po-Wei Chou and Roberto Calandra  “TACTO: A Fast, Flexible, and Open-source Simulator for High-resolution Vision-based Tactile Sensors”  In _IEEE Robotics and Automation Letters (RA-L)_ 7.2, 2022, pp. 3930–3937  DOI: 10.1109/LRA.2022.3146945