STI

    short-term imperfection
RIS

    Reconfigurable intellgent surface
NMSE

    normalized mean square error
HOSVD

    higher order singular value decomposition

# Tensor-Based Channel Estimation for Reconfigurable intellgent surface
(RIS)-Assisted Networks Operating Under Imperfections

Paulo R. B. Gomes  Wireless Telecom Research Group
Federal University of Ceará
Fortaleza, Brazil
<EMAIL_ADDRESS>Gilderlan T. de Araújo  Wireless Telecom Research Group
Federal University of Ceará
Fortaleza, Brazil
<EMAIL_ADDRESS>Bruno Sokal  Wireless Telecom Research Group
Federal University of Ceará
Fortaleza, Brazil
<EMAIL_ADDRESS>André L. F. de Almeida  Wireless Telecom Research
Group
Federal University of Ceará
Fortaleza, Brazil
<EMAIL_ADDRESS>Behrooz Makki  Ericsson Research
Göteborg, Sweden
<EMAIL_ADDRESS>Gábor Fodor  Ericsson Research and KTH Royal
Institute of Technology
Stockholm, Sweden
<EMAIL_ADDRESS>
###### Abstract

Reconfigurable intelligent surface (RIS) is a candidate technology for future
wireless networks. It enables to shape the wireless environment to reach
massive connectivity and enhanced data rate. The promising gains of RIS-
assisted networks are, however, strongly depends on the accuracy of the
channel state information. Due to the passive nature of the RIS elements,
channel estimation may become challenging. This becomes most evident when
physical imperfections or electronic impairments affect the RIS due to its
exposition to different environmental effects or caused by hardware
limitations from the circuitry. In this paper, we propose an efficient and
low-complexity tensor-based channel estimation approach in RIS-assisted
networks taking different imperfections into account. By assuming a short-term
model in which the RIS imperfections behavior, modeled as unknown amplitude
and phase shifts deviations, is non-static with respect to the channel
coherence time, we formulate a closed-form higher order singular value
decomposition based algorithm for the joint estimation of the involved
channels and the unknown impairments. Furthermore, the identifiability and
computational complexity of the proposed algorithm are analyzed, and we study
the effect of different imperfections on the channel estimation quality.
Simulation results demonstrate the effectiveness of our proposed tensor-based
algorithm in terms of the estimation accuracy and computational complexity
compared to competing tensor-based iterative alternating solutions.

###### Index Terms:

Reconfigurable intelligent surface, channel estimation, imperfections
detection, hardware, MIMO, HOSVD.

##  I Introduction

Wireless communications have become a necessity in our daily lives providing
significant benefits to our modern society. Its popularization occurs thanks
to frequent efforts and proliferation of technological innovations in order to
improve system capacity and diverse quality of service requirements for
different applications. However, such achievement has led to the massive
increment in the number of connected devices as well as their rate
requirements [1, 2]. As the network size and capabilities increase, energy
consumption and implementation costs become challenging [3]. Therefore, to
guarantee green sustainable wireless networks, one needs to carefully take the
energy consumption and the hardware cost into account [4].

Thanks to the recent development of meta-materials, reconfigurable intelligent
surface (RIS) has emerged as a potential technology applicable in future
wireless networks through the novel smart and programming environment
paradigm. Unlike conventional networks, RIS-assisted networks enable the
system to shape the wireless environment to be suitable for communication [5].
With a well-configured RIS, the propagation conditions can be improved by
controlling the scattering characteristics to create passive and active
beamforming at the RIS and the wireless transceivers to achieve high
beamforming gains [6]. In addition, a passive RIS does not require active
radio frequency chains for signal transmission, reception and processing since
it simply relies on passive signal reflection, which makes it a power-
efficient, cost-effective, and low-complexity technology [7, 8].

Despite its potential benefits, the performance gains achieved in RIS-assisted
networks are strongly dependent on the quality of the channel state
information (CSI), among others. Also, cost reduction may affect the hardware
capability and, consequently, its performance. Therefore, to have a realistic
view on the performance and usefulness of RIS, one needs to carefully take the
hardware impairments as well as the environmental effects, such as water
precipitations, flying debris, air particles, snowflakes, ice stones, dry/damp
sand particles and dirt into account, as shown in Fig. 1.

Figure 1: An illustration of a RIS-assisted network operating under
imperfections.

In practice, such blocking objects as well as the hardware impairments induce
unwanted attenuation and phase shifts on the reflected signals by the RIS
introducing time-varying distortions in the received signal, which directly
affect the channel estimation accuracy and consequently the network
performance. In this sense, typical channel estimation methods may not be able
to deal with different imperfections and may fail to properly estimate the
channel. Therefore, it is necessary to continuously monitor the channel and
compensate for imperfections in order to maintain robust network operation.

The effects of hardware impairments and environmental imperfections have been
rarely studied in the literature. For instance, [9, 10, 11] consider RIS
operating under finite resolution of the phase shifts or phase estimation
errors from imperfect channel estimation. More completely, [12] and [13]
consider different environmental effects on the RIS and propose methods to
jointly estimate the channel and array blockage parameters in millimeter wave
(mmWave) RIS-assisted systems.

In this paper, we propose an efficient and low-complexity tensor-based
algorithm for the joint estimation of the involved communication channels and
imperfections in RIS-assisted networks. First, we show that the received
signal under the short-term imperfection (STI) model in which the RIS
imperfections behavior is non-static with respect to the channel coherence
time can be recast as a tensor following a quadrilinear parallel factor
(PARAFAC) model. Exploiting the multi-linear structure of this model, we
derive a closed-form higher order singular value decomposition (HOSVD)-based
algorithm to solve the joint channel and RIS imperfections estimation problem.
Moreover, we also study the identifiability of the proposed estimator, discuss
its computational complexity and investigate the effect of imperfections on
the network performance. Compared to the typical channel estimation methods
such as [14], the key features of the proposed HOSVD-STI algorithm are its
ability to properly estimate the channel and its robustness to different kinds
of real-world imperfections at the RIS with low computational complexity,
since no iteration is required.

Notations. We use the following notations and properties. Scalars are denoted
by lower-case letters ($a,\ldots$), column vectors by bold lower-case letters
($\mathbf{a},\ldots$), matrices by bold upper-case letters
($\mathbf{A},\ldots$) and tensors are represented by upper-case calligraphic
letters ($\mathcal{A},\ldots$). Then, $\mathbf{A}^{\text{T}}$ and
$\mathbf{A}^{{\dagger}}$ stand for the transpose and Moore-Penrose pseudo-
inverse of $\mathbf{A}$, respectively. The operator $\text{vec}(\cdot)$
vectorizes its matrix argument by stacking its columns on top of each other,
while $\text{vecd}(\cdot)$ forms a vector out of the diagonal of its matrix
argument. Also, $\|\cdot\|_{\text{F}}$ represents the Frobenius norm of a
matrix or a tensor, while $\times_{n}$ denotes the $n$-mode product. Moreover,
$\mathbf{I}_{M}$ is the $M\times M$ identity matrix. The operator
$\mathbf{D}_{i}\left(\mathbf{A}\right)$ forms a diagonal matrix from the
$i$-th row of $\mathbf{A}$, while the operator $\text{diag}(\mathbf{a})$ forms
a diagonal matrix from $\mathbf{a}$. The Kronecker, Hadamard and the outer
product operators are denoted by $\otimes$, $\odot$ and $\circ$, respectively.
The Khatri-Rao product between two matrices is defined as

$\mathbf{A}\diamond\mathbf{B}=\left[\mathbf{D}_{1}\left(\mathbf{A}\right)\mathbf{B}^{\text{T}},\ldots,\mathbf{D}_{Q}\left(\mathbf{A}\right)\mathbf{B}^{\text{T}}\right]^{\text{T}},$
(1)

where
$\mathbf{A}=\left[\mathbf{a}_{1},\ldots,\mathbf{a}_{Q}\right]\in\mathbb{C}^{I\times
Q}$ and
$\mathbf{B}=\left[\mathbf{b}_{1},\ldots,\mathbf{b}_{Q}\right]\in\mathbb{C}^{J\times
Q}$.

In this paper, the following property of the Kronecker product will be used

$\displaystyle\mathbf{a}\otimes\mathbf{b}\otimes\mathbf{c}$ $\displaystyle=$
$\displaystyle\text{vec}\left(\mathbf{c}\circ\mathbf{b}\circ\mathbf{a}\right),\forall\mathbf{a},\mathbf{b},\mathbf{c}.$
(2)

Furthermore, the definitions and operations involving tensors are in
accordance with references [15] and [16].

##  II System Model

Figure 2: Structured pilot pattern in the time domain. Figure 3: An
illustration of the STI model with time-varying RIS fluctuations during the
channel coherence time.

We consider a single-user narrowband RIS-assisted MIMO network in which the
transmitter (Tx) and the receiver (Rx) are equipped with arrays composed of
$M$ and $L$ antennas, respectively, and the RIS has $N$ passive reflecting
elements, as illustrated in Fig. 1. The direct Tx-Rx link is assumed to be too
weak or unavailable due to unfavorable propagation conditions. We assume a
block-fading channel and the channel estimation occurs after the Tx sends
pilot symbols which are reflected by the RIS using a known activation pattern
and the receiver collects $p=1,\dots,P$ frames. Each frame is composed by $K$
blocks, $k=1,\dots,K$, each of which contains $T$, $t=1,\dots,T$ symbol
periods. Figure 2 illustrates this structured pilot pattern in the time
domain, while Fig. 3 illustrates the collected frames for channel estimation.
The division of the reception time into $P$ frames is motivated by possible
short-term variations caused by the RIS imperfections during the channel
coherence time. This STI model indicates a more realistic aspect where the RIS
elements are subjected to hardware impairments or environmental effects that
induces both amplitude and phase perturbations in the RIS elements that are
static within each small frame but are non-static from frame-to-frame.

By considering an RIS operating under this STI model, the RIS activation
pattern is modified in an undesired manner leading to the following impaired
reflection pattern in the $k$-th time-block at the $p$-th frame

$\bar{\mathbf{s}}\left[p,k,t\right]=\left[e_{n,p}\beta_{1,k,t}e^{j\phi_{1,k,t}},\ldots,e_{N,p}\beta_{N,k,t}e^{j\phi_{N,k,t}}\right]^{\text{T}},$
(3)

where $\phi_{n,k,t}\in(0,2\pi]$ and $\beta_{n,k,t}\in\\{0,1\\}$. Equivalently,

$\bar{\mathbf{s}}\left[p,k,t\right]=\mathbf{e}\left[p\right]\odot\mathbf{s}\left[k,t\right]\mathbb{C}^{N\times
1},$ (4)

where $\mathbf{s}[k,t]\in\mathbb{C}^{N\times 1}$ is the known activation
pattern and
$\mathbf{e}\left[p\right]=\left[e_{1,p},\ldots,e_{N,p}\right]^{\text{T}}\in\mathbb{C}^{N\times
1}$ $\forall p=1,\ldots,P$, depends on the $p$-th received frame and models
the unknown non-static amplitude and phase fluctuations along the training
time. The entries of the random vector $\mathbf{e}\left[p\right]$ associated
with the $p$-th frame are defined as

$e_{n,p}=\left\\{\begin{array}[]{ll}1,&\quad{\color[rgb]{0,0,0}\text{non-
impaired case}}\\\ \alpha_{n,p}\cdot
e^{j\theta_{n,p}},&\quad{\color[rgb]{0,0,0}\text{otherwise}},\end{array}\right.$
(5)

where $0\leq\alpha_{n,p}\leq 1$ and $0\leq\theta_{n,p}\leq 2\pi$ $\forall
n=1,\ldots,N$, and $\forall p=1,\ldots,P$, denote the unwanted amplitude
attenuation and phase shift perturbations that affect the $n$-th RIS element
at the $p$-th frame, respectively. It is important to note that the model in
(5) captures different kinds of real-world imperfections at the RIS. For
example, we can note the following situations:

  * •

$\alpha_{n,p}\neq 0$ and $\theta_{n,p}\neq 0$ represent the amplitude
absortion and phase shift caused by an object suspended on the $n$-th RIS
element, or caused by hardware impairments in the electronic circuits that
make up the RIS.

  * •

$\alpha_{n,p}=1$ and $\theta_{n,p}\neq 0$ represent the phase noise
perturbations from low-resolution phase shifts or phase errors from imperfect
channel estimation.

  * •

$\alpha_{n,p}=0$ represents the maximum absorption i.e., the $n$-th RIS
element is completely blocked.

  * •

$\alpha_{n,p}=1$ and $\theta_{n,p}=0$ represents the non-impaired RIS in which
no imperfection affects its $n$-th element. Note that in this ideal case
$\bar{\mathbf{s}}\left[p,k,t\right]=\mathbf{s}\left[k,t\right]$ holds since
$e_{n,p}=1$ $\forall n=1,\ldots,N$, and $\forall p=1,\ldots,P$.

Therefore, the baseband received pilot signal
$\mathbf{y}[p,k,t]\in\mathbb{C}^{L\times 1}$ associated with the $p$-th frame,
$k$-th block and $t$-th symbol period can be expressed by

$\mathbf{y}[p,k,t]=\mathbf{G}\textrm{diag}(\mathbf{e}\left[p\right]\odot\mathbf{s}\left[k,t\right])\mathbf{H}^{\textrm{T}}\mathbf{x}[k,t]+\mathbf{v}[p,k,t].$
(6)

According Fig. 2, we assume that the activation pattern remains constant
within the $k$-th block but may vary between different time-blocks, i,e.,
$\mathbf{s}[k,t]=\mathbf{s}[k],\,\,\text{for}\,\,t=1,\dots,T$. Furthermore,
the pilot symbol $\mathbf{x}\left[k,t\right]\in\mathbb{C}^{M\times 1}$
transmitted at the $t$-th symbol period within the $k$-th time-block is reused
for each $k=1,\ldots,K$, i.e.,
$\mathbf{x}[k,t]=\mathbf{x}[t],\,\,\text{for}\,\,k=1,\dots,K$. Based on this
training protocol, by collecting the received signals during the $T$ symbol
periods at the $p$-th frame and $k$-th time-block, the model in (6) can be
rewritten as

$\mathbf{Y}\left[p,k\right]=\mathbf{G}\text{diag}\left(\mathbf{e}\left[p\right]\odot\mathbf{s}\left[k\right]\right)\mathbf{H}^{\text{T}}\mathbf{X}+\mathbf{V}\left[p,k\right],$
(7)

where
$\mathbf{Y}\left[p,k\right]=\left[\mathbf{y}\left[p,k,1\right],\ldots,\mathbf{y}\left[p,k,T\right]\right]\in\mathbb{C}^{L\times
T}$ $\forall p=1,\ldots,P$, and $\forall k=1,\ldots,K$. The matrices
$\mathbf{H}\in\mathbb{C}^{M\times N}$ and $\mathbf{G}\in\mathbb{C}^{L\times
N}$ denote the Tx-RIS and RIS-Rx channels, respectively, while
$\mathbf{X}=\left[\mathbf{x}\left[1\right],\ldots,\mathbf{x}\left[T\right]\right]\in\mathbb{C}^{M\times
T}$ collects the pilot signals transmitted within the $k$-th time-block, and
$\mathbf{V}\left[p,k\right]=\left[\mathbf{v}\left[p,k,1\right],\ldots,\mathbf{v}\left[p,k,T\right]\right]\in\mathbb{C}^{L\times
T}$ is the additive white Gaussian noise (AWGN) matrix with zero mean and unit
variance elements. In order to simplify our formulation and analysis, without
loss of generality, we assume the transmission of the pilot signal
$\mathbf{X}=\mathbf{I}_{M}$.

In a more convenient form for our formulation, the received signal in (7) can
be written in its complete matrix and decoupled format as

$\mathbf{Y}\left[p,k\right]=\mathbf{G}\mathbf{D}_{p}\left(\mathbf{E}\right)\mathbf{D}_{k}\left(\mathbf{S}\right)\mathbf{H}^{\text{T}}+\mathbf{V}\left[p,k\right],$
(8)

where
$\mathbf{S}=\left[\mathbf{s}[1],\dots,\mathbf{s}[K]\right]^{\text{T}}\in\mathbb{C}^{K\times
N}$ collects in its rows the known activation pattern used accross $K$ blocks
specifically configured for the channel estimation. Each row of
$\mathbf{E}=\left[\mathbf{e}\left[1\right],\ldots,\mathbf{e}\left[P\right]\right]^{\text{T}}\in\mathbb{C}^{P\times
N}$ collects the unknown amplitude and phase parameters for the RIS elements
impaired at the $p$-th frame. Throughout this work, we assume that a number of
$N_{B}=NR_{B}$ random elements at the RIS are subject to imperfections, where
$R_{B}\in[0,1]$ denotes its occurrence probability.

##  III PARAFAC Modeling

According to [15], the noiseless signal part of (8) expresses the ($p,k$)-th
frontal slice of a fourth-order tensor $\mathcal{Y}\in\mathbb{C}^{L\times
M\times K\times P}$ that follows the PARAFAC decomposition and its
representation in terms of $n$-mode product notation is given by

$\mathcal{Y}=\mathcal{I}_{4,N}\times_{1}\mathbf{G}\times_{2}\mathbf{H}\times_{3}\mathbf{S}\times_{4}\mathbf{E}.$
(9)

Here, $\mathcal{I}_{4,N}$ denotes the fourth-order identity tensor of size
$N\times N\times N\times N$, while $\mathbf{G}$, $\mathbf{H}$, $\mathbf{S}$
and $\mathbf{E}$ are the 1,2,3,4-mode factor matrices of the decomposition,
respectively.

By stacking column-wise the noiseless received signal in (8) for the $K$ time-
blocks at frame $p$ as the matrix
$\mathbf{Y}_{p}=\left[\mathbf{Y}\left[p,1\right],\ldots,\mathbf{Y}\left[p,K\right]\right]\in\mathbb{C}^{L\times
MK}$, we have

$\mathbf{Y}_{p}=\mathbf{G}\mathbf{D}_{p}\left(\mathbf{E}\right)\left[\mathbf{D}_{1}\left(\mathbf{S}\right)\mathbf{H}^{\text{T}},\ldots,\mathbf{D}_{K}\left(\mathbf{S}\right)\mathbf{H}^{\text{T}}\right],$
(10)

$\forall p=1,\ldots,P$. Applying the property (1) to the right-hand side of
(10), a more compact form is obtained as

$\mathbf{Y}_{p}=\mathbf{G}\mathbf{D}_{p}\left(\mathbf{E}\right)\left(\mathbf{S}\diamond\mathbf{H}\right)^{\text{T}}\in\mathbb{C}^{L\times
MK}.$ (11)

From (11), we can define the new column-wise collection
$\left[\mathcal{Y}\right]_{(1)}=\left[\mathbf{Y}_{1},\ldots,\mathbf{Y}_{P}\right]\in\mathbb{C}^{L\times
MKP}$ as the 1-mode matrix unfolding of the received signal tensor
$\mathcal{Y}\in\mathbb{C}^{L\times M\times K\times P}$ in (9), which is given
by

$\left[\mathcal{Y}\right]_{(1)}=\mathbf{G}\left[\mathbf{D}_{1}\left(\mathbf{E}\right)\left(\mathbf{S}\diamond\mathbf{H}\right)^{\text{T}},\ldots,\mathbf{D}_{P}\left(\mathbf{E}\right)\left(\mathbf{S}\diamond\mathbf{H}\right)^{\text{T}}\right].$
(12)

By applying property (1) to the right-hand side of (12), we finally obtain

$\left[\mathcal{Y}\right]_{(1)}=\mathbf{G}\left(\mathbf{E}\diamond\mathbf{S}\diamond\mathbf{H}\right)^{\text{T}}\in\mathbb{C}^{L\times
MKP}.$ (13)

The remaining 2-mode, 3-mode and 4-mode matrix unfoldings can be deduced using
a similar procedure by permuting the factor matrices in (8). This leads to the
following factorizations to the other unfoldings

$\displaystyle\left[\mathcal{Y}\right]_{(2)}$ $\displaystyle=$
$\displaystyle\mathbf{H}\left(\mathbf{E}\diamond\mathbf{S}\diamond\mathbf{G}\right)^{\text{T}}\in\mathbb{C}^{M\times
LKP},$ (14) $\displaystyle\left[\mathcal{Y}\right]_{(3)}$ $\displaystyle=$
$\displaystyle\mathbf{S}\left(\mathbf{E}\diamond\mathbf{H}\diamond\mathbf{G}\right)^{\text{T}}\in\mathbb{C}^{K\times
LMP},$ (15) $\displaystyle\left[\mathcal{Y}\right]_{(4)}$ $\displaystyle=$
$\displaystyle\mathbf{E}\left(\mathbf{S}\diamond\mathbf{H}\diamond\mathbf{G}\right)^{\text{T}}\in\mathbb{C}^{P\times
LMK}.$ (16)

##  IV Proposed HOSVD-STI Algorithm for Joint Channel and Imperfections
Estimation

We now derive a HOSVD-based closed-form solution for joint estimation of the
involved communication channels $\mathbf{G}$ and $\mathbf{H}$ and the RIS
imperfections embedded in $\mathbf{E}$. According to (15), the transpose of
the 3-mode unfolding of $\mathcal{Y}$ is denoted by

$\left[\mathcal{Y}\right]^{\text{T}}_{(3)}=\left(\mathbf{E}\diamond\mathbf{H}\diamond\mathbf{G}\right)\mathbf{S}^{\text{T}}.$
(17)

The first processing step at the receiver is to apply a bilinear time-domain
matched-filtering by multiplying both sides in (17) by the pseudo-inverse of
$\mathbf{S}^{\text{T}}$, resulting in

$\tilde{\mathbf{Y}}=\mathbf{E}\diamond\mathbf{H}\diamond\mathbf{G}\in\mathbb{C}^{LMP\times
N},$ (18)

where
$\tilde{\mathbf{Y}}=\left[\mathcal{Y}\right]^{\text{T}}_{(3)}\left(\mathbf{S}^{\text{T}}\right)^{{\dagger}}$.
From (18), decoupled estimates of the channel matrices and RIS imperfections
can be obtained by separating each factor matrix in the Khatri-Rao product. In
this sense, the estimates can be obtained by minimizing the following cost
function

$\underset{\mathbf{G},\mathbf{H},\mathbf{E}}{\text{min}}\left\|\tilde{\mathbf{Y}}-\mathbf{E}\diamond\mathbf{H}\diamond\mathbf{G}\right\|_{\text{F}}^{2}.$
(19)

We propose to solve this problem by means of multiple rank-one tensor
approximations via the HOSVD. To this end, let us define
$\tilde{\mathbf{Y}}=\left[\tilde{\mathbf{y}}_{1},\ldots,\tilde{\mathbf{y}}_{N}\right]\in\mathbb{C}^{LMP\times
N}$. The $n$-th column of $\tilde{\mathbf{Y}}$ can be written as

$\tilde{\mathbf{y}}_{n}=\mathbf{e}_{n}\otimes\mathbf{h}_{n}\otimes\mathbf{g}_{n}\in\mathbb{C}^{LMP\times
1},$ (20)

where $\mathbf{e}_{n}\in\mathbb{C}^{P\times 1}$,
$\mathbf{h}_{n}\in\mathbb{C}^{M\times 1}$ and
$\mathbf{g}_{n}\in\mathbb{C}^{L\times 1}$ denote the $n$-th column of
$\mathbf{E}$, $\mathbf{H}$ and $\mathbf{G}$, respectively. Using the
equivalence property in (2) that relates the Kronecker product to the outer
product, we can rewrite (20) as

$\tilde{\mathbf{y}}_{n}=\text{vec}\left(\mathbf{g}_{n}\circ\mathbf{h}_{n}\circ\mathbf{e}_{n}\right)\in\mathbb{C}^{LMP\times
1},$ (21)

that represents the vectorized form of the following third-order rank-one
tensor

$\tilde{\mathcal{Y}}_{n}=\mathbf{g}_{n}\circ\mathbf{h}_{n}\circ\mathbf{e}_{n}\in\mathbb{C}^{L\times
M\times P}.$ (22)

Thus, the optimization problem in (19) is equivalent to finding the estimates
of $\mathbf{H}$, $\mathbf{G}$ and $\mathbf{E}$ that minimize a set of $N$
rank-one tensor approximations, i.e,

$\left(\hat{\mathbf{G}},\hat{\mathbf{H}},\hat{\mathbf{E}}\right)=\underset{\mathbf{G},\mathbf{H},\mathbf{E}}{\text{argmin}}\sum_{n=1}^{N}\left\|\tilde{\mathcal{Y}}_{n}-\mathbf{g}_{n}\circ\mathbf{h}_{n}\circ\mathbf{e}_{n}\right\|_{\text{F}}^{2}.$
(23)

for $n=1,\ldots,N$

1 1. Rearrange the $n$-th column of $\tilde{\mathbf{Y}}$ in Equation (18)

as the rank-one tensor $\tilde{\mathcal{Y}}_{n}$ in Equation (22);

2. HOSVD procedure

2.1 Compute $\mathbf{U}_{n}^{(1)}$ as the $L$ left singular vectors of
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(1)}$:
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(1)}=\mathbf{U}_{n}^{(1)}\cdot\mathbf{\Sigma}_{n}^{(1)}\cdot\mathbf{V}_{n}^{(1)\text{H}};$

2.2 Compute $\mathbf{U}_{n}^{(2)}$ as the $M$ left singular vectors of
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(2)}$:
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(2)}=\mathbf{U}_{n}^{(2)}\cdot\mathbf{\Sigma}_{n}^{(2)}\cdot\mathbf{V}_{n}^{(2)\text{H}};$

2.3 Compute $\mathbf{U}_{n}^{(3)}$ as the $P$ left singular vectors of
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(3)}$:
$\left[\tilde{\mathcal{Y}}_{n}\right]_{(3)}=\mathbf{U}_{n}^{(3)}\cdot\mathbf{\Sigma}_{n}^{(3)}\cdot\mathbf{V}_{n}^{(3)\text{H}};$

2.4 Compute the HOSVD core tensor $\mathcal{G}_{n}$ as:
$\mathcal{G}_{n}=\tilde{\mathcal{Y}}_{n}\times_{1}\mathbf{U}_{n}^{(1)\text{H}}\times_{2}\mathbf{U}_{n}^{(2)\text{H}}\times_{3}\mathbf{U}_{n}^{(3)\text{H}};$

end procedure

2 3. Obtain the estimates for $\hat{\mathbf{g}}_{n}$, $\hat{\mathbf{h}}_{n}$
and $\hat{\mathbf{e}}_{n}$ from Equations

(25), (26) and (27), respectively;

end

3 4. Return the matrices
$\hat{\mathbf{G}}=\left[\hat{\mathbf{g}}_{1},\ldots,\hat{\mathbf{g}}_{N}\right]$,
$\hat{\mathbf{H}}=\left[\hat{\mathbf{h}}_{1},\ldots,\hat{\mathbf{h}}_{N}\right]$

and
$\hat{\mathbf{E}}=\left[\hat{\mathbf{e}}_{1},\ldots,\hat{\mathbf{e}}_{N}\right]$.

Algorithm 1 HOSVD-STI Algorithm

Let us introduce the HOSVD of $\tilde{\mathcal{Y}}_{n}$ as

$\tilde{\mathcal{Y}}_{n}=\mathcal{G}_{n}\times_{1}\mathbf{U}_{n}^{(1)}\times_{2}\mathbf{U}_{n}^{(2)}\times_{3}\mathbf{U}_{n}^{(3)}\in\mathbb{C}^{L\times
M\times P},$ (24)

where $\mathbf{U}_{n}^{(1)}\in\mathbb{C}^{L\times L}$,
$\mathbf{U}_{n}^{(2)}\in\mathbb{C}^{M\times M}$ and
$\mathbf{U}_{n}^{(3)}\in\mathbb{C}^{P\times P}$ are unitary matrices, while
$\mathcal{G}_{n}\in\mathbb{C}^{L\times M\times P}$ denotes the HOSVD core
tensor. The estimates of the vectors $\mathbf{g}_{n}$, $\mathbf{h}_{n}$ and
$\mathbf{e}_{n}$ that solve the LS problem in (23) can be obtained by
truncating the HOSVD of $\tilde{\mathcal{Y}}_{n}$ to its dominant rank-one
component, yielding

$\displaystyle\hat{\mathbf{g}}_{n}$ $\displaystyle=$
$\displaystyle\sqrt[3]{\left(\mathcal{G}_{n}\right)_{1,1,1}}\cdot\mathbf{u}^{(1)}_{1,n},$
(25) $\displaystyle\hat{\mathbf{h}}_{n}$ $\displaystyle=$
$\displaystyle\sqrt[3]{\left(\mathcal{G}_{n}\right)_{1,1,1}}\cdot\mathbf{u}^{(2)}_{1,n},$
(26) $\displaystyle\hat{\mathbf{e}}_{n}$ $\displaystyle=$
$\displaystyle\sqrt[3]{\left(\mathcal{G}_{n}\right)_{1,1,1}}\cdot\mathbf{u}^{(3)}_{1,n},$
(27)

where $\mathbf{u}^{(1)}_{1,n}\in\mathbb{C}^{L\times 1}$,
$\mathbf{u}^{(2)}_{1,n}\in\mathbb{C}^{M\times 1}$ and
$\mathbf{u}^{(3)}_{1,n}\in\mathbb{C}^{P\times 1}$ are the first higher order
singular vectors, i.e., the first column of $\mathbf{U}_{n}^{(1)}$,
$\mathbf{U}_{n}^{(2)}$ and $\mathbf{U}_{n}^{(3)}$, respectively. Here,
$\left(\mathcal{G}_{n}\right)_{1,1,1}$ is the first element of the core tensor
$\mathcal{G}_{n}$. The estimates of $\hat{\mathbf{G}}$, $\hat{\mathbf{H}}$ and
$\hat{\mathbf{E}}$ are obtained by repeating the procedure of (22)-(27) for
the $N$ columns of $\tilde{\mathbf{Y}}$ in (18). In other words, a total of
$N$ rank-one tensor approximations via HOSVD are necessary to obtain the full
estimates of the matrices
$\hat{\mathbf{G}}=\left[\hat{\mathbf{g}}_{1},\ldots,\hat{\mathbf{g}}_{N}\right]$,
$\hat{\mathbf{H}}=\left[\hat{\mathbf{h}}_{1},\ldots,\hat{\mathbf{h}}_{N}\right]$
and
$\hat{\mathbf{E}}=\left[\hat{\mathbf{e}}_{1},\ldots,\hat{\mathbf{e}}_{N}\right]$
in a closed-form manner. The implementation steps of the proposed closed-form
HOSVD-STI algorithm are summarized in Algorithm 1.

###  IV-A Indentifiability and Computational Complexity

The proposed HOSVD-STI algorithm is a closed-form solution requiring only that
the RIS activation pattern matrix $\mathbf{S}\in\mathbb{C}^{K\times N}$ have
full column-rank in order to guarantee the uniqueness in the LS sense when the
bilinear time-domain matched-filtering preprocessing is performed at the
receiver side, as indicated in (18). This leads to $K\geq N$ as a necessary
and sufficient condition. It can be satisfied under the truncated discrete
Fourier transform (DFT) design for the RIS activation pattern matrix since it
has orthonormal columns, i.e.,
$\mathbf{S}^{\textrm{S}}\mathbf{H}=K\mathbf{I}_{N}$. Therefore, by satisfying
$K\geq N$, the estimated matrices $\hat{\mathbf{G}}$, $\hat{\mathbf{H}}$ and
$\hat{\mathbf{E}}$ are column scaled versions of their true values. This
scaling ambiguity can be eliminated with a simple normalization procedure.

Regarding the computational complexity of the HOSVD-STI algorithm, it is
dominated by the HOSVD computation of the third-order rank-one tensor in (22),
which is equivalent to compute the truncated singular value decompositions
(SVDs) of its 1-mode, 2-mode and 3-mode unfolding matrices to rank-one. These
truncated SVDs are repeated $N$ times. Therefore, the HOSVD-STI algorithm has
a complexity $\mathcal{O}\left(NMLP\right)$ flops. However, the execution of
the HOSVD-STI algorithm can be parallelized if more than one processor is
available leading to greater reduction in the total processing time for
channel estimation, making it attractive when low processing delay is needed.

##  V Simulation Results

(a)

(b)

(c)

Figure 4: NMSE of HOSVD-STI algorithm versus SNR (dB) assuming i.i.d. Rayleigh
channels.

We present a set of simulation results for performance evaluation of the
proposed tensor-based HOSVD-STI algorithm. The results presented here are
averaged over $\Omega=3000$ independent Monte Carlo runs. Each run corresponds
to a different realization of the Tx-RIS and RIS-Rx channels, RIS patterns,
impairment parameters and noise. We have designed the RIS pattern matrix
$\mathbf{S}$ as a DFT matrix. The amplitude and phase impairments parameters
embedded in the matrix $\mathbf{E}$ follow a uniform distribution between 0
and 1 and between 0 and $2\pi$, respectively. The location of the $N_{B}$
impaired elements are assumed to be random with occurrence probability
$R_{B}$, totalizing $N_{B}=NR_{B}$ impaired elements at the RIS. The metric
used to evaluate the estimation accuracy is the normalized mean square error
(NMSE) between the true and estimated matrices that provides a relative
measure for the estimation error of the proposed algorithm. For the estimated
channel $\hat{\mathbf{H}}$ we define
$\text{NMSE}(\hat{\mathbf{H}})=\frac{1}{\Omega}\sum_{\omega=1}^{\Omega}\frac{\|\mathbf{H}^{(\omega)}-\hat{\mathbf{H}}^{(\omega)}\|_{\text{F}}^{2}}{\|\mathbf{H}^{(\omega)}\|_{\text{F}}^{2}},$
where $\mathbf{H}^{(\omega)}$ and $\hat{\mathbf{H}}^{(\omega)}$ denote the
true channel and its estimate both related to the $\omega$-th run,
respectively. Similar definitions apply to the estimates of $\hat{\mathbf{G}}$
and $\hat{\mathbf{E}}$.

We examine, in Figs. 4(a), 4(b) and 4(c), the estimation performance of the
proposed HOSVD-STI algorithm. We compare our approach with the iterative
bilinear alternating least squares based algorithm proposed in [14]. It is a
tensor-based pilot-assisted method formulated to the ideal case in which no
impairments affect the RIS. Additionaly, as a lower-bound, we also plot the
clairvoyant estimator that assumes perfect knowledge of the channels and
impairments. By considering different blockage probabilities, we can observe
that the method in [14] is not suitable to tackle the channel estimation
problem when impairments are present. In contrast, our proposed HOSVD-STI
algorithm provides accurate channel estimates that are not sensitive to the
number of impaired elements and closely of the lower-bound, confirming the
effectiveness of the proposed algorithm. However, in terms of imperfections
estimation, we can see that the HOSVD is sensitive to SNR values providing
accurate estimates very close to the lower-bound in the medium-to-right SNR
regime, while some performance degradation is observed in the low-to-medium
SNR regime.

Figure 5: Average runtime (in seconds) of HOSVD-STI algorithm versus SNR (dB)
assuming i.i.d. Rayleigh channels.

In Fig. 5, we evaluate the performance of the HOSVD-STI algorithm in terms of
runtime (in seconds). It can be seen that the runtime grows when the number of
RIS elements increases, which is an expected result. However, the runtime
required by HOSVD-STI algorithm is not sensitive to the SNR since it is a
closed-form approach. Note that the closed-form HOSVD-STI algorithm requires
less runtime in all the SNR regime, compared to [14]. On the other hand, the
iterative method in [14] has a higher computational complexity compared to our
proposed algorithm, providing to be less attractive than our approach in both
estimation accuracy and complexity performance.

##  VI Conclusion

We have proposed an efficient and low-complexity tensor-based algorithm for
joint channel and imperfections estimation in impaired RIS-assisted networks.
By assuming the STI model in which the behavior of the RIS impairments are
non-static with respect to the channel coherence time, and resorting to the
inherent tensor algebraic structure of the received signal, we have formulated
the HOSVD-STI algorithm. It is a closed-form solution that presents lower
computational complexity and parallel processing capability compared to the
competing tensor-based iterative alternating solution. Our proposed HOSVD-STI
algorithm proves to be robust when operating in more realistic scenarios in
which the RIS elements are subject to hardware impairments and/or
environmental effects providing accurate channel estimates in this non-
idealized case.

## Acknowledgment

This work was supported in part by the Ericsson Research, Technical
Cooperation UFC.48, by the Coordenação de Aperfeiçoamento de Pessoal de Nível
Superior - Brasil (CAPES) and by CNPq. Gábor Fodor was partially supported by
the Digital Futures project PERCy.

## References

  * [1] G. Forecast, “Cisco visual networking index: global mobile data traffic forecast update, 2017–2022,” _Update_ , vol. 2017, p. 2022, Feb. 2019.
  * [2] ITU, “IMT traffic estimates for the years 2020 to 2030 ,” ITU-R M, TR 2370-0, Jan. 2015, V.15.0.0.
  * [3] S. Zhang, Q. Wu, S. Xu, and G. Y. Li, “Fundamental green tradeoffs: Progresses, challenges, and impacts on 5G networks,” _IEEE Commun. Surveys Tuts._ , vol. 19, no. 1, pp. 33–56, Jul. 2016.
  * [4] Q. Wu, G. Y. Li, W. Chen, D. W. K. Ng, and R. Schober, “An overview of sustainable green 5G networks,” _IEEE Wireless Communications_ , vol. 24, no. 4, pp. 72–80, Aug. 2017.
  * [5] M. Di Renzo, M. Debbah, D.-T. Phan-Huy, A. Zappone, M.-S. Alouini, C. Yuen, V. Sciancalepore, G. C. Alexandropoulos, J. Hoydis, H. Gacanin _et al._ , “Smart radio environments empowered by reconfigurable ai meta-surfaces: An idea whose time has come,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2019, no. 1, pp. 1–20, May. 2019.
  * [6] N. Rajatheva, I. Atzeni, S. Bicais, E. Bjornson, A. Bourdoux, S. Buzzi, C. D’Andrea, J.-B. Dore, S. Erkucuk, M. Fuentes, K. Guan, Y. Hu, X. Huang, J. Hulkkonen, J. M. Jornet, M. Katz, B. Makki, R. Nilsson, E. Panayirci, K. Rabie, N. Rajapaksha, M. Salehi, H. Sarieddeen, S. Shahabuddin, T. Svensson, O. Tervo, A. Tolli, Q. Wu, and W. Xu, “Scoring the terabit/s goal:broadband connectivity in 6G,” 2022, arXiv:2008.07220 [eess.SP].
  * [7] E. Basar, M. Di Renzo, J. De Rosny, M. Debbah, M.-S. Alouini, and R. Zhang, “Wireless communications through reconfigurable intelligent surfaces,” _IEEE access_ , vol. 7, pp. 116 753–116 773, Aug. 2019.
  * [8] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network,” _IEEE Communications Magazine_ , vol. 58, no. 1, pp. 106–112, Jan. 2020.
  * [9] X. Qian, M. Di Renzo, J. Liu, A. Kammoun, and M.-S. Alouini, “Beamforming through reconfigurable intelligent surfaces in single-user MIMO systems: SNR distribution and scaling laws in the presence of channel fading and phase noise,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 1, pp. 77–81, Jan. 2021.
  * [10] S. Zhou, W. Xu, K. Wang, M. Di Renzo, and M.-S. Alouini, “Spectral and energy efficiency of IRS-assisted MISO communication with hardware impairments,” _IEEE wireless commun. lett._ , vol. 9, no. 9, pp. 1366–1369, Sep. 2020.
  * [11] K. Zhi, C. Pan, H. Ren, and K. Wang, “Uplink achievable rate of intelligent reflecting surface-aided millimeter-wave communications with low-resolution ADC and phase noise,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 3, pp. 654–658, Mar. 2021.
  * [12] B. Li, Z. Zhang, Z. Hu, and Y. Chen, “Joint array diagnosis and channel estimation for RIS-aided mmwave MIMO system,” _IEEE Access_ , vol. 8, pp. 193 992–194 006, Oct. 2020.
  * [13] B. Li, Z. Zhang, and Z. Hu, “Channel estimation for reconfigurable intelligent surface-assisted multiuser mmwave MIMO system in the presence of array blockage,” _Transactions on Emerging Telecommunications Technologies_ , vol. 32, no. 11, p. e4322, Jun. 2021.
  * [14] G. T. de Araújo, A. L. de Almeida, and R. Boyer, “Channel estimation for intelligent reflecting surface assisted MIMO systems: A tensor modeling approach,” _IEEE J. Sel. Topics Signal Process._ , vol. 15, no. 3, pp. 789–802, Apr. 2021.
  * [15] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” _SIAM review_ , vol. 51, no. 3, pp. 455–500, Aug. 2009.
  * [16] L. De Lathauwer, B. De Moor, and J. Vandewalle, “A multilinear singular value decomposition,” _SIAM journal on Matrix Analysis and Applications_ , vol. 21, no. 4, pp. 1253–1278, Apr. 2000.

  *[RIS]: Reconfigurable intellgent surface
  *[NMSE]: normalized mean square error


# Systematic Analysis of Music Representations
from BERT

Sangjun Han
LG AI Research
<EMAIL_ADDRESS>
&Hyeongrae Ihm
LG AI Research
<EMAIL_ADDRESS>
&Woohyung Lim
LG AI Research
<EMAIL_ADDRESS>

###### Abstract

There have been numerous attempts to represent raw data as numerical vectors
that effectively capture semantic and contextual information. However, in the
field of symbolic music, previous works have attempted to validate their music
embeddings by observing the performance improvement of various fine-tuning
tasks. In this work, we directly analyze embeddings from BERT and BERT with
contrastive learning trained on bar-level MIDI, inspecting their musical
information that can be obtained from MIDI events. We observe that the
embeddings exhibit distinct characteristics of information depending on the
contrastive objectives and the choice of layers. Our code is available at
https://github.com/sjhan91/MusicBERT.

##  1 Introduction

Music consists of many repetitive components from motifs to phrases, and they
have been conceptualized as forms of musical knowledge or atmosphere that
humans are capable of understanding. For instance, at the note level,
performing successive multiple notes can convey harmonies and rhythmic
dynamics for a short time. At the bar-level, the performance can be expressed
in chords, with chords being arranged in relationships among various bars. At
the song level, several features can serve as an overview of the composition,
including played instruments, tempo, and genre. Our focus is to understand
bar-level symbolic music since it provides versatile capabilities for music
analysis such as estimating musical similarity, extracting chords, and
comprehending the whole structure of music.

Triggered by the field of natural language processing, there have been
numerous attempts to represent raw data as numerical vectors that effectively
capture semantic and contextual information. Based on the Transformer blocks
[1], text embeddings can be extracted from encoder-only designs that
incorporate bidirectional context [2], decoder-only designs that facilitate
text sequence generation [3], and encoder-decoder designs that combine both
functionalities [4]. In the speech, wav2vec series (wav2vec 2.0 [5], HuBERT
[6], and vq-wav2vec [7]) have applied BERT [2] for speech representations with
embedding discretization. Also in computer vision, Vision Transformer (ViT)
[8] has transformed input images into multiple grid patches and introduced the
use of class token embeddings for the classification problem.

After suggested in [9], event-based representation of MIDI for machine
learning has become widespread. In this approach, each event token serves as
an indicator of a specific musical action such as event changes of pitch, time
shift, or velocity. As in the following other research domains, the process of
tokenization for MIDI enables us to utilize Transformer models to capture
contextual information effectively. MIDIBERT-Piano [10] has adopted a super
token-level masking strategy for BERT pre-training and demonstrated promising
performance in fine-tuning performance across four tasks. Similarly, MusicBERT
[11] has proposed an efficient concept of super-token, employing a bar-level
masking strategy for BERT. MuseBERT [12] also has adopted BERT model, but
factorized MIDI representations into attribute sets and event relation matrix.
The aforementioned previous works have attempted to validate their music
embeddings by observing the performance improvement of various fine-tuning
tasks. However, when dealing with music tasks at the bar-level (_e.g._ music
information retrieval), it is more desirable to evaluate the embeddings based
on their association with musical properties and semantics.

In this work, we directly analyze BERT embeddings trained on bar-level MIDI by
inspecting their musical information that can be obtained from MIDI events.
Additionally, we compare BERT models that employ different contrastive
objectives. These models are trained with BERT loss and contrastive loss at
the same time, enabling them to take into account longer contexts among bars.
This ensures that the models can be effectively adjusted to the user’s
intention or downstream tasks [13, 14, 15]. Our results demonstrate that BERT
embeddings can capture important musical features and semantic information in
the bar-level MIDI. Furthermore, we observe that the embeddings exhibit
distinct characteristics of information depending on the contrastive
objectives and the choice of layers.

##  2 Method

Figure 1: Left: The basic structure of BERT with masked prediction, Right:
Various contrastive objectives that are trained in conjunction with masked
prediction.

In this section, we introduce the process of data preparation, several design
concepts for music embedding models, and the evaluation protocol.

### Data Preparation

Among symbolic music datasets, Lakh MIDI Dataset (LMD) [16] is widely used
since it comprises 176,581 MIDI files spanning diverse genres and tracks. We
convert each MIDI from LMD into REMI+ representation [17], an extended version
of REMI [18] that enables the expression of multiple tracks. Our vocabulary
contains 556 events for 8 categories; 1 <bar>, 32 <tempo>, 129 <instrument>,
128 <pitch>, 128 <pitch drum>, 48 <position>, 58 <duration>, and 32
<velocity>. We adopt the same configuration for REMI+ as described in [17]. In
the end, we collect a total of 9,971,616 bars from the LMD.

### Model Descriptions

Our embedding models are following BERTbase model configuration (the number of
layers=12, the hidden size=768, the number of self-attention heads=12). During
the training process, we utilize masked language modeling loss (MLM loss)
which involves masking a portion of input tokens and predicting those tokens.
At each iteration, 15% of input tokens are selected, among which 80% of the
tokens are masked, 10% of the tokens are randomly replaced, and the remaining
10% of tokens remain unchanged. We remove the next sentence prediction task
from the original BERT.

We introduce three variant models derived from BERT; BERT-aug, BERT-neighbor,
and BERT-dropout. These models are trained MLM loss and contrastive loss (NT-
Xent loss) formulated at SimCLR [19] simultaneously. For a minibatch $N$, the
NT-Xent loss can be defined as

$L_{NT-
Xent}(z^{\prime},z^{\prime\prime})=-log\frac{exp(sim(z^{\prime},z^{\prime\prime})/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{[z_{k}\neq
z^{\prime}]}exp(sim(z^{\prime},z_{k})/\tau)}$ (1)

where $z^{\prime}$ and $z^{\prime\prime}$ represent a positive pair that is
semantically identical, while $z_{k}$ is sampled from a negative set. Given a
$N$ batch, we generate a $N$ new batch of positive views using predefined
functions (_e.g._ augmentation), resulting in 2$N$ samples in a batch. When
considering a single positive pair, the remaining $2(N-1)$ samples are
regarded as the negative set. In this context, $Sim(\cdot,\cdot)$ denotes the
cosine similarity, $\mathbbm{1}$ does the indicator function, and $\tau$ does
the temperature parameter. Then, our total loss can be described as

$L=L_{MLM}+\alpha\cdot L_{NT-Xent}$ (2)

where $\alpha$ controls the degree of NT-Xent loss. We set $\tau$ and $\alpha$
to 0.1 for all experiments. Unlike previous studies [13, 14, 15], since we
train the two objectives (MLM loss and NT-Xent loss) concurrently, masked
inputs ($x^{mask}$) inevitably are involved in the training process of
contrastive learning as shown in Figure 1. In other words, $x^{mask}$ can be
regarded as one of the augmented views from the predefined functions. It is
analogous to Mask Contrast [20] for image representations in that both the
masked view and augmented view are participating in the contrastive loss.
Below, we provide a comprehensive explanation of the design principles behind
our variant models.

  * •

BERT-aug: To generate the positive view of samples, we apply data augmentation
to original sample $x$ by shifting randomly all pitches {-6, -5, … , 5, 6} and
velocities {-3, -2, … , 2, 3} in a sequence, resulting in $x^{aug}$. It still
maintains the melodic contour, not undermining musical semantics. Similar
strategies for data augmentation can be found in [21, 22].

  * •

BERT-neighbor: This is motivated by NNCLR [23] which is a contrastive model
regarding nearest neighbors of the augmented view as positives. In our
setting, a sample $x^{neigh}$ is considered to be a neighbor of $x$ if they
belong to the same music.

  * •

BERT-dropout: This is motivated by SimCSE [13] which adopts Dropout [24] as a
stochastic augmentation. The model receives the same $x^{mask}$ twice in the
forward pass and generates two different embeddings in a positive relation. We
place the Dropout mask on attention maps and feed-forward networks in
Transformer blocks and set the masking rate to 0.1.

Table 1 compares the MLM accuracy and NT-Xent accuracy of each model for the
training and validation set. Exceptionally, BERT-neighbor exhibits significant
disparities between the training and validation accuracy even after adjusting
the values of $\tau$ and $\alpha$. We can speculate that neighbors within the
same music do not extensively share musical information.

Table 1: The training and validation accuracy of four models, indicating the masked language modeling loss (MLM) and the normalized temperature-scaled cross-entropy loss (NT-Xent). Model | MLM Acc | NT-Xent Acc
---|---|---
BERT | 0.889 / 0.869 | -
BERT-aug | 0.892 / 0.869 | 1.000 / 0.918
BERT-neighbor | 0.900 / 0.868 | 1.000 / 0.059
BERT-dropout | 0.890 / 0.868 | 1.000 / 1.000

### Evaluation Methods

We evaluate the bar-level BERT embeddings on their alignment with human
interpretable domain knowledge. Referring to [17], the metrics can be listed
as follows; chords, groove patterns, instruments, tempo, mean velocity, mean
duration, and song clustering. The evaluation entails assessing the
performance of linear probing tasks, including multi-class classification with
a Ridge classifier, multi-label classification with a Ridge classifier,
regression with a Ridge regressor, and clustering with K-means. We provide a
detailed explanation of each of these metrics and evaluation methods.

  * •

Chords (C): As following [17, 18], we extract chords using an adapted version
of the Viterbi algorithm. They consist of 12 root notes and 7 qualities,
resulting in a total of 84 possible chords. Since multiple chords can be
placed on a bar, we evaluate the performance of multi-label classification for
the chords.

  * •

Groove Patterns (GP): We label a position as 1 in a bar if any note is played
and as 0 if no note is present. We evaluate the performance of multi-label
classification for the groove patterns.

  * •

Instruments (I): We label an instrument as 1 if the instrument appears. We
evaluate the performance of multi-label classification for the instruments.

  * •

Tempo (T): Tempos are quantized into 32 bins. We evaluate the performance of
multi-class classification for the tempos.

  * •

Mean Velocity (MV): We compute the average of all velocity values within a
bar. We evaluate the performance of regression for the mean velocity.

  * •

Mean Duration (MD): We compute the average of all duration values within a
bar. We evaluate the performance of regression for the mean duration.

  * •

Song Clustering (SC): Using K-means, we compute the average of entropy based
on the number of classes assigned to the bars in the same music. The low
entropy means that the bars in the same music tend to be clustered to the same
class. This metric can verify how shared semantics permeating through music
can be extracted.

##  3 Experiments

We empirically demonstrate quantitative evaluations of musical information
from the BERT-variants models. First, we inspect the information from the last
layer of Transformer blocks and the changes in the amount of information
across different layers.

### The inspection of BERT embeddings from the last layer

Table 2: The performance of linear probing using embeddings from the last
layer of BERT models. Best values are marked in bold font.

Model | C ($\uparrow$) | GP ($\uparrow$) | I ($\uparrow$) | T ($\uparrow$) | MV ($\downarrow$) | MD ($\downarrow$) | SC ($\downarrow$)
---|---|---|---|---|---|---|---
BERT | 0.861 | 0.808 | 0.764 | 0.083 | 9.169 | 1.904 | 0.252
| (1.933e-4) | (4.430e-4) | (1.674e-4) | (1.593e-4) | (1.482e-2) | (1.728e-2) | (4.154e-4)
BERT-aug | 0.532 | 0.805 | 0.855 | 0.027 | 13.093 | 1.804 | 0.217
| (4.058e-4) | (4.098e-4) | (2.186e-4) | (3.661e-4) | (1.432e-2) | (1.401e-2) | (1.442e-3)
BERT-neighbor | 0.671 | 0.768 | 0.823 | 0.946 | 7.107 | 2.371 | 0.072
| (5.486e-4) | (4.142e-4) | (1.951e-4) | (2.277e-4) | (1.393e-2) | (1.790e-2) | (4.454e-4)
BERT-dropout | 0.797 | 0.776 | 0.750 | 0.042 | 8.943 | 1.937 | 0.226
| (3.031e-4) | (4.789e-4) | (2.566e-4) | (2.967e-4) | (1.052e-2) | (1.702e-2) | (8.667e-4)

Table 3: The best performance of linear probing using embeddings across all
BERT layers. We indicate the performance as (the layer number, the performance
of the layer). Best values are marked in bold font.

Model | C ($\uparrow$) | GP ($\uparrow$) | I ($\uparrow$) | T ($\uparrow$) | MV ($\downarrow$) | MD ($\downarrow$) | SC ($\downarrow$)
---|---|---|---|---|---|---|---
BERT | (12, 0.861) | (5, 0.863) | (4, 0.882) | (1, 0.933) | (2, 4.905) | (1, 1.571) | (3, 0.169)
BERT-aug | (1, 0.566) | (4, 0.839) | (12, 0.855) | (2, 0.337) | (2, 9.823) | (6, 1.453) | (12, 0.219)
BERT-neighbor | (8, 0.716) | (2, 0.852) | (4, 0.941) | (4, 0.990) | (2, 5.159) | (2, 1.584) | (6, 0.062)
BERT-dropout | (12, 0.797) | (7, 0.804) | (5, 0.807) | (2, 0.664) | (5, 5.985) | (5, 1.664) | (12, 0.226)

Table 2 reports linear probing tasks for the embeddings of the last layer in
terms of four models and seven metrics. Remarkably, the original BERT exhibits
the highest performance for the chord classification. It can be inferred that
during the augmentation process (or neighbor sampling), the other models
augment pitch events in positive pairs so that they obtain invariant features
for the chords. Similarly, BERT-aug shows the lowest performance for the mean
velocity since it specifically modifies the velocity values during the
augmentation. BERT-dropout controls all factors so that it decreases all
performance metrics compared to the best performances.

The advantage of contrastive learning lies in its ability to extract shared
information in a positive view. For BERT-aug, the mean duration is not a
variable factor between bars in a positive relationship, which may improve its
performance. For BERT-neighbor, it tends to successfully classify bars into
their respective songs in terms of song clustering.

The results of groove pattern, instrument, and tempo classification are
inconsistent with subsequent results that analyze the performance across all
BERT layers. We supplement the explanation in the following section.

### The inspection of BERT embeddings across all layers

Table 3 reports the best performance and its layer number for the linear
probing tasks. Except for BERT-neighbor, the models with significantly lower
association with the tempo show the performance improvement depending on the
choice of layers. The information on groove patterns and instruments from all
models is distributed across all layers. Figure 2 illustrates the layer-wise
performance of the BERT-variants model.

Figure 2: The layer-wise performance of the BERT, BERT-aug, BERT-neighbor, and
BERT-dropout (from top to bottom).

##  4 Discussion

Several studies have analyzed BERT embeddings for various NLP tasks [14, 25,
26]. They indicate that BERT does not follow the classical NLP pipeline
(simple to complex) or exhibits distributed information across its layers.
Nevertheless, our research provides consistent evidence for several factors
and demonstrates the effectiveness of contrastive learning. Especially for
BERT-neighbor, it can be utilized to extract a musical theme effectively. As
mentioned in [25], the integration of information from various layers will be
important for enhancing the quality of information.

In this paper, we perform a systemic analysis of bar-level music embeddings
from BERT and BERT with contrastive learning models. For seven metrics, our
linear probing tasks can assess the amount of musical information, revealing
the effectiveness of specific models for certain metrics. The bar-level
embedding models will contribute to various musical tasks, including obtaining
chord extraction, music similarity analysis, and music structure
understanding.

## References

  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Ni et al. [2021] Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  _arXiv preprint arXiv:2108.08877_ , 2021.
  * Baevski et al. [2020a] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460, 2020a.
  * Hsu et al. [2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Baevski et al. [2020b] Alexei Baevski, Steffen Schneider, and Michael Auli.  vq-wav2vec: Self-supervised learning of discrete speech representations.  In _International Conference on Learning Representations_ , 2020b.  URL https://openreview.net/forum?id=rylwJxrYDS.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Oore et al. [2020] Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan.  This time with feeling: Learning expressive musical performance.  _Neural Computing and Applications_ , 32:955–967, 2020\.
  * Chou et al. [2021] Yi-Hui Chou, I Chen, Chin-Jui Chang, Joann Ching, Yi-Hsuan Yang, et al.  Midibert-piano: Large-scale pre-training for symbolic music understanding.  _arXiv preprint arXiv:2107.05223_ , 2021.
  * Zeng et al. [2021] Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tie-Yan Liu.  MusicBERT: Symbolic music understanding with large-scale pre-training.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 791–800, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.70.  URL https://aclanthology.org/2021.findings-acl.70.
  * Wang and Xia [2021] Ziyu Wang and Gus Xia.  Musebert: Pre-training music representation for music understanding and controllable generation.  In _ISMIR_ , pages 722–729, 2021.
  * Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.552.  URL https://aclanthology.org/2021.emnlp-main.552.
  * Kim et al. [2021] Taeuk Kim, Kang Min Yoo, and Sang-goo Lee.  Self-guided contrastive learning for BERT sentence representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2528–2540, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.197.  URL https://aclanthology.org/2021.acl-long.197.
  * Su et al. [2021] Peng Su, Yifan Peng, and K Vijay-Shanker.  Improving bert model using contrastive learning for biomedical relation extraction.  _arXiv preprint arXiv:2104.13913_ , 2021.
  * Raffel [2016] Colin Raffel.  _Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching_.  Columbia University, 2016.
  * von Rütte et al. [2022] Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann.  Figaro: Generating symbolic music with fine-grained artistic control.  _arXiv preprint arXiv:2201.10936_ , 2022.
  * Huang and Yang [2020] Yu-Siang Huang and Yi-Hsuan Yang.  Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1180–1188, 2020.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Zhao et al. [2021] Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha.  Self-supervised visual representations learning by contrastive mask prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10160–10169, 2021.
  * Huang et al. [2019] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck.  Music transformer.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rJe4ShAcF7.
  * Shih et al. [2022] Yi-Jen Shih, Shih-Lun Wu, Frank Zalkow, Meinard Muller, and Yi-Hsuan Yang.  Theme transformer: Symbolic music generation with theme-conditioned transformer.  _IEEE Transactions on Multimedia_ , 2022.
  * Dwibedi et al. [2021] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9588–9597, 2021.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * de Vries et al. [2020] Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim.  What’s so special about BERT’s layers? a closer look at the NLP pipeline in monolingual and multilingual models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4339–4350, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.389.  URL https://aclanthology.org/2020.findings-emnlp.389.
  * Jawahar et al. [2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah.  What does BERT learn about the structure of language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1356.  URL https://aclanthology.org/P19-1356.