# Channel Orthogonalization in Panel-Based LIS

Juan Vidal Alegría1, Ove Edfors1
<EMAIL_ADDRESS><EMAIL_ADDRESS>1Department of
Electrical and Information Technology, Lund University, Lund, Sweden

###### Abstract

Large intelligent surface (LIS) has gained momentum as a potential 6G-enabling
technology that expands the benefits of massive multiple-input multiple-output
(MIMO). On the other hand, orthogonal space-division multiplexing (OSDM) may
give a promising direction for efficient exploitation of the spatial
resources, analogous as what is achieved with orthogonal frequency-division
multiplexing (OFDM) in the frequency domain. To this end, we study how to
enforce channels orthogonality in a panel-based LIS scenario. Our proposed
method consists of having a subset of active LIS-panels coherently serving a
set of users, and another subset of LIS-panels operating in semi-passive mode
by implementing a receive and re-transmit (RRTx) process. This results in an
inter-symbol interference (ISI) channel, where we characterize the semi-
passive processing required to achieve simultaneous orthogonality in time and
space. We then employ the remaining degrees of freedom (DoFs) from the
orthogonality constraint to minimize the semi-passive processing power, where
we derive a closed-form global minimizer, allowing for efficient
implementation of the proposed scheme.

###### Index Terms:

Large intelligent surface (LIS), Orthogonal space-division multiplexing
(OSDM), Channel orthogonalization, Inter-symbol interference (ISI) channel.

##  I Introduction

Large intellignet surface (LIS) [1] is a technology which extends the concept
of massive multiple-input multiple-output (MIMO) [2, 3] by considering whole
walls covered with electromagnetic active material serving a set of user
equipments (UEs) within the same time-frequency resource. The main potential
of LIS mainly derives from its increased spatial resolution, allowing to
multiplex users not only in the angular domain, but also in the depth
domain[1, 4], thus leading to improved spectral efficiency beyond that of
massive MIMO.

The main theoretical results on LIS—also found under the term holographic
MIMO—have relied upon modeling it as a continuous surface of
electromagnetically active material [1]. However, practical implementations of
LIS may potentially consist of a discrete set of antennas deployed throughout
large surfaces, which has firm equivalence with the continuous counterpart as
proved in [1, 5]. Moreover, practical deployments may also favor panel-based
LIS approaches, where a LIS is divided into several panels of antennas which
may be distributed throughout large areas [6, 7].

As remarked in the literature [8, 9, 4], having distributed panels of antennas
coherently serving a set of UEs leads to critical issues in terms of
interconnection bandwidth and centralized processing complexity. These issues
may be tackled by employing decentralized approaches where the panels may
preprocess the received signals before combining them and/or sending them to a
central processing unit (CPU) [10, 11, 12, 13]. In this work, we propose an
alternative approach to relieve these issues which consists of using only a
subset of active LIS-panels for jointly decoding the data from the UEs, while
the remaining panels operate in a semi-passive mode with the aim of easing the
decoding task of the active panels. Specifically, we consider employing the
semi-passive panels to enforce channel orthogonality in space—as well as
time—allowing for orthogonal space-division multiplexing (OSDM).

The idea of OSDM was considered in [14] as a counterpart to orthogonal
frequency-division multiplexing (OFDM) for the spatial domain, which offers a
convenient framework for efficient decoding and resource allocation in MIMO-
based systems. Initial results from [14] show how to achieve orthogonal multi-
user MIMO channels through the use of passive reconfigurable surfaces, where
the freedom in the orthogonality constraint is employed to assure the passive
nature of reconfigurable surfaces. In the current work, we however consider
the use of LIS-panels for achieving channel orthogonality by implementing a
receive and retransmit (RRTx) process using a subset of the LIS-panels in
semi-passive mode. The main difference is that, in order to maintain coherence
and synchronization using baseband processing at the LIS-panels, the signals
retransmitted through the RRTx process are delayed one time-slot before
arriving the active LIS-panels, thus generating an inter-symbol interference
(ISI) channel. One important benefit of the current approach over those using
passive surfaces [14, 15] is that the narrowband channel assumption is now
accurate for OFDM-based systems since the baseband processing units allow to
perform independent processing per subcarrier—unlike passive networks.

As happened in [14, 16], the orthogonality restriction still allows for some
of degrees of freedom (DoFs), which may be hereby employed to minimize the
power of the semi-passive processing—similar to what was done in [14, 16] when
trying to fulfill the passive constraints. In the current framework, we have
however been able to characterize in closed-form the global minimizers of the
semi-passive processing power. This also gives an important advancement over
[14, 16], where the solution to the analogous optimization problems employed
costly minimization algorithms whose convergence to a global minimizer could
not be formally assured.

The rest of the paper is organized as follows. Section II introduces the
considered system model. In Section III, we show how to perform channel
orthogonalization in the considered framework. In Section IV, we derive the
closed-form solution to the power minimization problem. Section V includes a
numerical evaluation of the proposed methods. Section VI concludes the paper
with some final remarks.

##  II System Model

Let us consider a panel-based LIS scenario111Equivalence with cell-free
massive MIMO scenarios is evident by noting the correspondence between a LIS-
panel and a distributed multi-antenna AP. where a total of $3$ LIS-panels are
employed to decode the uplink transmission of $K$ UEs through a narrow-band
channel (e.g., one OFDM subcarrier). For simplicity, all LIS-panels are
assumed to have the same number of antennas $M\geq K$, although extension of
this work to LIS-panels with different combinations of antenna numbers is
straightforward. On the other hand, we believe that the $3$-panel scenario
captures the essence of the proposed solution, but in the extended version we
may formalize the results for an arbitrary number of LIS-panels.

Out of the $3$ LIS-panels, only $1$ panel is used for actively receiving and
decoding the data from the UEs. The $2$ remaining LIS-panels operate in semi-
passive mode, where they jointly implement a receive and retransmit (RRTx)
process. In this process one of the semi-passive LIS-panels—referred to as the
semi-passive receive (SP-Rx) panel—is used for receiving the signals being
exchanged between the UEs and the active panel, while the other one—referred
to as the semi-passive transmit (SP-Tx) panel—is used for simultaneously
transmitting the signals previously received at the respective SP-Rx panel. In
the RRTx process, the semi-passive panels may also apply some linear
processing which may be physically performed at the baseband processing unit
of either (or both) of the panels. Effectively, the processing achieved by the
semi-passive panels is analogous to that achieved by the fully-reconfigurable
intelligent surface (FRIS) model with relaxed power constraints, defined in
[14]. However, we now have the extra limitation that the reflected signals are
delayed due to the RRTx process. The received vector at the active panel at
time-slot $t$, is then given by

$\boldsymbol{y}[t]=\boldsymbol{H}_{0}\boldsymbol{s}[t]+\boldsymbol{H}_{1}\boldsymbol{\Theta}\boldsymbol{r}[t-1]+\boldsymbol{n}[t],$
(1)

where $\boldsymbol{H}_{0}$ is the $M\times K$ channel matrix between the UEs
and the active panel, $\boldsymbol{H}_{1}$ is the $M\times M$ channel between
the SP-Tx panel and the active panel, $\boldsymbol{r}[t]$ is the received
vector at the SP-Rx panel at time-slot $t$, $\boldsymbol{\Theta}$ is the
equivalent linear-processing applied in throughout the RRTx process, and
$\boldsymbol{n}[t]\sim\mathcal{CN}(\mathbf{0},N_{0}\mathbf{I}_{M})$ models the
additive white Gaussian noise (AWGN) added at the active panel at time-slot
$t$. We may express $\boldsymbol{r}[t]$ recurrently by

$\boldsymbol{r}[t]=\boldsymbol{H}_{2}\boldsymbol{s}[t]+\boldsymbol{H}_{12}\boldsymbol{r}[t-1]+\tilde{\boldsymbol{n}}[t],$
(2)

where $\boldsymbol{H}_{2}$ is the $M\times K$ channel between the UEs and the
SP-Rx panel, $\boldsymbol{H}_{12}$ is the $M\times M$ channel between the SP-
Tx and the SP-Rx panels, and
$\tilde{\boldsymbol{n}}[t]\sim\mathcal{CN}(\mathbf{0},\tilde{N}_{0}\mathbf{I}_{M})$
models the AWGN added at the SP-Rx.

##  III ISI Channel Orthogonalization

We begin this section by considering two simplifying assumptions that will
make the problem more tractable, while still matching to a great extend
realistic scenarios. The validity of the first of these assumptions will be
further reinforced by the results from Sections IV and V.

###### Assumption 1

If we consider the fact that the semi-passive processing may be optimized such
that it does not require large amounts of amplification, we may assume that
the noise term added in the RRTx process is negligible in (1). Note that this
extra noise term appears multiplied by $\boldsymbol{\Theta}$ in (1) so it may
have non-diagonal covariance—i.e., leading to colored noise—while a small norm
for $\boldsymbol{\Theta}$ will also reduce the impact of the noise added by
the RRTx process compared to the white noise term $\boldsymbol{n}[t]$.
Assuming negligible noise from the RRTx process thus means that the resulting
noise will remain approximately white.

###### Assumption 2

Considering that the panels may be conveniently placed within scenario, we may
assume that the semi-passive panels are located such that the channel between
them is negligible. For example, we could have one facing the main user
locations, and the other one facing the active panel. We may thus take the
assumption $\boldsymbol{H}_{12}\approx\boldsymbol{0}$ such that we can get rid
of the recurrence associated to (2).222Note that, even if
$\boldsymbol{H}_{12}$ is not exactly $\boldsymbol{0}$, every step in the
recursion would attenuate its contribution due to the repeated path-loss being
experienced.

Considering Assumptions 1 and 2, we may rewrite (1) as

$\boldsymbol{y}[t]=\boldsymbol{H}_{0}\boldsymbol{s}[t]+\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})\boldsymbol{s}[t-1]+\boldsymbol{n}[t],$
(3)

where the $M\times K$ matrix
$\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})=\boldsymbol{H}_{1}\boldsymbol{\Theta}\boldsymbol{H}_{2}$
corresponds to the aggregate channel associated to the RRTx process. Note
that, assuming full-rank $\boldsymbol{H}_{1}$ and $\boldsymbol{H}_{2}$, as
well as unrestricted semi-passive processing $\boldsymbol{\Theta}$, we have
full freedom in designing $\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$.
For example, this could be achieved by canceling $\boldsymbol{H}_{2}$ in the
SP-Rx panel—multiplying the left pseudo-inverse—and then cancelling
$\boldsymbol{H}_{1}$ in the SP-Tx panel—multiplying the inverse—after applying
the desired processing on the signal coming from the SP-Rx panel. We may also
note that, if Assumption 1 does not apply, the noise term in (3) would be
colored, hence compromising the optimality of symbol-by-symbol detection from
the orthogonal channels—it would correspond to a pseudo-zero forzing solution
instead. However, the results that we show next would still lead to channel
orthogonality in the desired signal part of (3).

As can be seen from (3), due to the RRTx process, the received vector suffers
from ISI, thus compromising the orthogonality property in the time domain.
Considering $T$ consecutive symbol transmissions within the coherence time,
and assuming $\boldsymbol{s}[t]=\boldsymbol{0}$ for $t=0$ and $t=T+1$, we may
write the complete received vector as

$\boldsymbol{y}_{\text{C}}=\boldsymbol{\mathcal{H}}\boldsymbol{s}_{\text{C}}+\boldsymbol{n}_{\text{C}},$
(4)

where $\boldsymbol{y}_{\text{C}}$ is a $(T+1)M\times 1$ vector stacking up the
received vectors $\boldsymbol{y}[t]$ at $t=1,\dots,T+1$,333We assume that the
receiver may be active for one more time-slot than the transmission duration
so that it can listen to the last retransmission. For large $T$, this would
have negligible effect on the overall performance. $\boldsymbol{s}_{\text{C}}$
is a $TK\times 1$ vector stacking up the transmitted symbol vectors
$\boldsymbol{s}[t]$ at $t=1,\dots,T$, $\boldsymbol{n}_{\text{C}}$ is the
corresponding $(T+1)M\times 1$ AWGN vector, and $\boldsymbol{\mathcal{H}}$ is
the resulting $(T+1)M\times TK$ ISI channel matrix, which considers
simultaneously the space and the time domain. From (3), we may express
$\boldsymbol{\mathcal{H}}$ as

$\boldsymbol{\mathcal{H}}=\begin{bmatrix}\boldsymbol{H}_{0}&\boldsymbol{0}&\cdots&\boldsymbol{0}\\\
\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})&\boldsymbol{H}_{0}&\ddots&\vdots\\\
\boldsymbol{0}&\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})&\ddots&\boldsymbol{0}\\\
\vdots&\ddots&\ddots&\boldsymbol{H}_{0}\\\
\boldsymbol{0}&\cdots&\boldsymbol{0}&\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})\\\
\end{bmatrix}.$ (5)

In order to achieve simultaneous orthogonality in time an space, the ISI
channel should fulfill

$\boldsymbol{\mathcal{H}}^{\text{H}}\boldsymbol{\mathcal{H}}=\beta\mathbf{I}_{TK}.$
(6)

Note that we may also relax (6) by allowing the right-hand side (RHS) to be an
arbitrary diagonal matrix instead of a scaled identity. However, this would
lead to having different channel capacity for different UEs, or under
different time-slots, which is generally not as desirable as enforcing a fair
rate distribution by considering (6) (see [14]). Nevertheless, many of our
results may be trivially extended to that case. Considering (5), we may
operate the left-hand side (LHS) of (6) to reach

$\boldsymbol{\mathcal{H}}^{\text{H}}\boldsymbol{\mathcal{H}}=\begin{bmatrix}\boldsymbol{G}(\boldsymbol{\Theta})&\boldsymbol{Z}^{\text{H}}(\boldsymbol{\Theta})&\boldsymbol{0}&\cdots&\boldsymbol{0}\\\
\boldsymbol{Z}(\boldsymbol{\Theta})&\boldsymbol{G}(\boldsymbol{\Theta})&\boldsymbol{Z}^{\text{H}}(\boldsymbol{\Theta})&\ddots&\vdots\\\
\boldsymbol{0}&\boldsymbol{Z}(\boldsymbol{\Theta})&\boldsymbol{G}(\boldsymbol{\Theta})&\ddots&\boldsymbol{0}\\\
\vdots&\ddots&\ddots&\ddots&\boldsymbol{Z}^{\text{H}}(\boldsymbol{\Theta})\\\
\boldsymbol{0}&\cdots&\boldsymbol{0}&\boldsymbol{Z}(\boldsymbol{\Theta})&\boldsymbol{G}(\boldsymbol{\Theta})\end{bmatrix},$
(7)

where we have defined the $K\times K$ matrices

$\boldsymbol{G}(\boldsymbol{\Theta})=\boldsymbol{H}_{0}^{\text{H}}\boldsymbol{H}_{0}+\widetilde{\boldsymbol{H}}^{\text{H}}(\boldsymbol{\Theta})\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$
(8a)
$\boldsymbol{Z}(\boldsymbol{\Theta})=\boldsymbol{H}_{0}^{\text{H}}\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta}).$
(8b)

The orthogonality constraint, given in (6), can then be translated into the
following conditions

$\boldsymbol{G}(\boldsymbol{\Theta})=\beta\mathbf{I}_{K}$ (9a)
$\boldsymbol{Z}(\boldsymbol{\Theta})=\boldsymbol{0}_{K\times K}.$ (9b)

Taking into account the expression for $\boldsymbol{Z}(\boldsymbol{\Theta})$
in (8b), in order to fulfill (9b) $\boldsymbol{\Theta}$ should be selected
such that the columns of $\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$
fall in the null-space of $\boldsymbol{H}_{0}^{\text{H}}$. Since we have
unrestricted $\boldsymbol{\Theta}$,
$\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$ may be arbitrarily selected
to fulfill said restriction. Let us express the singular value decomposition
(SVD) of $\boldsymbol{H}_{0}$ as

$\boldsymbol{H}_{0}=\boldsymbol{U}_{0}\begin{bmatrix}\boldsymbol{\Lambda}_{0}^{\frac{1}{2}}\\\
\boldsymbol{0}\end{bmatrix}\boldsymbol{V}_{0},$ (10)

where $\boldsymbol{U}_{0}$ and $\boldsymbol{V}_{0}$ are the respective
$M\times M$ and $K\times K$ unitary matrices, and
$\boldsymbol{\Lambda}_{0}^{\frac{1}{2}}$ is the $K\times K$ diagonal matrix
with the singular values along the diagonal. We may then select

$\widetilde{\boldsymbol{H}}(\boldsymbol{B})\triangleq\widetilde{\boldsymbol{H}}\big{(}\boldsymbol{\Theta}_{\text{s}}(\boldsymbol{B})\big{)}=\boldsymbol{U}_{0}\begin{bmatrix}\boldsymbol{0}_{K\times
K}\\\ \boldsymbol{B}\end{bmatrix},$ (11)

where $\boldsymbol{B}$ corresponds to an $(M-K)\times K$ matrix which may be
arbitrarily selected. Note that (11) generalizes any $M\times K$ matrix
falling in the null-space of $\boldsymbol{H}_{0}^{\text{H}}$. Moreover,
$\boldsymbol{\Theta}_{\text{s}}(\boldsymbol{B})$ may be trivially obtained
from (11) by inverting $\boldsymbol{H}_{1}$ and $\boldsymbol{H}_{2}$ as

$\boldsymbol{\Theta}_{\text{s}}(\boldsymbol{B})=\boldsymbol{H}_{1}^{-1}\widetilde{\boldsymbol{H}}(\boldsymbol{B})\boldsymbol{H}_{2}^{\dagger},$
(12)

where $\boldsymbol{H}_{2}^{\dagger}$ corresponds to the left pseudo-inverse of
$\boldsymbol{H}_{2}$—which may have multiple solutions. Alternative solutions
may consider the inversion of the cascaded channel
$(\boldsymbol{H}_{2}^{\text{T}}\otimes\boldsymbol{H}_{1})$ associated to the
vectorized version of (12).444The reader may refer to [14] for detailed
results on the connection between $\boldsymbol{\Theta}$ and
$\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$ under architectures leading
to various resctrictions on $\boldsymbol{\Theta}$.

Selecting $\widetilde{\boldsymbol{H}}(\boldsymbol{\Theta})$ by (11) allows to
fulfill the restriction (9b), while maintaining the remaining DoF in the
selection of $\boldsymbol{B}$. Next, we would like to determine
$\boldsymbol{B}$ such that (9a) can be simultaneously fulfilled, leading to
channel orthogonalization. Considering (11) and (8a), equation (9a) leads to

$\boldsymbol{B}^{\text{H}}\boldsymbol{B}=\beta\mathbf{I}_{K}-\boldsymbol{H}_{0}^{\text{H}}\boldsymbol{H}_{0}.$
(13)

The previous equation is only solvable whenever we can match the rank of the
two sides. For an generic $\boldsymbol{H}_{0}$—i.e., excluding those in a set
of measure 0—and assuming $M\geq K$, the right-hand side is of full-rank $K$
with probability 1. Thus, we also need the LHS to be of full-rank $K$, which
leads to the orthogonalization condition

$M\geq 2K,$ (14)

where we considered that
$\text{rank}(\boldsymbol{B}^{\text{H}}\boldsymbol{B})\leq\min(M-K,K)$.
Assuming (14) is fulfilled, (13) can be solved by selecting

$\boldsymbol{B}(\widetilde{\boldsymbol{U}})=\widetilde{\boldsymbol{U}}\big{(}\beta\mathbf{I}_{K}-\boldsymbol{\Lambda}_{0}\big{)}^{\frac{1}{2}}\boldsymbol{V}_{0}^{\text{H}},$
(15)

where $\widetilde{\boldsymbol{U}}$ is a semi-unitary matrix of dimension
$(M-K)\times K$, i.e.,
$\widetilde{\boldsymbol{U}}^{\text{H}}\widetilde{\boldsymbol{U}}=\mathbf{I}_{K}$,
which may be arbitrarily selected. The result in (15) corresponds to the
selection
$\boldsymbol{B}=(\beta\mathbf{I}_{K}-\boldsymbol{H}_{0}^{\text{H}}\boldsymbol{H}_{0})^{\frac{1}{2}}$,
but we have expressed it in terms of its SVD to outline the DoFs left after
enforcing the orthogonality constrain, which are captured in the selection of
$\widetilde{\boldsymbol{U}}$. In fact, these DoFs can be used to reduce the
power requirements for performing the RRTx process, as will be discussed in
the next section. On the other hand, the selection (15) also leads to an extra
orthogonalization condition given by

$\beta\geq\lambda_{0,\max},$ (16)

where $\lambda_{0,\max}$ corresponds to the greatest eigenvalue of
$\boldsymbol{H}_{0}^{\text{H}}\boldsymbol{H}_{0}$—i.e., the greatest diagonal
element of $\boldsymbol{\Lambda}_{0}$. This comes from avoiding square roots
of negative diagonal elements in (15), which would not allow to fulfill (13)
since the resulting matrix should be positive semi-definite. The condition
(16) may be understood as the minimum channel gain required to be able to
compensate the direct channel, which may be matched to a minimum power
requirement for the semi-passive panels.

##  IV Power Minimization

In Section III, we have shown how to achieve perfect orthogonalization of the
ISI channel resulting from using two isolated LIS-panels for relaying an
uplink transmission to an active LIS-panel. Condition (14) delimits the
possibility to perform channel orthogonalization in this framework, so let us
assume this condition is fulfilled—which is reasonable in typical LIS
scenarios where ${M\gg K}$. Let us also assume that $\beta$ is fixed such that
(16) is fulfilled, which may be achieved by adjusting the amplification in the
RRTx process. As previously mentioned, the orthogonalization restriction still
allows for DoFs which may be suitably exploited to enforce desirable
properties on the processing applied throughout the RRTx process. We will next
study how to make use of these DoFs.

A major goal of the proposed scheme is to increase the overall energy
efficiency by employing LIS-panels in a semi-passive (low-power) operation
mode. Hence, we propose to use the DoFs available from the channel
orthogonalization to minimize the power of the processing applied at the semi-
passive panels. Said power may be measured in terms of the squared Frobenius
norm of the RRTx processing, leading to the following optimization problem

$\displaystyle\min_{\widetilde{\boldsymbol{U}}}$
$\displaystyle\;\|\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\|^{2}_{\text{F}}$
(17) s.t.
$\displaystyle\;\widetilde{\boldsymbol{U}}^{\text{H}}\widetilde{\boldsymbol{U}}=\mathbf{I}_{K},$

where we have
$\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\triangleq\boldsymbol{\Theta}_{\text{s}}\big{(}\boldsymbol{B}(\widetilde{\boldsymbol{U}})\big{)}$,
i.e., the RRTx processing is obtained from (12), using (11) and (15). Note
that (17) is also relevant for increasing the validity of Assumption 1, since
$\|\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\|^{2}_{\text{F}}$
multiplies the power of the noise associated to the RRTx process reaching (1).
We may simplify the objective function in (17) by

$\|\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\|^{2}_{\text{F}}\triangleq\text{tr}\big{(}\boldsymbol{\Theta}^{\text{H}}_{\text{s}}(\widetilde{\boldsymbol{U}})\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\big{)}\\!=\text{tr}\big{(}\widecheck{\boldsymbol{H}}_{1}\widetilde{\boldsymbol{U}}\widecheck{\boldsymbol{H}}_{2}\widecheck{\boldsymbol{H}}_{2}^{\text{H}}\widetilde{\boldsymbol{U}}^{\text{H}}\widecheck{\boldsymbol{H}}_{1}^{\text{H}}\big{)},$
(18)

where we have defined

$\widecheck{\boldsymbol{H}}_{1}\triangleq\boldsymbol{H}_{1}^{-1}\boldsymbol{U}_{0}\begin{bmatrix}\boldsymbol{0}_{K\times(M-K)}\\\
\mathbf{I}_{M-K}\end{bmatrix}$ (19a)
$\widecheck{\boldsymbol{H}}_{2}\triangleq\big{(}\beta\mathbf{I}_{K}-\boldsymbol{\Lambda}_{0}\big{)}^{\frac{1}{2}}\boldsymbol{V}_{0}^{\text{H}}\boldsymbol{H}_{2}^{\dagger}.$
(19b)

Since we want to solve (17), we may select without loss
$\boldsymbol{H}_{2}^{\dagger}$ as the Moore-Penrose inverse [17], since any
other selection corresponds to adding a matrix in the left null-space of
$\boldsymbol{H}_{2}$, which would give a positive additive term in
$\|\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\|^{2}_{\text{F}}$,
i.e., leading to a suboptimal result.

The minimization problem defined in (17) is non-convex due to the semi-unitary
constraint on $\widetilde{\boldsymbol{U}}$. However, this constraint
corresponds to restricting the search space to the Stiefel manifold
$\mathcal{S}\big{(}M-K,K\big{)}$, which, when equipped with a Riemannian
metric, gives a Riemannian manifold. We may then employ the Riemannian
structure to solve (17) within the Stiefel manifold defined from its
restrictions. Specifically, we will consider the results from [18], which
studies the Riemannian struture of the unitary group under the bi-invariant
Riemannian metric induced by the inner product
$\langle\boldsymbol{U},\boldsymbol{V}\rangle=\frac{1}{2}\text{tr}(\boldsymbol{U}\boldsymbol{V}^{\text{H}})$.
Note that the particularization of results from the unitary group
$\mathcal{U}(M-K)$ to the Stiefel manifold $\mathcal{S}\big{(}M-K,K\big{)}$
can be done by considering that any
$\widetilde{\boldsymbol{U}}\in\mathcal{S}\big{(}M-K,K\big{)}$ corresponds to a
projection of a unitary matrix $\boldsymbol{U}\in\mathcal{U}(M-K)$ given by
$\widetilde{\boldsymbol{U}}=\boldsymbol{U}\begin{bmatrix}\mathbf{I}_{K}&\boldsymbol{0}\end{bmatrix}^{\text{T}}$.

Considering the Riemannian structure studied in [18] for the unitary group, we
may express the Riemannian gradient at an arbitrary point
$\boldsymbol{U}\in\mathcal{U}(M-K)$ as

$\tilde{\nabla}\mathcal{J}(\boldsymbol{U})=\boldsymbol{\Gamma}_{\mathcal{J}}(\boldsymbol{U})-\boldsymbol{U}\boldsymbol{\Gamma}^{\text{H}}_{\mathcal{J}}(\boldsymbol{U})\boldsymbol{U},$
(20)

where
$\boldsymbol{\Gamma}_{\mathcal{J}}(\boldsymbol{U})\triangleq\frac{\partial\mathcal{J}(\boldsymbol{U})}{\partial\boldsymbol{U}^{*}}$
corresponds to the Euclidean gradient of the objective function
$\mathcal{J}(\boldsymbol{U})$.555We consider differentiating only over the
frame $\partial\boldsymbol{U}^{*}$, which does not span by itself the whole
complex tangent space. However, since our analysis depends upon finding
stationary points, while a stationary point under said frame equivalently
corresponds to a stationary point under the frame $\partial\boldsymbol{U}$
[19] (and vice-versa), we have no loss in differentiating only over
$\boldsymbol{U}^{*}$. Since we are interested in solving (17), the objective
function corresponds to
$\mathcal{J}(\boldsymbol{U})=\big{\|}\boldsymbol{\Theta}_{\text{s}}\big{(}\boldsymbol{U}\begin{bmatrix}\mathbf{I}_{K}&\boldsymbol{0}\end{bmatrix}^{\text{T}}\big{)}\big{\|}^{2}_{\text{F}}$.
We may note that, since the objective function $\mathcal{J}(\boldsymbol{U})$
does not depend on the last $(M-2K)$ columns of $\boldsymbol{U}$, the
respective entries in $\boldsymbol{\Gamma}_{\mathcal{J}}(\boldsymbol{U})$ are
$0$. Through basic optimization arguments [20], the solution to (17) should be
given either by a stationary point or by a boundary point. When
particularizing the gradient to the underlying Riemannian structure, all the
points now correspond to interior points, so the solution to (17) should be
exclusively given by stationary points, i.e., such that the Riemannian
gradient in (20) vanishes. The stationary points may then be found by solving
the equation $\tilde{\nabla}\mathcal{J}(\boldsymbol{U})=0$. We then note that
the $(M-2K)$ columns of $\boldsymbol{U}$ not appearing in
$\widetilde{\boldsymbol{U}}$ have no impact on the stationary points of (20)
since an equivalent equation to the previous one is obtained by multiplying by
$\boldsymbol{U}^{\text{H}}$ both sides from the right, so that in the
resulting equation the respective columns of $\boldsymbol{U}$ appear
multiplied by the $0$ entries in
$\boldsymbol{\Gamma}_{\mathcal{J}}(\boldsymbol{U})$. We thus reach the
equivalent equation

$\boldsymbol{\Gamma}_{\mathcal{J}}(\widetilde{\boldsymbol{U}})\widetilde{\boldsymbol{U}}^{\text{H}}=\widetilde{\boldsymbol{U}}\boldsymbol{\Gamma}^{\text{H}}_{\mathcal{J}}(\widetilde{\boldsymbol{U}}),$
(21)

where the Euclidean gradient
$\boldsymbol{\Gamma}_{\mathcal{J}}(\widetilde{\boldsymbol{U}})\triangleq\frac{\partial\mathcal{J}(\widetilde{\boldsymbol{U}})}{\partial\widetilde{\boldsymbol{U}}^{*}}$
corresponds to the non-zero part of
$\boldsymbol{\Gamma}_{\mathcal{J}}(\boldsymbol{U})$. From (18), we may get
[21]

$\boldsymbol{\Gamma}_{\mathcal{J}}(\widetilde{\boldsymbol{U}})=\widecheck{\boldsymbol{G}}_{1}\widetilde{\boldsymbol{U}}\widecheck{\boldsymbol{G}}_{2},$
(22)

where we have defined
$\widecheck{\boldsymbol{G}}_{1}=\widecheck{\boldsymbol{H}}_{1}^{\text{H}}\widecheck{\boldsymbol{H}}_{1}$
and
$\widecheck{\boldsymbol{G}}_{2}=\widecheck{\boldsymbol{H}}_{2}\widecheck{\boldsymbol{H}}_{2}^{\text{H}}$.
Substituting in (21) gives the following matrix equation

$\widecheck{\boldsymbol{G}}_{1}\widetilde{\boldsymbol{U}}\widecheck{\boldsymbol{G}}_{2}\widetilde{\boldsymbol{U}}^{\text{H}}=\widetilde{\boldsymbol{U}}\widecheck{\boldsymbol{G}}_{2}^{\text{H}}\widetilde{\boldsymbol{U}}^{\text{H}}\widecheck{\boldsymbol{G}}_{1}^{\text{H}}.$
(23)

The following lemma characterizes the unique set of solutions to (23) for any
full-rank $\widecheck{\boldsymbol{G}}_{1}$ and
$\widecheck{\boldsymbol{G}}_{2}$ with distinct eigenvalues. Note that this
condition is fulfilled with probability 1 for generic $\boldsymbol{H}_{1}$ and
$\boldsymbol{H}_{2}$ matrices. However, whenever it is not fulfilled, the
solution to (17) described next may be extended by analogously exploiting the
extra DoFs.

###### Lemma 1

Consider two arbitrary positive-definite matrices
$\widecheck{\boldsymbol{G}}_{1}$ and $\widecheck{\boldsymbol{G}}_{2}$, each of
them containing distinct eigenvalues, and where we may assume without loss
that the dimension of $\widecheck{\boldsymbol{G}}_{1}$ is larger than that of
$\widecheck{\boldsymbol{G}}_{2}$. The solutions to (23) with semi-unitary
constraint on $\widetilde{\boldsymbol{U}}$ fulfill

$\widetilde{\boldsymbol{U}}=\widecheck{\boldsymbol{U}}_{1}\boldsymbol{P}_{1}\begin{bmatrix}\boldsymbol{\Lambda}_{\boldsymbol{\varphi}}\\\
\boldsymbol{0}\end{bmatrix}\boldsymbol{P}_{2}\widecheck{\boldsymbol{U}}_{2}^{\text{H}},$
(24)

where $\widecheck{\boldsymbol{U}}_{1}$ and $\widecheck{\boldsymbol{U}}_{2}$
are the unitary matrices from the eigenvalue decomposition (with decreasing
eigenvalues) of $\widecheck{\boldsymbol{G}}_{1}$ and
$\widecheck{\boldsymbol{G}}_{2}$, respectively, $\boldsymbol{P}$ is an
arbitrary permutation matrix, and
$\boldsymbol{\Lambda_{\boldsymbol{\varphi}}}$ is a diagonal matrix with unit
modulus entries along the diagonal.

### Sketch of Proof

The proof consists of expressing $\widehat{\boldsymbol{G}}_{1}$ and
$\widehat{\boldsymbol{G}}_{2}$ through their eigenvalue decompositions, and
absorbing in $\widetilde{\boldsymbol{U}}$ their respective unitary matrices of
eigenvectors by having
$\widetilde{\boldsymbol{U}}=\widecheck{\boldsymbol{U}}_{1}\widetilde{\widetilde{\boldsymbol{U}}}\widecheck{\boldsymbol{U}}_{2}^{\text{H}}$.
The resulting equation is then solved by any
$\widetilde{\widetilde{\boldsymbol{U}}}$ that simultaneously diagonalizes
$\widecheck{\boldsymbol{\Lambda}}_{1}$ and
$\widecheck{\boldsymbol{\Lambda}}_{2}$, corresponding to the respective
diagonal eigenvalue matrices of $\widecheck{\boldsymbol{G}}_{1}$ and
$\widecheck{\boldsymbol{G}}_{2}$. Assuming $\widecheck{\boldsymbol{G}}_{1}$
has greater dimension than $\widecheck{\boldsymbol{G}}_{2}$, and both matrices
have distinct eigenvalues, the solutions are then trivially captured by (24).
A complete proof may be included in the extended version. $\square$

From Lemma 1, we may substitute (24) into (18), which leads to

$\|\boldsymbol{\Theta}_{\text{s}}\|^{2}_{\text{F}}=\sum_{i=1}^{K}\check{\lambda}_{1,\pi_{1}(i)}\check{\lambda}_{2,\pi_{2}(i)},$
(25)

where $\check{\lambda}_{1,i}$ and $\check{\lambda}_{2,i}$ correspond to the
$i$th eigenvalue of $\widecheck{\boldsymbol{G}}_{1}$ and
$\widecheck{\boldsymbol{G}}_{2}$, respectively, while $\pi_{1}$ and $\pi_{2}$
correspond to the ${(M-K)}$\- and $K$-element permutations of associated to
$\boldsymbol{P}_{1}$ and $\boldsymbol{P}_{2}$, respectively. From (25), we can
note that the diagonal values of $\boldsymbol{\Lambda}_{\boldsymbol{\varphi}}$
have no impact whatsoever on the objective function since the corresponding
phases cancel each other after being multiplied by their conjugates. We may
thus select $\boldsymbol{\Lambda}_{\boldsymbol{\varphi}}=\boldsymbol{I}_{K}$
without loss of generality. On the other hand, since only $K$ of the $M-K$
eigenvalues of $\widecheck{\boldsymbol{G}}_{1}$ appear in (25), we can
trivially select $\pi_{1}$—through the respective $\boldsymbol{P}_{1}$—such
that eigenvalues of $\widecheck{\boldsymbol{G}}_{1}$ appearing in (25) are the
$K$ eigenvalues with lower value. Alternatively, we may equivalently
substitute $\widetilde{\boldsymbol{U}}_{1}$ in (24), associated to the
eigenvalue decompostion of $\widecheck{\boldsymbol{G}}_{1}$ for increasing
eigenvalues, with the respective unitary matrix associated to the eigenvalue
decompositon for decreasing eigenvalues and select
$\boldsymbol{P}_{1}=\mathbf{I}_{M-K}$. In any case, we may trivially fix
$\boldsymbol{P}_{1}$ without loss, and the remaining DoFs are captured
exclusively by $\pi_{2}$ (or correspondingly $\boldsymbol{P}_{2}$). Hence,
with the previous considerations, the search space for the global minimum of
(17) is reduced to evaluating the $K!$ possible options for $\pi_{2}$ in (25),
and choosing the one attaining lowest value. However, me may further obtain a
closed form optimal selection of $\pi_{2}$ by considering the following lemma.

###### Lemma 2

Considering a finite increasing sequence $\\{\check{\lambda}_{1,i}\\}_{1\leq
i\leq K}$ and an arbitrary permutation $\pi_{2}$ of another finite increasing
sequence $\\{\check{\lambda}_{2,i}\\}_{1\leq i\leq K}$, the solution to

$\min_{\pi_{2}}\sum_{i=1}^{K}\check{\lambda}_{1,i}\check{\lambda}_{2,\pi_{2}(i)}$
(26)

is given by the permutation $\pi_{2}^{\text{opt}}$ attaining a decreasing
sequence $\\{\check{\lambda}_{2,\pi_{2}^{\text{opt}}(i)}\\}_{1\leq i\leq K}$.

### Sketch of Proof

The proof can be extended from the $K=2$ case, where we can express
$\check{\lambda}_{1,1}=\check{\lambda}_{1,2}+\epsilon_{1}$ and
$\check{\lambda}_{2,1}=\check{\lambda}_{2,2}+\epsilon_{2}$ for some
$\epsilon_{1},\epsilon_{2}>0$. If we select $\pi_{2}$ such that
$\\{\check{\lambda}_{2,\pi_{2}(i)}\\}_{1\leq i\leq 2}$ is increasing instead
of dectreasing we get an extra additive term $\epsilon_{1}\epsilon_{2}$, which
is positive from the definition of $\epsilon_{1}$ and $\epsilon_{2}$. Hence,
the optimal choice is to have $\pi_{2}$ such that
$\\{\check{\lambda}_{2,\pi_{2}^{\text{opt}}(i)}\\}_{1\leq i\leq 2}$ is
decreasing. A formal proof may be included in the extended version. $\square$

If we put together Lemmas 1 and 2, as well as our observation on how to fix
$\boldsymbol{P}_{1}$, we can obtain the closed form global optimum
$\widetilde{\boldsymbol{U}}^{\text{opt}}$ to the minimization problem from
(17)—up to some extra DoFs with no impact on
$\|\boldsymbol{\Theta}_{\text{s}}(\widetilde{\boldsymbol{U}})\|^{2}_{\text{F}}$.

##  V Numerical Results

Due to the channel power requirement for the orthogonalization condition in
(16), the power of the direct channel $\boldsymbol{H}_{0}$, which is linearly
related to the maximum eigenvalue of
$\boldsymbol{H}_{0}^{\text{H}}\boldsymbol{H}_{0}$, is one of the limiting
factors to be able to efficiently implement the proposed scheme. Specifically,
it is the relation between this channel and the gain of the channels
$\boldsymbol{H}_{1}$ and $\boldsymbol{H}_{2}$ that matters, since these also
scale the power of the retransmitted signals consequently processed to
compensate the direct channel. Thus, we model $\boldsymbol{H}_{0}$,
$\boldsymbol{H}_{1}$, and $\boldsymbol{H}_{2}$ as normalized IID Rayleigh
fading channels [22], where the power of $\boldsymbol{H}_{0}$ is scaled by a
variable $\eta$ giving the relation between the direct channel and the RRTx
channel.666We are aware that the IID Rayleigh channel model may not be
accurate when considering panel-based LIS scenarios since it disregards the
inherent spatial correlation of panels [5], as well as the presence of strong
line-of-sight paths [1]. However, it still allows us to assess the main
limitations of our methods, which are mainly related to the rank and power of
the considered channels, while it detaches them from specific
geometries/implementations.

In Fig. 1 (left), we show the average minimum semi-passive processing power
required to achieve channel orthogonalization for different values of $M$ and
$K$. We include results for the optimized DoF selection from Section IV, as
well as for a random seletion, i.e., where $\widetilde{\boldsymbol{U}}$ in
(15) is isotropically selected, which outlines the importance of our
optimization method that allow to reduce the required power by roughly $10$
dB. On the other hand, we include as comparison the optimization results from
[16], which solves an equivalent optimization problem through geodesic
gradient descent for two reflective surfaces, namely amplitude-reconfigurable
intelligent surface (ARIS) and FRIS. For equal antenna combinations, the new
results outperform those of ARIS, and, for weak direct channels, they also
outperform FRIS. However, for greater direct channel gains, FRIS still
requires lower processing power, which makes sense since applying per-
subcarrier baseband processing makes our solution more robust, but it comes at
the extra cost of having to compensate an ISI channel. Moreover, as outlined
in [14], the complication of practical implementations of FRIS makes it mostly
suitable as a theoretical construction that upper limits the capabilities of
such systems, while our solution still performs close to it. On the other
hand, we should not disregard the corresponding channel gain associated to
these processing power curves, shown in Fig. 1 (right). We may note that the
proposed method attains the same minimum $\beta$ as that of FRIS, ARIS for the
same antenna configurations, hinting that a similar rule to (16) may apply to
such systems, while this was not formally proved yet in [16, 14].

Figure 1: Semi-passive power (left) for minimum channel gain per UE (right)
with respect to the direct channel gain.

Fig. 2 shows the sum capacity of the proposed method with and without
considering Assumption 1. We have assumed that the power of the noise added at
the semi-passive panels is the same as that of the noise added at the active
panel, i.e., $\tilde{N}_{0}=N_{0}$. However, $\tilde{N}_{0}$ could be actually
lower when operating the semi-passive panels at low amplification, so that
Assumption 1 may have better accuracy in reality than what is seen from Fig.
2. In any case, we see that this assumption is still accurate as long as the
direct channel gain is not much greater than the gain of channels experienced
by the RRTx process, while in general scenarios we may expect them to be in
the same order. We have plotted for comparison the capacity achieved when the
three panels are actively serving the UEs, giving a performance upper bound,
as well as when the active LIS-panel serves the UEs by itself, giving a
performance lower bound. However, we should note that our method has the extra
advantage of achieving a perfectly orthogonal channel, which is not fulfilled
by the other two cases. As expected, having 3 active LIS-panels has an
important advantage when the direct channel gain is low, but comes at the cost
of higher interconnection bandwidth, while our method still gets reasonably
close performance as gain increases.

Figure 2: Ergodic sum capacity with respect to direct channel gain for $M=8$
antennas per panel and $K=2$ UEs.

##  VI Conclusions

We have studied channel orthogonalization in a panel-based LIS scenario with a
single active LIS-panel and two semi-passive LIS panels jointly implementing
an RRTx scheme. We have derived the semi-passive processing required to
perfectly orthogonalize the resulting channel, and characterized the
conditions under which this is possible. We have also derived closed-form
results on how to utilize the DoFs available from the orthogonality constraint
to obtain (globally) minimum semi-passive processing power. The numerical
results showcase the potential of the methods studied in this work.

## References

  * [1] S. Hu, F. Rusek, and O. Edfors, “Beyond massive MIMO: The potential of data transmission with large intelligent surfaces,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 10, pp. 2746–2758, May 2018.
  * [2] T. L. Marzetta, “Noncooperative cellular wireless with unlimited numbers of base station antennas,” _IEEE Transactions on Wireless Communications_ , vol. 9, no. 11, pp. 3590–3600, November 2010.
  * [3] F. Rusek, D. Persson, B. K. Lau, E. G. Larsson, T. L. Marzetta, O. Edfors, and F. Tufvesson, “Scaling up MIMO: Opportunities and challenges with very large arrays,” _IEEE Signal Processing Magazine_ , vol. 30, no. 1, pp. 40–60, Jan 2013.
  * [4] E. Björnson, L. Sanguinetti, H. Wymeersch, J. Hoydis, and T. L. Marzetta, “Massive MIMO is a reality—what is next?: Five promising research directions for antenna arrays,” _Digital Signal Processing_ , vol. 94, pp. 3–20, 2019, special Issue on Source Localization in Massive MIMO. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1051200419300776
  * [5] A. Pizzo, T. L. Marzetta, and L. Sanguinetti, “Spatially-stationary model for holographic mimo small-scale fading,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 9, pp. 1964–1979, 2020.
  * [6] A. Pereira, F. Rusek, M. Gomes, and R. Dinis, “Deployment strategies for large intelligent surfaces,” _IEEE Access_ , vol. 10, pp. 61 753–61 768, 2022\.
  * [7] M. Jung, W. Saad, Y. Jang, G. Kong, and S. Choi, “Performance analysis of large intelligent surfaces (LISs): Asymptotic data rate and channel hardening effects,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 3, pp. 2052–2065, 2020.
  * [8] G. Interdonato, E. Björnson, H. Quoc Ngo, P. Frenger, and E. G. Larsson, “Ubiquitous cell-free massive mimo communications,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2019, no. 1, pp. 1–13, 2019\.
  * [9] S. Elhoushy, M. Ibrahim, and W. Hamouda, “Cell-free massive mimo: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 24, no. 1, pp. 492–523, 2022.
  * [10] E. Björnson and L. Sanguinetti, “Scalable cell-free massive mimo systems,” _IEEE Transactions on Communications_ , vol. 68, no. 7, pp. 4247–4261, 2020\.
  * [11] J. Rodríguez Sánchez, F. Rusek, O. Edfors, and L. Liu, “Distributed and scalable uplink processing for lis: Algorithm, architecture, and design trade-offs,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 2639–2653, 2022.
  * [12] J. Vidal Alegría, F. Rusek, and O. Edfors, “Trade-offs in decentralized multi-antenna architectures: The WAX decomposition,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 3627–3641, 2021.
  * [13] F. Wiffen, W. H. Chin, and A. Doufexi, “Distributed dimension reduction for distributed massive MIMO c-ran with finite fronthaul capacity,” in _2021 55th Asilomar Conference on Signals, Systems, and Computers_ , 2021, pp. 1228–1236.
  * [14] J. Vidal Alegría, J. Thunberg, and O. Edfors, “Channel orthogonalization with reconfigurable surfaces: General models, theoretical limits, and effective configuration,” 2024.
  * [15] T. Jiang and W. Yu, “Interference nulling using reconfigurable intelligent surface,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 5, pp. 1392–1406, 2022.
  * [16] J. Vidal Alegría and F. Rusek, “Channel orthogonalization with reconfigurable surfaces,” in _2022 IEEE Globecom Workshops (GC Wkshps)_ , 2022, pp. 37–42.
  * [17] R. A. Horn and C. R. Johnson, _Matrix Analysis_. Cambridge University Press, 1985.
  * [18] T. E. Abrudan, J. Eriksson, and V. Koivunen, “Steepest descent algorithms for optimization under unitary matrix constraint,” _IEEE Transactions on Signal Processing_ , vol. 56, no. 3, pp. 1134–1147, 2008.
  * [19] D. H. Brandwood, “A complex gradient operator and its application in adaptive array theory,” in _IEE Proceedings H (Microwaves, Optics and Antennas)_ , vol. 130, no. 1. IET Digital Library, 1983, pp. 11–16.
  * [20] S. P. Boyd and L. Vandenberghe, _Convex optimization_. Cambridge university press, 2004.
  * [21] A. Hjorungnes and D. Gesbert, “Complex-valued matrix differentiation: Techniques and key results,” _IEEE Transactions on Signal Processing_ , vol. 55, no. 6, pp. 2740–2746, 2007.
  * [22] A. Paulraj, R. Nabar, and D. Gore, _Introduction to Space-Time Wireless Communications_ , 1st ed. USA: Cambridge University Press, 2008.


# How does a spontaneously speaking conversational agent affect user behavior?

Takahisa Iizuka
Graduate School of Regional Development and Creativity
Utsunomiya University
7-1-2 Yoto, Utsunomiya, 321-8585 Japan
<EMAIL_ADDRESS>
Hiroki Mori
Faculty of Engineering
Utsunomiya University
7-1-2 Yoto, Utsunomiya, 321-8585 Japan
<EMAIL_ADDRESS>

(April 30, 2022)

###### Abstract

This study investigated the effect of synthetic voice of conversational agent
trained with spontaneous speech on human interactants. Specifically, we
hypothesized that humans will exhibit more social responses when interacting
with conversational agent that has a synthetic voice built on spontaneous
speech. Typically, speech synthesizers are built on a speech corpus where
voice professionals read a set of written sentences. The synthesized speech is
clear as if a newscaster were reading a news or a voice actor were playing an
anime character. However, this is quite different from spontaneous speech we
speak in everyday conversation. Recent advances in speech synthesis enabled us
to build a speech synthesizer on a spontaneous speech corpus, and to obtain a
near conversational synthesized speech with reasonable quality. By making use
of these technology, we examined whether humans produce more social responses
to a spontaneously speaking conversational agent. We conducted a large-scale
conversation experiment with a conversational agent whose utterances were
synthesized with the model trained either with spontaneous speech or read
speech. The result showed that the subjects who interacted with the agent
whose utterances were synthesized from spontaneous speech tended to show
shorter response time and a larger number of backchannels. The result of a
questionnaire showed that subjects who interacted with the agent whose
utterances were synthesized from spontaneous speech tended to rate their
conversation with the agent as closer to a human conversation. These results
suggest that speech synthesis built on spontaneous speech is essential to
realize a conversational agent as a social actor.

_Keywords_ Conversational agents, Spontaneous speech synthesis, Human-computer
interaction

##  1 Introduction

Nass and Moon[1] stated that humans interacting computers perceive them as
social actors, and unconsciously behave socially towards them. In their
several socio-psychological experiments, they observed that humans react to
computers in the same social manner as they do to humans. These findings,
however, do not mean that humans always respond to computers as they do to
humans. Rather, people rarely exhibit anthropocentric reactions to computers
that surround us today. They treat voice assistants[2, 3] as mere machines
that can be controlled by voice commands. People rarely respond to voice
agents with backchannels such as “Uh-huh,” interjections such as “Wow!,” or
emotional expressions such as laughs, unlike they do to humans. To make human-
computer interaction closer to human-human interaction, we think it necessary
to change the current human attitude toward computers to be more social. In
the “Computers are social actors” theory[4], cues that are closely associated
with the human prototype, such as spoken words and interactivity, trigger
prescribed behaviors in human-human interactions, which in turn cause mindless
social responses for machines. This suggests the possibility that more human-
like cues from machines encourage more social responses from humans. We are
approaching the realization of such a conversational agent as a social actor
from the viewpoint of speech synthesis. We assume that the way a
conversational machine has _social patiency_ [5] (= the capacity to have its
_face_[6] threatened or affirmed by social action) reflects its _paralanguage_
(= the way of speaking) as an _observable_ that constitutes the human
interactant’s _levels of abstraction_ (LoA) [7]. In other words, we assume
that humans do not take social actions to a conversational machine as they do
to a human partly because current speech synthesis lacks some paralinguistic
property that constitutes their LoA at which the machine is perceived as
having social patiency.

Nonverbal social behavior such as backchannels, laughs, expressive
interjections, and repetitions is an indication of attending conversations,
and helps a lot in facilitating human-human conversations, which Den and his
colleagues[8] called _response tokens_. Backchannels are short utterances
spoken by a listener such as “Uh-huh” or “Yeah”. Maynard defined it as a brief
expression sent by the listener while the speaker is speaking[9]. The role of
the backchannel of a listener is to display that she/he understands what the
speaker is saying or is paying attention to the speaker [10]. Expressive
interjections are non-lexical speech sounds which indicate the speaker’s
cognitive or affective state changes. They are distinguished from another type
of interjections, filled pauses, in their unique morphological and pragmatic
properties[11]. Repetitions have social functions, such as involvement in
conversations, showing interest, concerning and surprising, and eliciting
backchannel-like responses [12]. In addition to these response tokens, filled
pauses play an important role in conversations. Studies on filled pauses
suggest that they show hesitation or intention to take a turn[13, 14].

In a human-human conversation, the speaker takes advantage of these social
responses as indicators of the listener’s state of understanding and
dynamically redesigns her or his speech accordingly. Such social responses
could be useful for smoother interaction between human and computer as well,
for example, regulating computer’s speech rate and response timing, once
humans begin to show social responses to computers.

Studies on spoken dialogue systems involve dialogue management[15, 16], speech
recognition[17, 18, 19], and speech synthesis[20, 21, 22, 23, 24, 25, 26].
Recent studies on speech synthesis[24, 25, 26] have dramatically improved the
quality of synthesized speech to the point where it is difficult to
distinguish the synthesized speech from a real human voice. However, this does
not mean that current speech technology has reached a status where it can
fully reproduce all speech sounds that humans may utter in everyday
conversation. So far, the construction of speech synthesizers has been assumed
to involve written sentences read by voice professionals such as newscasters
or voice actors. For this reason, current speech synthesizers speak as if a
newscaster were reading a news or a voice actor were playing an anime
character. In daily conversation, however, we speak spontaneously without a
script. Spontaneous speech has quite different nature from read speech,
especially in its prosody [27, 28]. Speech in conversation may exhibit some
characteristic tone patterns including the final rise-fall[29, 30, 31]. These
prosodic cues are considered to be crucial not only to sound like spontaneous
speech, but also play a meta-communicative role such as turn coordination,
which is intrinsic to real-time interactions of humans. Using a read speech
corpus on scripted text for building a speech synthesizer is to ignore such an
important aspect of speech in conversation.

A number of studies aim to apply the nonverbal aspects of human-human
interactions, such as turn-taking[32], anthropomorphization[33],
backchannel[34], and gaze[35], to enhance human-computer interactions.
Regarding nonverbal aspects of synthesized speech for spoken dialogue systems,
Chiba et al.[36] showed the effects of emotional speech synthesis in non-task-
oriented dialogue systems. The study showed that the use of appropriate
emotional expression could improve subjective impressions of humans such as
dialogue richness and likeness to the agent. James et al.[37] showed that
adding appropriate emotion to the speech of the robot could express empathy
for users and be preferred to a typical voice of the robot. Misu et al.[38]
investigated whether the dialogue- or monologue-style induced more
backchannels from humans as training data for speech synthesizers. They showed
that synthesized speech in dialogue-style induced more natural backchannels
and nods than in monologue-style.

In almost all previous studies on spoken dialogue system including the above,
the speech synthesizer used was the one trained with read speech. Despite the
discrepancy between read and spontaneous speech as pointed out above, there
have been quite few attempts to synthesize conversational speech using a
spontaneous speech corpus. After an attempt at HMM-based speech synthesis
using conversational speech[39], no work on spontaneous conversational speech
synthesis was found in Interspeech conferences or SSW (Speech Synthesis
Workshops) presentations until Ben-David and Shechtman[40], except for the
authors’ works [41, 42]. One reason for this might be that it has been unclear
what a spontaneously speaking machine could be useful for. The current study
is the first one to demonstrate that speech synthesis based on spontaneous
speech is actually useful for making human-machine interaction more like
human-human interaction.

Our study investigates the effect of synthetic voice of conversational agent
trained with spontaneous speech on human interactants. We hypothesize that the
synthetic voices of current conversational agents, built on a read speech
dataset, cause humans to behave as if they are mere machines rather than
_social actors_. We also hypothesize that humans will exhibit more social
responses when interacting with a conversational agent that has a synthetic
voice built on a spontaneous speech dataset.

To test these hypotheses, we focused on nonverbal behavior of humans as
listeners, which include backchanneling, interjections, laughing, and nodding
because humans rarely exhibit this behavior while interacting with the
existing spoken dialogue systems. We also observed the response time of human
interactants. Timing is important because, in human-human communication,
people attend closely to the turn-taking timing of their own and their
partner, trying to keep the right timing when it goes awry[43]. On the other
hand, if people do not regard a conversational agent as a social actor, they
would not care about when to speak, and they would not mind keeping the agent
waiting all the time. So far, the nonverbal behavior of spoken dialogue
systems has been studied extensively [32, 33, 34, 35]. However, studies
regarding nonverbal behavior of users have been very few[38, 44, 45, 46]. The
conversation experiment designed in this paper focuses on the nonverbal
behavior of human interactants. This allows us to examine whether humans
produce more nonverbal responses to a spontaneously speaking conversational
agent in shorter delays, i.e., their tendency to behave as _social agents_[5].

For the experiment, we set up two conversational agents. One is an agent whose
utterances were synthesized using spontaneous speech, and the other is an
agent whose utterances were synthesized using read speech. Two groups of
subjects, assigned to either the “spontaneous” or “read” condition,
participated in a chat with the conversational agent that follows an identical
scenario. Then, the frequency of the subjects’ response tokens and the
distribution of response time are compared for the two conditions. A
subjective evaluation using a questionnaire on the impressions of the
conversational agent was also conducted.

##  2 Speech synthesis

###  2.1 Corpus

####  2.1.1 Spontaneous speech corpus

For the spontaneous speech corpus, we used the Utsunomiya University Spoken
Dialogue Database (UUDB) [47], where participants (12 females and 2 males)
were engaged in the “four-frame cartoon sorting task” to estimate the original
order of the shuffled frames. This corpus consists of 27 sessions and lasts
about 130 min. In this study, utterances of a female speaker FTS were used
because her recorded speech was longest in total duration in this corpus
(about 18 min.).

####  2.1.2 Read speech corpus

For the read speech corpus, we used the Japanese speech corpus of Saruwatari-
lab, the University of Tokyo (JSUT) [48]. JSUT contains speech data of a
female who is not a professional speaker but has experience working with
voices. We used all the subcorpora of JSUT, which are approximately 10 hours
in length in total.

###  2.2 Method

As the speech synthesizer, we used Tacotron 2 [25]. Tacotron 2 is a
combination of the Tacotron-style model[24] that generates mel-spectrograms
from a sequence of characters and the modified WaveNet [49, 50] that
conditions on mel-spectrograms to generate the waveform. Since this approach
can directly learn the correspondence between characters and waveforms, it can
generate speech of such high quality that it is difficult to distinguish from
a real human voice.

In this study, we replaced the neural vocoder with MelGAN [51]. To train
MelGAN, we used spontaneous monologue speech of 361 speakers in the Corpus of
Spontaneous Japanese [52].

Two Tacotron 2 models were trained for synthesizing the agent’s utterances to
be used in the conversation experiment. The “read” model was built on the read
speech corpus, JSUT. Likewise, the “spontaneous” model was built on the
spontaneous speech corpus, UUDB. However, model training could not be done
straightforwardly unlike the “read” model, because UUDB is a natural dialogue
corpus. Treatment of nonverbal sounds such as laughter was an issue. For the
present study, we simply ignore them; utterance that contains nonlinguistic
sounds was split so as to include spoken contents only. Another unique
phenomenon in spontaneous speech is filled pauses and expressive
interjections. Because these interjections have different acoustic properties
from ordinary lexical sounds [53, 11], a dedicated vowel set was defined to
transcribe these sounds for UUDB.

Another issue in using a natural dialogue corpus to build speech synthesizers
is its insufficiency in size. To overcome this, the pretraining and fine-
tuning approach was adopted. Starting from “read” model trained from JSUT with
sufficient amount of data, the “spontaneous” model was trained by fine-tuning
the initial model using UUDB. This allowed us to obtain a near conversational
synthesized speech with reasonable quality, even with a small amount of data.

In this study, the mel spectrograms were calculated using a short-time Fourier
transform using 50 ms frame size, 12.5 ms frame hop, and a Hann window
function, as in the original Tacotron 2 paper [25]. The model hyperparameters
were set to the default values of the NVIDIA’s implementation [54], except for
the threshold to generate the stop token to be 0.1, which was necessary to
obtain stable outputs.

###  2.3 Analysis of prosody

The “spontaneous” model produces speech that gives a quite different
impression than conventional speech synthesizers built on read speech corpora,
primarily due to its prosody. Specifically, the “spontaneous” model reproduces
tone patterns characteristic of conversational speech, particularly phrase-
final tones. To quantitatively compare the prosody synthesized by the
“spontaneous” and “read” models, a prosodic labelling based on the J_ToBI [55]
(Japanese Tones and Break Indices) was performed for the synthesized speech.
J_ToBI describes prosody from two aspects; prosodic pitch (Tone) and prosodic
boundary strength (BI) in Japanese. It defines a set of phrase-final boundary
tone labels L%, L%H%, and L%HL%, which roughly correspond to fall, rise, and
rise-fall, and the latter two combined tones constitute the boundary pitch
movements (BPMs). Because most of the sentences in read speech corpora are
declarative, and voice professionals generally avoid using BPMs except at the
end of a sentence when reading texts out loud, conventional speech
synthesizers are exclusively capable of producing speech without BPMs, unless
interrogative sentences are specially handled. Contrastively, spontaneous
speech contains a lot of BPMs. Therefore, we assumed that the “spontaneous”
model produces speech with a larger number of BPMs than the “read” model for a
fixed set of text.

Table 1: Frequencies of the phrase-final boundary tones appeared in the conversational agent’s synthesized utterances. | “read” model | “spontaneous” model
---|---|---
L% | 133 | 105
L%H% | 9 | 20
L%HL% | 0 | 17

The distribution of phrase-final boundary tones in the synthesized utterances
using the two models is shown in Table 1. Note that since the set of input
texts was exactly the same, the total number of phrase-final boundary tones
(corresponding to Break Index 2 [55]) was also the same for the two models.
This result shows that there was no rise-fall pattern in the utterances
synthesized with the “read” model, in contrast to the “spontaneous” model.
This means that utterances synthesized with the “spontaneous” model reflected
the prosodic properties of natural conversational speech used to train the
model.

###  2.4 Subjective evaluation of synthesized speech

To compare the overall impressions of synthesized speech built from “read” and
“spontaneous” models, a subjective evaluation test was performed. In addition
to speech _clarity_ as a common criterion in evaluating speech synthesizers,
we also evaluated speech _spontaneity_ , the degree to which the synthesized
speech sounds like it was uttered on the spot without a script. The Likert
scales for assessing clarity and spontaneity in the questionnaire were:

#### Clarity

Excellent

Good

Fair

Poor

Bad

#### Spontaneity

I am convinced that she was speaking what came to her mind on the spot.

I feel that she was speaking what came to her mind on the spot.

I am not sure whether she was speaking what came to her mind on the spot or a
script.

I feel that she was speaking from a script.

I am convinced that she was speaking from a script.

The subjective evaluation test was conducted as a follow-up to the
conversation experiment described in Sect. 3. The subjects were 26
undergraduate and graduate students who also participated in the conversation
experiment and agreed to participate the additional experiment. The stimulus
set consisted of 50 synthesized utterances, which was identical to that used
in the conversation experiment described in the next section, per model. The
experiment was conducted in a within-subjects design, i.e., each subject
evaluated a total of $50\times 2=100$ utterances.

The results of the subjective evaluation test are shown in Fig. 1. In the box-
and-whiskers plot, the lower and upper hinges correspond to the first and
third quartiles, the lower and upper whiskers extend from the hinge to the
smallest and largest not-outlying values, and individual points correspond to
the outlying values. The mean clarity was 4.34 and 2.93 for the “read” and
“spontaneous” models, respectively (Fig. 1(a)). From this result, it can be
said that synthesized speech built on the read speech corpus was perceived
clearer as a whole than that built on the spontaneous speech corpus.

Figure 1: Comparison of synthesized speech with the “read” and “spontaneous”
models from the viewpoints of clarity and spontaneity.

Note, however, that this does not necessarily mean the inferiority of the
speech synthesis based on spontaneous speech. Rather, we should argue whether
the synthesized speech is intelligible enough for smooth communication. It is
natural to assume that our daily speech is inherently less clear than read-
aloud speech in, for example, news or dramas, since we try to save as much
speech effort as possible to the extent that the speech act can achieve its
goal. Conversely, one would feel unnatural if someone else spoke to her/him as
clearly as a newscaster reads a news. From this perspective we think that the
clarity of the “spontaneous” model is acceptable for the conversational
agent’s voice.

The mean spontaneity was 1.80 and 3.78 for the “read” and “spontaneous”
models, respectively (Fig. 1(b)). This result indicates that the “spontaneous”
model tends to produce speech that sounds as if it was uttered on the spot
more than the “read” model.

From these results, it was found that the speech synthesized with the
“spontaneous” model indeed had a quality of spontaneous speech. By using a
spontaneous speech corpus for training, it is possible to synthesize speech
that is close to our everyday speech, at least in some aspects. Therefore, one
might expect a human-machine interaction that is closer to a human-human
interaction, by using “spontaneous” synthesized speech as the machine’s voice.

##  3 Conversation experiment with a spontaneously speaking agent

###  3.1 Overview of the conversational agent

The conversational agent used for this experiment was implemented using the
MMDAgent [56]. MMDAgent is a platform for building spoken dialogue systems
that have modules of speech recognition, speech synthesis, dialogue
management, and 3D model motion management. Fig. 2 shows an overview of the
conversational agent used in this experiment.

Figure 2: Overview of the conversational agent (adapted from [56].)

In this study, we did not use the default speech recognizer but instead
applied the Wizard of Oz (WoZ) technique [57], where an experimenter operates
the agent behind the subject. The reason for employing WoZ was to prevent bad
impressions to the agent due to speech recognition errors or unnatural speech
timings. The agent was designed to speak by playing back pre-synthesized
utterances according to the wizard’s operation.

The dialogue scenario was designed so that the agent speaks almost
unilaterally. In the scenario, the agent asks the human interactant if she/he
is interested in traveling abroad, right after the initial greeting. No matter
she/he is interested or not, the agent talks about countries she loves to go.
After this, the agent continues to talk about various trivia about countries
around the world. Sometimes she quizzes the human interactant, such as “Do you
know which country is most famous for pyramids?” Basically, answering to these
quizzes are only opportunity for the human interactant to take turns, forcing
she/he to be a listener for the rest of the time.

Instead of writing the scenario by hand, we first recorded a dialogue between
one of the authors and his close relative, where the author improvised the
role of the agent, with no script at all. The recorded dialogue was then
transcribed and transformed to an FST (finite-state transducer) for MMDAgent
by an in-house tool. We consider it crucial to avoid handwritten scripts for
the utterances of conversational agents, because actual words of spontaneous
utterances have different linguistic characteristics from “imaginary” words,
and human interlocutors tend to behave differently in response [58].

The wizard manipulated the agent’s behaviors, which include triggering the
next utterance in the scenario, determining whether the subject’s answer to a
quiz is correct or not, triggering a backchannel, triggering an utterance to
encourage the subject to speak friendly, triggering a confirmation that the
subject is attending, and triggering an expression to get the conversation
back on track (such as “Anyway,”) when it is going to break down. Determining
the correctness of the answers to the quiz was necessary to reflect on the
agent’s next action. Sending backchannel was intended to make the agent behave
more human-like when the subject is speaking. A previous study revealed that
randomly generating acoustically different backchannels improves the
naturalness of dialogue compared to repeatedly generating an identical
backchannel [59]. Therefore, three similar but different backchannels were
prepared and randomly selected for playback. The purpose of encouraging to
speak friendly was to induce a relaxed and natural mood, as if the subject
were talking with a friend. The purpose of asking if the subject was listening
was to prevent the subject from becoming a mere listener and to encourage
reactions. However, this operation was limited to twice at most in each
conversation.

The appearance of the agent was replaced to a silhouette in order to prevent
any inconsistency between the appearance of the agent and the individuality of
the synthesized speech.

The system displayed subtitles simultaneously with the agent’s utterance. This
prevented subjects from missing utterances even when the quality of
synthesized speech was not sufficient.

###  3.2 Method

The subjects were 50 undergraduate and graduate students who were not engaged
in speech research. They received both verbal and written explanations of the
experiment, and provided written informed consent before the experiment. The
experiment was approved by the Ethics Committee on Research Involving Humans,
Utsunomiya University.

They were assigned to either the “spontaneous” or “read” condition, namely,
the experiment was conducted in a between-subjects design. In the
“spontaneous” condition, the subject had a conversation with the agent whose
utterances were synthesized by the “spontaneous” model described in Sect. 2.2.
The “read” condition was identical to the “spontaneous” condition except that
the agent’s utterances were synthesized by the “read” model. A video excerpt
of a conversation in the “spontaneous” condition is included as supplemental
material.

It is not considered fair to have a subject participate in both conditions. If
a subject interacted with both “spontaneous” and “read” agents, she/he would
easily notice that the objective of the experiment was to test the effect of
the agent’s voice. Eventually, the subject might also notice that one agent
was speaking spontaneously unlike existing dialogue systems and, in an effort
to be a “good subject,” might try to behave more favorably in her/his
interaction with the “proposed” agent. Therefore, the conversation experiment
should not be conducted in a within-subjects design, but in a between-subjects
design.

As evaluation indices of how close the human-agent interaction and human-human
interaction are, we examined the response time of subjects to the agent, the
number of subjects’ response tokens (backchannels, expressive interjections,
laughs, and filled pauses), and the number of nods. By investigating these
nonverbal behaviors, it is possible to determine whether humans talking with a
conversational agent behave as if it were a human-like social actor and not
just a machine.

After the session was over, each subject was asked to rate her/his impressions
of the agent and the quality of the conversation using a 6-item questionnaire.
As a debriefing after the evaluation, the subjects were told that the
conversational agent was not automated but human-operated.

##  4 Result

###  4.1 Nonverbal behavior

Fig. 3 shows the distribution of the indices for nonverbal behaviors during
the interaction between the conversational agent and subjects. In the
following paragraphs, summary statistics are shown in the form of mean and 95%
confidence intervals.

Figure 3: Nonverbal behavior indices.

The mean response time was $1.16\pm 0.07$ sec. for the “read” condition and
$0.99\pm 0.05$ sec. for the “spontaneous” condition (Fig. 3(a)), and the
difference was significant (Welch’s $t$-test, $t(957.8)=3.80$, $p=0.00016$).
This result indicates that subjects who interacted with the agent whose
utterances were synthesized from spontaneous speech data tended to respond
faster than those who interacted with the agent whose speech was synthesized
from read speech data.

The average number of backchannels was $6.96\pm 3.92$ for the “read” condition
and $19.00\pm 6.91$ for the “spontaneous” condition (Fig. 3(b)), and the
difference was significant ($t(38.0)=-3.13$, $p=0.0034$). This result
indicates that subjects who interacted with the agent whose utterances were
synthesized from spontaneous speech data tended to show a larger number of
backchannels than those who interacted with the agent whose speech was
synthesized from read speech data.

The average number of expressive interjections was $8.92\pm 2.68$ for the
“read” condition and $10.12\pm 2.02$ for the “spontaneous” condition (Fig.
3(c)), and the difference was not significant ($t(44.6)=-0.71$, $p=0.48$).

The average number of filled pauses was $3.60\pm 1.09$ for the “read”
condition and $3.92\pm 1.01$ for the “spontaneous” condition (Fig. 3(d)), and
the difference was not significant ($t(47.8)=-0.45$, $p=0.66$).

The average number of laughs was $5.72\pm 3.26$ for the “read” condition and
$6.04\pm 2.82$ for the “spontaneous” condition (Fig. 3(e)), and the difference
was not significant ($t(47.0)=-0.15$, $p=0.88$).

The average number of nods was $27.58\pm 8.23$ for the “read” condition and
$36.42\pm 10.20$ for the “spontaneous” condition (Fig. 3(f)), and the
difference was not significant ($t(44.1)=-1.40$, $p=0.17$).

In summary, humans interacting with the agent whose utterances were
synthesized from spontaneous speech data tended to exhibit shorter response
times and more response tokens, which can be interpreted that they behaved
more like interacting with a human.

###  4.2 Questionnaire

Table 2 summarizes the subjects’ impressions of the agent and the quality of
their conversations. Each section of the table shows the question, the meaning
of the scale (5–1), and a contingency table (columns correspond to the
response options and rows correspond to the conditions to which the subjects
were assigned). The most notable result is for the question “How close was
your conversation with Mei-chan to a conversation with a human?”. The mean was
3.60 for the “read” condition and 4.08 for the “spontaneous” condition. The
Brunner-Munzel test showed that the difference in the response distributions
for the two conditions was significant ($p=0.049$). This result indicates that
subjects who interacted with the agent whose utterances were synthesized from
spontaneous speech data tended to evaluate their conversation as closer to a
human conversation.

Table 2: Questions and responses asking subjects’ impression of the agent and
the conversation quality.

The differences for other items were not significant.

##  5 Discussion

Subjects in the “spontaneous” condition tended to respond faster than in the
“read” condition. This result suggests that subjects who interacted with the
spontaneously speaking conversational agent are more likely to feel pressure
to respond at the right timing. Since one is less likely to feel such time
pressure when interacting with mere a machine, this implies that the subjects
tended to view the agent as a social actor rather than mere a machine.

Subjects in the “spontaneous” condition tended to show a larger number of
backchannels than in the “read” condition. Since one is less likely to show
backchannels to mere a machine, this may also be evidence that the subjects
tended to view the agent as a social actor rather than mere a machine.

Subjects in the “spontaneous” condition tended to evaluate their conversation
with the agent as closer to a human conversation than in the “read” condition.
This result supports the above interpretation of the nonverbal behavior
results, i.e., spontaneously speaking agent tend to be more viewed as a social
actor.

These results suggest that speech synthesis built on spontaneous speech is
essential to realize a conversational agent as a social actor.

At this time, however, the specific features of spontaneous speech that
explain these results still remain unknown. One candidate for such a feature
is the phrase-final boundary tones. The analysis described in Sect. 2.3 showed
a clear difference between utterances synthesized with the “spontaneous” model
and those synthesized with the “read” model in terms of phrase-final boundary
tones. Considering that the final rise-fall tones are characteristic of
conversational speech [29, 30, 31], and that they constitute important cues
that determine the occurrence of backchannels in a computational model of
dialogue prosody [60], it is natural to attribute the occurrence of
backchannel in the “spontaneous” condition to the L%HL% (rise-fall) pattern.
To prove this, we tried to find the relationship between the rise-fall
patterns of the agent’s utterances and the subjects’ subsequent responses, but
could not find any direct relationship. Additional experiments will be needed,
such as controlling the phrase-final boundary tones of the agent’s utterances
and seeing the effect on human behavior.

Another issue is the speaker individuality. If we had a pair of read speech
corpus and spontaneous speech corpus of a same speaker, we could eliminate the
extraneous variable, but building such a dataset would be very costly. We
think the effect of speaker on the current experiment was minimal because the
JSUT speaker and the UUDB speaker were both female and of the same generation.

This research is the antithesis of the conventional speech synthesis that has
placed supreme importance on naturalness as professional speech. The current
study suggests that speech synthesis for conversational agents should also aim
to produce speech that sounds as if it were uttered on the spot. In the
future, conversational agents will be used on a daily basis and will
increasingly be treated as a partner or a friend, rather than just a tool.
Speech synthesizers built on spontaneous speech will help to realize such a
conversational agent. The current study clarifies the significance of using
spontaneous speech for speech synthesis in the field of human-machine
interaction research.

##  6 Conclusions

In this paper, we investigated the effect of synthetic voice of conversational
agent trained with spontaneous speech on humans who interact with it. To
quantitatively compare the prosody synthesized by the model trained with
spontaneous speech and the model trained with read speech, a prosodic labeling
was performed for the synthesized speech, and revealed that utterances
synthesized with the model trained with spontaneous speech reflected the
prosodic properties of natural conversational speech. A subjective evaluation
test was also performed to assess the clarity and spontaneity of the
synthesized speech, and the result showed that the model trained with
spontaneous speech tended to produce speech that sounds more like spontaneous
speech uttered on the spot. A large-scale conversation experiment was
conducted with a conversational agent whose utterances were synthesized with
either the model trained on spontaneous speech or that trained on read speech.
This revealed that subjects who interacted with the agent whose utterances
were synthesized from spontaneous speech tended to show shorter response times
and a larger number of backchannels. Furthermore, the subjects who interacted
with the agent whose utterances were synthesized from spontaneous speech
tended to rate their conversation with the agent as closer to a human
conversation.

In summary, it can be concluded that humans exhibit more social responses when
interacting with a conversational agent that has a synthetic voice built on
spontaneous speech, and such an agent is more likely to be viewed as a social
actor.

## References

  * [1] C. Nass and Y. Moon, “Machines and mindlessness: Social responses to computers,” Journal of Social Issues, vol. 56, no. 1, pp. 81–103, 2000\.
  * [2] F. Bentley, C. Luvogt, M. Silverman, R. Wirasinghe, B. White, and D. Lottridge, “Understanding the long-term use of smart speaker assistants,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 2, no. 3, 2018.
  * [3] M. B. Hoy, “Alexa, Siri, Cortana, and more: An introduction to voice assistants,” Medical reference services quarterly, vol. 37, no. 1, pp. 81–88, 2018.
  * [4] C. Nass, J. Steuer, and E. R. Tauber, “Computers are social actors,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 72–78, Association for Computing Machinery, 1994.
  * [5] R. B. Jackson and T. Williams, “A theory of social agency for human-robot interaction,” Frontiers in Robotics and AI, vol. 8, pp. 1–15, 2021.
  * [6] P. Brown and S. C. Levinson, Politeness : Some universals in language usage.  No. 4 in Studies in Interactional Sociolinguistics, Cambridge University Press, 1987.
  * [7] L. Floridi, “The method of levels of abstraction,” Minds and Machines, vol. 18, pp. 303–329, 2008.
  * [8] Y. Den, H. Koiso, K. Takanashi, and N. Yoshida, “Annotation of response tokens and their triggering expressions in Japanese multi-party conversations,” in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pp. 1332–1337, 2012.
  * [9] S. K. Maynard, Kaiwabunseki.  Tokyo: Kurosio Publishers, 1993.
  * [10] J. Horiguchi, “Komyunikeshon ni okeru kikite no gengokodo,” Journal of Japanese Language Teaching, vol. 64, pp. 13–26, 1988.
  * [11] H. Mori, “Morphology of vocal affect bursts: Exploring expressive interjections in Japanese conversation,” in Proc. Interspeech 2015, pp. 1309–1313, 2015.
  * [12] R. J. Beun, “The function of repetitions in information dialogues,” IPO Annual Progress Report, vol. 20, pp. 91–98, 1995.
  * [13] C. Yamane, Nihongo no danwa ni okeru fira.  Tokyo: Kurosio Publishers, 2002.
  * [14] E. Mizukami and K. Yamashita, “An examination of a function of fillers as maintaining the speaker’s right to speak,” Cognitive Studies, vol. 14, no. 4, pp. 588–603, 2007.
  * [15] B. Thomson and S. Young, “Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems,” Computer Speech and Language, vol. 24, no. 4, pp. 562–588, 2010.
  * [16] Y. Xu, P. Huang, J. Tang, Q. Huang, Z. Deng, W. Peng, and J. Lu, “Policy optimization of dialogue management in spoken dialogue system for out-of-domain utterances,” in 2016 International Conference on Asian Language Processing (IALP), pp. 10–13, 2016.
  * [17] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in Proc. ICML ’14, p. II–1764–II–1772, 2014\.
  * [18] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Advances in Neural Information Processing Systems (C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, eds.), vol. 28, pp. 577–585, Curran Associates, Inc., 2015.
  * [19] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc. ICASSP 2016, pp. 4960–4964, 2016.
  * [20] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP 2000, vol. 3, pp. 1315–1318, 2000.
  * [21] H. Zen, K. Tokuda, and A. W. Black, “Statistical parametric speech synthesis,” Speech Communication, vol. 51, no. 11, pp. 1039–1064, 2009\.
  * [22] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using deep neural networks,” in Proc. ICASSP 2013, pp. 7962–7966, 2013\.
  * [23] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura, “Speech synthesis based on hidden Markov models,” Proceedings of the IEEE, vol. 101, no. 5, pp. 1234–1252, 2013.
  * [24] Y. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in Proc. Interspeech 2017, pp. 4006–4010, 2017.
  * [25] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu, “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP 2018, pp. 4779–4783, 2018.
  * [26] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “FastSpeech 2: Fast and high-quality end-to-end text to speech,” 2020, arXiv:2006.04558.
  * [27] T. Koriyama, T. Nose, and T. Kobayashi, “On the use of extended context for HMM-based spontaneous conversational speech synthesis,” in Proc. Interspeech 2011, pp. 2657–2660, 2011.
  * [28] T. Koriyama, T. Nose, and T. Kobayashi, “An F0 modeling technique based on prosodic events for spontaneous speech synthesis,” in Proc. ICASSP 2012, pp. 4589–4592, 2012.
  * [29] J. Pierrehumbert and M. Beckman, Japanese tone structure.  MIT Press, 1988.
  * [30] J. J. Venditti, K. Maeda, and J. P. H. van Santen, “Modeling Japanese boundary pitch movements for speech synthesis,” in Proc. 3rd ESCA Speech Synthesis Workshop, pp. 317–322, 1998.
  * [31] C. T. Ishi, “The functions of phrase final tones in Japanese: Focus on turn-taking,” Journal of the Phonetic Society of Japan, vol. 10, no. 3, pp. 18–28, 2006.
  * [32] G. Skantze, “Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks,” in Proc. SIGdial 2017, (Saarbrücken, Germany), pp. 220–230, Aug. 2017.
  * [33] D. Kontogiorgos, A. Pereira, O. Andersson, M. Koivisto, E. Gonzalez Rabal, V. Vartiainen, and J. Gustafson, “The effects of anthropomorphism and non-verbal social behaviour in virtual assistants,” in Proc. IVA ’19, p. 133–140, 2019.
  * [34] A. Krogsager, N. Segato, and M. Rehm, “Backchannel Head Nods in Danish First Meeting Encounters with a Humanoid Robot: The Role of Physical Embodiment,” in Proc. HCI AIMT 2014, vol. 8511 of Lecture Notes in Computer Science, pp. 651–662, Springer International Publishing, 2014.
  * [35] G. Skantze, A. Hjalmarsson, and C. Oertel, “Exploring the effects of gaze and pauses in situated human-robot interaction,” in Proc. SIGdIAL 2013, pp. 163–172, 2013.
  * [36] Y. Chiba, T. Nose, T. Kase, M. Yamanaka, and A. Ito, “An analysis of the effect of emotional speech synthesis on non-task-oriented dialogue system,” in Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pp. 371–375, 2018.
  * [37] J. James, C. I. Watson, and B. MacDonald, “Artificial empathy in social robots: An analysis of emotions in speech,” in 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 632–637, 2018.
  * [38] T. Misu, E. Mizukami, Y. Shiga, S. Kawamoto, H. Kawai, and S. Nakamura, “Toward construction of spoken dialogue system that evokes users’ spontaneous backchannels,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences (Japanese Edition), vol. J95-A, no. 1, pp. 16–26, 2012.
  * [39] S. Andersson, J. Yamagishi, and R. Clark, “Utilising spontaneous conversational speech in HMM-based speech synthesis,” in Proc. 7th ISCA Speech Synthesis Workshop, 2010.
  * [40] A. Ben-David and S. Shechtman, “Acquiring conversational speaking style from multi-speaker spontaneous dialog corpus for prosody-controllable sequence-to-sequence speech synthesis,” in Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), pp. 66–71, 2021.
  * [41] T. Nagata, H. Mori, and T. Nose, “Robust estimation of multiple-regression HMM parameters for dimension-based expressive dialogue speech synthesis,” in Proc. Interspeech 2013, pp. 1549–1553, 2013.
  * [42] M. Yokoyama, T. Nagata, and H. Mori, “Effects of dimensional input on paralinguistic information perceived from synthesized dialogue speech with neural network,” in Proc. Interspeech 2018, pp. 3053–3056, 2018.
  * [43] H. H. Clark, “Speaking in time,” Speech Commun., vol. 36, no. 1–2, p. 5–13, 2002.
  * [44] A. Bliek, S. Bensch, and T. Hellström, “How can a robot trigger human backchanneling?,” in 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 96–103, 2020.
  * [45] A. Hjalmarsson and C. Oertel, “Gaze direction as a backchannel inviting cue in dialogue,” in Proceedings of the IVA 2012 workshop on Realtime Conversational Virtual Agents (RCVA 2012), (Santa Cruz, CA, USA), 2012.
  * [46] Y. Arimoto, N. Nomura, and N. Kamo, “Quantitative evaluation of agency identification against conversational character agent,” in Proceedings of the 2019 Spring Meeting of the Acoustical Society of Japan, pp. 813–816, 3 2019.
  * [47] H. Mori, T. Satake, M. Nakamura, and H. Kasuya, “Constructing a spoken dialogue corpus for studying paralinguistic information in expressive conversation and analyzing its statistical/acoustic characteristics,” Speech Communication, vol. 53, no. 1, pp. 36–50, 2011.
  * [48] R. Sonobe, S. Takamichi, and H. Saruwatari, “JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,” 2017, arXiv:1711.00354.
  * [49] A. Gibiansky, S. Arik, G. Diamos, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou, “Deep Voice 2: Multi-speaker neural text-to-speech,” in Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, pp. 2963–2971, Curran Associates, Inc., 2017.
  * [50] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda, “Speaker-dependent WaveNet vocoder,” in Proc. Interspeech 2017, pp. 1118–1122, 2017.
  * [51] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), vol. 32, Curran Associates, Inc., 2019.
  * [52] K. Maekawa, “Corpus of Spontaneous Japanese: Its design and evaluation,” Proceedings of The ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition (SSPR 2003), pp. 7–12, 2003.
  * [53] K. Maekawa and H. Mori, “Comparison of voice quality between the vowels in filled pauses and ordinary lexical items,” Journal of the Phonetic Society of Japan, vol. 21, pp. 53–62, 2017.
  * [54] NVIDIA Corporation, “Tacotron 2.” https://github.com/NVIDIA/tacotron2.
  * [55] J. J. Venditti, “The J_ToBI model of Japanese intonation,” in Prosodic Typology: The Phonology of Intonation and Phrasing (S.-A. Jun, ed.), pp. 172–200, Oxford University Press, 2005.
  * [56] A. Lee, K. Oura, and K. Tokuda, “MMDAgent: A fully open-source toolkit for voice interaction systems,” in Proc. ICASSP 2013, pp. 8382–8385, 1 2013\.
  * [57] N. M. Fraser and G. Gilbert, “Simulating speech systems,” Computer Speech and Language, vol. 5, no. 1, pp. 81–99, 1991.
  * [58] Y. Takamatsuya and H. Mori, “Effects of agent’s pre-recorded vs live speech on the appearance of listener’s response tokens,” in Proc. Human Communication Symposium 2020, IEICE, 2020.
  * [59] H. Mori, “Dynamic aspects of aizuchi and its influence on the naturalness of dialogues,” Acoustical Science and Technology, vol. 34, no. 2, pp. 147–149, 2013.
  * [60] H. Koiso, Y. Horiuchi, S. Tutiya, A. Ichikawa, and Y. Den, “An analysis of turn-taking and backchannels based on prosodic and syntactic features in Japanese map task dialogs,” Language and Speech, vol. 41, no. 3-4, pp. 295–321, 1998.