11institutetext: School of Sciences and Humanities, Nazarbayev University,
Nur-Sultan, Kazakhstan
11email<EMAIL_ADDRESS>
# Squashed Shifted PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces

Zhenisbek Assylbekov  0000-0003-0095-9409 Alibi Jangeldin

###### Abstract

We show that removing sigmoid transformation in the skip-gram with negative
sampling (SGNS) objective does not harm the quality of word vectors
significantly and at the same time is related to factorizing a squashed
shifted PMI matrix which, in turn, can be treated as a connection
probabilities matrix of a random graph. Empirically, such graph is a complex
network, i.e. it has strong clustering and scale-free degree distribution, and
is tightly connected with hyperbolic spaces. In short, we show the connection
between static word embeddings and hyperbolic spaces through the squashed
shifted PMI matrix using analytical and empirical methods.

###### Keywords:

Word vectors PMI Complex networks Hyperbolic geometry

##  1 Introduction

Modern word embedding models (McCann et al., 2017; Peters et al., 2018; Devlin
et al., 2019) build vector representations of words in context, i.e. the same
word will have different vectors when used in different contexts (sentences).
Earlier models (Mikolov et al., 2013b; Pennington et al., 2014) built the so-
called static embeddings: each word was represented by a single vector,
regardless of the context in which it was used.

Despite the fact that static word embeddings are considered obsolete today,
they have several advantages compared to contextualized ones. Firstly, static
embeddings are trained much faster (few hours instead of few days) and do not
require large computing resources (1 consumer-level GPU instead of 8–16 non-
consumer GPUs). Secondly, they have been studied theoretically in a number of
works (Levy and Goldberg, 2014b; Arora et al., 2016; Hashimoto et al., 2016;
Gittens et al., 2017; Tian et al., 2017; Ethayarajh et al., 2019; Allen et
al., 2019; Allen and Hospedales, 2019; Assylbekov and Takhanov, 2019; Zobnin
and Elistratova, 2019) but not much has been done for the contextualized
embeddings (Reif et al., 2019). Thirdly, static embeddings are still an
integral part of deep neural network models that produce contextualized word
vectors, because embedding lookup matrices are used at the input and output
(softmax) layers of such models. Therefore, we consider it necessary to
further study static embeddings.

With all the abundance of both theoretical and empirical studies on static
vectors, they are not fully understood, as this work shows. For instance, it
is generally accepted that good quality word vectors are inextricably linked
with a low-rank approximation of the pointwise mutual information (PMI) matrix
or the Shifted PMI (SPMI) matrix, but we show that vectors of comparable
quality can also be obtained from a low-rank approximation of a Squashed SPMI
matrix (Section 2). Thus, a Squashed SPMI matrix is a viable alternative to
standard PMI/SPMI matrices when it comes to obtaining word vectors.

At the same time, it is easy to interpret the Squashed SPMI matrix with
entries in $[0,1)$ as a connection probabilities matrix for generating a
random graph. Studying the properties of such a graph, we come to the
conclusion that it is a so-called complex network, i.e. it has a strong
clustering property and a scale-free degree distribution (Section 3).

It is noteworthy that complex networks, in turn, are dual to hyperbolic spaces
(Section 4) as was shown by Krioukov et al. (2010). Hyperbolic geometry has
been used to train word vectors (Nickel and Kiela, 2017; Tifrea et al., 2018)
and has proven its suitability — in a hyperbolic space, word vectors need
lower dimensionality than in the Euclidean space.

Thus, to the best of our knowledge, this is the first work that establishes
simultaneously a connection between word vectors, a Squashed SPMI matrix,
complex networks, and hyperbolic spaces. Figure 1 summarizes our work and
serves as a guide for the reader.

Squashed SPMIComplex NetworksWord EmbeddingsHyperbolic SpacesSection 2Section
3Section 4 Figure 1: Summary of our work

### Notation

We let $\mathbb{R}$ denote the real numbers. Bold-faced lowercase letters
($\mathbf{x}$) denote vectors, plain-faced lowercase letters ($x$) denote
scalars, $\langle\mathbf{x},\mathbf{y}\rangle$ is the Euclidean inner product,
$(a_{ij})$ is a matrix with the $ij$-th entry being $a_{ij}$. ‘i.i.d.’ stands
for ‘independent and identically distributed’. We use the sign $\propto$ to
abbreviate ‘proportional to’, and the sign $\sim$ to abbreviate ‘distributed
as’.

Assuming that words have already been converted into indices, let
$\mathcal{W}:=\\{1,\ldots,n\\}$ be a finite vocabulary of words. Following the
setup of the widely used word2vec model (Mikolov et al., 2013b), we use two
vectors per each word $i$: (1) $\mathbf{w}_{i}\in\mathbb{R}^{d}$ when
$i\in\mathcal{W}$ is a center word, (2) $\mathbf{c}_{i}\in\mathbb{R}^{d}$ when
$i\in\mathcal{W}$ is a context word; and we assume that $d\ll n$.

In what follows we assume that our dataset consists of co-occurence pairs
$(i,j)$. We say that “the words $i$ and $j$ co-occur” when they co-occur in a
fixed-size window of words. The number of such pairs, i.e. the size of our
dataset, is denoted by $N$. Let $\\#(i,j)$ be the number of times the words
$i$ and $j$ co-occur, then
$N=\sum_{i\in\mathcal{W}}\sum_{j\in\mathcal{W}}\\#(i,j)$.

##  2 Squashed SPMI and Word Vectors

A well known skip-gram with negative sampling (SGNS) word embedding model of
Mikolov et al. (2013b) maximizes the following objective function

$\textstyle\sum_{i\in\mathcal{W}}\sum_{j\in\mathcal{W}}\\#(i,j)\left(\log\sigma(\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle)+k\cdot\mathbb{E}_{j^{\prime}\sim
p}[\log\sigma(-\langle\mathbf{w}_{i},\mathbf{c}_{j^{\prime}}\rangle)]\right),$
(1)

where $\sigma(x)=\frac{1}{1+e^{-x}}$ is the logistic sigmoid function, $p$ is
a smoothed unigram probability distribution for words111The authors of SGNS
suggest $p(i)\propto\\#(i)^{3/4}$., and $k$ is the number of negative samples
to be drawn. Interestingly, training SGNS is approximately equivalent to
finding a low-rank approximation of a Shifted PMI matrix (Levy and Goldberg,
2014b) in the form $\log\frac{p(i,j)}{p(i)p(j)}-\log
k\approx\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle$, where the left-hand side
is the $ij$-th element of the $n\times n$ shifted PMI matrix, and the right-
hand side is an element of a matrix with rank $\leq d$ since
$\mathbf{w}_{i},\mathbf{c}_{j}\in\mathbb{R}^{d}$. This approximation (up to a
constant shift) was later re-derived by Arora et al. (2016); Assylbekov and
Takhanov (2019); Allen et al. (2019); Zobnin and Elistratova (2019) under
different sets of assuptions. In this section we show that constraint
optimization of a slightly modified SGNS objective (1) leads to a low-rank
approximation of the Squashed Shifted PMI ($\sigma$SPMI) matrix, defined as
$\sigma\mathrm{SPMI}_{ij}:=\sigma(\mathrm{PMI}_{ij}-\log k)$.

###### Theorem 2.1

Assuming $0<\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle<1$, the following
objective function

$\mathcal{L}=\sum_{i\in\mathcal{W}}\sum_{j\in\mathcal{W}}\underbrace{\\#(i,j)\left(\log\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle+k\cdot\mathbb{E}_{j^{\prime}\sim
P}[\log(1-\langle\mathbf{w}_{i},\mathbf{c}_{j^{\prime}}\rangle)]\right)}_{\ell(\mathbf{w}_{i},\mathbf{c}_{j})},$
(2)

reaches its optimum at
$\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle=\sigma\mathrm{SPMI}_{ij}$.

###### Proof

Expanding the sum and the expected value in (2) as in Levy and Goldberg
(2014b), and defining $p(i,j):=\frac{\\#(i,j)}{N}$, $p(i):=\frac{\\#(i)}{N}$,
we have

$\mathcal{L}=N\sum_{i\in\mathcal{W}}\sum_{j\in\mathcal{W}}p(i,j)\cdot\log\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle+p(i)\cdot
p(j)\cdot k\cdot\log(1-\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle).$ (3)

Thus, we can rewrite the individual objective
$\ell(\mathbf{w}_{i},\mathbf{c}_{j})$ in (2) as

$\ell=N\left[p(i,j)\cdot\log\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle+p(i)\cdot
p(j)\cdot k\cdot\log(1-\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle)\right].$
(4)

Differentiating (4) w.r.t. $\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle$ we
get

$\frac{\partial\ell}{\partial\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}=N\left[\frac{p(i,j)}{\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}-\frac{p(i)\cdot
p(j)\cdot k}{1-\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}\right].$

Setting this derivative to zero gives

$\frac{p(i,j)}{p(i)p(j)}\cdot\frac{1}{k}=\frac{\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}{1-\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}\quad\Rightarrow\quad\log\frac{p(i,j)}{p(i)p(j)}-\log
k=\log\frac{\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}{1-\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle}\\\
\Leftrightarrow\quad\log\frac{p(i,j)}{p(i)p(j)}-\log
k=\operatorname*{logit}\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle\\\
\Leftrightarrow\quad\sigma\left(\log\frac{p(i,j)}{p(i)p(j)}-\log
k\right)=\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle,$ (5)

where $\operatorname*{logit}(q):=\log\frac{q}{1-q}$ is the logit function
which is the inverse of the logistic sigmoid function, i.e.
$\sigma(\operatorname*{logit}(q))=q$. From (5) we have
$\sigma\mathrm{SPMI}_{ij}=\langle\mathbf{w}_{i},\mathbf{c}_{j}\rangle$, which
concludes the proof.

###### Remark 1

Since $\sigma(x)$ can be regarded as a smooth approximation of the Heaviside
step function $H(x)$, defined as $H(x)=1$ if $x>0$ and $H(x)=0$ otherwise, it
is tempting to consider a binarized SPMI (BSPMI) matrix
$H(\mathrm{PMI}_{ij}-\log k)$ instead of $\sigma$SPMI. Being a binary matrix,
BSPMI can be interpreted as an adjacency matrix of a graph, however our
empirical evaluation below (Table 1) shows that such strong roughening of the
$\sigma$SPMI matrix degrades the quality of the resulting word vectors. This
may be due to concentration of the SPMI values near zero (Figure 5), while
$\sigma(x)$ is approximated by $H(x)$ only for $x$ away enough from zero.

###### Remark 2

The objective (2) differs from the SGNS objective (1) only in that the former
does not use the sigmoid function (keep in mind that
$\sigma(-x)=1-\sigma(x)$). We will refer to the objective (2) as Nonsigmoid
SGNS.

### Direct Matrix Factorization

Optimization of the Nonsigmoid SGNS (2) is not the only way to obtain a low-
rank approximation of the $\sigma$SPMI matrix. A viable alternative is
factorizing the $\sigma$SPMI matrix with the singular value decomposition
(SVD): $\sigma\mathrm{SPMI}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{\top}$, with
orthogonal $\mathbf{U},\mathbf{V}\in\mathbb{R}^{n\times n}$ and diagonal
$\mathbf{\Sigma}\in\mathbb{R}^{n\times n}$, and then zeroing out the $n-d$
smallest singular values, i.e.

$\sigma\mathrm{SPMI}\approx\mathbf{U}_{1:n,1:d}\mathbf{\Sigma}_{1:d,1:d}\mathbf{V}^{\top}_{1:d,1:n},$
(6)

where we use $\mathbf{A}_{a:b,c:d}$ to denote a submatrix located at the
intersection of rows $a,a+1,\ldots,b$ and columns $c,c+1,\ldots,d$ of
$\mathbf{A}$. By the Eckart-Young theorem (Eckart and Young, 1936), the right-
hand side of (6) is the closest rank-$d$ matrix to the $\sigma$SPMI matrix in
Frobenius norm. The word and context embedding matrices can be obtained from
(6) by setting
$\mathbf{W}^{\text{SVD}}:=\mathbf{U}_{1:n,1:d}\sqrt{\mathbf{\Sigma}_{1:d,1:d}}$,
and
$\mathbf{C}^{\text{SVD}}:=\sqrt{\mathbf{\Sigma}_{1:d,1:d}}\mathbf{V}^{\top}_{1:d,1:n}$.
When this is done for a positive SPMI (PSPMI) matrix, defined as
$\max(\mathrm{PMI}_{ij}-\log k,0)$, the resulting word embeddings are
comparable in quality with those from the SGNS (Levy and Goldberg, 2014b).

### Empirical Evaluation of the $\sigma$SPMI-based Word Vectors

To evaluate the quality of word vectors resulting from the Nonsigmoid SGNS
objective and $\sigma$SPMI factorization, we use the well-known corpus,
text8.222http://mattmahoney.net/dc/textdata.html. We ignored words that
appeared less than 5 times, resulting in a vocabulary of 71,290 words. The
SGNS and Nonsigmoid SGNS embeddings were trained using our custom
implementation.333https://github.com/zh3nis/SGNS The SPMI matrices were
extracted using the hyperwords tool of Levy et al. (2015) and the truncated
SVD was performed using the scikit-learn library of Pedregosa et al. (2011).

Table 1: Evaluation of word embeddings on the analogy tasks (Google and MSR) and on the similarity tasks (the rest). For word similarities evaluation metric is the Spearman’s correlation with the human ratings, while for word analogies it is the percentage of correct answers. Method | WordSim | MEN | M. Turk | Rare Words | Google | MSR
---|---|---|---|---|---|---
SGNS | .678 | .656 | .690 | .334 | .359 | .394
Nonsigm. SGNS | .649 | .649 | .695 | .299 | .330 | .330
PMI + SVD | .663 | .667 | .668 | .332 | .315 | .323
SPMI + SVD | .509 | .576 | .567 | .244 | .159 | .107
PSPMI + SVD | .638 | .672 | .658 | .298 | .246 | .207
$\sigma$SPMI + SVD | .657 | .631 | .661 | .328 | .294 | .341
BSPMI + SVD | .623 | .586 | .643 | .278 | .177 | .202

The trained embeddings were evaluated on several word similarity and word
analogy tasks: WordSim (Finkelstein et al., 2002), MEN (Bruni et al., 2012),
M.Turk (Radinsky et al., 2011), Rare Words (Luong et al., 2013), Google
(Mikolov et al., 2013a), and MSR (Mikolov et al., 2013c). We used the Gensim
tool of Řehůřek and Sojka (2010) for evaluation. For answering analogy
questions ($a$ is to $b$ as $c$ is to $?$) we use the 3CosAdd method of Levy
and Goldberg (2014a) and the evaluation metric for the analogy questions is
the percentage of correct answers. We mention here that our goal is not to
beat state of the art, but to compare SPMI-based embeddings (SGNS and
SPMI+SVD) versus $\sigma$SPMI-based ones (Nonsigmoid SGNS and
$\sigma$SPMI+SVD). The results of evaluation are provided in Table 1.

As we can see the Nonsigmoid SGNS embeddings in general underperform the SGNS
ones but not by a large margin. $\sigma$SPMI shows a competitive performance
among matrix-based methods across most of the tasks. Also, Nonsigmoid SGNS and
$\sigma$SPMI demonstrate comparable performance as predicted by Theorem 2.1.
Although BSPMI is inferior to $\sigma$SPMI, notice that such aggressive
compression as binarization still retains important information on word
vectors.

Figure 2: Spectral distribution of the $\sigma$SPMI-induced graphs (left and
middle columns), and of scale-free random graphs with strong clustering
property (right top: Goh et al. (2001), right bottom: Farkas et al. (2001)).
When generating several random graphs from the same $\sigma$SPMI matrix, their
eigenvalue distributions are visually indistinguishable, thus we display the
results of one run per each matrix.

Figure 3: Degree distributions of the $\sigma$SPMI-induced graphs. The axes are on logarithmic scales. Table 2: Clustering coefficients of the $\sigma$SPMI-induced graphs. For each corpus–window combination we generate ten graphs and report 95% confidence intervals across these ten runs. | text8 | enwik9
---|---|---
| window $=2$ | window $=5$ | window $=2$ | window $=5$
$C$ | $.1341\pm.0006$ | $.1477\pm.0005$ | $.1638\pm.0006$ | $.1798\pm.0004$
$\bar{k}/n$ | $.0014\pm.0000$ | $.0030\pm.0000$ | $.0006\pm.0000$ | $.0012\pm.0000$

##  3 $\sigma$SPMI and Complex Networks

$\sigma$SPMI matrix has the following property: its entries
$\sigma\text{SPMI}_{ij}\in[0,1)$ can be treated as connection probabilities
for generating a random graph. As usually, by a graph $\mathcal{G}$ we mean a
set of vertices $\mathcal{V}$ and a set of edges
$\mathcal{E}\subset\mathcal{V}\times\mathcal{V}$. It is convenient to
represent graph edges by its adjacency matrix $(e_{ij})$, in which $e_{ij}=1$
for $(i,j)\in\mathcal{E}$, and $e_{ij}=0$ otherwise. The graph with
$\mathcal{V}:=\mathcal{W}$ and
$e_{ij}\sim\text{Bernoulli}(\sigma\mathrm{SPMI}_{ij})$ will be referred to as
$\sigma$SPMI-induced Graph.

###  3.1 Spectrum of the $\sigma$SPMI-induced Graph

First of all, we look at the spectral properties of the $\sigma$SPMI-induced
Graphs.444We define the graph spectrum as the set of eigenvalues of its
adjacency matrix. For this, we extract SPMI matrices from the text8 and enwik9
datasets using the hyperwords tool of Levy et al. (2015). We use the default
settings for all hyperparameters, except the word frequency threshold and
context window size. We ignored words that appeared less than 100 times and
250 times in text8 and enwik9 correspondingly, resulting in vocabularies of
11,815 and 21,104 correspondingly. We additionally experiment with the context
window size 5, which by default is set to 2. We generate random graphs from
the $\sigma$SPMI matrices and compute their eigenvalues using the TensorFlow
library (Abadi et al., 2016), and the above-mentioned threshold of 250 for
enwik9 was chosen to fit the GPU memory (11GB, RTX 2080 Ti). The eigenvalue
distributions are provided in Figure 2.

The distributions seem to be symmetric, however, the shapes of distributions
are far from resembling the Wigner semicircle law
$x\mapsto\frac{1}{2\pi}\sqrt{4-x^{2}}$, which is the limiting distribution for
the eigenvalues of many random symmetric matrices with i.i.d. entries (Wigner,
1955, 1958). This means that the entries of the $\sigma$SPMI-induced graph’s
adjacency matrix are dependent, otherwise we would observe approximately
semicircle distributions for its eigenvalues. We observe some similarity
between the spectral distributions of the $\sigma$SPMI-induced graphs and of
the so-called complex networks which arise in physics and network science
(Figure 2).

Notice that the connection between human language structure and complex
networks was observed previously by Cancho and Solé (2001). A thorough review
on approaching human language with complex networks was given by Cong and Liu
(2014). In the following subsection we will specify precisely what we mean by
a complex network.

###  3.2 Clustering and Degree Distribution of the $\sigma$SPMI-induced Graph

We will use two statistical properties of a graph – degree distribution and
clustering coefficient. The degree of a given vertex $i$ is the number of
edges that connects it with other vertices, i.e.
$\deg(i)=\sum_{j\in\mathcal{V}}e_{ij}$. The clustering coefficient measures
the average fraction of pairs of neighbors of a vertex that are also neighbors
of each other. The precise definition is as follows.

Let us indicate by $\mathcal{G}_{i}=\\{j\in\mathcal{V}\mid e_{ij}=1\\}$ the
set of nearest neighbors of a vertex $i$. By setting
$l_{i}=\sum_{j\in\mathcal{V}}e_{ij}\left[\sum_{k\in\mathcal{G}_{i};\
j<k}e_{jk}\right],$ we define the local clustering coefficient as
$C(i)=\frac{l_{i}}{{|\mathcal{G}_{i}|\choose 2}}$, and the clustering
coefficient as the average over $\mathcal{V}$:
$C=\frac{1}{n}\sum_{i\in\mathcal{V}}C(i)$.

Let $\bar{k}$ be the average degree per vertex, i.e.
$\bar{k}=\frac{1}{n}\sum_{j\in\mathcal{V}}e_{ij}$. For random binomial graphs,
i.e. graphs with edges
$e_{ij}\,\,{\stackrel{{\scriptstyle\text{iid}}}{{\sim}}}\,\,\mathrm{Bernoulli}(p)$,
it is well known (Erdős and Rényi, 1960) that $C\approx\frac{\bar{k}}{n}$ and
$\deg(i)\,\,\sim\,\,\mathrm{Binomial}(n-1,p)$. A complex network is a graph,
for which $C\gg\frac{\bar{k}}{n}$ and
$p(\deg(i)=k)\propto\frac{1}{k^{\gamma}}$, where $\gamma$ is some constant
(Dorogovtsev, 2010). The latter property is referred to as scale-free (or
power-law) degree distribution.

We constructed $\sigma$SPMI-induced Graphs from the text8 and enwik9 datasets
using context windows of sizes 2 and 5 and ignoring words that appeared less
than 5 times, and computed their clustering coefficients (Table 2) as well as
degree distributions (Figure 3) using the NetworKit tool (Staudt et al.,
2016). NetworKit uses the algorithm of Schank and Wagner (2005) to compute the
clustering coefficient. As we see, the $\sigma$SPMI-induced graphs are complex
networks, and this brings us to the hyperbolic spaces.

##  4 Complex Networks and Hyperbolic Geometry

Complex networks are “dual” to hyperbolic spaces as was shown by Krioukov et
al. (2010). They showed that any complex network, as defined in Section 3, has
an effective hyperbolic geometry underneath. Apart from this, they also showed
that any hyperbolic geometry implies a complex network: they placed randomly
$n$ points (nodes) into a hyperbolic disk of radius $R$, and used
$p_{ij}:=\sigma\left(c[R-x_{ij}]\right)$ as connection probability for
connecting nodes $i$ and $j$, where $x_{ij}$ is the hyperbolic distance
between $i$ and $j$, and $c$ is a constant. An example of such random graph is
shown in Figure 5.

Figure 4: Rand. hyperbolic graph.

Figure 5: SPMI values distr’n (top) vs $R-X$.

Krioukov et al. (2010) showed that the resulting graph is a complex network.
They establish connections between the clustering coefficient $C$ and the
power-law exponent $\gamma$ of a complex network and the curvature of a
hyperbolic space.

Comparing the construction of Krioukov et al. (2010) to the way we generate a
random graph from the $\sigma$SPMI matrix, and taking into account that both
methods produce similar structures (complex networks), we conclude that the
distribution of the SPMI values should be similar to the distribution of
$R-x_{ij}$, i.e. $\text{PMI}_{ij}-\log k\sim R-x_{ij}$. To verify this claim
we compare the distribution of SPMI values with the p.d.f. of a random
variable $R-X$, where $X$ is a hyperbolic distance between two random points
on the hyperbolic disk (the exact form of this p.d.f. is given in the Appendix
0.A). $R$ was chosen according to the formula $R=2\ln[8n/(\pi\bar{k})]$
(Krioukov et al., 2010), where $\bar{k}$ is the average degree of the
$\sigma$SPMI-induced Graph. The results are shown in Figure 5. As we can see,
the two distributions are indeed similar and the main difference is in the
shift—distribution of $R-X$ is shifted to the left compared to the
distribution of the SPMI values. This allows us reinterpreting the pointwise
mutual information as the negative of hyperbolic distance (up to scaling and
shifting).

##  5 Conclusion

It is noteworthy that the seemingly fragmented sections of scientific
knowledge can be closely interconnected. In this paper, we have established a
chain of connections between word embeddings and hyperbolic geometry, and the
key link in this chain is the Squashed Shifted PMI matrix. Claiming that
hyperbolicity underlies word vectors is not novel (Nickel and Kiela, 2017;
Tifrea et al., 2018). However, this work is the first attempt to justify the
connection between hyperbolic geometry and the word embeddings. In the course
of our work, we discovered novel objects—Nonsigmoid SGNS and Squashed Shifted
PMI matrix—which can be investigated separately in the future.

## Acknowledgements

This work is supported by the Nazarbayev University faculty-development
competitive research grants program, grant number 240919FD3921. The authors
would like to thank Zhuldyzzhan Sagimbayev for conducting preliminary
experiments for this work, and anonymous reviewers for their feedback.

## References

  * Abadi et al. (2016) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale machine learning.  In: Proceedings of OSDI. pp. 265–283 (2016)
  * Allen et al. (2019) Allen, C., Balazevic, I., Hospedales, T.: What the vec? towards probabilistically grounded embeddings.  In: Advances in Neural Information Processing Systems. pp. 7465–7475 (2019)
  * Allen and Hospedales (2019) Allen, C., Hospedales, T.: Analogies explained: Towards understanding word embeddings.  In: International Conference on Machine Learning. pp. 223–231 (2019)
  * Arora et al. (2016) Arora, S., Li, Y., Liang, Y., Ma, T., Risteski, A.: A latent variable model approach to pmi-based word embeddings.  Transactions of the Association for Computational Linguistics 4, 385–399 (2016)
  * Assylbekov and Takhanov (2019) Assylbekov, Z., Takhanov, R.: Context vectors are reflections of word vectors in half the dimensions.  Journal of Artificial Intelligence Research 66, 225–242 (2019)
  * Bruni et al. (2012) Bruni, E., Boleda, G., Baroni, M., Tran, N.K.: Distributional semantics in technicolor.  In: Proceedings of ACL. pp. 136–145. Association for Computational Linguistics (2012)
  * Cancho and Solé (2001) Cancho, R.F.I., Solé, R.V.: The small world of human language.  Proceedings of the Royal Society of London. Series B: Biological Sciences 268(1482), 2261–2265 (2001)
  * Cong and Liu (2014) Cong, J., Liu, H.: Approaching human language with complex networks.  Physics of life reviews 11(4), 598–618 (2014)
  * Devlin et al. (2019) Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding.  In: Proceedings of NAACL-HLT. pp. 4171–4186 (2019)
  * Dorogovtsev (2010) Dorogovtsev, S.: Lectures on Complex Networks.  Oxford University Press, Inc., USA (2010)
  * Eckart and Young (1936) Eckart, C., Young, G.: The approximation of one matrix by another of lower rank.  Psychometrika 1(3), 211–218 (1936)
  * Erdős and Rényi (1960) Erdős, P., Rényi, A.: On the evolution of random graphs.  Publ. Math. Inst. Hung. Acad. Sci 5(1), 17–60 (1960)
  * Ethayarajh et al. (2019) Ethayarajh, K., Duvenaud, D., Hirst, G.: Towards understanding linear word analogies.  In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 3253–3262 (2019)
  * Farkas et al. (2001) Farkas, I.J., Derényi, I., Barabási, A.L., Vicsek, T.: Spectra of “real-world” graphs: Beyond the semicircle law.  Physical Review E 64(2), 026704 (2001)
  * Finkelstein et al. (2002) Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., Ruppin, E.: Placing search in context: The concept revisited.  ACM Transactions on information systems 20(1), 116–131 (2002)
  * Gittens et al. (2017) Gittens, A., Achlioptas, D., Mahoney, M.W.: Skip-gram- zipf+ uniform= vector additivity.  In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 69–76 (2017)
  * Goh et al. (2001) Goh, K.I., Kahng, B., Kim, D.: Spectra and eigenvectors of scale-free networks.  Physical Review E 64(5), 051903 (2001)
  * Hashimoto et al. (2016) Hashimoto, T.B., Alvarez-Melis, D., Jaakkola, T.S.: Word embeddings as metric recovery in semantic spaces.  Transactions of the Association for Computational Linguistics 4, 273–286 (2016)
  * Krioukov et al. (2010) Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., Boguná, M.: Hyperbolic geometry of complex networks.  Physical Review E 82(3), 036106 (2010)
  * Levy and Goldberg (2014a) Levy, O., Goldberg, Y.: Linguistic regularities in sparse and explicit word representations.  In: Proceedings of CoNLL. pp. 171–180 (2014a)
  * Levy and Goldberg (2014b) Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization.  In: Proceedings of NeurIPS. pp. 2177–2185 (2014b)
  * Levy et al. (2015) Levy, O., Goldberg, Y., Dagan, I.: Improving distributional similarity with lessons learned from word embeddings.  Transactions of the Association for Computational Linguistics 3, 211–225 (2015)
  * Luong et al. (2013) Luong, T., Socher, R., Manning, C.: Better word representations with recursive neural networks for morphology.  In: Proceedings of CoNLL. pp. 104–113 (2013)
  * McCann et al. (2017) McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: Contextualized word vectors.  In: Advances in Neural Information Processing Systems. pp. 6294–6305 (2017)
  * Mikolov et al. (2013a) Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781 (2013a)
  * Mikolov et al. (2013b) Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality.  In: Advances in neural information processing systems. pp. 3111–3119 (2013b)
  * Mikolov et al. (2013c) Mikolov, T., Yih, W.t., Zweig, G.: Linguistic regularities in continuous space word representations.  In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 746–751 (2013c)
  * Nickel and Kiela (2017) Nickel, M., Kiela, D.: Poincaré embeddings for learning hierarchical representations.  In: Advances in neural information processing systems. pp. 6338–6347 (2017)
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12, 2825–2830 (2011)
  * Pennington et al. (2014) Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation.  In: Proceedings of EMNLP. pp. 1532–1543 (2014)
  * Peters et al. (2018) Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations.  In: Proceedings of NAACL-HLT. pp. 2227–2237 (2018)
  * Radinsky et al. (2011) Radinsky, K., Agichtein, E., Gabrilovich, E., Markovitch, S.: A word at a time: computing word relatedness using temporal semantic analysis.  In: Proceedings of the 20th international conference on World wide web. pp. 337–346. ACM (2011)
  * Řehůřek and Sojka (2010) Řehůřek, R., Sojka, P.: Software Framework for Topic Modelling with Large Corpora.  In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. pp. 45–50. ELRA, Valletta, Malta (May 2010), http://is.muni.cz/publication/884893/en
  * Reif et al. (2019) Reif, E., Yuan, A., Wattenberg, M., Viegas, F.B., Coenen, A., Pearce, A., Kim, B.: Visualizing and measuring the geometry of bert.  In: Advances in Neural Information Processing Systems. pp. 8592–8600 (2019)
  * Schank and Wagner (2005) Schank, T., Wagner, D.: Approximating clustering coefficient and transitivity.  Journal of Graph Algorithms and Applications 9(2), 265–275 (2005)
  * Staudt et al. (2016) Staudt, C.L., Sazonovs, A., Meyerhenke, H.: Networkit: A tool suite for large-scale complex network analysis.  Network Science 4(4), 508–530 (2016)
  * Tian et al. (2017) Tian, R., Okazaki, N., Inui, K.: The mechanism of additive composition.  Machine Learning 106(7), 1083–1130 (2017)
  * Tifrea et al. (2018) Tifrea, A., Bécigneul, G., Ganea, O.E.: Poincaré glove: Hyperbolic word embeddings.  arXiv preprint arXiv:1810.06546 (2018)
  * Wigner (1955) Wigner, E.P.: Characteristic vectors of bordered matrices with infinite dimensions.  Annals of Mathematics pp. 548–564 (1955)
  * Wigner (1958) Wigner, E.P.: On the distribution of the roots of certain symmetric matrices.  Annals of Mathematics pp. 325–327 (1958)
  * Zobnin and Elistratova (2019) Zobnin, A., Elistratova, E.: Learning word embeddings without context vectors.  In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019). pp. 244–249 (2019)

##  Appendix 0.A Auxiliary Results

###### Proposition 1

Let $X$ be a distance between two points that were randomly uniformly placed
in the hyperbolic disk of radius $R$. The probability distribution function of
$X$ is given by

$f_{X}(x)=\int_{0}^{R}\int_{0}^{R}\frac{\sinh(x)}{\pi\sqrt{1-A(r_{1},r_{2},x)}\sinh(r_{1})\sinh(r_{2})}\rho(r_{1})\rho(r_{2})dr_{1}dr_{2},$
(7)

where
$A(r_{1},r_{2},x)=\frac{\cosh(r_{1})\cosh(r_{2})-\cosh(x)}{\sinh(r_{1})\sinh(r_{2})}$,
and $\rho(r)=\frac{\sinh r}{\cosh R-1}$.

The proof is by direct calculation and is omitted due to page limit.


# Digital Twins for Autonomous Driving: A Comprehensive Implementation and
Demonstration

Kui Wang1, Tao Yu1, Zongdian Li1, Kei Sakaguchi1, Omar Hashash2, and Walid
Saad2  1Department of Electrical and Electronic Engineering, Tokyo Institute
of Technology, Tokyo, Japan.  2Wireless@VT, Bradley Department of Electrical
and Computer Engineering, Virginia Tech, Arlington, VA, USA.  Emails: 1{kuiw,
yutao, lizd<EMAIL_ADDRESS>2{omarnh<EMAIL_ADDRESS>
###### Abstract

The concept of a digital twin (DT) plays a pivotal role in the ongoing digital
transformation and has achieved significant strides for various wireless
applications in recent years. In particular, the field of autonomous vehicles
is a domain that is ripe for exploiting the concept of DT. Nevertheless, there
are many challenges that include holistic consideration and integration of
hardware, software, communication methods, and collaboration of edge/cloud
computing. In this paper, an end-to-end (E2E) real-world smart mobility DT is
designed and implemented for the purpose of autonomous driving. The proposed
system utilizes roadside units (RSUs) and edge computing to capture real-world
traffic information, which is then processed in the cloud to create a DT
model. This DT model is then exploited to enable route planning services for
the autonomous vehicle to avoid heavy traffic. Real-world experimental results
show that the system reliability can reach $99.53\%$ while achieving a latency
that is $3.36\%$ below the 3GPP recommended value of $100$ ms for autonomous
driving. These results clearly validate the effectiveness of the system
according to practical 3GPP standards for sensor and state map sharing (SSMS)
and information sharing.

###### Index Terms:

Digital twin, autonomous driving, roadside unit, route planning,
implementation.

##  I Introduction

The realm of the Internet of Things (IoT) is rapidly expanding beyond the
conventional frameworks, branching out to encapsulate a broader spectrum of
digital connectivity [1]. This emerging paradigm facilitates intelligent data
sharing among entities, thereby advancing our ability to monitor, control, and
optimize the physical world. As a prominent manifestation of IoT, digital twin
(DT) technology has begun to establish its foothold across different
industries [2]. The potential of DTs lies in their capabilities to construct
comprehensive digital representations, thereby enabling bidirectional
interaction between the physical space and cyber space [3]. Henceforth, DTs
empower real-time decision-making and enhance efficiency, productivity, and
adaptability in a multitude of applications [4].

With the rapid advancement in autonomous driving technologies, the convergence
of DTs and autonomous vehicles presents an exciting new frontier. The
implementation of DTs within the autonomous driving ecosystem creates vast
possibilities for improved safety, efficiency, and robustness of the system
[5]. As such, leveraging real-time synchronization between the digital and
physical worlds, autonomous vehicles can access global information to ”see
more and see further”, enhancing their awareness and understanding of the
traffic environment [6]. Furthermore, a DT allows for effective testing and
validation of autonomous driving algorithms in a controllable and scalable
virtual environment, accelerating the development and deployment of autonomous
driving [7].

Hence, there has been a growing interest in introducing the concept of DTs in
smart mobility [8, 9, 10, 11, 12, 13, 14, 15, 16]. The mainstream roles of
vehicular DTs are discussed and summarized in [8]. The authors in [9] proposed
a DT-enabled scheduling architecture to help multiple vehicle users fulfill
their personalized requirements for path planning. Nevertheless, this prior
work [9] was only conducted and evaluated based on simulation results, without
realizing path planning for vehicles in a real-world setting. Recent works in
[10, 11, 12] investigated the generation of highly accurate DT models by using
various onboard and infrastructure sensors with edge computing to collect and
process real-time information from the physical space. However, the decision
feedback from cyber space to entities in physical space is not properly
captured. In [13, 14, 15], the implementation of DT systems was done in order
to provide situational awareness and cooperative driving services for _human_
drivers, which fails to consider the _autonomy_ of vehicles. To the best of
our knowledge, there is a lack of the holistic integration of _DTs with
autonomous vehicles_ , as well as an absence of any practical end-to-end (E2E)
system designs and implementation of such a system. Hence, in our previous
work [16], we proposed a system architecture for the smart mobility DT and
conducted a preliminary proof-of-concept (PoC) test. Our PoC provides a route
planning service that allows a vehicle to avoid overcrowded traffic. However,
this fundamental work in [16] only considered offline collection of sensor
data and overlooked the route planning decisions and actions that were to be
executed at the vehicle level.

In contrast to these prior works, the main contribution of this paper is a
real-time smart mobility DT framework for autonomous driving, as well as a
comprehensive real-world implementation of this system. The proposed system
requires a realistic virtual representation of real-life traffic in autonomous
driving scenarios as well as the establishment of an effective and stable
request-feedback loop for the autonomous vehicle to obtain cloud-based
decisions. Thus, from a system design perspective, the challenge lies in
holistic consideration and integration of hardware, software, communication
methods, and collaboration of edge/cloud computing. In addition, it is
important to implement and test the whole system in real-time traffic
conditions, which allows us to evaluate and validate the performance of the
designed DT. Hence, the system testing is carried out in different scenarios
to provide a thorough analysis on the impact of various factors. In summary,
our key contributions include:

Figure 1: System architecture of smart mobility DT.

  * •

We design a smart mobility DT that reflects objects of interest in the context
of road driving scenarios, e.g., roads, infrastructures, vehicles, and
pedestrians.

  * •

We implement the DT platform for autonomous driving by employing the roadside
units (RSUs) and the autonomous vehicle equipped with sensors, communication
modules, and edge computing capabilities.

  * •

We design a cloud-based route planning service for the autonomous vehicle
based on real-time traffic information.

  * •

Our results show the realization of the DT modeling and demonstrate the
effectiveness of the route planning service, whereby the planned routes are
successfully transmitted to the vehicle for real-time execution.

  * •

We validate the system performance in terms of reliability and latency based
on 3GPP standards for sensor and state map sharing (SSMS) and information
sharing.

The rest of the paper is organized as follows. Section II provides a concise
overview of the system design. Details about hardware deployment, software
installation, and route planning are shown in Section III. Section IV
discusses the experimental evaluation. Section V concludes the paper.

##  II Design for Smart Mobility DT

We first develop and propose our DT system architecture that is tailored for
implementation in a specific but generalizable environment. The proposed smart
mobility DT system is essentially based on our work in [16, 17]. This proposed
system enables real-time DT modeling and real-time feedback services to the
autonomous vehicle.

Fig. 1 gives the system architecture, consisting of autonomous vehicles and
RSUs in the physical world and a DT system maintained within the cloud server.
We exploit the computing capabilities of the edge and cloud by distributing
various functions and services on these two planes. The edge, referring to
RSUs and autonomous vehicles, provides better real-time operation in order to
address delay-sensitive tasks such as environmental perception and vehicle
maneuvering. Meanwhile, the cloud offers robust data processing capabilities
with higher latency, making it suitable for handling large-scale and
computation-intensive tasks that need global information, such as DT modeling
and deriving cloud-based services. Thus, our system design incorporates edge
and cloud computing to enhance efficiency and safety for autonomous driving.

In the physical world, diverse sensors on the vehicle and RSUs capture real-
time data from their surroundings. Then the vehicle can achieve a high level
of autonomy, incorporating functional modules such as navigation, perception,
localization, motion planning, and control. Similarly, RSUs use sensors for
object detection, tracking, and prediction, upload the perception data to the
cloud, and share it with nearby vehicles to enhance their safety during the
driving process.

The smart mobility DT within the cloud plane is dynamically updated using
real-time perception data from the physical system, thereby accurately
mirroring the current state of the real-world traffic. After receiving the
real-time perception data from RSU edges, the cloud server synchronizes
incoming edge channels and locates detected objects based on their relative
coordinates to different edge sensors. By integrating with static HD maps and
3D models, we can realize the generation of a virtual representation from the
physical space to cyber space. By utilizing the smart mobility DT, we can
facilitate the provision of a variety of derivative services for autonomous
vehicles that include route planning, risk prediction, and congestion
alerting. Since these services are predicated on the comprehensive global
information from the DT, they can assist vehicles in evading high-risk and
congested areas outside onboard sensing horizons and improving their overall
commuting efficiency.

It is important to note that the communication within this system is based on
a heterogeneous vehicle-to-everything (V2X) network [18], which includes
vehicle-to-infrastructure (V2I), vehicle-to-cloud (V2C), and infrastructure-
to-cloud (I2C) communications. Considering the large communication distance
between vehicles and the cloud, the cellular network is applied for V2C
communication. Depending on the application scenario, RSUs may send data of
different levels to the cloud server and nearby vehicles, such as raw data or
processed data, which have distinct requirements on communication coverage and
bandwidth. Hence, we use wired networks for I2C communication and employ both
dedicated short-range communications (DSRC) and millimeter wave (mmWave)
technologies for V2I.

TABLE I: Communication requirements and performances Function | Requirements | Message contents | Methods | Performances
---|---|---|---|---
V2C |  | Speed: $\geq 80$ Mbps
---
Coverage: $\geq 1$ km
| Upload: vehicle position and motion states.
---
Download: cloud-based decisions
WiMAX |  |  $\geq 120$ Mbps
---
$\geq 50$ km
V2I |  | Speed: $\geq 1$ Mbps
---
Coverage: $\geq 50$ m
| Cooperative perception (processed data,
---
e.g., detection and tracking result)
Wi-Fi router |  |  $\geq 10$ Mbps
---
$\geq 200$ m
| Speed: $\geq 1$ Gbps
---
Coverage: $\geq 20$ m
| Cooperative perception (raw sensor data, e.g.
---
LiDAR point cloud and camera images)
WiGig |  |  $\geq 1$ Gbps
---
$\geq 120$ m
I2C | Speed: $\geq 1$ Gbps | Environmental perception (both raw data and processed data) | Ethernet |  $\geq 1$ Gbps

##  III Implementation for Smart Mobility DT

In this section, we will discuss the implementation of our system in the Tokyo
Tech smart mobility field, as well as the route planning workflow involved.

###  III-A Hardware Deployment

Our testing field in Tokyo Tech Ookayama campus, as well as our hardware
devices, are shown in Fig. 2. In the field setting, we have three RSUs and one
autonomous vehicle. The three RSUs are located at the corners of a square road
section within the campus. Each RSU is equipped with an 80-layer LiDAR. These
sensors collect raw data from the physical world, capturing information about
dynamic objects in the traffic environment. Additionally, each RSU is also
outfitted with edge computing devices, i.e., NVIDIA Jetson AGX Orin, where we
launch functional modules for functions like object detection, tracking, and
prediction, among others. The communication between RSUs and the cloud server
relies on the campus Ethernet network. To cater to the requirements of
different application scenarios, we upload varying levels of perception data,
such as raw point cloud data, detection and tracking results to cloud.

Figure 2: Hardware components and Tokyo Tech. smart mobility field.

The autonomous vehicle is equipped with a 32-layer LiDAR sensor positioned on
the rooftop to sense its surroundings. A dedicated Autoware PC is utilized for
processing the LiDAR data and performing environmental perception,
localization, motion planning, and motion control. The control signals are
then transmitted to the vehicle’s onboard unit (OBU) to enable autonomous
driving. Additionally, the vehicle is equipped with communication modules,
including worldwide interoperability for microwave access (WiMAX) for V2C
communication, a Wi-Fi router as a replacement for dedicated short-range
communications (DSRC), and a WiGig antenna for millimeter wave (mmWave)
communication. The selection of communication equipment is based on specific
requirements and functions. Table I shows the required communication speeds
and coverage in our experiment field, as well as the performance of selected
communication methods. These devices effectively facilitate information
sharing among edge and cloud planes.

The cloud server works as the central hub for data aggregation and storage,
global information processing, and providing feedback services to the
autonomous vehicle. Therefore, the performance requirements for the cloud
server are very high, which encompass scalability, processing power, and
reliability to efficiently and securely handle large volumes of real-time
data. Hence, we opted for a computer equipped with a GeForce RTX 5000 GPU,
ensuring ample memory and storage capacity to meet the demanding computational
requirements.

###  III-B Software Installation

The software in our platform is centered around two key open-source software,
i.e., Autoware [19] and Robot Operating System (ROS).

In the RSU edges and cloud server, we deploy Autoware Universe and version 2
of the ROS (ROS2). Autoware Universe serves as the primary software framework
for our system. It provides comprehensive functionality for the development of
autonomous driving systems, making it well-suited for DT modeling. Autoware
facilitates the detection and tracking of traffic participants within the RSU
sensor range. The detection module applies the CenterPoint framework[20],
which can detect, identify, and visualize 3D objects from the LiDAR point
clouds in real-time. Then, a tracking algorithm, named Multi-object Tracker
[21], is responsible for assigning the detected objects with IDs and
estimating their velocities. Moreover, Autoware Universe is built on ROS2,
which enables easy deployment and seamless communication among distributed
computers. In our case, both the cloud server and the RSU edge computing
devices are connected to the campus Ethernet network and assigned IP addresses
within a local area network (LAN). Hence, we can subscribe to all the ROS2
topics on the cloud server. The utilization of ROS2 greatly facilitates the
modeling of the DT and further enhances the interoperability and scalability
of the DT system. After acquiring object tracking data from three RSUs, the
cloud server utilizes objects’ positions to align them with the corresponding
road segments, which facilitates traffic monitoring and the computation of
congestion level on each road section.

The Autoware PC onboard the autonomous vehicle is installed with an Ubuntu
16.04 operating system, enabling the implementation of autonomous driving
functionalities using Autoware AI based on ROS. Autoware AI is responsible for
driving the vehicle’s LiDAR sensor and utilizes the normal distributions
transform (NDT) algorithm for LiDAR scan matching with the 3D point cloud map,
which enables real-time localization with centimeter-level accuracy. The route
from origin to destination is in the form of a prerecorded waypoint file,
consisted of a set of route points with position, velocity, and orientation
information. The automated vehicle can sequentially track these waypoints to
follow any given route. To perform local motion planning and motion control, a
velocity planner adjusts the velocity plan based on the waypoints to
decelerate or accelerate in response to nearby objects and road
characteristics, including stop lines and traffic lights. Then, a pure pursuit
algorithm is employed to generate coordinated sets of velocities and steering
angles that guide the ego vehicle to follow the target waypoints.

###  III-C Route Planning Scenario and Workflow

Figure 3: Sequence diagram of route planning service.

Based on the hardware and software implementation, we design the workflow of
the route planning service, as shown in Fig. 3. There are two main sequences
in the workflow, i.e., DT modeling and route requisition. The DT modeling
sequence is a loop fragment for real-time edge sensing, detection, tracking,
and data acquisition on the cloud plane. Three RSUs run the LiDAR driver,
object detection, and tracking. The detection and tracking results are
transmitted from the RSUs to the cloud. Then the cloud computing will help the
ego vehicle choose the optimal path to circumvent traffic congestion and to
improve its efficiency, using the shortest-paths algorithm like Dijkstra’s
algorithm and A* algorithm. Since the specific route planning is not the
primary focus of this paper, it is not elaborated upon in detail here.

The route requisition sequence is an alternative combination fragment, obeying
the “if then else” logic. During the vehicle’s movement from the starting
point to the destination, it utilizes LiDAR-based localization and the road
vector map to calculate the distance between itself and upcoming
intersections. When the vehicle approaches an intersection, within a
predetermined threshold $D_{\textrm{thre}}$, it initiates a path request to
the cloud server via the hypertext transfer protocol (HTTP). The HTTP server
running on the cloud receives the request and sends the uniform resource
locator (URL) of the optimal path to the vehicle. Then the autonomous vehicle
downloads the route and tracks it for vehicle navigation.

##  IV Experimental Demonstration

In this section, we will show the outdoor test results and the evaluation from
the aspects of functionality and reliability.

###  IV-A Field Experiment Results

Figure 4: Real-time DT modeling: (a) Scenario #A: no congestion in the road
network, (b) Scenario #B: congestion occurs on the straight route

Figure 5: Autonomous driving operation results: (a) Result #A: driving on
default route, (b) Result #B: driving on alternative route

The real-time DT modeling results on the cloud plane are illustrated in Fig.
4. As discussed in Section III.A, we upload diverse types of perception data
in different application scenarios. In this case, we only transmit object-
tracking data to alleviate communication loads. The tracking result has a
significantly lower data rate, ranging from $50$ to $300$ Kb/s, depending on
the number of detected objects, compared to the raw point cloud data
(approximately $46$ Mb/s). Within the LiDAR detection range of the RSU,
various entities are identified and visualized using different colored
bounding boxes, indicating their categories, positions, and approximate
shapes. In the figure, pink boxes represent pedestrians, while blue boxes
represent vehicles. The tracking frequency can achieve a rate of $30$ Hz. The
cloud server performs fusion and processing of these recognized objects to
choose the best global route for the vehicle. We also mark the origin and
destination of the autonomous vehicle in this figure, i.e., the vehicle drives
from the bottom-left road segment to the bottom-right road segment. In
scenario #A, where no congestion is observed on any road segment, the cloud
server selects the straight route indicated by the blue dashed line. In
scenario #B, congestion occurs within the straight-line segment due to the
presence of pedestrians and vehicles. In such cases, the route planner selects
the path represented by the yellow dashed line, which has a longer distance
but effectively helps the autonomous vehicle avoid heavily congested road
segments, enabling an efficient and smooth driving process.

The results of the autonomous driving operation, from real-world images and
Autoware visualization tool Rviz, are shown in Fig. 5. During the
initialization phase, the vehicle follows the default straight route in the
road network, so in both results, the vehicle initially tracks the straight
trajectory. When the ego vehicle arrives at the predetermined threshold
distance $D_{\textrm{thre}}$ from the intersection area, it will send a route
planning request to the cloud. Result #A corresponds to scenario #A, where the
vehicle downloads the same default path from the cloud and successfully
traverses the intersections in a straight manner. Result #B corresponds to
scenario #B, where congestion is observed on the straight-line segment. In
this case, the vehicle downloads the alternative path from the cloud. The
vehicle makes a left turn and proceeds along the alternative route.

In our experiment setting, the threshold distance from the ego vehicle to the
intersection is determined with the consideration of safety and comfort
issues. The ego vehicle should have enough time to process the planned route
before reaching the intersection. Given the presence of pedestrians crossing
at intersections, it is necessary for the vehicle to comfortably decelerate
and stop before reaching the intersection. In our system, the ego vehicle’s
free-flow speed $v_{\textrm{f}}$ is set to $15$ km/h. Braking can be
classified into emergency and comfortable types. In the presence of unexpected
objects, the majority of emergency braking deceleration is more than $4.5$
m/s2 [22]. According to Institute of Transportation Engineers (ITE)
recommendations, a comfortable deceleration rate $a_{\textrm{comfy}}$ should
be less than $10$ ft/s2, equivalent to $3.048$ m/s2 [23]. Thus the threshold
distance is determined as $D_{\textrm{thre}}=0.039\times
v_{\textrm{f}}^{2}/a_{\textrm{comfy}}=2.9$ m.

###  IV-B Evaluation

The proposed platform is evaluated from _reliability and latency_. In DT
modeling progress, congestion caused by heavy traffic flow is not a typical
concern in the campus environment. Instead, congestion may arise during
specific periods due to high pedestrian volumes or gatherings. In our
demonstration, the frequency of changes in the path selection does not exceed
0.92 changes per minute. Hence, real-time route planning decisions exhibit low
sensitivity to time variations. On the other hand, reliability, i.e., the
packet delivery rate (PDR) in communication from RSUs to the cloud server, is
critical as severe packet loss of tracking results could lead to incorrect
planning decisions. If only the tracking results are uploaded from the RSUs,
the PDR approaches $100\%$. When simultaneously uploading other levels of
data, such as raw LiDAR point clouds, the PDR is around $99.53\%$, which meets
the reliability requirement for SSMS, i.e., higher than $95\%$ [24].

Figure 6: Latency in route planning service.

Fig. 6 shows the latency in the route request process, including delays caused
by localization and position check $T_{\textrm{local}}$, path execution
$T_{\textrm{exe}}$ on Autoware PC, and communication $T_{\textrm{comm}}$ with
cloud server. The maximum total latency $T_{\textrm{max}}$ is $96.61$ ms,
which is $3.39\%$ below the threshold of the max E2E latency requirements for
information sharing (less than $100$ ms) proposed by 3GPP [24]. Assuming the
autonomous vehicle maintains the constant free-flow speed during this period,
it is still approximately $2.4$ m away from the intersection. This distance
provides sufficient room for the vehicle to smoothly maneuver through the
intersection without deceleration or, in the presence of pedestrians, to
appropriately decelerate ($3.62$ m/s2 on average, smaller than the emergency
braking deceleration $4.5$ m/s2 [22]) and stop in front of the intersection.

##  V Conclusion

In this paper, we have designed a smart mobility DT for autonomous driving.
Our system utilizes RSUs to capture real-world traffic information, which is
processed in the cloud to create a real-time DT model, enabling route planning
services for the autonomous vehicle. We have implemented and demonstrated the
proposed system in the Tokyo Tech smart mobility field. Test results show that
the PDR of DT modeling can reach $99.53\%$ and the latency of route planning
service is smaller than $96.61$ ms, which validates the effectiveness of the
system in terms of reliability and latency. The latency performance currently
leaves room for improvement, indicating that future work should focus on
optimizing the communication system.

## Acknowledgment

This work was supported by the MEXT Doctoral Program for World-leading
Innovative & Smart Education for Super Smart Society (WISE-SSS), JST SPRING
(Grant Number JPMJSP2106), NICT-JUNO (Promotion of Commissioned Research for
Advanced Communications and Broadcasting Research and Development (#22404)),
and by the U.S. National Science Foundation under JUNO3 Grant CNS-2210254.

## References

  * [1] S. Nižetić, P. Šolić, D. L.-d.-I. Gonzalez-De, and L. Patrono, “Internet of Things (IoT): Opportunities, issues and challenges towards a smart and sustainable future,” _Journal of Cleaner Production_ , vol. 274, p. 122877, 2020.
  * [2] Z. Wang, R. Gupta, K. Han, H. Wang, A. Ganlath, N. Ammar, and P. Tiwari, Prashant, “Mobility digital twin: Concept, architecture, case study, and future challenges,” _IEEE Internet of Things Journal_ , vol. 9, no. 18, pp. 17 452–17 467, 2022.
  * [3] O. Hashash, C. Chaccour, W. Saad, T. Yu, K. Sakaguchi, and M. Debbah, “The seven worlds and experiences of the wireless metaverse: Challenges and opportunities,” _arXiv preprint arXiv:2304.10282_ , 2023.
  * [4] H. X. Nguyen, R. Trestian, D. To, and M. Tatipamula, “Digital twin for 5G and beyond,” _IEEE Communications Magazine_ , vol. 59, no. 2, pp. 10–15, 2021\.
  * [5] S. Almeaibed, S. Al-Rubaye, A. Tsourdos, and N. P. Avdelidis, “Digital twin analysis to promote safety and security in autonomous vehicles,” _IEEE Communications Standards Magazine_ , vol. 5, no. 1, pp. 40–46, 2021.
  * [6] O. Hashash, C. Chaccour, and W. Saad, “Edge continual learning for dynamic digital twins over wireless networks,” in _Proc. of IEEE 23rd International Workshop on Signal Processing Advances in Wireless Communication (SPAWC)_ , Oulu, Finland, Jul. 2022, pp. 1–5.
  * [7] O. Veledar, V. Damjanovic-Behrendt, and G. Macher, “Digital twins for dependability improvement of autonomous driving,” in _Proc. of European Conference on software process improvement_ , Cham, Switzerlad: Springer, 2019, pp. 415–426.
  * [8] C. Schwarz and Z. Wang, “The role of digital twins in connected and automated vehicles,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 14, no. 6, pp. 41–51, 2022.
  * [9] Y. Hui, Q. Wang, N. Cheng, R. Chen, X. Xiao, and T. H. Luan, “Time or reward: Digital-twin enabled personalized vehicle path planning,” in _Proc. of IEEE Global Communications Conference (GLOBECOM)_ , Madrid, Spain, Feb. 2021, pp. 1–6.
  * [10] L. Kloeker, A. Kloeker, F. Thomsen, A. Erraji, and L. Eckstein, “How to build a highly accurate digital twin–intelligent infrastructure in the corridor for new mobility-accord,” in _Proc. of 30th Aachen Colloquium Sustainable Mobility 2021_ , vol. 5, Oct. 2021, pp. 651–680.
  * [11] V. Tihanyi, A. and Rövid, V. Remeli, Z. Vincze, M. Csonthó, Z. Pethő, M. Szalai, B. Varga, A. Khalil, and Z. Szalay, “Towards cooperative perception services for ITS: Digital twin in the automotive edge cloud,” _Energies_ , vol. 14, no. 18, p. 5930, 2021.
  * [12] R. Sell, E. Malayjerdi, M. Malayjerdi, and B. C. Baykara, “Safety toolkit for automated vehicle shuttle-practical implementation of digital twin,” in _Proc. of International Conference on Connected Vehicle and Expo (ICCVE)_ , Lakeland, FL, USA, Mar. 2022, pp. 1–6.
  * [13] H. Xu, A. Berres, S. B. Yoginath, H. Sorensen, P. J. Nugent, J. Severino, S.A. Tennille, A. Moore, W. Jones, and J. Sanyal, “Smart mobility in the cloud: Enabling real-time situational awareness and cyber-physical control through a digital twin for traffic,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 24, no. 3, pp. 3145–3156, 2023.
  * [14] X. Liao, Z. Wang, X. Zhao, K. Han, P. Tiwari, M. J. Barth, and G. Wu, “Cooperative ramp merging design and field implementation: A digital twin approach based on vehicle-to-cloud communication,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 5, pp. 4490–4500, 2021.
  * [15] J. Dong, Q. Xu, J. Wang, C. Yang, M. Cai, C. Chen, Y. Liu, J. Wang, and K. Li, “Mixed cloud control testbed: Validating vehicle-road-cloud integration via mixed digital twin,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 4, pp. 2723–2736, 2023.
  * [16] K. Wang, T. Yu, Z. Li, and K. Sakaguchi, “Cloud and edge computing empowered mobility digital twin for automated driving: Design and proof-of-concept,” in _Proc. of IEEE 97th Vehicular Technology Conference (VTC2023-Spring)_ , Florence, Italy, Jun. 2023.
  * [17] T. Yu, Z. Li, K. Sakaguchi, O. Hashash, W. Saad, and M. Debbah, “Internet of federated digital twins (iofdt): Connecting twins beyond borders for society 5.0,” _arXiv preprint arXiv:2312.06432_ , 2023.
  * [18] Z. Li, K. Wang, T. Yu, and K. Sakaguchi, “Het-SDVN: SDN-based radio resource management of heterogeneous V2X for cooperative perception,” _IEEE Access_ , vol. 11, pp. 76 255–76 268, 2023.
  * [19] Autoware universe documentation. [Online]. Available: https://autowarefoundation.github.io/autoware.universe/main/.
  * [20] T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in _Proc. of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 11 784–11 793.
  * [21] Autoware multi-object tracker. [Online]. Available: https://autowarefoundation.github.io/autoware.universe/main/perception/multi_object_tracker/.
  * [22] P. Greibe, “Braking distance, friction and behaviour,” _Trafitec, Scion-DTU_ , 2007.
  * [23] Institute of Transportation Engineers. Technical Council Committee 4A-16, “Determining vehicle change intervals: A proposed recommended practice,” _Institute of Transportation Engineers_ , 1985.
  * [24] 3GPP, “Study on enhancement of 3GPP support for 5G V2X services (Release 16),” _Tech. Rep. 22.886 V16.2.0_ , 2018.


# BWSNet: Automatic Perceptual Assessment Of Audio Signals

This paper introduces BWSNet, a model that can be trained from raw human
judgements obtained through a Best-Worst scaling (BWS) experiment. It maps
sound samples into an embedded space that represents the perception of a
studied attribute. To this end, we propose a set of cost functions and
constraints, interpreting trial-wise ordinal relations as distance comparisons
in a metric learning task. We tested our proposal on data from two BWS studies
investigating the perception of speech social attitudes and timbral qualities.
For both datasets, our results show that the structure of the latent space is
faithful to human judgements.

Index Terms—  Automatic Perceptual Assessment, Best-Worst Scaling, Metric
Learning, Social Attitudes, Timbre

##  1 Introduction

Access to a perceptual representation of data typically involves an experiment
in which stimuli are judged by human participants according to a specific
criterion. In particular, several methods to subjectively assess audio stimuli
have been proposed, including pairwise comparison, MUSHRA and rating scales,
the latter being widely praised in experimental psychology. With the advent of
synthesis algorithms and the subsequent need to finely assess their outputs,
such rating scales are widely employed to complement, or even replace,
objective measures such as MCD or RMSE that are not necessarily correlated
with human perception [1]. Typically, the Mean Opinion Score (MOS) has been
used to assess speech synthesis models’ performance by asking participants to
rate the quality or naturalness of output samples [2] and their similarity to
a reference, according to specific speech attributes, e.g., speaker identity
[2, 3], emotion [4], or attitude [5]. Although favoured, scale-based methods
present potential biases [6, 7], making the choice of evaluation method an
important issue in experimental psychology.

Recently, attention was given to Best-Worst Scaling (BWS) [8], another method
in which participants are presented with trials of $N$ (e.g. $N$=4,5) items
and asked to judge which ones are the best and worst according to a studied
attribute. Once the experiment completed, each item receives a score -
computed using more or less elaborate techniques on the basis of raw
judgements - representing how it is perceived in regards with this attribute.
BWS has proven to yield more reliable results than rating scales to gather
perceptual scores [9, 10, 11] and has been found effective for various audio-
related tasks such as the perception of speech emotions [4] and attitudes [12]
as well as timbral qualities [13].

Unfortunately, all these subjective assessment methods always require a large
number of human ratings to be reliable, making them costly and time-consuming.
Facing this challenge, an entire field of research has emerged with the aim of
automating perceptual evaluation. Thus, several methods have been proposed to
learn a regression model that predicts MOS scores such as AutoMOS [14],
Quality-Net [15], and MOSNet [16, 17, 18]. To our knowledge and despite its
aforementioned benefits, there is no existing method for predicting BWS
judgements.

In this paper, we introduce BWSNet, a model for automatic perceptual
assessment based on BWS data. In contrast with existing MOSNet approaches, we
do not seek to predict perceptual scores in a regression task. Indeed, the
determination of BWS scores entails a dimensional reduction of the perceptual
space underlying raw judgements, which involves a potential loss of
information. To predict these judgements, that infer ordinal relations between
items in each trial, we design a metric learning task in which these relations
(see Fig. 1) are interpreted as distances comparisons. BWSNet is thus trained
to learn a function that maps sound samples into a latent space in which the
distance represents samples dissimilarity with respect to the studied
attribute.

We present two contributions, firstly BWSNet, which consists of a set of cost
functions adapted to the ordinal and relative nature of BWS judgements.
Secondly, we apply it in the specific context of two studies previously led by
the authors, investigating the perception of speech social attitudes [12] and
timbral concepts [13].

##  2 BWSNet

###  2.1 Problem Positioning

A BWS trial is a tuple of $N$ sounds $t^{a}=\\{x^{1},...,x^{N}\\}$ set for
judgement. For each trial, a judgment consists of choosing the best and worst
items in the tuple, the $N-2$ remaining items are then considered neutrals. We
denote $\mathcal{T}^{a}$ the set containing all the trials considered for the
BWS experiment that investigates $a$.

Each sample is renamed with respect to the trial it lies in and the judgement
it has been assigned. The best, worst and neutrals of trial $t^{a}$ can be
indexed as $t_{b}^{a}$, $t_{w}^{a}$ and $t_{n_{i}}^{a}$ with
$i\in\\{1,N-2\\}$, respectively. We denote $\succ_{a}$, the relation such that
$x\succ_{a}y$ is equivalent to ”$x$ is more perceived as $a$ than $y$”. Then,
$t^{a}$ judgement is represented by $2(N-1)$ relations - ordinal in nature -
between sample triplets, as shown on the right part of Fig. 1.

Fig. 1: A BWS trial $t^{a}\in\mathcal{T}^{a}$ of $N=4$ sounds judged with
respect to the attribute $a$ (left) and the derived relations (right).

###  2.2 Concept of a BWSNet

Fig. 2: The Mel-Spectrograms of the $N=4$ samples related to a BWS trial
$t^{a}$ investigating the attribute $a$ are passed to BWSNet. The model yields
BWS embeddings of which relative position is changed over training.

Our goal is to design a model able to learn perceptual representations
underlying human BWS judgements. We use mel-spectrograms as sound
representations, as it has proven to account for many perceptually relevant
aspects of speech. As shown in Fig. 2, for any sample $x$, the BWSNet takes a
mel-spectrogram $\mathbf{X}$ as input and produce a BWS embedding
$\mathbf{h}^{x}$. Relative positioning of embeddings, in the latent space they
lie in, must be true to BWS judgments. To achieve this, ordinal relations
between sound samples in BWS trials are translated into distances comparisons
of the corresponding embeddings. Thus, introducing a distance
$\lVert.\lVert:\mathbb{R}^{d}\rightarrow\mathbb{R}$, the relations for a trial
$t^{a}$ displayed in Fig. 1 can be translated into the following inequalities
for $i\in\\{1,...,N-2\\}$:

$\displaystyle\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}$
$\displaystyle\rVert\geq\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert$
(1) $\displaystyle\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}$
$\displaystyle\rVert\geq\lVert\mathbf{h}^{t_{w}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert$
(2)

###  2.3 Losses and Optimisation

To train the model to match trials relations within its latent space, we
designed our own training criterion, a cost function inspired by the metric
learning literature and notably the triplet loss proposed in [19].

The relations to match are ordinal and, like in triplet loss, each one
involves three items in a trial as formalized in inequalities 1 and 2.
However, they only prevail within a given trial, making the problem even more
complex. To avoid mode collapse, i.e., the model turning all samples into one
single point in the latent space, aforementioned inequalities must be
considered strict. We thus introduce a positive margin $\alpha$ and define the
RC loss $\mathcal{L}_{t^{a}}$ for any trial $t^{a}\in\mathcal{T}^{a}$ as:

$\small\begin{split}\mathcal{L}_{rc}^{t^{a}}=\frac{1}{n_{v}^{t^{a}}}\sum_{i=1}^{N-2}&\max{(\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert-\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}\rVert+\alpha,0)}+\\\
\frac{1}{n_{v}^{t^{a}}}\sum_{i=1}^{N-2}&\max{(\lVert\mathbf{h}^{t_{w}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert-\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}\rVert+\alpha,0)}\end{split}$
(3)

where $n_{v}^{t^{a}}$ is the number of relations that remain to be fulfilled
within trial $t^{a}$.

Perceptual differences represented in BWS trials can be more or less
substantial, quantifying them requires dynamic margins rather than fixed ones.
We thus introduce a network $\mathcal{M}$ dedicated to margin learning. It
takes two parameters as arguments, a mean $\mu$ and an amplitude $\delta$
value, such that learnt margin lies between $\mu-\delta$ and $\mu+\delta$.
With all trials’ embeddings in the batch as input, it produces $N-2$ distinct
margins $\\{\alpha_{b,n_{i}},\alpha_{w,n_{i}}\\}_{i\in\\{1,N-2\\}}$ related to
each trial relation. This impacts the RC loss formulated in equation 3 as it
takes learnt margins as additional input. Thus, for a given trial $t^{a}$, the
Dynamic margin (Dm)-RC loss can be expressed as follows:

$\small\begin{split}\mathcal{L}_{drc}^{t^{a}}=\frac{1}{n_{v}^{t^{a}}}\sum_{i=1}^{N-2}&\max{(\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert-\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}\rVert+\alpha_{b,n_{i}},0)}+\\\
\frac{1}{n_{v}^{t^{a}}}\sum_{i=1}^{N-2}&\max{(\lVert\mathbf{h}^{t_{w}^{a}}-\mathbf{h}^{t_{n_{i}}^{a}}\rVert-\lVert\mathbf{h}^{t_{b}^{a}}-\mathbf{h}^{t_{w}^{a}}\rVert+\alpha_{w,n_{i}},0)}\end{split}$
(4)

We assume a Gaussian distribution of margins and propose an additional
constraint to penalize our model’s tendency to learn low-value margins. This
constraint is formalized through a function $\gamma$ that takes the learned
margins as an argument and outputs a scalar loss. Various functions can be
tested, resulting in different learnt margin distributions. For a trial
$t^{a}$, the Dynamic Margin Constraint (DMC) can be formulated as:

$\mathcal{L}_{dmc}^{t^{a}}=\sum_{i=1}^{N-2}\gamma(\alpha_{b,n_{i}}-\mu)+\gamma(\alpha_{w,n_{i}}-\mu)$
(5)

Decrease in the Dm-RC loss does not guarantee an increase in the number of
fulfilled relations. Since margins can decrease overall without affecting
order relationships. To avoid this, we have added a final loss directly
derived from the Dm-RC loss which accounts, for a given trial $t^{a}$, to the
number of unfulfilled relations within the trial divided by the number $N$ of
elements in the trial.

$\displaystyle\mathcal{L}_{fr}^{t^{a}}=\frac{n_{v}^{t^{a}}}{N}$ (6)

We introduce the scalars $\lambda_{dmc}$, $\lambda_{fr}\geq 0$ and define the
global BWSNet loss for a trial $t^{a}$ as:

$\displaystyle\mathcal{L}^{t^{a}}=\mathcal{L}_{dmrc}^{t^{a}}+\lambda_{dmc}\mathcal{L}_{dmc}^{t^{a}}+\lambda_{fr}\mathcal{L}_{fr}^{t^{a}}$
(7)

##  3 Experiments

We used data from two previous BWS studies to train and test our model. We
provide short descriptions of these in the following.

###  3.1 Data for Experiment

####  3.1.1 Study I: Speech Social Attitude

We carried out this first study on Att-HACK [20], a 30-hour speech dataset
composed of 9 male and 11 female actors portraying four different social
attitudes - friendliness, dominance, distance and seduction \- over 100
sentences in French. 96 participants (48 women) were recruited to evaluate the
perception of its related sounds for each a priori attitude, i.e. the ones
already produced with aim of conveying this specific attitude. Model trainings
were thus conducted on the four attitudes separately. For the sake of
feasibility, only a fourth of the dataset has been assessed with each sound
being evaluated eight times.

####  3.1.2 Study II: Instrumental Timbre

With the purpose of giving acoustic portraits of timbre concepts, sound expert
participants (N=16, sound engineers and conductors) evaluated a dataset of
musical instruments (N=520) on four well-known timbral concepts, namely
brightness, warmth, roundness and roughness. The sounds showcase a great
diversity of instruments, playing techniques, dynamics and registers. Unlike
Study I, each sample is evaluated with respect to all concepts.

###  3.2 Implementation Details

####  3.2.1 Input Pipeline

Mel-Spectrograms are obtained through computing Short-Term-Fourier-Transform
(STFT) with FFT $2048$, hop $200$, window $800$ and $80$ mel filters. Our
custom RC loss takes two tensors as input, in addition to the BWS embeddings:
a tensor made of the corresponding trial names - which ensures the loss is
computed trial-wise even if several trials lie in a single batch - and a
tensor made of each element’s corresponding judgement labels ($b$, $w$, $n$).

####  3.2.2 Architecture Design

The BWSNet architecture was inspired by a promising version of ACRNN [21]
initially proposed for speech emotion recognition. We previously assessed the
role of its various components for speech attitude recognition in [12] using
the same attitudinal dataset [20]. Based on this study, the best configuration
used $2$ convolutional layers with $64$ filters, temporal and feature kernel
of sizes $5$ and $3$ respectively, and an $8$ head attention mechanism with
$512$ dimensions. Here, we reuse the same architecture for our task - which is
close to the one for which it has proved optimal - without further
experimenting. Finer adjustments to the architecture, e.g. latent space
dimension $d$, could improve performance depending on the input data,
especially for timbral concepts. Here we choose $d=32$ and focus on the
relative importance of the aforementioned cost functions and constraints.

####  3.2.3 Training Procedure

In both experiments, we split data into three groups, first we selected $10\%$
of the samples and dropped out any trial they were involved in. Then, we split
the remaining data trial-wise into training ($80\%$) and validation ($20\%$)
trial sets. We fed our model with batches of size $80$ with as many data of
each concept for Experiment II and used ADAM optimizer with $0.0001$ as
learning rate. We found the euclidean distance for $\lVert.\lVert$,
$\gamma:x\rightarrow ReLU(-x)$ for DMC and $\mu=\gamma=1$ as margin learning
module $\mathcal{M}$ parameters to yield the best results.

####  3.2.4 Evaluation Criteria

To evaluate the performance of BWSNet, we used two metrics, reflecting the
arrangement of speech samples in the latent space at two levels: FR and WAT \-
respectively the percentages of fulfilled relations and well-arranged trials
within the set - both computed on relations from dropped trial set, involving
at least one unseen sample. We chose lowest FR value as early stopping
criterion.

##  4 Results & Discussion

We carried out an ablation study to demonstrate the influence of each cost
function and constraint on the model’s performance. Then, we sought to explore
the latent spaces yielded by BWSNet for both studies.

###  4.1 Ablation Study

Table 1 displays FR and WAT mean and standard deviation values for various
BWSNet configurations evaluated on unseen samples across both BWS studies’
sets of attributes. First, as expected, the fixed-margin configuration (A-f)
did not generalize well, as some trials’ best and worst could be either very
distant in the latent space or rather close. Then, when considering learnt
margin configurations, it appeared that with no constraint on margins (A-l)
the latent space was collapsing into one single point, turning distances
between any pair of points null. Adding such a constraint (A-l-d) tended to
help the model converge and fulfill just under one relations in two on
attitudes and slightly more than one relations in two for timbre, which is
insufficient to claim that our model accurately predicts BWS judgements.
However, we found that BWSNet alternatively uses two strategies to lower Dm-RC
loss: it could seek to fulfill more relations within trials otherwise it could
reduce the margins. The addition of FR loss as training criterion (A-l-d-fr)
appeared to prevent the model from engaging in the second strategy and led to
improvements by $27.7\%$ and $27.5\%$ in both FR and WAT respectively for
attitudes and by $4.7\%$ in FR for timbre. The model performed better for
attitudinal than for timbral data, which may be due to the choice of neural
architecture that we deliberately adapted to the former.

|  |  | Study I: Attitudes | Study II: Instrument Timbre
---|---|---|---|---
Model | $\lambda_{dmc}$ | $\lambda_{fr}$ |  FR (%) |  WAT (%) |  FR (%) |  WAT (%)
A-f | - | - | $21.4\pm 8.5$ | $5.8\pm 3.6$ | $37.5\pm 2.5$ | $4.7\pm 0.5$
A-l | 0 | 0 | $1.0\pm 0.4$ | $0.2\pm 0.0$ | $26.1\pm 2.1$ | $\textbf{26.0}\pm\textbf{1.3}$
A-l-d | 1 | 0 | $40.1\pm 18.7$ | $22.4\pm 17.1$ | $51.6\pm 3.5$ | $19.9\pm 2.6$
A-l-d-fr | 1 | 1 | $\textbf{67.7}\pm\textbf{4.5}$ | $\textbf{49.9}\pm\textbf{8.9}$ | $\textbf{56.3}\pm\textbf{2.4}$ | $23.9\pm 1.2$

Table 1: FR & WAT mean and standard deviation values for various BWSNet
configurations evaluated on unseen samples across both BWS studies’ sets of
attributes.

###  4.2 Exploring BWSNet’s Latent Space

To further analyse our results, we investigated the relation between item
scores, obtained with the original scoring algorithm [10] used in both
studies, with our latent space’s dimensions. Figure 3 shows the BWSNet latent
spaces corresponding to each attitude (left) and timbre qualities (right) with
the original scores characterised with color and size respectively.

By assessing attitudes independently in the BWS experiment, we obtained four
distinct latent spaces with different degrees of polarisation with regard to
BWS scores. For instance, for friendliness and seductiveness \- that were
found the most consensual [12] \- high and low score samples are located in
opposite ends of the space. Furthermore, regarding the latent spaces obtained
for distance and dominance, BWS scores may be an inaccurate depiction of their
multi-dimensional nature.

As for timbral concepts, the original study sought to uncover their mutual
interactions which leads to have all samples lying in the same space. To
report on these interactions, we associated each sound with its most salient
attribute (i.e., the highest scored concept for each sound). We observe
similar interactions to those in the original study, e.g. warmth and roundness
are blended together in the timbral space while showing strong opposition to
brighthness.

Fig. 3: BWSNet’s latent space UMAP vizualization for social attitudes (left)
and timbral qualities (right). Each point is a sample whose BWS score is
represented by its colour (left) and size (right) respectively.

##  5 Conclusion

This paper presents BWSNet, a model dedicated to automatic audio perceptual
assessment based on BWS judgements. Distances between learnt sample embeddings
in the resulting latent space represent their perceptual similarity. By
fulfilling almost 70% of relations involving an unseen sample for attitudinal
speech data, BWSNet provides a rather accurate estimation of how this sample
is perceived based on its distance with previously judged samples in the
latent space. Its performance on timbral data (56% fulfilled relations) also
indicates potential for application to a manifold of judgement tasks. These
results, obtained on two very different datasets, mark a first step towards
automating the BWS perceptual assessment. Furthermore, as the analysis of its
latent space suggests, BWSNet could be a relevant tool for representing and
understanding human perception. Provided that participants agree to a certain
extent, using more BWS judgements for training would likely yield a more
comprehensive map of the perceptual space for a chosen sound attribute.

## References

  * [1] Lucas Theis, Aäron van den Oord, and Matthias Bethge,  “A note on the evaluation of generative models,” 2016.
  * [2] Mingyang Zhang, Xin Wang, Fuming Fang, Haizhou Li, and Junichi Yamagishi,  “Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet,” 2019.
  * [3] Hirokazu Kameoka, Kou Tanaka, Damian Kwaśny, Takuhiro Kaneko, and Nobukatsu Hojo,  “Convs2s-vc: Fully convolutional sequence-to-sequence voice conversion,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1849–1863, 2020.
  * [4] Kun Zhou, Berrak Sisman, Rajib Rana, B. W. Schuller, and Haizhou Li,  “Speech synthesis with mixed emotions,” 2022.
  * [5] Clément Le Moine, Nicolas Obin, and Axel Roebel,  “Towards end-to-end F0 voice conversion based on Dual-GAN with convolutional wavelet kernels,”  in EUSIPCO, Dublin (virtual ), Ireland, 2021.
  * [6] Hans Baumgartner and Jan-Benedict EM Steenkamp,  “Response styles in marketing research: A cross-national investigation,”  Journal of marketing research, vol. 38, no. 2, pp. 143–156, 2001\.
  * [7] Howard Schuman and Stanley Presser,  Questions and answers in attitude surveys: Experiments on question form, wording, and context,  Sage, 1996.
  * [8] Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley,  Best-Worst Scaling: Theory, Methods and Applications,  Cambridge University Press, 2015.
  * [9] Svetlana Kiritchenko and Saif Mohammad,  “Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation,”  in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, July 2017, pp. 465–470, Association for Computational Linguistics.
  * [10] Geoff Hollis,  “Scoring best-worst data in unbalanced many-item designs, with applications to crowdsourcing semantic judgments,”  Behavior research methods, vol. 50, no. 2, pp. 711–729, 2018.
  * [11] Victor Rosi, Aliette Ravillion, Olivier Houix, and Patrick Susini,  “Best-worst scaling, an alternative method to assess perceptual sound qualities,”  The Journal of the Acoustical Society of America, vol. 2, pp. 064404, 06 2022.
  * [12] Clément Le Moine Veillon,  Neural Conversion of Social Attitudes in Speech Signals,  Ph.D. thesis, 2023,  Thèse de doctorat dirigée par Roebel, Axel et Obin, Nicolas Informatique Sorbonne université 2023.
  * [13] Victor Rosi, Pablo Arias Sarah, Olivier Houix, Nicolas Misdariis, and Patrick Susini,  “Shared mental representations underlie metaphorical sound concepts,”  Scientific Reports, vol. 13, no. 1, pp. 5180, 2023.
  * [14] Brian Patton, Yannis Agiomyrgiannakis, Michael Terry, Kevin Wilson, Rif A. Saurous, and D. Sculley,  “Automos: Learning a non-intrusive assessor of naturalness-of-speech,”  in NIPS 2016 End-to-end Learning for Speech and Audio Processing Workshop, 2016.
  * [15] Szu-Wei Fu, Yu Tsao, Hsin-Te Hwang, and Hsin-min Wang,  “Quality-net: An end-to-end non-intrusive speech quality assessment model based on BLSTM,”  09 2018, pp. 1873–1877.
  * [16] Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang,  “MOSNet: Deep learning-based objective assessment for voice conversion,”  in Interspeech 2019. sep 2019, ISCA.
  * [17] Yeunju Choi, Youngmoon Jung, and Hoirin Kim,  “Deep MOS predictor for synthetic speech using cluster-based modeling,”  in Interspeech 2020. oct 2020, ISCA.
  * [18] Yeunju Choi, Youngmoon Jung, and Hoirin Kim,  “Neural MOS prediction for synthesized speech using multi-task learning with spoofing detection and spoofing type classification,”  in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 462–469.
  * [19] Elad Hoffer and Nir Ailon,  “Deep metric learning using triplet network,”  in Similarity-Based Pattern Recognition, Aasa Feragen, Marcello Pelillo, and Marco Loog, Eds., Cham, 2015, pp. 84–92, Springer International Publishing.
  * [20] Clément Le Moine and Nicolas Obin,  “Att-HACK: An Expressive Speech Database with Social Attitudes,”  in Speech Prosody, Tokyo, Japan, May 2020.
  * [21] Yuanchao Li, Tianyu Zhao, and Tatsuya Kawahara,  “Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning,”  09 2019, pp. 2803–2807.