# Joint Multi-scale Cross-lingual Speaking Style Transfer with Bidirectional
Attention Mechanism for Automatic Dubbing

Jingbei Li, Sipan Li, Ping Chen, Luwen Zhang, Yi Meng, Zhiyong Wu, Helen Meng,
Qiao Tian, Yuping Wang, Yuxuan Wang  Manuscript received April 19, 2021;
revised August 16, 2021. This work is supported by National Natural Science
Foundation of China (62076144), Shenzhen Science and Technology Innovation
Committee (WDZC20220816140515001), AMiner.Shenzhen SciBrain fund and Shenzhen
Key Laboratory of next generation interactive media innovative technology
(ZDSYS20210623092001004). (Corresponding author: Zhiyong Wu)Jingbei Li, Sipan
Li, Ping Chen, Luwen Zhang, Yi Meng and Zhiyong Wu are with Tsinghua-CUHK
Joint Research Center for Media Sciences, Technologies and Systems, Shenzhen
International Graduate School, Tsinghua University, Shenzhen, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>zlw20@
mails.tsinghua.edu.cn<EMAIL_ADDRESS><EMAIL_ADDRESS>
cn). Jingbei Li and Sipan Li equally contributes to this work.Helen Meng is
with Department of Systems Engineering and Engineering Management, The Chinese
University of Hong Kong, Hong Kong SAR, China (e-mail:
hmmeng@se.cuhk.edu.hk).Qiao Tian, Yuping Wang and Yuxuan Wang are with
ByteDance, Shanghai, China. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>wangyuxuan.11@bytedance.com).

###### Abstract

Automatic dubbing, which generates a corresponding version of the input speech
in another language, could be widely utilized in many real-world scenarios
such as video and game localization. In addition to synthesizing the
translated scripts, automatic dubbing needs to further transfer the speaking
style in the original language to the dubbed speeches to give audiences the
impression that the characters are speaking in their native tongue. However,
state-of-the-art automatic dubbing systems only model the transfer on duration
and speaking rate, neglecting the other aspects in speaking style such as
emotion, intonation and emphasis which are also crucial to fully perform the
characters and speech understanding. In this paper, we propose a joint multi-
scale cross-lingual speaking style transfer framework to simultaneously model
the bidirectional speaking style transfer between languages at both global
(i.e. utterance level) and local (i.e. word level) scales. The global and
local speaking styles in each language are extracted and utilized to predicted
the global and local speaking styles in the other language with an encoder-
decoder framework for each direction and a shared bidirectional attention
mechanism for both directions. A multi-scale speaking style enhanced
FastSpeech 2 is then utilized to synthesize the predicted the global and local
speaking styles to speech for each language. Experiment results demonstrate
the effectiveness of our proposed framework, which outperforms a baseline with
only duration transfer in both objective and subjective evaluations.

###### Index Terms:

Automatic dubbing, cross-lingual speaking style transfer, multi-scale speaking
style transfer, bidirectional attention mechanism, text-to-speech synthesis.

Figure 1: Cross-lingual speaking style transfer between two languages at
multiple scales.

##  I Introduction

Text-to-speech (TTS) synthesis which aims to generate natural speech from
given text [1], has been widely researched and utilized in many real-world
application. With the development of speech processing and neural network
technologies, state-of-the-art TTS systems [2, 3, 4, 5, 6] have achieved end-
to-end speech synthesis by merging different components in conventional TTS
systems into one trainable framework to synthesize high quality speeches while
requiring less feature engineering. Such end-to-end neural network based TTS
models have been successfully implemented in various languages [7], and widely
used as the backbones for downstream researches such as emotional speech
synthesis [8, 9], speaking style transfer [10, 11, 12], cross-lingual TTS
system [13, 14, 15] and automatic dubbing [16].

Among these downstream researches of TTS, automatic dubbing [16] which
converts the speeches in films, televisions or games to their corresponding
versions in a different language rendition, is a potential research area and
could be utilized in film and game localization to simplify the laborious
human dubbing process. Differing from standard TTS synthesis [2, 3, 4, 5, 6]
which directly synthesize speech from the given text, automatic dubbing need
to further consider the speaking styles in speech and transfer them to the
dubbed speech during synthesis. In film or game production, actors are using
various speaking styles to present the personalities, emotions and intentions
of the different characters, including the speaking styles at the global (i.e.
utterance level) and local scale (i.e. word level). Global speaking style [10]
is used to control the global emotion and intention for each utterance, while
local speaking style [17] is used to control the emphasis and intonation for
each word to highlight the details in each utterance. Both kinds of speaking
style are crucial for speech understanding and should be transferred in
automatic dubbing to give the audiences the impression that the characters are
speaking in their native tongue.

State-of-the-art automatic dubbing systems [18] have successfully transferred
the duration and speaking rate in local speaking style to dubbed speech. These
researches employ algorithms based on attention mechanism [16] or hidden
Markov model (HMM) [19, 20] to match the prosodic phrases in the two
languages. Then several prosodic phrase alignment methods [16] are proposed to
adjust the duration [18] and speaking rate for each prosodic phrase [21] to
match the duration of the corresponding prosodic phrase in the speech in the
original language.

However, these existing automatic dubbing systems are only applicable to few
simple dubbing scenarios and may cause pauses and duration in other scenarios.
These approaches require that the same numbers of prosodic phrases in the
source and the dubbing language. For those complicated scenarios that the
number of prosodic phrases are often different such as dubbing between Chinese
and English, these approaches may synthesize improper pauses in the dubbed
speeches. Furthermore, these approaches can only adjust the duration and
speaking rate of each prosodic phrase according to the prosodic phrases in the
other language at the same position, ignoring if they have same semantic
meanings. Such design could produce improper duration and speaking rate when
the word orders are significantly different in the original and dubbed
languages.

Moreover, existing approaches in automatic dubbing have insufficient modeling
on the speaking style transfer. At the local scale, the modeling on the other
aspects of local speaking styles rather than duration and speaking rate, such
as pitch, energy and emotion which are also crucial to present the character
is largely missing. In addition, the modeling on the speaking style transfer
at the global scale is also rarely considered. The transfer of global speaking
style is vital for utterances that are sense-for-sense translation in which
the meanings of each words in the two languages are complete different. Such
insufficient modeling on the transfer of both local and global speaking styles
will synthesis speeches without those particular emotions and intentions and
seriously decrease the expressiveness of the dubbed speech.

In this paper, to address these issues and improve the speaking style transfer
in automatic dubbing, we propose a multi-scale cross-lingual speaking style
transfer framework which jointly model and optimizes the speaking style
transfer for both dubbing directions between two languages with multi-task
learning. The global and local speaking styles of the speeches in each
language are extracted and utilized to predict the corresponding global and
local speaking styles in the other language. The predicted global and local
speaking styles for the other language are then synthesized to speech by a
corresponding multi-scale speaking style enhanced FastSpeech 2 (hereafter,
MST-FastSpeech 2) [5, 17] for each language. The source code of our work will
be published at GitHub when the paper is accepted.

Experiments results demonstrated the effectiveness of our proposed model. The
proposed approach outperform the baseline approach with only duration transfer
in both objective and subjective evaluations, with MOS increased by $0.2$ and
$0.013$, and the preference rate exceeded by $39.64\%$ and $3.58\%$ on the two
dubbing directions.

The rest of the paper is organized as follows. We introduce related work in
Section II. We then conduct data observation in Section III. In Section IV, we
introduce the details of proposed approach. Experimental results and ablation
studies are presented in Section V. We discuss our work in Section VI.
Finally, Section VII concludes the paper.

Figure 2: Architecture of proposed joint multi-scale cross-lingual speaking
style transfer framework.

##  II Related work

###  II-A Text-to-speech synthesis

Text-to-speech synthesis is still a challenging task despite decades of
investigation [1]. Conventional TTS systems usually employ complicated
pipelines[22], including linguistic feature generation, duration prediction,
acoustic feature prediction and vocoder[23, 24]. These components complete the
respective subtasks in the conventional TTS pipelines, but are developed and
trained in separate, suffering from errors compounding in steps [25, 2].

With the development of deep learning, end-to-end TTS techniques are proposed
to integrate the components in conventional TTS systems [2, 3] into one
trainable framework. Such end-to-end TTS systems, such as Tacotron [2, 3] and
FastSpeech [5], are benefiting from joint training to alleviate the errors
compounding and provide more robust synthesized speeches with less feature
engineering. These end-to-end TTS systems are powered by the encoder-decoder
structure [26] with attention mechanism [27], using encoder to produce
linguistic encodings and decoder to directly generate raw spectrogram.
Attention mechanism such as location-sensitive attention in [28] is applied
with decoder to increase both performance and interpretability by learning the
alignment between the text and spectrogram. Then neural vocoder is utilized to
generate waveform from the spectrogram to further improve the overall natural
ness and quality of the synthesized speech [3].

The improvement of end-to-end TTS systems reveals a growing opportunity for a
number of applications, such as audiobook narration, news readers, and
conversational assistants, which also brings the challenge to end-to-end TTS
systems to improve the controllablity of choosing appropriate speaking style
for different application scenarios. Global style token [10] is firstly
proposed to represent the attributes that related to the speaking style among
the whole utterance such as emotion and intention as a single vector, which
have been widely utilized in robust emotion and global speaking style transfer
[10]. An adversarial branch [29] is further proposed to disentangle the
speaker information from global style tokens for cross-speaker global speaking
style transfer [10]. At the word level, local style token [17] is proposed to
represent the attributes related to the speaking style of each word such as
emphasis, rhythm and intonation with the help of a neural network based force
aligner [30].

###  II-B Automatic dubbing

Dubbing is a complex process that converts speeches in films or games into
another language while preserving the speaking styles in the original
language, which can be regarded as a special kind of speech-to-speech
translation[31, 32, 33]. Traditionally, such dubbing process are all done by
professional teams and voice talents. However, with such amount of manpower
spent, some dubs are still controversial among the audiences due to the
unsatisfied speaking style performance in dubbed speeches.

With the development of TTS synthesis, neural network based automatic dubbing
is of possibility to be developed on the top of end-to-end TTS synthesis
technologies to simplify and standardize such dubbing process in film and game
production. Several automatic dubbing systems have been proposed to transfer
the duration and speaking rate to the dubbed speech. A prosodic alignment
based automatic dubbing system is firstly proposed [16] to transfer the
duration of each prosodic phrase. This approach employs a prosodic alignment
algorithm to extract the mapping between the prosodic phrases in two languages
from the attention weights extracted from a pretrained machine translation
model. Then it adjusts the duration of each prosodic phrase to be same to the
duration of its mapped prosodic phrase in the original language. However, such
design could generate too long or too short duration when the numbers of
phonemes are largely different between the mapped prosodic phrases in two
languages.

An hidden Markov model (HMM) [34] based prosodic alignment algorithm [35] is
further proposed to segment the dubbed speech into prosodic phrases that match
the pauses in the original speech by maximize the probability of pause
positions in the dubbed speech. An improvement [21] to this HMM based prosodic
alignment algorithm is proposed to also transfer the speaking rate. An
additional phoneme level duration predictor is proposed [18] to further
improve the predicted duration and speaking rate in both prosodic alignment
and the following TTS system. But these approaches ignore the semantic meaning
of each word when segment the prosodic phrases. So it’s very likely that the
original and corresponding segmented prosodic phrases have completely
different semantic meanings. In this case, forcibly transfer the duration and
speaking rate in the original language to an unrelated prosodic phrase in the
dubbed language may seriously lowering the naturalness of the synthesized
speech.

Moreover, the existing automatic dubbing systems have insufficient modeling on
the speaking style transfer between two languages. Speaking styles are how
people speak an utterance and each word in it to express their emotions and
intentions which should be also transferred in automatic dubbing.
Traditionally, speaking style could be only represented as the combination of
many features defined with prior knowledge. With the development of neural
network technology, global style token [10] is firstly proposed to represent
all those attributes that related to the speaking style among the whole
utterance such as emotion and intention as a single vector. Such global style
tokens extracted from well-trained models have been widely utilized in many
researches [10] for robust emotion and global speaking style transfer. An
adversarial branch [29] is further proposed to disentangle the speaker
information from global style tokens for cross-speaker global speaking style
transfer [10]. Local style token [17] is further proposed to represent the
attributes related to the local speaking style such as emphasis, rhythm and
intonation as a vector for each word with the help of a neural network based
force aligner [30].

###  II-C Contributions

Compared with the previous works, the contributions of our work include: (1)
to our best knowledge, this is the first work that introduces speaking style
transfer to automatic dubbing, in which we advocate the modeling of speaking
style transfer at both global and local scales to significantly improve the
expressiveness in automatic dubbing; (2) this is also the first work that
employs multi-task learning to jointly optimize the modeling of cross-lingual
speaking style transfer for both dubbing directions between two languages; (3)
we propose an effective approach to infer the speaking styles at both global
and local scales for automatic dubbing.

##  III Data observation

We first conduct subjective observation to analyze the speaking style transfer
among the real-world professional dubs. We randomly collect 25 pairs of
parallel utterances in the English and Chinese dubs of the game Borderlands 3
for the observation, which is a well-known game with excellent and highly
acclaimed dubs in many languages. 25 listeners are invited to listen to the
English and Chinese speeches of each pair of these collected utterances and
answer the following questions: (1) Is there a special global speaking style
in each utterance which differs to the normal speaking style in declarative
sentences? (2) Are the global speaking styles of this pair of utterances in
the two languages same, similar, irrelevant or opposite? (3) Is there any
special local speaking style in the words of each utterance which differs to
the normal speaking style in the words of declarative sentences? (4) If a word
has a word in the other language which have same or similar meaning and local
speaking style to this word, we call that this word can be matched to the
corresponding word in the other language. Then for each language, about how
much words can be matched to the words in another language?

The first two question aim to demonstrate the necessity and possibility of
modeling the global speaking style transfer in automatic dubbing. The
aggregated responses from the listeners (by averaging) indicate that $76.13\%$
and $78.06\%$ of the dubbed utterances in English and Chinese have global
speaking styles different to the speaking style in common declarative
sentences. This shows that various global speaking styles are commonly used in
dubbing which should be considered and transferred to other languages.
Besides, the aggregated responses further indicate that in $96.76\%$ of the
pairs of the parallel utterances in these two languages, their global speaking
styles are same or similar to the global speaking style to each other. This
demonstrate the possibility of modeling the global speaking style transfer in
automatic dubbing.

On the other side, the last two question are designed to demonstrate the
necessity and possibility of modeling the speaking style transfer at local
scale. The aggregated responses to the third question show that $83.12\%$ and
$83.68\%$ of the dubbed utterances in English and Chinese have words with
local speaking styles different to the speaking style in common declarative
sentences, which reflects the importance of modeling the local speaking style
transfer for automatic dubbing. And in the final question, the aggregated
responses show that $86.7\%$ of the words in the English dubs can be matched
to the words in Chinese, which means that they have similar semantic
information and local speaking styles. And for the other direction, $87.09\%$
of the words in the Chinese dubs can also be matched to the words in English.
This reveals that the similarity on the semantic meanings is related to the
similarity of local speaking styles between parallel utterances in different
languages, and shows the possibility of modeling local speaking style transfer
across languages.

The above observation demonstrates the need to consider the both global and
local speaking styles in automatic dubbings (Question 1, 3), and the
possibility to model the speaking style transfer between the utterances at the
global scale (Question 2) and local scale (Question 4).

##  IV Methodology

Based on the phenomenon found in data observation, we propose a multi-scale
cross-lingual speaking style transfer framework for automatic dubbing, which
jointly learns the speaking style transfer between two languages at both
global and local scales.

###  IV-A Multi-scale feature extraction

To support the modeling of multi-scale cross-lingual speaking style transfer,
the textual features and multi-modal features that fuse the textual and
speaking style information in the two languages are extracted at both the
global and local scales.

####  IV-A1 Textural feature extraction

A pretrained BERT model is employed for each language to extract the sentence
level and word level BERT embeddings. The sentence level BERT embedding is
directly used as the textual feature at global scale for each language. The
word level BERT embeddings are converted by a local text encoder for each
language as the textual features at local scale, which consists of the pre-net
and CBHG networks in Tacotron [2]. The above process can also be formulated
as:

$\displaystyle g_{1}^{t}$ $\displaystyle=SBERT_{1}$ (1) $\displaystyle
g_{2}^{t}$ $\displaystyle=SBERT_{2}$ (2) $\displaystyle s_{1}^{t}$
$\displaystyle=E_{1}^{t}\left(BERT_{1}\right)$ (3) $\displaystyle s_{2}^{t}$
$\displaystyle=E_{2}^{t}\left(BERT_{2}\right)$ (4)

where $SBERT_{1}$ and $SBERT_{2}$, $BERT_{1}$ and $BERT_{2}$, $E_{1}^{t}$ and
$E_{2}^{t}$, $g_{1}^{t}$ and $g_{2}^{t}$, $s_{2}^{t}$ and $s_{1}^{t}$ are the
sentence level BERT embeddings, word level BERT embeddings, local text
encoder, global textual feature and local textual features, for each language.

####  IV-A2 Speaking style extraction

Global and local speaking style encoders [17] are then employed to extract the
speaking styles at global and local scales for each language. The global
speaking style encoder consists of 6 strided convolutional neural networks
(CNNs) composed of $3\times 3$ kernels with 32, 32, 64, 64, 128, 128 filters
respectively and $2\times 2$ stride, a 256 dimensional GRU layer and a 128
dimensional style attention layer. The mel-spectrograms of the input speech
are first processed by CNNs and GRU. The final state of GRU is further sent to
the style attention layer to derive the global speaking style vector $GST$ as
the weights for 10 automatically learnt base global speaking style embeddings.
The process can be formulated as:

$\displaystyle q$ $\displaystyle=final(GRU(CNN(speech)))$ (5) $\displaystyle
GST$ $\displaystyle=softmax\left(q^{T}GST^{table}\right)$ (6)

where $final$ returns the final state of GRU, $q$ is the query for the style
attention layer, $GST^{table}$ contains the 10 automatically learnt base
global speaking style embeddings.

The architecture of the local speaking style encoder is same to the global
speaking style encoder, except the stride is now $1\times 2$, and the GRU
layer now returns the output for each input frame. The outputs of GRU are then
summarized for each word in the utterance by multiplying them with the speech-
to-text attention weights extracted from a pretrained neural network based
forced aligner (NeuFA) [30]. NeuFA employs bidirectional attention mechanism
[30] to learn the bidirectional information mapping between a pair of text and
speech. The learnt attention weights at the speech-to-text direction could be
used to summarize the frame-level information for each word in the utterance,
deriving the local speaking style sequence $LST$. The process of local
reference encoder can be formulated as:

$\displaystyle q^{\prime}$ $\displaystyle=W_{ASR}^{T}GRU(CNN(speech))$ (7)
$\displaystyle LST$ $\displaystyle=softmax\left(q^{\prime
T}LST^{table}\right)$ (8)

where $W_{ASR}$ is the attention weights at the speech-to-text direction
obtained by NeuFA, $q^{\prime}$ is the query for the local style attention
layer, $LST^{table}$ contains 10 automatically learnt base local speaking
style embeddings.

####  IV-A3 Multi-modal feature fusion

To fuse the extracted textual and features, The sentence level BERT embedding
and the global speaking style of each language are then concatenated as the
multi-modal feature at the global scale. The word level BERT embeddings and
local speaking styles are concatenated and consumed by a local encoder for
each language to produce the multi-modal feature sequence at the local scale,
which has the same architecture to the local textual encoder. The above
process can also be formulated as:

$\displaystyle g_{1}^{m}$ $\displaystyle=\left[SBERT_{1};GST_{1}\right]$ (9)
$\displaystyle g_{2}^{m}$ $\displaystyle=\left[SBERT_{2};GST_{2}\right]$ (10)
$\displaystyle s_{1}^{m}$
$\displaystyle=E_{1}\left(\left[BERT_{1};LST_{1}\right]\right)$ (11)
$\displaystyle s_{2}^{m}$
$\displaystyle=E_{2}^{l}\left(\left[BERT_{2};LST_{2}\right]\right)$ (12)

where $GST_{1}$ and $GST_{2}$, $LST_{1}$ and $LST_{2}$, $E_{1}$ and $E_{2}$,
$g_{1}^{m}$ and $g_{2}^{m}$, $s_{1}^{m}$ and $s_{2}^{m}$ are the global
speaking style, local speaking styles, local encoder, global multi-modal
feature and local multi-modal features, for each language.

Figure 3: Joint cross-lingual speaking style transfer at global scale. Figure
4:  Bidirectional attention mechanism[30].  Figure 5: Joint cross-lingual
speaking style transfer at local scale.

###  IV-B Joint multi-scale cross-lingual speaking style transfer

####  IV-B1 Speaking style transfer at global scale

As shown in Figure 3, the global multi-modal feature of each language is
concatenated to the global textual feature of the other language and consumed
by a global linear projection to predict the global speaking style in the
other language:

$\displaystyle GST^{\prime}_{1}$
$\displaystyle=f_{2to1}^{g}([g_{2}^{m};g_{1}^{t}])$ (13) $\displaystyle
GST^{\prime}_{2}$ $\displaystyle=f_{1to2}^{g}([g_{1}^{m};g_{2}^{t}])$ (14)

where $f_{2to1}^{g}$ and $f_{1to2}^{g}$, $GST^{\prime}_{1}$ and
$GST^{\prime}_{2}$ are the global linear projection for each dubbing direction
and the predicted global speaking style for each language.

####  IV-B2 Speaking style transfer at local scale

As shown in Figure 5, an architecture with two encoder-decoder frameworks and
a shared bidirectional attention mechanism is employed to learn the speaking
style transfer between two languages at the local scale.

The bidirectional attention mechanism is extended from conventional attention
mechanism [36] to learn the bidirectional relations between two sets of key-
value pairs [30], which is also shown in Figure 4. With two sets of key-value
pairs are given as the inputs, each set of keys serves as the queries for the
other set of key-value pairs. Then a shared compatibility function are modeled
for both sets of keys to calculate the attention weights for each direction
which are further utilized to summarize the two sets of values as the outputs
for both directions. The bidirectional attention mechanism is formulated as:

$\displaystyle A$ $\displaystyle=f(K_{1},K_{2})$ (15) $\displaystyle W_{12}$
$\displaystyle=softmax(A)$ (16) $\displaystyle W_{21}$
$\displaystyle=softmax(A^{T})$ (17) $\displaystyle O_{1}$
$\displaystyle=W_{12}^{T}V_{1}$ (18) $\displaystyle O_{2}$
$\displaystyle=W_{21}^{T}V_{2}$ (19)

where $K_{1}\in R^{n_{1}\times d_{k1}}$, $V_{1}\in R^{n_{1}\times d_{v1}}$ and
$K_{2}\in R^{n_{2}\times d_{k2}}$, $V_{2}\in R^{n_{2}\times d_{v2}}$ are the
two sets of key-value pairs, $n_{1}$ and $n_{2}$ are the numbers of the key-
value pairs, $d_{k1}$, $d_{v1}$, $d_{k2}$ and $d_{v2}$ are feature dimensions,
$f$ is the compatibility function, $A\in R^{n_{1}\times n_{2}}$ is the shared
attention matrix, $W_{12}\in R^{n_{1}\times n_{2}}$ and $W_{21}\in
R^{n_{2}\times n_{1}}$ are the attention weights for two directions
respectively, $O_{1}\in R^{n_{2}\times d_{v1}}$ is the weighted sum of $V_{1}$
for each key in $K_{2}$ and $O_{2}\in R^{n_{1}\times d_{v2}}$ is the weighted
sum of $V_{2}$ for each key in $K_{1}$.

We employ the multiplicative form of bidirectional attention mechanism to
jointly model speaking style transfer at the local scale and summarize the
multi-modal features for both dubbing directions. Particularly, the local
textual features and the local multi-modal features of the two languages are
respectively used as the the two sets of keys and values, and the calculation
of the attention weights and outputs are formulated as:

$\displaystyle A_{1to2}$ $\displaystyle=f_{1}(s_{1}^{t})\times
f_{2}(s_{2}^{t})^{T}$ (20) $\displaystyle A_{2to1}$
$\displaystyle=f_{2}(s_{2}^{t})\times f_{1}(s_{1}^{t})^{T}=A_{1to2}^{T}$ (21)
$\displaystyle W_{1to2}$ $\displaystyle=softmax(A_{1to2})$ (22) $\displaystyle
W_{2to1}$ $\displaystyle=softmax(A_{2to1})$ (23) $\displaystyle O_{1to2}$
$\displaystyle=W_{1to2}^{T}s_{1}^{m}$ (24) $\displaystyle O_{2to1}$
$\displaystyle=W_{2to1}^{T}s_{2}^{m}$ (25)

where $f_{1}$ and $f_{2}$ are two linear projections, $A_{1to2}$ and
$A_{2to1}$ are the shared attention matrices, $W_{1to2}$ and $W_{2to1}$ are
the learnt attention weights for the two dubbing directions, $O_{1to2}$ and
$O_{2to1}$ are the summarized the multi-modal features of each language.

The summarize the multi-modal features of each language are then concatenated
with the local textual features of the other language, and consumed by a local
linear projection to the predicted local speaking styles of the other
language:

$\displaystyle LST^{\prime}_{1}$
$\displaystyle=f_{2to1}^{l}([O_{2to1};s_{1}^{t}])$ (26) $\displaystyle
LST^{\prime}_{2}$ $\displaystyle=f_{1to2}^{l}([O_{1to2};s_{2}^{t}])$ (27)

where $f_{2to1}^{l}$ and $f_{1to2}^{l}$, $LST^{\prime}_{1}$ and
$LST^{\prime}_{2}$ are the local linear projection for each dubbing direction
and the predicted local speaking style for each language.

###  IV-C Speech synthesis with predicted multi-scale speaking styles

A MST-FastSpeech 2 [17] is trained for each language to synthesize speeches
with global and local speaking styles inferred by the proposed framework.

The speaker embedding and the predicted global and local speaking styles of
each utterance are upsampled to phoneme level and concatenated with the
encoder outputs of FastSpeech 2. Then the pitch, duration and energy of each
phoneme are inferred from the concatenated encoder outputs by the variance
adaptor, and further converted to Mel-spectrogram by the decoder of FastSpeech
2.

A well-trained HiFi-GAN [24] is used as the vocoder to synthesize speech from
the predicted Mel-sprectrogram with desired speaking styles transferred from
the other language.

Figure 6: Pretraining the global and local speaking style encoders and
FastSpeech 2 for each language[17].

###  IV-D Training strategy

Before the training of the proposed joint cross-lingual multi-scale speaking
style transfer framework, the global and local speaking style encoders and
their corresponding MST-FastSpeech 2 for each language are previously trained
to ensure the extracted global and local speaking styles are compatible with
the TTS backbone. To improve the robustness of extracted speaking styles and
their cross-lingual transfer, the text and speaker information are also
disentangled from the global and local speaking styles with gradient reversal
layer (GRL) [37] and domain adversarial learning [29, 38] at this stage, which
are also formulated as:

$\displaystyle speaker^{\prime}_{GST}$
$\displaystyle=f_{speaker}^{GST}\left(GRL(GST)\right)$ (28) $\displaystyle
speaker^{\prime}_{LST}$ $\displaystyle=f_{speaker}^{LST}\left(GRL(LST)\right)$
(29) $\displaystyle BERT^{\prime}$
$\displaystyle=f_{BERT}\left(GRL(LST)\right)$ (30)

where $f_{speaker}^{GST}$, $f_{speaker}^{LST}$ and $f_{BERT}$ are linear
projections serving as the adversarial speaker predictors and text predictor,
GRL reverses the gradients of these adversarial predictors,
$speaker^{\prime}_{GST}$, $speaker^{\prime}_{LST}$ are the adversarially
predicted speaker embeddings, $BERT^{\prime}$ is the adversarially predicted
textual embedding of this utterance. The loss for pretraining is the sum of
reconstruction loss and adversarial losses, where the former one is the mean
squared error (MSE) between the predicted and ground-truth mel-spectrograms,
and the latter ones are the MSEs between the predicted and ground-truth
speaker and BERT embeddings.

The pretrained global and local encoders and FastSpeech 2 of each language are
then frozen and utilized to extract the speaking styles and synthesize speech
with predicted speaking styles in the proposed framework. The proposed joint
cross-lingual multi-scale speaking style transfer framework is then trained
and back-propagated with the sum of the MSEs between the predicted and ground-
truth global and local speaking styles on the two languages:

$\displaystyle loss=$ $\displaystyle
MSE(GST_{1},GST^{\prime}_{1})+MSE(LST_{1},LST^{\prime}_{1})$
$\displaystyle+MSE(GST_{2},GST^{\prime}_{2})+MSE(LST_{2},LST^{\prime}_{2})$
(31)

TABLE I: Subjective and objective evaluations for different approaches. *The predicted mel-spectrograms are resized to the size of ground-truths with nearest-neighbor interpolation.  Speaking style transfer method | Direction | MSE* | MOS$\pm$95%CI | Preference rate
---|---|---|---|---
No speaking style transfer [5] | en-zh | $4.694$ | $3.161\pm 0.075$ | $7.67\%$
Duration transfer [18] | $3.695$ | $3.923\pm 0.073$ | $25.83\%$
Multi-scale speaking style transfer (Proposed) | $\mathbf{1.392}$ | $\mathbf{4.123\pm 0.071}$ | $\mathbf{65.47\%}$
No speaking style transfer [5] | zh-en | $5.385$ | $3.358\pm 0.077$ | $14.83\%$
Duration transfer [18] | $3.301$ | $3.992\pm 0.069$ | $40.41\%$
Multi-scale speaking style transfer (Proposed) | $\mathbf{2.140}$ | $\mathbf{4.005\pm 0.073}$ | $\mathbf{43.99\%}$

##  V Experiments

###  V-A Baselines

To demonstrate the effectiveness of the proposed joint cross-lingual multi-
scale speaking style transfer framework, we employ 2 approaches with different
speaking style transfer methods as the baselines, which also employ FastSpeech
2 as the TTS backbone.

####  V-A1 No speaking style transfer

The first baseline approach is a vanilla FastSpeech 2 [5] with no speaking
style transfer, which is also a representative of state-of-the-art
conventional TTS and speech-to-speech translation systems with ideal
translated scripts.

####  V-A2 Duration transfer

Based on the proposed approach, we implement another approach with only
duration transfer as the second baseline. This approach is inspired by a
state-of-the-art automatic dubbing system [18] and should have similar or
better performance with the encoder-decoder frameworks and attention
mechanism.

In this approach, the modeling on global speaking styles transfer is omitted,
and only the duration of each word is transferred to the speech in the other
language. Specifically, a sequence of the duration for each word is used to
replaced the local speaking style in the proposed approach. And Equation (11),
(12), (26) and (27) are respectively changed to:

$\displaystyle s_{1}^{m}$
$\displaystyle=E_{1}\left(\left[BERT_{1};d_{1}\right]\right)$ (32)
$\displaystyle s_{2}^{m}$
$\displaystyle=E_{2}^{l}\left(\left[BERT_{2};d_{2}\right]\right)$ (33)
$\displaystyle d^{\prime}_{1}$ $\displaystyle=f_{1}^{l}([O_{2to1};s_{1}^{t}])$
(34) $\displaystyle d^{\prime}_{2}$
$\displaystyle=f_{1}^{l}([O_{1to2};s_{2}^{t}])$ (35)

where $d_{1}$, $d_{2}$, $d^{\prime}_{1}$, $d^{\prime}_{2}$ are the sequences
of duration and predicted duration for each word in the each language. Then
the sequence of predicted duration for each language is utilized by a
FastSpeech 2 with precise duration control for each language to synthesize the
dubbed speech.

###  V-B Training setup

We collect more parallel utterances from the English and Chinese dubs of the
game Borderlands 3 as the corpus for our research. As introduced in Section
III, the game Borderlands 3 is an action role-playing first-person shooter
video game from the Borderlands game franchise in a space Western science
fiction setting, which provides excellent and highly acclaimed dubs in many
languages with various speaking styles. Borderlands 3 has 4 playable
characters and dozens of non-playable characters (NPCs) in which there are
about 20 main NPCs who are frequently speaking with each other to tell the
background stories or guide the players in the missions. Each one of these
playable and non-playable characters has their unique speaking style to
present their different personality. Moreover, the characters are further
changing their speaking styles for different scenarios during the 35 gameplay
hours of main along with other side missions.

We collect 4,914 pairs of parallel utterances from Borderlands 3, including 5
hours of English speeches and 5 hours of Chinese speeches along with their
corresponding subtitles. The sample rate of all the speeches in the corpus is
48,000Hz. For the English speeches in the corpus, the length of each utterance
is ranged from 0.212s to 12.629s, with a mean of 3.801s and a standard
deviation of 2.350s. And the number of the words of each utterance is ranged
from 1 to 46, with a mean of 11.397 and a standard deviation of 7.715. For the
Chinese speeches in the corpus, the length of each utterance is ranged from
0.309s to 12.630s, with a mean of 3.913s and a standard deviation of 2.439s.
And the number of the words of each utterance is ranged from 1 to 70, with a
mean of 17.360 and a standard deviation of 11.918.

We extract Mel-spectrogram for each utterance in the corpus with a window
length of 25ms and a shift of 10ms, which will be used as the inputs for the
global and local speaking style encoders and the ground-truths for the MST-
FastSpeech 2. We employ a pretrained BERT [39] model to extract the sentence
and word level BERT embeddings, in which the BERT embedding of each word is
the average of the BERT embeddings of the corresponding sub-words.

We then employ the whole corpus to pretrain the MST-FastSpeech 2 and the
global and local speaking style encoders for both languages. We follow the
training setups of FastSpeech 2 [5] to train the model of each language for
1,000,000 iterations with a batch size of 32.

We randomly employ 4,414 pairs of parallel utterances in the two languages as
the training set to train the proposed multi-scale speaking style transfer
framework. The proposed framework is trained for 10 epochs with a batch size
of 32 and a learning rate of $10^{-4}$.

All the models are implemented with PyTorch [40] and trained on an NVIDIA
Tesla V100 GPU.

###  V-C Evaluation

We employ the rest 500 pairs of parallel utterances as the test set for
evaluation. We adopt the mean squared error (MSE) between the predicted and
ground-truth mel-spectrograms as the metric for the objective evaluation. The
predicted mel-spectrogram is resized to match the length of the ground-truth
mel-spectrogram with the nearest-neighbour interpolation.

For the subjective evaluation, 20 conversation chunks111Some synthesized
samples are available at https://thuhcsi.github.io/
automatic-dubbing/ . are further randomly selected and evaluated by 25
listeners. The listeners are asked to rate on how the speaking styles of
synthesized speeches are transferred from the speech in the other language on
a scale from 1 to 5 with 1 point interval, from which subjective mean opinion
scores (MOS) are calculated. Meanwhile, the listeners are asked to choose a
preferred dubbed speech from the speeches generated by different approaches,
from which preference rates are calculated.

###  V-D Experiment results

The results of the objective and subjective evaluations are shown in Table I.
Both the approaches with duration transfer and the proposed multi-scale
speaking style transfer have outperformed the baseline with no speaking style
transfer. The approach with duration transfer increases the MOS by $0.762$ and
$0.634$ for the two directions than the baseline with no speaking style
transfer, and the corresponding preference rate exceeds by $18.16\%$ and
$25.58\%$. Compared with the baseline with duration transfer, the proposed
approach further improves the MOS by $0.2$ and $0.013$ for the two directions,
and the corresponding preference rate exceeds by $29.83\%$ and $3.58\%$. Such
improvements on performance demonstrate the necessity and effectiveness of
modeling the transfer on all aspects in speaking style than just duration.

We illustrate the Mel spectrogram of the synthesized speech by each approach
in Figure 7 for the utterance ‘With this ship, we can kill bandits all over
the world.’ from the test set. We highlight the comparison between the words
‘bandits’ and ’world’ in blue and red frames these utterances. In the Figure 7
(a) and (b), these words in the original English and Chinese dubs are
emphasized with local speaking styles to express the excitement of the
characters with the new ship. And in the Figure 7 (g) and (h), we can see
their corresponding words in the other language have also been more clearly
emphasized than those synthesized by the vanilla FastSpeech 2 and the baseline
approach with only duration transfer.

TABLE II: Ablation studies. | English to Chinese | Chinese to English
---|---|---
Approach | MSE (Mel) | MSE (GST) | MSE (LST) | MSE (Mel) | MSE (GST) | MSE (LST)
Proposed | $\mathbf{1.392}$ | $4.220\times 10^{-3}$ | $7.983\times 10^{-3}$ | $\mathbf{2.140}$ | $2.012\times 10^{-3}$ | $\mathbf{1.075\times 10^{-2}}$
\- without local scale | $1.675$ | $3.274\times 10^{-3}$ | - | $2.699$ | $1.401\times 10^{-2}$ | -
\- without global scale | $1.660$ | - | $7.373\times 10^{-3}$ | $2.666$ | - | $1.092\times 10^{-2}$
\- without en-zh direction | - | - | - | $2.368$ | $2.024\times 10^{-3}$ | $1.080\times 10^{-2}$
\- without zh-en direction | $1.402$ | $4.228\times 10^{-3}$ | $8.097\times 10^{-3}$ | - | - | -

###  V-E Ablation study

Based on the proposed approach, we implement several variant approaches for
ablation studies. In addition to the MSEs between the predicted and ground-
truth Mel-spectrogram in both languages, we further consider the MSEs between
the predicted and ground-truth global and local speaking styles in ablation
studies.

The first two variants respectively ignore the speaking style transfer at the
global or local scale. The corresponding results are shown in the second and
third lines of Table II. Although these tow variants may have MSE of global or
local speaking styles which outperforms the MSE of the proposed approach, they
are suffering from missing scale in speaking style transfer and resulting in
poorer MSEs of the Mel-spectrograms than the proposed approach.

The last two variants respectively ignore one dubbing direction when modeling
the speaking style transfer. The results of these two variant are shown in the
last two lines of Table II. The proposed approach which jointly considers both
directions slightly outperforms these two variants, which showing the
effectiveness of jointly modeling the speaking style transfer in both
directions.

##  VI Discussion

Automatic dubbing is a potential research area in TTS and could be utilized to
simplify the dubbing process in real world game and film production. While
several efforts have been performed towards automatic dubbing and transferred
the duration of each prosodic phrase to the dubbed language, no previous work
has fully considered and transferred the speaking styles in automatic dubbing.
In this paper, for the first time, a joint multi-scale cross-lingual speaking
style transfer framework is proposed to simultaneously model the bidirectional
speaking style transfer between languages at both global and local scales.
Extensive experiments show that the proposed framework is able to jointly
consider the speaking style of each word and the speaking style of the whole
utterance in the source language and synthesize speech with appropriate
speaking styles in the target language.

To demonstrate the necessity of multi-scale cross lingual speaking style
transfer in automatic dubbing, subjective observation is conducted on the
cross-lingual speaking style transfer in a real game, Borderlands 3 which
contains excellent dubs in different languages with rich speaking styles. The
observation results reveal the the high correlation between the speaking
styles of parallel utterances in different languages. Particularly, at
utterance level, the global speaking styles of these parallel utterances are
always same or similar. And at word level, the local speaking styles of each
word could be often matched to a word in the other language, which means these
words have both same or similar semantic information and local speaking
styles.

The difficulty of collecting speeches in different languages which are
parallel in both meaning and speaking styles may be a reason to why there is
no approach to cross-lingual speaking style transfer modeling in automatic
dubbing yet. To overcome this issue, we collect parallel speeches from the
English and Chinese dubs in the game Borderlands 3 as the corpus for our
research. These speeches are performed by experienced voice talents and
recorded with professional equipment, which are ideal for learning the
speaking style transfer pattern across different languages. However, these
speeches can only be used in private and might be a gap to reproduce our work.
We wish that game companies can publicly release their dubs in different
languages to support the research of automatic dubbing in the future.

It is vital to extract clean and robust speaking style information for
learning speaking style transfer patterns. We employ the MST-FastSpeech 2 to
extract the ground-truth global and local speaking styles for each utterance
in the corpus and also build a TTS module with multi-scale speaking style
control for each language at the same time. For global speaking styles, it
utilizes the global style token [10] learning branch for Tacotron [2] which
has been widely used in speaking style and emotion transfer. And for local
speaking styles, we map the frame-level acoustic features to word-level with
the help of the neural network based forced aligner, NeuFA [30]. NeuFA is a
young, experimental model which could be used as a soft forced alignment model
to extract the attention weights between the given text and frame-level
acoustic feature sequence. To remove unwanted information and improve the
robustness of the extracted speaking styles, domain adversarial learning is
employed to disentangle the semantic and speaker information from speaking
styles.

The relevance between the semantic and speaking style information in different
languages is the key to achieve interpretive cross-lingual speaking style
transfer for automatic dubbing, which has been already demonstrated in our
data observation. We present a novel framework to model cross-lingual speaking
style transfer in global scale, which predicts the global speaking style for
the dubbed language from the global multi-modal features in the source
language and the global textual features in the dubbed language. At local
scale, we employ bidirectional attention mechanism to transfer the multi-modal
information in the source to the dubbed language according to the attention
weights learnt between the semantic meanings of the words in two languages.
Moreover, we unify the learning of cross-lingual speaking style transfer for
both dubbing directions with multi-task learning to jointly optimize the
attention weights at the local scale. However, the effectiveness of the multi-
task learning for both dubbing directions may be limited by the size of the
corpus. We believe such design would bring more performance enhancement with
more parallel utterances.

(a) Input English speech (b) Input Chinese speech

(c) Vanilla FastSpeech 2 (d) Vaniila FastSpeech 2

(e) Duration transfer (f) Duration transfer

(g) Multi-scale speaking style transfer (Proposed) (h) Multi-scale speaking
style transfer (Proposed)

Figure 7: Comparison between the Mel spectrograms for the words ‘bandits’ and
‘worlds’ in the synthesized speech ‘With this ship, we can kill bandits all
over the worlds.’ of each approach in English to Chinese and Chinese to
English dubbing directions.

##  VII Conclusion

In this paper, to properly transfer the speaking styles at both global (i.e.
utterance level) and local (i.e. word level) scales in automatic dubbing, we
propose a joint multi-scale cross-lingual speaking style transfer framework to
model the speaking style transfer between languages at both scales with multi-
task learning. The global and local speaking styles in each language are
extracted and utilized to predict the corresponding global and local speaking
styles in the other language, which are further synthesized to speech by MST-
FastSpeech 2. The effectiveness of our proposed framework is demonstrated by
experiments and ablation studies.

## References

  * [1] P. Taylor, _Text-to-speech synthesis_. Cambridge university press, 2009.
  * [2] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio _et al._ , “Tacotron: Towards end-to-end speech synthesis,” _arXiv preprint arXiv:1703.10135_ , 2017.
  * [3] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 4779–4783.
  * [4] N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural speech synthesis with transformer network,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 6706–6713.
  * [5] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” _arXiv preprint arXiv:2006.04558_ , 2020.
  * [6] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5530–5540.
  * [7] J. Li, Z. Wu, R. Li, P. Zhi, S. Yang, and H. Meng, “Knowledge-based linguistic encoding for end-to-end mandarin text-to-speech synthesis.” in _INTERSPEECH_ , 2019, pp. 4494–4498.
  * [8] X. Cai, D. Dai, Z. Wu, X. Li, J. Li, and H. Meng, “Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5734–5738.
  * [9] Y. Lei, S. Yang, X. Wang, and L. Xie, “Msemotts: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 853–864, 2022.
  * [10] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5180–5189.
  * [11] X. Li, C. Song, J. Li, Z. Wu, J. Jia, and H. Meng, “Towards multi-scale style control for expressive speech synthesis,” _arXiv preprint arXiv:2104.03521_ , 2021.
  * [12] X. Li, C. Song, X. Wei, Z. Wu, J. Jia, and H. Meng, “Towards cross-speaker reading style transfer on audiobook dataset,” _arXiv preprint arXiv:2208.05359_ , 2022.
  * [13] L. Sun, H. Wang, S. Kang, K. Li, and H. M. Meng, “Personalized, cross-lingual tts using phonetic posteriorgrams.” in _INTERSPEECH_ , 2016, pp. 322–326.
  * [14] T. Tu, Y.-J. Chen, C.-c. Yeh, and H.-Y. Lee, “End-to-end text-to-speech for low-resource languages by cross-lingual transfer learning,” _arXiv preprint arXiv:1904.06508_ , 2019.
  * [15] M. Chen, M. Chen, S. Liang, J. Ma, L. Chen, S. Wang, and J. Xiao, “Cross-lingual, multi-speaker text-to-speech synthesis using neural speaker embedding.” in _Interspeech_ , 2019, pp. 2105–2109.
  * [16] A. Öktem, M. Farrús, and A. Bonafonte, “Prosodic phrase alignment for machine dubbing,” _arXiv preprint arXiv:1908.07226_ , 2019.
  * [17] J. Li, Y. Meng, X. Wu, Z. Wu, J. Jia, H. Meng, T. Qiao, Y. Wang, and Y. Wang, “Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , ser. MM ’22. Lisboa, Portugal: ACM, 2022. [Online]. Available: http://doi.acm.org/10.1145/3503161.3547831
  * [18] J. Effendi, Y. Virkar, R. Barra-Chicote, and M. Federico, “Duration modeling of neural tts for automatic dubbing,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 8037–8041.
  * [19] M. Federico, Y. Virkar, R. Enyedi, and R. Barra-Chicote, “Evaluating and optimizing prosodic alignment for automatic dubbing.” in _INTERSPEECH_ , 2020, pp. 1481–1485.
  * [20] Y. Virkar, M. Federico, R. Enyedi, and R. Barra-Chicote, “Prosodic alignment for off-screen automatic dubbing,” _arXiv preprint arXiv:2204.02530_ , 2022\.
  * [21] ——, “Improvements to prosodic alignment for automatic dubbing,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7543–7574.
  * [22] H. Zen, K. Tokuda, and A. W. Black, “Statistical parametric speech synthesis,” _speech communication_ , vol. 51, no. 11, pp. 1039–1064, 2009\.
  * [23] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [24] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 022–17 033, 2020.
  * [25] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _International conference on machine learning_. PMLR, 2014, pp. 1764–1772.
  * [26] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [27] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [28] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [29] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [30] J. Li, Y. Meng, Z. Wu, H. Meng, Q. Tian, Y. Wang, and Y. Wang, “Neufa: Neural network based end-to-end forced alignment with bidirectional attention mechanism,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 8007–8011.
  * [31] Y. Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y. Wu, “Direct speech-to-speech translation with a sequence-to-sequence model,” _arXiv preprint arXiv:1904.06037_ , 2019.
  * [32] A. Lee, H. Gong, P.-A. Duquenne, H. Schwenk, P.-J. Chen, C. Wang, S. Popuri, J. Pino, J. Gu, and W.-N. Hsu, “Textless speech-to-speech translation on real data,” _arXiv preprint arXiv:2112.08352_ , 2021.
  * [33] G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto, “Creating corpora for speech-to-speech translation,” in _Eighth European Conference on Speech Communication and Technology_ , 2003.
  * [34] L. Rabiner and B. Juang, “An introduction to hidden markov models,” _ieee assp magazine_ , vol. 3, no. 1, pp. 4–16, 1986.
  * [35] M. Federico, R. Enyedi, R. Barra-Chicote, R. Giri, U. Isik, A. Krishnaswamy, and H. Sawaf, “From Speech-to-Speech Translation to Automatic Dubbing,” in _Proceedings of the 17th International Conference on Spoken Language Translation_. Online: Association for Computational Linguistics, 2020, pp. 257–264. [Online]. Available: https://aclanthology.org/2020.iwslt-1.31
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [37] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [38] Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Z. Chen, R. J. Skerry-Ryan, Y. Jia, A. Rosenberg, and B. Ramabhadran, “Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning,” _arXiv:1907.04448 [cs, eess]_ , Jul. 2019, arXiv: 1907.04448. [Online]. Available: http://arxiv.org/abs/1907.04448
  * [39] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.


#  Detecting Owner-member Relationship with Graph Convolution Network in
Fisheye Camera System

Zizhang Wu<EMAIL_ADDRESS>Jason Wang  Tianhao Xu  tianhao.xu@tu-
braunschweig.de Fan Wang  Zongmu Technology, Shanghai, China  Braunschweig
University of Technology, Braunschweig, Germany

###### Abstract

The owner-member relationship between wheels and vehicles contributes
significantly to the 3D perception of vehicles, especially in embedded
environments. However, to leverage this relationship we must face two major
challenges: i) Traditional IoU-based heuristics have difficulty handling
occluded traffic congestion scenarios. ii) The effectiveness and applicability
of the solution in a vehicle-mounted system is difficult. To address these
issues, we propose an innovative relationship prediction method, DeepWORD, by
designing a graph convolutional network (GCN). Specifically, to improve the
information richness, we use feature maps with local correlation as input to
the nodes. Subsequently, we introduce a graph attention network (GAT) to
dynamically correct the a priori estimation bias. Finally, we designed a
dataset as a large-scale benchmark which has annotated owner-member
relationship, called WORD. In the experiments we learned that the proposed
method achieved state-of-the-art accuracy and real-time performance. The WORD
dataset is made publicly available at
https://github.com/NamespaceMain/ownermember-relationship-dataset.
Keywords: autonomous driving; vehicle-mounted perception system; owner-member
relationship of wheels and vehicles; GCN; GAT

††journal: Pattern Recognition Letters

##  1 INTRODUCTION

Autonomous vehicles Grigorescu et al. (2020) are rapidly evolving, and various
aspects of their applications are attracting increasing attention, such as 3D
perception in vehicle-mounted systems Häne et al. (2017); Gregor et al.
(2002), which is critical for localisation Wang et al. (2017); Javanmardi et
al. (2019), planning, obstacle avoidance Minguez et al. (2008); Antonelli et
al. (1999), etc., as shown in Figure 1. The method Arrospide et al. (2010);
Chu et al. (2018); Vivet et al. (2012) using the homography transformation
Arrospide et al. (2010) of wheel grounding points is currently dominant. They
estimate the localization of the target vehicle by detecting the wheels in a
limited low-cost environment. In complex scenarios where different vehicles
overlap each other, the use of wheel and vehicle owner-member relationships
can effectively counteract this confusion. IoU is a method commonly used to
predict relationships between different objects, but the accuracy of this
method is very low for complex scenarios, as shown in Figure 2, in which one
wheel belongs to two different vehicle bounding boxes at the same time.

Fig. 1: The visualization of the owner-member relationship between wheels and
vehicles on 3D information acquisition. The estimation of four BEV corners
come into being after obtaining the relationship between wheels and vehicles.
In this way, the projection position of surrounding vehicles is helpful for
the autonomous driving system to make decision.

In this study, to address the problem of vehicle overlap in complex scenarios
and to constraint the shortcomings of prior analyses, we propose a model with
a graph convolutional network(GCN) structure to predict the owner-member
relationship, which can implicitly learn the relationship between the wheels
and vehicles.

##  2 RELATED WORK AND OUR CONTRIBUTIONS

###  2.1 Related work

#### Traditional relationship prediction methods

Models designed using traditional methods usually rely on statistical and
design thresholds through data such as semantics, pose, and order constraints
of objects and predict the relationship between objects. Desai et al. (2011)
employed information about the spatial layout of detected objects to predict
the relationship between objects. To predict the correlation between objects,
Dai et al. (2017) uses a priori statistics of the owner-member relationship
between objects obtained through Conditional Random Field (CRF) Sutton and
McCallum (2010). Zellers et al. (2018) used a priori knowledge to train the
model to learn objects that frequently occur in pairs, and LSTM (Long short-
term memory (LSTM) Staudemeyer and Morris (2019) to encode the images to
enhance contextual information and thus improve the performance of
relationship prediction.

#### Graph convolutional network methods

By further exploration of the method Shuman et al. (2013); Bruna et al.
(2013); Defferrard et al. (2016); Kipf and Welling (2016), GCN can be used in
a frequency domain based mathematical representation paradigm, so that graph
convolution has the properties of general convolutional structures can be used
in deep learning training. Owner-member and relative location relationship
prediction is the main target of GCN relationship prediction. Hamilton et al.
(2017) combines the understanding of aggregation from a spatial domain
perspective into the training of graph neural networks to update the graph
convolution. This approach is called graph SAGE. In addition, it generalizes
the expressions in the spatial domain and optimizes the difficulties
encountered in the training of large-scale graph data institutions. Veličković
et al. (2017) added attention mechanism in graph neural network to improve the
interaction between nodes. Xu et al. (2019) introduces a spatial-aware graph
relationship network (SGRN) to encode graph relationships to automatically
identify relationships between targets. It combines important semantic
information and object space relations to get better relationship predictions

Fig. 2: The visualization of complex scenarios. It is obvious that the
bounding-boxes of vehicles 4 and 5 completely contain wheel 0 as shown in (a),
and the bounding-boxes of vehicles 7 and 8 also completely contain wheel 9 as
shown in (b). In these scenarios, the prediction of the owner-member
relationship is not available with the IoU alone.

###  2.2 Our motivations and contributions

After analysing the above methods, there is still great potential in how to
improve the prediction accuracy of the owner-member relationship between the
wheel and the vehicle.

The first challenge is that autonomous driving does not yet have a available
public dataset in the field of relationship prediction, which is also an
impossible task for individuals. Secondly, the application of a priori
statistics to predict the owner-membership relationships of wheels and
vehicles, as mentioned in the above approach, is only applicable to single
scenario and cannot solve complex real-world scenarios. Therefore, it is also
difficult to produce positive effects in practical autonomous driving.
Moreover, the stacked logical judgments will reduce the computational
efficiency of the system, thus increasing the safety risks.

In this study, we draw on previous methods and make corresponding improvements
and optimizations to enhance the prediction accuracy. Specifically, we propose
a GCN-based owner-membership relationship prediction method for wheels and
vehicles, which models the owner-membership relationship between wheels and
vehicles by regarding the geometric relative positions between wheels and
vehicles as a priori statistics with the GCN structure. In addition, to
improve the information richness of the nodes in the GCN, we use feature
vectors of local correlations as the input to the nodes. Finally, we introduce
the graph attention network module (GAT) Veličković et al. (2017) to
counteract the effect of noise on accuracy, because GAT can dynamically amend
the deviation of prior estimates of edges through the training process. Our
contributions are summarised as follows:

1) To attract the participation of more researchers in the field, we have
created a large-scale benchmark dataset WORD containing 9000 samples.

2) We propose a GCN-based wheel and vehicle owner-member relationship
prediction network that is more applicable to relationship prediction in
complex scenarios.

3) We validate the effectiveness of our proposed method using the WORD
dataset. The experimental results show that this method achieves superior
accuracy, especially real-time effects, in an embedded environment.

##  3 DEEPWORD: A GCN-BASED APPROACH FOR OWNER-MEMBER RELATIONSHIP DETECTION

Fig. 3: The overall framework of the proposed DeepWORD. The input is the
detected bounding box, and after ROI Align the images are the same size.
Whereafter, it generates corresponding feature vectors with MLP operation,
learning the owner-member relationship with GCN to update the feature vectors.
Further, we calculate the cosine distance between the feature vectors from the
wheels and vehicles, and retain the wheel-vehicle pairs greater than the
threshold as the final results.

In this paragraph, we present all the details included in the method, such as
the overall structure of the model, the GCN module, the principle of GAT.

###  3.1 Overall framework of the proposed method

The overall framework of the proposed DeepWORD is shown in Figure 3, which
consists of two parts, the detection network and the relationship prediction
network. The detection network is CenterNet Zhou et al. (2019), whose main
task is to detect the vehicle vehicle and tires from the images, and input the
detection results into the relationship prediction network. The final model
will output the owner-membership relationship between the wheels and the
vehicle obtained by prediction.

In the model, we use two Gaussian mixture distributions from prior statistics
to predict the initial values of the edges of the connected nodes in the
network, which represent the degree of association between the nodes. To
improve the accuracy of these association, i.e., to reduce the bias of the
prior statistics, we introduce GAT in the GCN to dynamically update these
edges. The feature vector of each object is updated once after GCN modul, and
GAT associates each object with more global semantic information, thus
improving the association between the vehicle and wheels belonging to the same
vehicle. Finally, we use cosine distance to measure the similarity between
vehicle and wheels, i.e., the probability of belonging to the same vehicle,
and keep the pairs with a threshold above 0.5.

Fig. 4: The distance ratio distribution of wheel-vehicle pairs.

###  3.2 GCN-based relationship prediction network

In this section, we explain each part of the GCN-based relationship prediction
network.

#### Prior statistical relationship

The use of a priori statistical relationships between wheels and vehicles has
a significant positive effect on the update of nodes in GCN. The relationship
between wheels and vehicles after Gaussian distribution is shown in Figure 4.
Obviously, there is a spatially significant interrelationship between the
bounding boxes of wheels and vehicle belonging to the same vehicle. For
example, for most cases, the wheels belonging to the vehicle are at a shorter
distance from the vehicle than the wheels of other vehicles, and the wheels
are usually in the lower half of the vehicle’s bounding box.

As stated above, we conducted a statistical analysis of the distance ratio of
wheel-wheel and wheel-vehicle pairs. To eliminate the interference of
different sizes of objects, we normalise them as follows:

$\centering
D^{2}=(\frac{A_{\text{j}}}{W}-\frac{B_{\text{j}}}{W})^{2}+(\frac{A_{\text{i}}}{H}-\frac{B_{\text{i}}}{H})^{2}\@add@centering$
(1)

Fig. 5: The visualization of the owner-member relationship between wheels and
vehicles. (a) Front camera; (b) Left camera; (c) Right camera. Where the upper
left corner is the number of each object, the red line between two wheels
means that they are a couple. The green and blue lines respectively connect
the rear and front wheels and the vehicles that they belong to.

where $A$ denotes the vehicle and $B$ denotes the wheel in the wheel-vehicle
pair, whereas in the wheel-wheel pair, $A$ is the rear wheel, and $B$ is the
front wheel. $A_{\text{j}}$ and $B_{\text{j}}$ represent the horizontal
positions of $A$ and $B$, and $A_{\text{i}}$ and $B_{\text{i}}$ refer to the
vertical positions of $A$ and $B$, respectively. $W$ and $H$ are the width and
height of the input image, respectively.

Distortion of the image and the change of object scale from far to near can
seriously affect the prediction results, but by using 2 to obtain the distance
ratio between the vehicle and the wheels. We found that the priori statistics
are more consistent with the distribution of the Gaussian mixture model after
logarithmic transformation. Further, we apply this data processing method to
the matching of front and rear wheel pairs. Thus, the data after two Gaussian
mixture models, i.e., the distance ratio, is used as the prior statistic of
the adjacency matrix in the GCN to greatly improve the accuracy of the
relationship matching

$Ratio=\frac{2D}{W_{\text{B}}+H_{\text{B}}}$ (2)

where $W_{\text{B}}$ and $H_{\text{B}}$ represent the width and height of $B$,
respectively.

#### GCN structure

In the proposed GCN structure, we utilise the feature vectors obtained by
convolutional layers plus fully connected layers (Conv+fc) as nodes;
therefore, our model is more adept at expressing local spatial information. As
mentioned earlier a priori statistical relations are used to initialize the
adjacency matrix. Then the vehicle and wheel pairings updated by the GCN
structure are computed using the cosine distance. The final pairings with
scores greater than a threshold are retained, and in our experiments we set
the threshold to 0.5

#### GAT module

Given that the prior statistics are greatly related to the number of samples
and the scene richness, we introduce the GAT into GCN to amend the deviation
caused by the limited data. Specifically, we can weight GAT linearly for each
edge and secondly refine the weights of the edges in GCN, which can alleviate
the impact of noise in the available dataset and enhance the representation
ability of the network.

We set the node vector as
$\textbf{h}=\left\\{{\overrightarrow{h}_{\text{1}}},{\overrightarrow{h}_{\text{2}}},...,{\overrightarrow{h}_{\text{N}}}\right\\}$,
where ${\overrightarrow{h}_{\text{i}}}\in R^{F}$, $N$ represents the number of
nodes, and $F$ is the number of features in each node. In the GCN structure,
we input the features of each node $\overrightarrow{h}$ and their adjacent
nodes $\overrightarrow{h}_{\text{i}}$ into the GAT module, and concatenate
them through two fully connected layers. Then, we extract the weight matrix
$\textbf{W}=\left\\{w_{\text{1}},w_{\text{2}},...,w_{N\times N}\right\\}$ by
fully connected layers (FC) and nonlinear activation layers, where
$w_{\text{i}}\in R^{F\times F}$.

$Net(X_{\text{1}},X_{\text{2}})\rightarrow
FC(Relu(FC(Concat(X_{\text{1}},X_{\text{2}}))))$ (3)

where $Net$ denotes the procedure for dealing with the GCN,
$(X_{\text{1}},X_{\text{2}})$ represents the features of each node with their
adjacent nodes, and $FC$ is a fully connected layer.

Following, we utilize the softmax function to normalize the output and obtain
the attention coefficient $Scale_{i}$:

Next, we utilise the softmax function to normalise the output and obtain the
attention coefficient $Scale_{i}$:

$Scale_{\text{i}}=\frac{exp(Net(\overrightarrow{h},\overrightarrow{h}_{\text{i}}))}{\sum_{\text{j}}exp(Net(\overrightarrow{h},\overrightarrow{h}_{\text{i}}))}$
(4)

Finally, we update the original edge weight $w_{\text{i}}$ between the node
and the adjacent node by multiplying it with $Scale_{\text{i}}$ to obtain a
new weight $w_{\text{i}}^{{}^{\prime}}$ as follows:

$w_{\text{i}}^{{}^{\prime}}=w_{\text{i}}\times Scale_{\text{i}}$ (5)

In this manner, we can dynamically correct the prior estimation deviation of
the edge during the training process, and improve the prediction accuracy of
the owner-member relationship between wheels and vehicles.

#### Label preparation

We adjust the obtained feature maps of the vehicle and wheels to $H\times W$
and normalize them by [0, 1] to obtain the labels of the relationship
prediction network. The four coordinates of the object become a matrix of size
$H\times W$ after normalization. We concatenate the coordinate matrix with the
wheel and vehicle matrices to obtain $H\times W\times\text{7}$ and input it
into the relationship network. The Gaussian mixture distribution is created to
model the available data so that we can calculate the probabilities of wheel-
vehicle pairs and wheel-wheel pairs to generate the initial adjacency matrix.
In the adjacency matrix we adjust the values of unwanted objects to zero (e.g.
small objects)

##  4 EXPERIMENTS

###  4.1 Dataset overview

We constructed a dataset called WORD (Wheel and Vehicle Owner-Member
Relationship Dataset) as a benchmark. WORD mainly covers two typical scenarios
of autonomous driving: parking lots and highways. It contains about 9,000
images collected by a surround-view camera system consisting of surround-view
fisheye cameras. Where the frame ids of the images represent the order in
which they were taken, and images with the same frame id are taken by
different fisheye cameras at the same time.

###  4.2 Improvements on embedded platforms

To deploy our model to the embedded platform of an autonomous driving system,
we analysed the Qualcomm SNPE acceleration library and found that some
operations in the proposed model were not supported. These operations mainly
included RoI Align, GCN matrix multiplication operations, input of multiple
heads and sizes, etc. Therefore, we made some improvements to the proposed
model as follows:

(1) We removed the RoI Align and resized the feature map extracted from the
original image to a fixed size ($\text{56}\times\text{56}$).

(2) We used a fully connected layer instead of a matrix multiplication
operation in the GCN.

(3) To solve the problems arising from the input of multiple heads and sizes,
we removed the FC encoding module of the coordinate value and concatenated it
with the cropped vehicles and wheels from the original image to generate the
input features of size $H\times W\times\text{7}$.

###  4.3 Backbone selection

We experimented with a series of backbones, as shown in table ref table01, so
that we could better deploy the model into an embedded platform for autonomous
driving. Since speed and accuracy are always conflicting, after a trade-off we
use the Conv + fc structure as the final backbone choice to extract the
feature maps.

Table 1: Performance comparison of backbone selection. $AP_{v}$ represents the accuracy of image visualization. 10-speed and 1-speed indicate the speed at which the model processes the image when the batch size is 10 and 1, respectively. Backbone |  | Model size
---
(M)
| $AP_{v}$
---
(%)
| 10-Speed
---
(ms/10imgs)
| 1-Speed
---
(ms/img)
Conv+fc |  68 | 62.83 |  28 |  5
ResNet18 |  66 | 92.47 |  110 |  13

After determining the backbone we conducted more experiments to ensure the
best balance between speed and accuracy, as shown in table 2. From the
experimental results, we learned that we should instead reduce the number of
fully connected layers and increase the depth of the convolutional layers.
Finally, after the reduction, the computational size of the model is 28M, but
after image visualization the accuracy still reaches 95.7% of the best
performance, but the running speed is twice as fast.

###  4.4 Training of the relationship network

During the training stage, the Conv+fc structure extracts the features of the
input feature matrix. First, the input matrix goes through Conv+fc, and the
GCN structure updates these feature vectors. In addition, we normalise the
obtained features and calculate the cosine distance of each wheel and vehicle
as the final relationship prediction results. Moreover, we use the L2 loss and
manually adjust the weights of the positive and negative samples to optimise
the model. At the prediction stage, we multiply the predicted matrix with the
mask, and the position where the value is greater than 0.5 indicates that the
corresponding combination has an owner-member relationship. The mask is
primarily used to filter unnecessary objects. The visualisation of the owner-
member relationship between wheels and vehicles in highway scenes is shown in
Figure 5.

Table 2: Performance comparison of different parameters setting based on Conv+fc. The measures taken in turn for backbone are: 1) Neg0.1: reduces the weight of negative samples to 0.1; 2) -Neg: decreases the number of negative samples; 3) -Sma: uses a mask to filter too small objects; 4) 56: intercepts the input image size to 56; 5) 56_ex_4: deepens 4 convolutions and reduce the amount of fully connected layers. Backbone |  | Model size
---
(M)
| $AP_{v}$
---
(%)
| 10-Speed
---
(ms/10imgs)
| 1-Speed
---
(ms/img)
Conv+fc | 68 | 62.83 | 28 | 5
Conv+fc Neg0.1 | 68 | 69.34 | 28 | 5
Conv+fc-Neg | 68 | 73.42 | 28 | 5
Conv+fc-sma | 68 | 89.21 | 28 | 5
Conv+fc 56 | 53 | 90.33 | 18 | 4
Conv+fc 56_ex_4 | 28 | 95.70 | 13.1 | 3.2

###  4.5 Performance evaluation of DeepWORD

To demonstrate the effectiveness of our structure, we compared the proposed
DeepWORD with the previous logic model method. We selected 1000 images to form
an easy scene dataset in which each image contained no more than three
vehicles, 1000 images to form a difficult scene dataset where each image had
more than three vehicles, and a mixed dataset consisting of 500 easy images
and 500 difficult images. All samples in these three datasets were randomly
selected from the WORD. Some samples are shown in Figure 6.

Table 3: Performance comparison of DeepWORD and logic model method. Methods |  Easy |  Hard |  Mixed
---|---|---|---
Logic model |  92.91 |  71.83 |  79.90
DeepWORD |  99.17 |  94.35 |  95.14
Fig. 6: (a) and (b) are easy and hard samples of WORD.

The performance of DeepWORD in simple and difficult scenarios is shown in
Figures 5 and 7, respectively. We find that DeepWORD performs very well in
difficult scenes, such as in a parking lot, where cars are parked parallel and
close to each other. The owner-member relationship can be correctly identified
even if the bounding boxes of two different cars intersect at the same wheel,
or the bounding boxes of different vehicles completely contain the same wheel,
which is very difficult to solve using the logic model based on IoU.

Fig. 7: (a), (b), (c) and (d) are difficult scenes taken by camera while the
vehicle is moving in the underground parking lot. The bounding-boxes of
vehicles 1 and 0 intersect at wheel 5 as shown in (a),and the bounding-boxes
of vehicles 6 and 2 intersect at wheel 1 as shown in (b). The bounding-boxes
of vehicles 0 and 2 completely contain wheel 3 as shown in (c), and the
bounding-boxes of vehicles 12 and 13 completely contain wheel 11 as shown in
(d).

Finally, we conducted experiments to compare the classification accuracy of
these three datasets. As shown in Table 3, our DeepWORD model improves the
accuracy by 6.26% in simple scenarios when compared with the traditional logic
model approach. In contrast, DeepWORD performs even better in complex
scenarios, with accuracy improvements of 22.52% and 15.24%, respectively.
After observing the prediction results, the traditional logic model is more
likely to predict the wrong owner-member relationship when the vehicles appear
to be densely arranged. It is worth noting that both the generalization
ability and prediction accuracy of DeepWORD perform excellent relative to the
traditional logic model approach.

##  5 CONCLUSION AND FUTURE WORK

In this paper, our main research result is a GCN-based owner-membership
relationship prediction network DeepWORD for complex scenarios, always with
good applicability and prediction accuracy. Moreover, we apply it to vehicle-
mounted surround-view camera systems and conclude that it is efficient and
effective after extensive experiments. In addition, to promote related
research, we have established and released the large-scale relational dataset
WORD as the first dataset for autonomous driving in the field of relation
prediction. In the future, we will continue to extend the WORD dataset for
more complex real-world scenarios and further optimize our owner-member
relationship prediction solution DeepWORD.

## References

  * Antonelli et al. (1999) Antonelli, G., Chiaverini, S., Finotello, R., Morgavi, E., 1999\.  Real-time path planning and obstacle avoidance for an autonomous underwater vehicle, in: Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).
  * Arrospide et al. (2010) Arrospide, J., Salgado, L., Nieto, M., Mohedano, R., 2010\.  Homography-based ground plane detection using a single on-board camera.  IET Intelligent Transport Systems .
  * Bruna et al. (2013) Bruna, J., Zaremba, W., et al., 2013\.  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203 .
  * Chu et al. (2018) Chu, H., Gong, K., Shao, Y., Chang, Z., Ni, J., 2018.  3d perception and reconstruction system based on 2d laser scanner, in: 2018 Chinese Automation Congress (CAC).
  * Dai et al. (2017) Dai, B., Zhang, Y., Lin, D., 2017.  Detecting visual relationships with deep relational networks, in: Proceedings of the IEEE conference on computer vision and Pattern recognition, pp. 3076–3086.
  * Defferrard et al. (2016) Defferrard, M., Bresson, X., et al., 2016\.  Convolutional neural networks on graphs with fast localized spectral filtering, in: Advances in neural information processing systems, pp. 3844–3852.
  * Desai et al. (2011) Desai, C., Ramanan, D., Fowlkes, C.C., 2011.  Discriminative models for multi-class object layout.  International journal of computer vision 95, 1–12.
  * Gregor et al. (2002) Gregor, R., Lutzeler, M., Pellkofer, M., et al., 2002.  Ems-vision: A perceptual system for autonomous vehicles.  Intelligent Transportation Systems, IEEE Transactions on .
  * Grigorescu et al. (2020) Grigorescu, S., Trasnea, B., et al., 2020\.  A survey of deep learning techniques for autonomous driving.  Journal of Field Robotics .
  * Hamilton et al. (2017) Hamilton, W., Ying, Z., Leskovec, J., 2017.  Inductive representation learning on large graphs, in: Advances in neural information processing systems, pp. 1024–1034.
  * Häne et al. (2017) Häne, C., Heng, L., , et al., 2017\.  3d visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection.  arXiv preprint arXiv:1708.09839 .
  * Javanmardi et al. (2019) Javanmardi, E., Gu, Y., Javanmardi, M., Kamijo, S., 2019\.  Autonomous vehicle self-localization based on abstract map and multi-channel lidar in urban area.  IATSS Research .
  * Kipf and Welling (2016) Kipf, T.N., Welling, M., 2016\.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907 .
  * Minguez et al. (2008) Minguez, J., Lamiraux, F., Laumond, J.P., 2008.  Motion planning and obstacle avoidance.  Springer Handbook of Robotics .
  * Shuman et al. (2013) Shuman, D.I., Narang, S.K., et al., 2013\.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.  IEEE signal processing magazine 30, 83–98.
  * Staudemeyer and Morris (2019) Staudemeyer, R.C., Morris, E.R., 2019\.  Understanding lstm – a tutorial into long short-term memory recurrent neural networks.  arXiv preprint arXiv:1909.09586 .
  * Sutton and McCallum (2010) Sutton, C., McCallum, A., 2010\.  An introduction to conditional random fields.  arXiv preprint arXiv:1011.4088 .
  * Veličković et al. (2017) Veličković, P., Cucurull, G., et al., 2017.  Graph attention networks.  arXiv preprint arXiv:1710.10903 .
  * Vivet et al. (2012) Vivet, D., Checchin, P., Chapuis, R., 2012.  Radar-only localization and mapping for ground vehicle at high speed and for riverside boat, in: 10.1109/ICRA.2012.6224573.
  * Wang et al. (2017) Wang, L., Zhang, Y., Wang, J., 2017.  Map-based localization method for autonomous vehicles using 3d-lidar.  IFAC-PapersOnLine .
  * Xu et al. (2019) Xu, H., Jiang, C., Liang, X., Li, Z., 2019.  Spatial-aware graph relation network for large-scale object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9298–9307.
  * Zellers et al. (2018) Zellers, R., Yatskar, M., et al., 2018\.  Neural motifs: Scene graph parsing with global context, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831–5840.
  * Zhou et al. (2019) Zhou, X., Wang, D., Krähenbühl, P., 2019.  Objects as points.  arXiv preprint arXiv:1904.07850 .

##  Appendix A WORD Dataset

WORD (Wheel and Vehicle Owner-Member Relationship Dataset) contains two
categories that are common in autonomous driving: parking scene and highway
scene. Examples of these two scenes are shown in Figure 8.

The WORD contains approximately 9,000 images, which were captured by a
surround-view camera system, composed of four fisheye cameras. Examples are
shown in Figure 9. There are approximately 2400 images from the front camera,
1800 images from the left camera, 2900 pictures from the right camera, and
2000 images from the back camera. The difference in the number of images from
the different cameras is due to the fact that images that do not contain
vehicles or do not have owner-member relationships are filtered out, which
improves the quality of the WORD dataset.

The label of the WORD dataset consists of two parts. One part is the
information of the edge boxes, containing seven attributes: x-coordinate and
y-coordinate of the upper left corner of the edge box, x-coordinate and
y-coordinate of the lower right corner of the edge box, score, class-ID, and
box-ID; the other part is the owner-member relationship label, indicating
which cars and wheels of the edge boxes have owner-member relationships. The
visualisation of the owner-member relationship between the wheels and vehicles
is shown in Figure 5.

(a) highway scene

(b) parking scene

Fig. 8: (a) and (b) show the highway scene and the parking scene. Each scene
is captured from top to bottom by the front camera, the left camera and the
right camera.

(a) front (b) left

(c) right (d) rear

Fig. 9: (a) , (b) , (c) and (d) are the fisheye images captured by the front
camera, left camera, right camera and rear camera while the vehicle is moving
in the parking lot.