# Discovering Representation Sprachbund For Multilingual Pre-Training

Yimin Fan1, Yaobo Liang2, Alexandre Muzio3, Hany Hassan3,
Houqiang Li1, Ming Zhou4 and Nan Duan2
1University of Science and Technology of China, 2Microsoft Research Asia,
3Microsoft, 4Sinovation Ventures
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{yalia, nanduan, alex.muzio<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work is done during internship at Microsoft Research
Asia.

###### Abstract

Multilingual pre-trained models have demonstrated their effectiveness in many
multilingual NLP tasks and enabled zero-shot or few-shot transfer from high-
resource languages to low-resource ones. However, due to significant
typological differences and contradictions between some languages, such models
usually perform poorly on many languages and cross-lingual settings, which
shows the difficulty of learning a single model to handle massive diverse
languages well at the same time. To alleviate this issue, we present a new
multilingual pre-training pipeline. We propose to generate language
representation from multilingual pre-trained models and conduct linguistic
analysis to show that language representation similarity reflect linguistic
similarity from multiple perspectives, including language family, geographical
sprachbund, lexicostatistics and syntax. Then we cluster all the target
languages into multiple groups and name each group as a representation
sprachbund. Thus, languages in the same representation sprachbund are supposed
to boost each other in both pre-training and fine-tuning as they share rich
linguistic similarity. We pre-train one multilingual model for each
representation sprachbund. Experiments are conducted on cross-lingual
benchmarks and significant improvements are achieved compared to strong
baselines.

##  1 Introduction

The use of pre-trained models is considered a milestone in the development of
NLP research. Though early works (Devlin et al., 2019; Radford et al., 2019)
on monolingual pre-training (pre-training one model for one language)
significantly boosts the performance on the target language, monolingual pre-
training can hardly be generalized to multilingual settings because of high
training cost and insufficient corpora resources for many languages.

Multilingual pre-training was proposed to resolve this issue. By using shared
vocabulary across languages and pre-training with corpora from multiple
languages, multilingual pre-trained models handle cross-lingual tasks in one
model. Large scale multilingual pre-trained models provide powerful
representation for languages worldwide, enabling significant advances in
various multilingual tasks. However, existing widely used multilingual pre-
trained models (Lample and Conneau, 2019; Conneau et al., 2020; Huang et al.,
2019) perform poorly on many languages and some cross-lingual tasks like
zero/few-shot cross-lingual transfer. _e.g._ the performance of zero shot
transfer on XNLI task from English data to Urdu language is 15%+ lower than to
English (Conneau et al., 2020). Such a huge performance gap is the result of
cross-lingual contradictions and differences. This phenomenon is also
recognized as negative transfer in transfer learning (Wang et al., 2019).

Many existing works in cross-lingual transfer (K et al., 2020; Pires et al.,
2019; Lin et al., 2019) and machine translation (Dabre et al., 2017; Tan et
al., 2019) have shown that cross-lingual transfer works best between
typologically similar languages. We believe that utilizing similarity between
languages is potentially beneficial for large-scale multilingual pre-training.
Motivated by this, we propose a new multilingual pre-training pipeline. First,
we design a fully data-driven end-to-end way to generate language
representation for all languages (108 languages) based on massive multilingual
corpora. We represent each language as a 768-dimension vector and use cosine
similarity as their similarity measure. With the similarities between
languages quantified by their language representation, we automatically divide
all languages into a small number of representation sprachbunds. Sprachbund is
a linguistic terminology in German that refers to a group of close languages
(Sprach means language and bund means federation in German, so literally it is
“language federation" in English). A representation sprachbund is defined as a
group of languages with similar language representation. We conduct extensive
linguistic analysis and show that languages with similar representation are
similar and related from many linguistic perspectives, including language
family, geographical sprachbund, lexicostatistics and syntax typology. We
believe that training with similar languages in pre-training and fine-tuning
are beneficial as they share similar linguistic properties. Second, we train
multiple multilingual pre-trained models. Each model is trained with corpora
from one representation sprachbund. When handling downstream tasks in one
specific language, we fine-tune the model pre-trained with the corresponding
representation sprachbund corpora. We conduct experiments on 8 representative
cross-lingual tasks from XGLUE (Liang et al., 2020) and XTREME (Hu et al.,
2020) including sentence classification, structure prediction, question
answering and sentence retrieval. Experiment results show that our model
significantly outperforms strong baselines.

Our contributions can be summarized as follows: i) We propose a way to
automatically generate language representation from multilingual pre-trained
models and massive multilingual corpora. ii) We conduct extensive analysis to
show language representation and representation sprachbunds can reflect
linguistic language similarity and relatedness from multiple perspectives,
therefore they can be considered as new paradigm for clustering similar
languages in linguistics. iii) We use representation sprachbunds in
multilingual pre-training to alleviate the cross-lingual contradiction and
differences, and obtain significant improvements compared with strong
baselines.

##  2 Related Work

Our approach presents a new pipeline of multilingual pre-training. Our
representation sprachbund is inspired by linguistic language clustering. Our
analysis is closely related to methodology in linguistics.

#### Multilingual Pre-Training

Multilingual pre-training was proposed to pre-train a single model with
hundreds of languages. Many works use a large amount of multilingual data
(_e.g._ , mC4 (Xue et al., 2020), CCNet (Wenzek et al., 2020)) to pre-train
large multilingual models like XLM (Lample and Conneau, 2019), XLM-R (Conneau
et al., 2020), Unicoder (Huang et al., 2019) and mT5 (Xue et al., 2020).
Several benchmarks are proposed to evaluate the cross-lingual ability of
multilingual pre-trained models, including XGLUE (Liang et al., 2020), XTREME
(Hu et al., 2020) and XTREME-R (Ruder et al., 2021)

Figure 1: The pipeline of our approach. We first generate language
representation for each language with multilingual pre-trained models (XLM-R)
and multilingual corpora. We cluster languages into several representation
sprachbunds composed of languages with similar representation. We pre-train
one model for each representation sprachbund with corpora from that
representation sprachbund. Best viewed in color.

#### Language Clustering in Linguistics

The linguists propose to classify languages in several ways from different
perspectives. There are two main kinds of language clustering: genealogical
clustering and typological clustering. In genealogical clustering, languages
are clustered into language families (Durbin, 1985; Marcantonio, 2002) by
their genetic relatedness. Languages in the same language family have the same
ancestral language. In typological clustering, languages are clustered by
their typological features, like word order, morphology (Dressler, 1986) and
lexicostatistics (Hymes, 1960). Geographical sprachbund (Emeneau, 1980) is
also a typological clustering method as it groups languages according to their
similar areal features coming from geographical proximity.

#### Language Clustering in Multilingual NLP

Several recent works utilize linguistic knowledge about language clustering in
multilingual pre-training and machine translation. Tan et al. (2019) uses
language family and language embedding to cluster languages and train machine
translation model for each cluster. The language embedding is the language-
specific tag added to the input of encoder. Their approach focuses on 23
relatively high resource languages. Fan et al. (2020) clusters languages into
several groups according to language family, cultural connection and
geographical proximity. They do not obtain any language representation and
their language groups are human annotated. Kudugunta et al. (2019) reveals the
connection between language SVCCA similarity from NMT models and language
family. Their evaluation relies on parallel data. Chung et al. (2020)
classifies languages into groups based on their token overlap. Only lexical
information of languages is used in their approach. Yu et al. (2021) uses
multilingual denoising autoencoder to generate language embeddings and analyze
the clusters derived from the embeddings. There are also a few earlier works
on generating and analyzing language representation Tiedemann (2018); Östling
and Tiedemann (2017).

Compared to existing works, our approach enjoys the following advantages.
First, our approach requires neither parallel data nor linguistic labeling,
while most existing works on clustering languages relies on parallel data and
linguistic knowledge. Second, our representation sprachbunds contain
linguistic features from various aspects, while most existing works focus on
language family.

##  3 Approach

Our proposed pipeline on representation sprachbund for multilingual pre-
training is illustrated in Figure 1. Our approach can be divided into two
stages: First, we quantify the similarity between languages with generated
language representation, and cluster all languages in our corpora into $K$
clusters based on their similarity. Each cluster is called one representation
sprachbund. $K$ is a hyper-parameter in our clustering algorithm. Second, we
separate our corpora into $K$ parts based on their corresponding
representation sprachbund, and pre-train one model with corpora from each
representation sprachbund.

###  3.1 Discovering Representation Sprachbund

Suppose we have $M$ languages in our multilingual corpora $C$ denoted as
$L=\\{l_{1},l_{2},...,l_{M}\\}$. The corpora of the $i$th language is denoted
as $C_{i}$. $C_{i}$ contains $n_{i}$ sentences, denoted as
$C_{i}=\\{s_{i1},s_{i2},...,s_{in_{i}}\\}$. Note that there is no need for the
sentences $\\{s_{ik}\\}_{k=1}^{n_{i}}$ of language $l_{i}$ and
$\\{s_{jk}\\}_{k=1}^{n_{j}}$ of $l_{j}$ to be aligned.

Choenni and Shutova (2020) reveals that the sentence representation from the
same language generated by the last layer of multilingual pre-trained models
will be very close and form a relatively independent cluster in the
representation space. Motivated by this, a centroid of all sentence
representation from the same language can be a reasonable language
representation. We employ a transformer-based multilingual pre-trained model,
denoted as $F$. Each language $l_{i}$ is represented by language
representation $v_{i}$. We denote the representation of the token $k$ of
sentence $s$ from the last layer of $F$ as $F_{k}(s)$. We then define

$v_{i}=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}F_{[CLS]}(s_{ij})$

We separate all $M$ languages into $K$ clusters via clustering algorithm with
input features $\\{v_{i}\\}_{i=1}^{M}$. The cosine similarity between language
representation $v_{i}$ and $v_{j}$ is used as a similarity metric for the
clustering algorithm. The output $K$ clusters are $K$ representation
sprachbunds.

Figure 2: Visualization of language representation (reduced to 2-dimension for
visualization). All languages are labeled with ISO 639-1 code. Languages from
the same language family are colored the same. We draw ellipse for 10 main
language family covering most languages they include. The distribution of
language representation has great overlap with the language family and several
geographical sprachbunds. Best viewed in color.

###  3.2 Representation Sprachbund for Multilingual Pre-Training

We denote our $K$ representation sprachbunds as $\\{L_{1},L_{2},...,L_{K}\\}$,
where $L_{k}=\\{l_{k1},...,l_{kk_{i}}\\}$. We have $\sum_{j=1}^{K}j_{i}=M$ as
our representation sprachbund is non-overlapping. The corresponding corpora is
denoted as
$C^{\prime}_{k}=\\{C^{\prime}_{k1},C^{\prime}_{k2},...,C^{\prime}_{kk_{i}}\\}$.
Note that the corpora of the $i$th language for discovering representation
sprachbund ($C_{i}$) and for training models ($C^{\prime}_{i}$) may be
different. We train $K$ separate models for $K$ representation sprachbunds.
When fine-tuning, we can use the data in language $\\{l_{kj}\\}_{j=1}^{k_{i}}$
to fine-tune the $k$th model if the data is available.

##  4 Representation Sprachbund Discovery and Analysis

###  4.1 Settings

We collect massive multilingual corpora for discovering representation
sprachbund (also for the following multilingual pre-training). We use
Wikipedia111https://en.wikipedia.org/wiki/Main_Page corpora (100 languages are
included, total size 101GB) and a clean version of Common Crawl
(CC)222https://commoncrawl.org/ (89 languages are included, total size 2500GB)
following Liang et al. (2020). 108 languages are included in our multilingual
corpora. Note that we do not use any parallel corpora. The pre-trained model
$F$ we use is XLM-R base model implemented by
HuggingFace333https://huggingface.co/xlm-roberta-base. As the size of the
whole corpora is very large, we use a random sampling strategy to get part of
the data for extracting representation. For those languages with less than
10GB of data, we use all the data for extracting representation; for those
languages with more than 10GB of data, we sample 10GB out of all the data. We
use the method mentioned in Section 3.1 to get the language representation.
The dimension of language representation is 768 as in XLM-R base model each
token is represented by a 768-dimension vector. We reduce the 768-dimension
vectors $v_{1:108}$ to 2-dimension (denoted as $\tilde{v}_{1:108}$) for
visualization with the t-SNE algorithm implemented in Scikit-learn Python
package 444https://scikit-learn.org/stable/ with default parameters. We use
min-max normalization to normalize $\tilde{v}_{1:108}$ to range $[0,1]$. The
2-dimension language representation are visualized in Figure 2.

(a) (Subject, Object, Verbal) Order

(b) Adjective Position

(c) Apposition Position

Figure 3: Visualization of the relationship of our language representation and
3 language syntactic features. languages with the same syntactic features
approximately fall in the same region. i.e., have similar language
representation. Note that syntactic feature data of several languages is not
available. Best viewed in color.

###  4.2 Linguistics Analysis

We find that our representation can reflect linguistic similarity and
relatedness between languages from different perspectives. We link language
representation with several linguistic language similarity measures, also with
some linguistic clustering methods. We believe that representation sprachbund
is a new paradigm for clustering similar and related languages in linguistics.
In Figure 2, each language corresponds to one point (2-dimension vector) on
that figure. We label all points with the ISO 639-1
code555http://www.infoterm.info/standardization/ISO_639.php of their
corresponding languages.

#### Relationship with Language Family

We find that the distribution of language representation has great overlap and
similarity with language family. In Figure 2, the color of each point
indicates the language family of its corresponding language. 108 languages are
categorized into 22 language families according to
Ethnologue666https://www.ethnologue.com/browse/families. All 108 languages and
their corresponding language family can be found in Appendix A. We only label
10 language families with ellipses for clarity. Though there are some special
cases, languages within the same language family approximately fall in the
same region.

#### Relationship with Geographical Sprachbund

Geographical sprachbund is a group of similar languages from geographical
proximity and language contact, while our representation sprachbund is a group
of similar languages from representation proximity. We find that our language
representation distribution is consistent with many geographical sprachbunds.
In Figure 2, on the top-right, Romance and Germanic language representation
closeness can be linked with the Western Europe sprachbund from WHORF (1944);
on the bottom-left, the closeness between Indo-Aryan, Dravidian and some Sino-
Tibetan languages aligns with Indian subcontinent sprachbund proposed in
Emeneau (1956); on the middle, the similar representation from Turkic, Uralic
and Mongolic (mn) also match Altaic sprachbund by BOSWORTH (1962).

#### Relationship with Lexicostatistics

We find that similarity in language representation also reflects lexical
similarity between languages. We collect lexical similarity data from
Ethnologue on several languages (en, fr, de, pt, ro, ru, es, ca), denoted as
$Sim_{\textrm{lex}}$. We denote the similarity quantified by our language
representation as $Sim_{\textrm{data}}$. We find that $Sim_{\textrm{lex}}$ and
$Sim_{\textrm{data}}$ are strongly linear correlated, with Pearson correlation
coefficient of 0.83. Lexicostatistics is a method to measure lexical
similarity by comparing the percentage of lexical cognates between languages
(Hymes, 1960), which is very time-consuming. Our language representation can
even further help linguists infer lexical similarity more easily (_e.g._
linear regression between representation similarity and lexical similarity).
The similarity data is shown in Appendix B.

#### Relationship with Language Syntax

Languages have diverse syntactic features defined by linguists and can be
classified through these features. We show that the distribution of our
language representation implies the syntactic features of corresponding
languages. We use the lang2vec Python package (Littell et al., 2017) to query
the URIEL database777http://www.cs.cmu.edu/~dmortens/uriel.html. We choose
three representative syntactic features: (subject, object, verbal) word order,
adjective position and adposition position. As shown in Figure 3, we find that
languages with the same syntactic features approximately have similar language
representation.

#### Surprise: Help for Exploring Linguistic Mystery

Coincidentally, we find that our language representation connect with an
existing under-explored linguistic mystery. In Figure 2, Uralic and
Austronesian languages (jv, id, ms) have similar language representation. To
the best of our knowledge, only a few linguistic works (Ohnishi, 2006, 2009)
discussed their similarity and relatedness. The reason for their similarity
cannot be explained by language family (genetic relationship) or geographical
sprachbund (geographical relationship). Their language representation
similarity may be a clue that motivates linguists to find more similarity
between them and further explain how their similarity formed.

With the above linguistic analysis, we show that our language representation
contain rich linguistic genealogical, geographical, typological, and lexical
features of languages, therefore the similarity between language
representation can be a good metric for clustering languages. With a
768-dimension vector as numerical feature for each language, we can implement
clustering algorithms to cluster similar languages into a representation
sprachbund. Our language representation will be released later.

##  5 Representation Sprachbund For Multilingual Pre-training

###  5.1 Datasets

We collect massive multilingual corpora for pre-training and use four datasets
for downstream task evaluation. The multilingual corpora has been described in
Section 4.1. We use XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019),
NER (Pan et al., 2017), Part of Speech Tagging (POS), MLQA (Lewis et al.,
2019), TydiQA (Clark et al., 2020), XQuAD(Artetxe et al., 2020) and cross-
lingual sentence retrieval (Artetxe and Schwenk, 2019) as downstream tasks.
For cross-lingual sentence retrieval, we collect 21 language pairs and extract
1000 sentence-pairs for each language-pair from
tatoeba888https://tatoeba.org/eng/downloads. This task aims to find the
nearest neighbor for each sentence in the other language.

#$i$ is the $i$th representation sprachbund
---
#1 |  | af als an ast bar br ca ceb da de en eo es el fr fy ga gd
---
gl ia it ku lb nds nl nn no oc pt ro scn sco sq sv tl ur war
#2 |  | ar arz bg bs cy fa hi hr id is mg mk ms ps ru sh sl so
---
sr su sw yi
#3 |  | am as be ckb cs et eu fi he hu ja jv km la lo lt lv mr my
---
ne or pa pl sa sd sk th uk wuu zh
#4 | az bn gu hy ka kk kn ko ky ml mn si ta te tt ug uz vi tr
Table 1: Components of 4 representation sprachbunds Tasks | XNLI(Acc) | PAWS-X(Acc) | POS(F1) | NER(F1) | TydiQA(F1) | XQuAD(F1) | MLQA(F1)
---|---|---|---|---|---|---|---
XLM-R | 75.0 | 84.2 | 79.7 | 60.8 | 45.9 | 70.3 | 65.0
XLM-R CT | 74.8 | 84.8 | 79.9 | 60.9 | 46.3 | 70.5 | 65.4
Random | 74.8 | 84.8 | 79.4 | 60.9 | 47.2 | 70.2 | 65.5
Ours | 75.7 | 85.4 | 80.1 | 63.5 | 47.8 | 70.9 | 66.3
Table 2: Performance of our model fine-tune with English on 7 cross-lingual understanding tasks. XLM-R: directly fine-tune on XLM-R model. XLM-R CT: continue to pre-train XLM-R base model with our corpora. Random: pre-train one model for each random language cluster. Ours: pre-train one model for each representation sprachbund respectively. Languages | de-en | pt-en | es-en | nl-en | tl-en | ur-en | el-en | af-en | fr-en | it-en | nl-de | de-it | es-pt | de-el | sv-da | da-no | fr-de | it-ro | ar-ru | zh-ja | pl-cs | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R | 89.0 | 78.7 | 72.0 | 77.4 | 29.9 | 33.4 | 54.1 | 53.1 | 73.2 | 65.9 | 67.0 | 55.1 | 75.9 | 54.5 | 81.1 | 89.6 | 76.0 | 51.2 | 43.4 | 52.6 | 73.8 | 64.1
XLM-R CT | 85.9 | 75.5 | 70.2 | 77.3 | 31.4 | 36.5 | 52.4 | 57.9 | 73.6 | 63.5 | 65.4 | 54.2 | 73.2 | 49.9 | 79.6 | 89.6 | 72.3 | 50.4 | 46.7 | 59.6 | 76.8 | 63.9
Ours | 90.3 | 80.6 | 75.3 | 79.0 | 32.2 | 39.8 | 54.5 | 57.4 | 74.6 | 67.9 | 68.2 | 57.8 | 76.7 | 55.3 | 81.0 | 89.9 | 77.5 | 53.3 | 58.7 | 61.7 | 82.1 | 67.3
Table 3: Performance (Accuracy) of our model on cross-lingual sentence
retrieval task without fine-tuning. Languages in different representation
sprachbunds are separated with vertical lines.

###  5.2 Settings

We use the XLM-R base model as our base model.
Fairseq999https://github.com/pytorch/fairseq is used as our pre-training code
base. The Huggingface Transformers101010https://huggingface.co/transformers/
is used as our fine-tuning code base. We use the hierarchical clustering
algorithm implemented by Scikit-learn Python package111111https://scikit-
learn.org/stable/ for clustering language representation. We cluster languages
into 4 representation sprachbunds. The reason for clustering 4 representation
sprachbunds is shown in Section 5.4. Components of representation sprachbunds
are shown in Table 1. We use the shared vocabulary of XLM-R base model for
reusing the pre-trained parameters and keeping the comparability with the
baseline model.

#### Pre-training Setting

We initialize our model with the XLM-R base model parameters and run continual
pre-training for 40000 updates on 8 Nvidia V100 GPUs with total batch size
8192. The experiment takes about 8 days. We use Adam optimizer with a linear
warm-up and set the learning rate to 3e-5. We pre-train 4 models according to
our 4 representation sprachbund corpora. We also randomly create 4 language
clusters (each with the same language number as the representation
sprachbund). We pre-train 4 models with random language clusters as baseline.
To evaluate the impact of continual pre-train corpora, we also continue to
pre-train XLM-R base model with our corpora.

#### Downstream Task Setting

For XNLI, we set the learning rate to 5e-6 and train 10 epochs with batch size
32. For POS tagging, we set the learning rate to 2e-5 and train 20 epochs with
batch size 32. For MLQA, we set the learning rate to 3e-5, train 2 epochs
following BERT for SQuAD with batch size 12. For PAWS-X, NER, TydiQA and
XQuAD, we follow the default settings in XTREME (Hu et al., 2020).The
downstream task performance on one specific language is measured by fine-
tuning the model pre-trained with the corresponding representation sprachbund
corpora. We select the checkpoint with the best performance on the dev set.
The results are averaged over three runs. For cross-lingual sentence
retrieval, we use the cosine similarity of the average middle layer (the 7th
layer of our 12-layer model) embedding for retrieval without fine-tuning.

Languages | en | fr | es | de | el | ur | bg | ru | ar | hi | sw | tr | vi | th | zh | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XNLI fine-tune with English
XLM-R | 84.8 | 78.9 | 79.2 | 77.7 | 76.6 | 67.0 | 78.6 | 76.6 | 72.3 | 71.1 | 66.4 | 73.3 | 75.6 | 72.4 | 74.7 | 75.0
XLM-R CT | 84.5 | 78.4 | 79.5 | 77.2 | 76.4 | 66.8 | 78.3 | 76.2 | 73.0 | 70.6 | 65.4 | 73.3 | 75.3 | 72.6 | 74.2 | 74.8
Random | 84.3 | 78.0 | 78.5 | 77.1 | 76.5 | 67.3 | 78.2 | 76.2 | 72.2 | 70.5 | 66.4 | 73.5 | 75.4 | 72.8 | 74.7 | 74.8
Ours | 84.1 | 78.4 | 80.0 | 77.8 | 77.3 | 68.2 | 78.4 | 76.7 | 74.5 | 72.1 | 68.6 | 74.9 | 75.3 | 74.1 | 74.6 | 75.7
XNLI fine-tune with pivot language
XLM-R | 83.6 | 79.5 | 80.0 | 79.8 | 77.4 | 69.3 | 80.1 | 78.4 | 74.2 | 73.3 | 66.9 | 74.2 | 78.8 | 76.1 | 76.3 | 76.5
Ours | 84.2 | 80.0 | 80.4 | 80.4 | 77.8 | 69.9 | 80.2 | 78.0 | 75.5 | 73.3 | 70.9 | 75.0 | 78.3 | 76.4 | 75.7 | 77.1
XNLI fine-tune with every language
XLM-R | 84.6 | 79.5 | 80.9 | 79.8 | 79.1 | 67.1 | 80.2 | 78.4 | 75.5 | 73.9 | 71.1 | 77.2 | 78.8 | 76.1 | 78.4 | 77.4
Ours | 84.1 | 80.4 | 81.1 | 80.4 | 78.9 | 66.5 | 80.3 | 78.0 | 77.2 | 74.3 | 72.6 | 76.9 | 78.3 | 76.4 | 78.0 | 77.6
Table 4: Performance (Accuracy) of our model on XNLI dataset on three
settings. Note that we do not use the result in (Conneau et al., 2020),
instead we fine-tune the model in our settings. Languages in different
representation sprachbunds are separated with vertical lines. Pivot languages
are bold in the first row.

#### Evaluation Setting

There are three main settings in the fine-tuning stage. (i) Fine-tune with
English. We fine-tune the model with English labeled data of downstream task.
(ii) Fine-tune with every language. _e.g._ We fine-tune the model with the
French labeled data and test its performance on the French test set. (iii)
Fine-tune with pivot language. We choose a pivot language $l_{i}$ in its
representation sprachbund $m$ based on similarity with other languages (if
$i=\mathop{\arg\max}_{i}\sum_{l_{j}\in L_{m}}cos(v_{i},v_{j})$), and fine-tune
the model with the pivot language labeled data.

###  5.3 Main Results

We conduct experiments in several settings on different types of tasks. We
find that our approach obtains significant improvements over XLM-R base model
and randomly clustered model when applied in pre-training and downstream
tasks.

#### Improving Pre-Training

Languages | en | fr | es | de | el | ur | bg | ru | ar | hi | sw | tr | vi | th | zh | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R+All | 84.8 | 81.3 | 82.0 | 80.5 | 80.0 | 71.7 | 81.6 | 79.1 | 78.2 | 75.6 | 73.1 | 78.1 | 79.4 | 77.2 | 79.8 | 78.8
XLM-R+RSB | 85.3 | 80.9 | 81.8 | 80.6 | 80.0 | 71.2 | 81.3 | 79.4 | 77.7 | 75.9 | 72.6 | 77.5 | 79.0 | 77.4 | 78.3 | 78.6
RSB+ | 85.6 | 81.4 | 81.7 | 80.5 | 80.9 | 70.9 | 82.0 | 79.5 | 78.0 | 76.8 | 73.8 | 77.6 | 78.7 | 77.3 | 79.7 | 79.0
Table 5: Downstream task (XNLI) data efficiency of our model. XLM-R+All: use
XNLI data in all languages to fine-tune XLM-R base model. XLM-R+RSB: use XNLI
data in the same representation sprachbund to fine-tune XLM-R base model.
RSB+: use XNLI data in the same representation sprachbund to fine-tune the
pre-trained model. Different representation sprachbunds are separated with
vertical lines.

In Table 2, We find that our approach significantly outperforms all baseline
models on 7 cross-lingual tasks. The 7 tasks are representative of almost all
kinds of cross-lingual understanding tasks, which shows the universal
effectiveness of our model. The detailed results of each task and each
language are shown in Appendix C.

#### Improving Fine-Tuning

In Table 4, we also show that without additional costly continual pre-
training, directly fine-tuning multiple models with representation sprachbund
also improves the performance. We fine-tune each model (XLM-R base) with the
labeled pivot language data on XNLI of each representation sprachbund. In
Table 4, we find that the improvement is significant (from 75.0 to 76.5). We
also show that fine-tuning the model with the labeled data from each language
yields significantly better results (77.4). We conclude that fine-tuning with
the language similar to target language is likely to boost the performance.

#### Improving Multilingual Embeddings

In Table 3, we show that the performance on cross-lingual sentence retrieval
greatly improves with our approach. Though our method is not designed for
improving multilingual embeddings, better multilingual embeddings are
generated without additional fine-tuning.

Task | XNLI | PAWS-X | POS | NER | TydiQA | XQuAD | MLQA
---|---|---|---|---|---|---|---
High | 0.2 | 0.6 | 0 | 1.3 | 1.9 | 0.5 | 1.0
Low | 1.2 | 0.6 | 1.2 | 3.1 | 1.4 | 0.8 | 1.5
Table 6: Low resource languages has more significant gains with our approach.
Low indicates the languages are low resource and isolated.

###  5.4 Analysis

#### Gaining More on Low Resource Languages

In Table 6, we show that our approach brings more gains to those low resource
and isolated languages (including ur, ar, sw, tr, vi, th, zh, pl, ja, ko, id,
fi, bn, te, tl, af, ms, fa, mr, et, he, jv, eu, yo, my, hu, ta, ml, kk ,kn,
ka) compared with those high resource languages (including en, de, es, ru, bg,
hi, it, fr, nl, pt) which will be beneficial for bridging the large
performance gap between high resource and low resource languages. An intuitive
explanation is that those low resource and isolated languages suffer from more
serious cross-lingual contradictions when trained with those dissimilar high
resource languages. When clustered with similar languages, those low resource
and isolated languages are likely to benefit a lot.

#### Achieving Data Efficiency in Downstream Tasks

We show that with the continual pre-training step with representation
sprachbund corpora, less data for downstream tasks is needed to achieve high
accuracy. In Table 5, on XNLI task, we find that fine-tuning our pre-trained
model with all the downstream task data from each representation sprachbund
(less than 30% of all the data) achieves better results than fine-tuning XLM-R
with all the data (from 78.8 to 79.0). We also find that fine-tuning XLM-R
with the representation sprachbund data achieves results comparable with fine-
tuning using all the data (78.6 and 78.8), which means that using data from
similar languages (though less) works well.

#### Choosing the Number of Representation Sprachbunds

We conduct experiments to choose the number of representation sprachbunds. We
cluster languages into 1,2,4,8 representation sprachbunds, and pre-train
1,2,4,8 models, respectively. We evaluate through fine-tuning with English on
the XNLI dataset. As shown in Figure 4, with the increase of the number of
representation sprachbunds, the performance also increases. We find that
clustering languages into 4 representation sprachbunds is a desirable choice,
as from 4 to 8 little gain is obtained but the cost doubles.

Figure 4: Impact of the number of representation sprachbunds on the
performance. Performance (Accuracy) is measured by fine-tuning with English
labeled data on XNLI task.

##  6 Conclusion

To reduce the cross-lingual contradictions in pre-training one model for all
languages, we propose to merge similar languages into a representation
sprachbund and pre-train one model for each representation sprachbund. Results
show that our approach outperforms strong baselines in various settings and
tasks. We also identify the relationship between our representation sprachbund
with linguistic theories. Applications of our representation sprachbund as a
paradigm for clustering languages in linguistics will be explored in
subsequent work.

## Acknowledgements

We thank Ning Wu, Xiaoze Jiang, Yuan Chai, Junhe Zhao, Shunyu Zhang for their
useful suggestions on writing this paper.

## References

  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the Cross-lingual Transferability of Monolingual Representations.  In _Proceedings of ACL 2020_.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * BOSWORTH (1962) C. E. BOSWORTH. 1962.  A turco-mongol practice amongst the early ghaznavids? _Central Asiatic Journal_ , 7(4):237–240.
  * Choenni and Shutova (2020) Rochelle Choenni and Ekaterina Shutova. 2020.  What does it mean to be language-agnostic? probing multilingual sentence encoders for typological properties.
  * Chung et al. (2020) Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020.  Improving multilingual models with language-clustered vocabularies.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  In _Transactions of the Association of Computational Linguistics_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _EMNLP_.
  * Dabre et al. (2017) Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. 2017.  An empirical study of language relatedness for transfer learning in neural machine translation.  In _Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation_ , pages 282–286. The National University (Phillippines).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dressler (1986) W.U. Dressler. 1986.  Morphology: a study of the relation between meaning and form: Joan l. bybee, (typological studies in language 9.) benjamins, amsterdam; 1985. vii+234pp.  _Lingua_ , 70(2):197 – 200.
  * Durbin (1985) Marshall Durbin. 1985.  A survey of the carib language family.  _South American Indian Languages. Retrospect and Prospect_ , 325:370.
  * Emeneau (1956) M. B. Emeneau. 1956.  India as a lingustic area.  _Language_ , 32(1):3–16.
  * Emeneau (1980) M. B. Emeneau. 1980.  _Language and linguistic area : essays / by Murray B. Emeneau ; selected and introduced by Anwar S. Dil_.  Stanford University Press Stanford, Calif.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.
  * Huang et al. (2019) Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019.  Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2485–2494, Hong Kong, China. Association for Computational Linguistics.
  * Hymes (1960) D. H. Hymes. 1960.  Lexicostatistics so far.  _Current Anthropology_ , 1(1):3–44.
  * K et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020.  Cross-lingual ability of multilingual bert: An empirical study.  In _International Conference on Learning Representations_.
  * Kudugunta et al. (2019) Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019.  Investigating multilingual NMT representations at scale.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1565–1575, Hong Kong, China. Association for Computational Linguistics.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _CoRR_ , abs/1901.07291.
  * Lewis et al. (2019) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019\.  Mlqa: Evaluating cross-lingual extractive question answering.  _ArXiv_ , abs/1910.07475.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019.  Choosing transfer languages for cross-lingual learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3125–3135, Florence, Italy. Association for Computational Linguistics.
  * Littell et al. (2017) Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017.  URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 8–14, Valencia, Spain. Association for Computational Linguistics.
  * Marcantonio (2002) Angela Marcantonio. 2002.  The uralic language family: facts, myths and statistics.
  * Ohnishi (2006) Koji Ohnishi. 2006.  Word-initial consonant correspondence laws between uralic and austronesian language families: The origin of the uralic language family from a language closely kin to western oceanic and sulawesi language group.
  * Ohnishi (2009) Koji Ohnishi. 2009.  Sound-correspondence laws of word-initial consonants between proto-indo-european and austronesian languages.  _Artificial Life and Robotics_ , 14(4):567–573.
  * Östling and Tiedemann (2017) Robert Östling and Jörg Tiedemann. 2017.  Continuous multilinguality with language vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 644–649, Valencia, Spain. Association for Computational Linguistics.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of ACL 2017_ , pages 1946–1958.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021.  Xtreme-r: Towards more challenging and nuanced multilingual evaluation.
  * Tan et al. (2019) Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019.  Multilingual neural machine translation with language clustering.
  * Tiedemann (2018) Jörg Tiedemann. 2018.  Emerging language spaces learned from massively multilingual corpora.
  * Wang et al. (2019) Zirui Wang, Zihang Dai, Barnabas Poczos, and Jaime Carbonell. 2019.  Characterizing and avoiding negative transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. 2020.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4003–4012.
  * WHORF (1944) BENJAMIN LEE WHORF. 1944.  The relation of habitual thought and behavior to language.  _ETC: A Review of General Semantics_ , 1(4):197–215.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of EMNLP 2019_ , pages 3685–3690.
  * Yu et al. (2021) Dian Yu, Taiqi He, and Kenji Sagae. 2021.  Language embeddings for typology and cross-lingual transfer learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7210–7225, Online. Association for Computational Linguistics.

##  Appendix A Linguistic Language Family of All Languages

Language Family | Languages
---|---
Germantic |  | af,als,bar,cy,da,de,en,fy,gd,is,
---
lb,nds,nl,nn,no,sco,sv,yi
Greek | el
Japonic | ja
Sino-Tibetan | my,wuu,zh
Turkic | az,kk,ky,tr,tt,ug,uz
Uralic | et,fi,hu
Austroasiatic | km,vi,war
Dravidian | kn,ml,ta,te
Slavic |  | be,bg,bs,cs,hr,lt,lv,mk,
---
pl,ru,sh,sk,sl,sr,uk
Kartvelian | ka
Niger-Congo | sw
Austronesian | ceb,id,jv,mg,ms,su,tl
Armenian | hy
Koreanic | ko
Albanian | sq
Tai-Kadai | lo,th
Romance |  | an,ast,br,ca,es,fr,gl,
---
it,la,oc,pt,ro,scn,eu
Constructed | eo,ia
Afro-Asiatic | am,ar,arz,he,so
Celtic | ga
Indo-Aryan |  | as,bn,ckb,fa,gu,hi,
---
ku,mr,ne,or,pa,ps
Mongolic | sa,sd,si,ur
Table 7: Languages and their corresponding language family

##  Appendix B Lexical similarity and Embedding similarity

Languages | ca | en | fr | de | pt | ro | ru | es
---|---|---|---|---|---|---|---|---
ca | 1.00 | - | 0.85 | - | 0.85 | 0.73 | - | 0.85
en | - | 1.00 | 0.27 | 0.60 | - | - | 0.24 | -
fr | 0.85 | 0.27 | 1.00 | 0.28 | 0.75 | 0.75 | - | 0.75
de | - | 0.60 | 0.28 | 1.00 | - | - | - | -
pt | 0.85 | - | 0.75 | - | 1.00 | 0.72 | - | 0.88
ro | 0.73 | - | 0.75 | - | 0.72 | 1.00 | 0.72 | 0.71
ru | - | 0.24 | - | - | - | - | 1.00 | -
es | 0.85 | - | 0.75 | - | 0.88 | 0.71 | - | 1.00
Table 8: Lexical similarity from Ethnologue. Some data is missing. Languages | ca | en | fr | de | pt | ro | ru | es
---|---|---|---|---|---|---|---|---
ca | 1.00 | 0.08 | 0.76 | 0.22 | 0.63 | 0.56 | 0.23 | 0.81
en | 0.08 | 1.00 | 0.26 | 0.38 | 0.28 | 0.17 | 0.31 | 0.00
fr | 0.76 | 0.26 | 1.00 | 0.49 | 0.63 | 0.65 | 0.47 | 0.68
de | 0.22 | 0.38 | 0.49 | 1.00 | 0.47 | 0.49 | 0.59 | 0.26
pt | 0.63 | 0.28 | 0.63 | 0.47 | 1.00 | 0.61 | 0.45 | 0.64
ro | 0.56 | 0.17 | 0.65 | 0.49 | 0.61 | 1.00 | 0.48 | 0.56
ru | 0.23 | 0.31 | 0.47 | 0.59 | 0.45 | 0.48 | 1.00 | 0.24
es | 0.81 | 0.00 | 0.68 | 0.26 | 0.64 | 0.56 | 0.24 | 1.00
Table 9: Language embedding similarity

##  Appendix C Detailed Results of Cross-lingual Tasks

Languages | ar | hi | de | en | es | zh | vi | Avg
---|---|---|---|---|---|---|---|---
XLM-R | 55.3 | 61.3 | 62.1 | 80.0 | 68.1 | 61.5 | 66.9 | 65.0
XLM-R CT | 57.1 | 61.7 | 61.9 | 80.1 | 68.0 | 61.5 | 67.3 | 65.4
Random | 57.2 | 62.3 | 62.0 | 81.0 | 67.6 | 61.6 | 67.2 | 65.5
Ours | 59.2 | 63.1 | 62.9 | 80.4 | 69.3 | 61.4 | 67.6 | 66.3
Table 10: Performance (F1 score) on MLQA dataset. We examine the results on MLQA on fine-tuning with English setting. Languages in different embedding sprachbunds are separated with vertical lines.  Languages | de | en | es | fr | ja | zh | ko | Avg
---|---|---|---|---|---|---|---|---
XLM-R | 87.5 | 94.4 | 88.5 | 88.5 | 75.9 | 80.1 | 74.7 | 84.2
XLM-R CT | 87.7 | 94.4 | 88.8 | 88.5 | 77.2 | 81.1 | 75.6 | 84.8
Random | 87.0 | 94.4 | 88.8 | 88.9 | 77.4 | 80.0 | 76.7 | 84.8
Ours | 88.4 | 94.9 | 89.1 | 89.6 | 78.2 | 80.1 | 77.2 | 85.4
Table 11: Performance (F1 score) on PAWS-X dataset. We examine the results on PAWS-X on fine-tuning with English setting. Languages in different embedding sprachbunds are separated with vertical lines.  Languages | ar | ru | bg | hi | ur | de | el | en | es | fr | it | nl | pt | th | zh | pl | tr | vi | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R | 69.5 | 86.6 | 88.5 | 72.2 | 59.8 | 92.3 | 87.6 | 96.4 | 88.8 | 89.3 | 92.1 | 88.6 | 90.0 | 58.6 | 59.7 | 84.0 | 73.6 | 56.7 | 79.7
XLM-R CT | 69.4 | 86.1 | 88.6 | 68.2 | 61.5 | 91.5 | 88.5 | 96.4 | 89.6 | 89.8 | 92.2 | 88.9 | 90.0 | 57.8 | 62.9 | 84.0 | 74.1 | 57.9 | 79.9
Random | 68.5 | 86.5 | 88.6 | 69.2 | 59.7 | 92.3 | 87.9 | 96.3 | 88.1 | 88.9 | 92.0 | 88.4 | 90.0 | 58.2 | 60.5 | 83.9 | 72.9 | 57.1 | 79.4
Ours | 69.3 | 85.4 | 88.2 | 70.8 | 60.1 | 92.4 | 88.6 | 95.6 | 89.1 | 89.4 | 92.3 | 88.9 | 90.9 | 58.5 | 63.0 | 84.7 | 75.5 | 58.5 | 80.1
Table 12: Performance (F1 score) on POS tagging dataset. We examine the results on POS tagging on fine-tuning with English setting. Languages in different embedding sprachbunds are separated with vertical lines.  Languages | en | ar | ru | sw | id | fi | bn | ko | te | Avg
---|---|---|---|---|---|---|---|---|---|---
XLM-R | 60.7 | 52.5 | 50.1 | 50.2 | 63.8 | 51.2 | 31.1 | 23.0 | 30.9 | 45.9
XLM-R CT | 60.1 | 53.7 | 50.6 | 54.5 | 66.1 | 52.4 | 31.0 | 21.6 | 26.7 | 46.3
Random | 61.2 | 52.4 | 50.1 | 51.1 | 65.5 | 50.1 | 38.3 | 22.2 | 33.5 | 47.2
Ours | 62.8 | 54.6 | 51.6 | 49.3 | 66.1 | 50.8 | 39.1 | 22.6 | 33.7 | 47.8
Table 13: Performance (F1 score) on TydiQA dataset. We examine the results on TydiQA on fine-tuning with English setting. Languages in different embedding sprachbunds are separated with vertical lines.  Languages | en | es | de | el | ru | hi | ar | th | zh | tr | vi | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R | 83.0 | 76.1 | 73.2 | 72.4 | 73.3 | 68.0 | 66.0 | 68.0 | 51.7 | 67.4 | 73.8 | 70.3
XLM-R CT | 83.4 | 75.9 | 73.4 | 72.4 | 73.7 | 69.2 | 65.8 | 68.1 | 52.8 | 66.7 | 74.0 | 70.5
Random | 82.8 | 76.0 | 72.8 | 71.9 | 73.2 | 68.9 | 65.5 | 67.4 | 52.6 | 66.8 | 74.1 | 70.2
Ours | 82.8 | 75.5 | 74.5 | 73.0 | 73.4 | 69.5 | 67.3 | 68.0 | 53.1 | 68.5 | 73.8 | 70.9
Table 14: Performance (F1 score) on XQuAD dataset. We examine the results on XQuAD on fine-tuning with English setting. Languages in different embedding sprachbunds are separated with vertical lines.  Languages | en | de | el | tl | af | nl | ur | fr | pt | es | it | ar | id | ms | hi | fa | bg | ru | sw | fa
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R | 83.0 | 74.3 | 72.5 | 71.4 | 74.6 | 80.4 | 50.1 | 76.9 | 77.9 | 70.8 | 77.2 | 45.1 | 50.1 | 56.4 | 66.1 | 40.7 | 77.3 | 63.6 | 66.9 | 40.7
XLM-R CT | 82.5 | 74.2 | 72.8 | 70.9 | 74.8 | 80.0 | 53.3 | 77.0 | 77.2 | 71.3 | 77.4 | 47.5 | 47.4 | 62.2 | 65.6 | 42.2 | 76.6 | 63.7 | 65.9 | 42.2
Random | 82.7 | 74.2 | 72.6 | 71.2 | 74.7 | 80.2 | 51.7 | 77.0 | 77.6 | 71.0 | 77.3 | 46.3 | 48.7 | 59.3 | 65.9 | 41.4 | 77.0 | 63.7 | 66.4 | 41.4
Ours | 83.1 | 76.4 | 74.2 | 73.1 | 78.2 | 81.8 | 47.2 | 78.7 | 78.2 | 74.5 | 78.4 | 51.1 | 49.6 | 68.7 | 65.2 | 45.4 | 78.0 | 64.1 | 67.0 | 45.4
Languages | mr | et | ja | zh | he | jv | eu | fi | yo | my | hu | ta | te | vi | ml | tr | ko | kk | bn | ka
XLM-R | 59.5 | 72.3 | 18.3 | 24.7 | 52.1 | 58.5 | 60.2 | 75.3 | 41.5 | 51.7 | 76.1 | 53.7 | 47.0 | 65.6 | 61.3 | 73.9 | 49.7 | 44.6 | 66.3 | 65.5
XLM-R CT | 60.9 | 71.7 | 18.1 | 21.9 | 50.8 | 58.2 | 60.1 | 74.5 | 38.3 | 51.2 | 75.2 | 52.9 | 48.2 | 65.7 | 60.7 | 74.5 | 49.0 | 49.6 | 66.2 | 64.7
Random | 60.2 | 72.0 | 18.2 | 23.3 | 51.5 | 58.3 | 60.2 | 74.9 | 39.9 | 51.4 | 75.7 | 53.3 | 47.6 | 65.7 | 61.0 | 74.2 | 49.4 | 47.1 | 66.3 | 65.1
Ours | 62.8 | 73.8 | 19.4 | 21.9 | 53.4 | 63.2 | 65.7 | 76.0 | 51.6 | 56.0 | 76.9 | 55.6 | 55.0 | 66.8 | 65.0 | 77.0 | 53.0 | 50.8 | 70.5 | 67.1
Table 15: Performance (F1 score) on NER dataset. We examine the results on NER
on fine-tuning with English setting. Languages in different embedding
sprachbunds are separated with vertical lines.


# BORM: Bayesian Object Relation Model for Indoor Scene Recognition

Liguang Zhou1,2, Jun Cen2,3, Xingchao Wang1,2, Zhenglong Sun1,2, Tin Lun
Lam1,2,†, Yangsheng Xu1,2 † Corresponding Author: Tin Lun Lam
(tllam@cuhk.edu.cn)This work was supported in part by the funding
AC01202101025 and 2019-INT007 from the Shenzhen Institute of Artificial
Intelligence and Robotics for Society.1 School of Science and Engineering, The
Chinese University of Hong Kong, Shenzhen2 Shenzhen Institute of Artificial
Intelligence and Robotics for Society, The Chinese University of Hong Kong,
Shenzhen.3 Hong Kong University of Science and Technology

###### Abstract

Scene recognition is a fundamental task in robotic perception. For human
beings, scene recognition is reasonable because they have abundant object
knowledge of the real world. The idea of transferring prior object knowledge
from humans to scene recognition is significant but still less exploited. In
this paper, we propose to utilize meaningful object representations for indoor
scene representation. First, we utilize an improved object model (IOM) as a
baseline that enriches the object knowledge by introducing a scene parsing
algorithm pretrained on the ADE20K dataset with rich object categories related
to the indoor scene. To analyze the object co-occurrences and pairwise object
relations, we formulate the IOM from a Bayesian perspective as the Bayesian
object relation model (BORM). Meanwhile, we incorporate the proposed BORM with
the PlacesCNN model as the combined Bayesian object relation model (CBORM) for
scene recognition and significantly outperforms the state-of-the-art methods
on the reduced Places365 dataset, and SUN RGB-D dataset without retraining,
showing the excellent generalization ability of the proposed method. Code can
be found at https://github.com/FreeformRobotics/BORM.

##  I Introduction

Scene understanding is a fundamental cognitive ability for robots to conduct
tasks in an unknown environment. To carry out the task commanded by
controllers, the most fundamental problem is recognizing the environment
robots are traveling in, such as the living room, bedroom, or kitchen.

Recognizing scene about the current environment is quite essential for robots
to make more intelligent behaviors. Fortunately, scene recognition has been an
important research area among computer vision and robotics for decades. There
have been many scene recognition algorithms proposed that using visual
information for scene understanding. Liu et al. [1][2] propose a color and
geometric features based adaptive descriptor for scene recognition. Also,
indoor scenes can be characterized by global spatial features [3]. However,
these methods are based on the traditional lower-level features, which are not
accurate enough, and not interpretable at the semantic level.

Figure 1: The left part of the figure shows the input image and the
corresponding scene parsing result. We have conducted a Bayesian probabilistic
analysis using the BORM on the Places365-14 dataset. The middle part shows the
conditional probability of $P(scene|object)$, and the top 6 scene given the
object “bed” and “lamp” are presented, respectively. The right part shows the
joint conditional probability of $P(scene|(object,object))$, and the top 6
scene given the object pair $(bed,lamp)$. As observed from the figure, the
proposed BORM suppresses the probability of scene-common object pairs while
highlights the probability of the scene-specific object pairs.

Recently, many deep learning based methods have demonstrated impressive
performance over various computer vision tasks. For example, ResNet has been
shown a convincing ability in image classification tasks for years [4].
However, the classification of the image using ResNet is like using a black
box to interpret the image, which is not really in an interpretable manner.
Chen et al. [5] consider the word-embedding model to reformulate the scene
understanding problem in a more semantic meaningful perspective. The output of
ResNet module, object detection module, and scene parsing module have been
encoded in the one global vector for scene understanding. However, the
performance of the word embedding method is just slightly higher than ResNet,
as reported. Besides, there are three streams used for scene recognition,
which is relatively inefficient. The utilization of semantic information for
scene recognition is important. To model the descriptors probabilities, a
Bayesian filtering method is proposed [6], but the object information is not
explicitly used. To use the object information, a object classifier is used to
classify low-level visual features to objects [7], while the relation among
objects is not exploited. To utilize the object relation for scene
recognition, spatial object-to-object relation is studied for RGB-D scene
recognition [8]. Besides, a Long Short-Term Memory modeling method is proposed
to investigate the object relation with ROI selection [9]. To utilize the
object information in the scene, the object model [10] is proposed as
complementary semantic information of the scene combined with ResNet to better
interpret the given scene. However, using only the object vector might not be
discriminative enough for scene understanding [11]. A Object-to-Scene model is
proposed, where the object features and object relation are learned by object
feature aggregation module and object attention module [12], respectively.
Overall, these above-mentioned method lack the statistically analysis of
object distribution in the scene. Specifically, there are some scene-common
objects across various scene and scene-specific objects only appear in the
particular scene. Therefore, we propose the Bayesian object relation model
(BORM) for improving the accuracy of scene recognition by highlighting the
scene-specific objects while preserving the scene-common objects for scene
understanding [13]. However, all of these methods neglect the probabilistic
relation among object pairs, which contains an essential message for scene
recognition.

In this paper, we consider the probabilistic relation between the object pairs
as shown in Fig. 1. We can observed that there is a high probability of the
specific object pair appear in the particular scene, e.g., the bed and lamp is
most likely (75.2%) appeared in the bedroom, and can be regarded as a scene-
specific object pair.

To utilize this conditional object pair relation w.r.t the various scenes, the
BORM that derives the joint conditional probability of the object pairs given
the various scenes. With the BORM, the joint conditional probability of the
scene-specific object pairs will be enhanced and the joint conditional
probability of scene-common object pairs will be suppressed.

In summary, our main contributions of this paper are as follows:

  * •

We utilize an IOM as baseline, based on scene parsing algorithm pretrained on
ADE20K dataset that contains more relevant object classes for indoor scene
representation. Surprisingly, the IOM surpasses the object model over 20.5%
accuracy on average.

  * •

Meanwhile, To conduct an in-depth study of object pair relations, we propose a
Bayesian object relation model (BORM) that enhances the scene-specific object
pair relation and suppresses the scene-common object pair relation in a
Bayesian probability manner for indoor scene representation.

  * •

We combine the proposed BORM and the PlacesCNN model as CBORM, which
significantly outperforms the state-of-the-art methods on the Places365-7 and
Places365-14 dataset over 2.0% and 2.1% accuracy, and SUN RGB-D dataset over
2.0% without retraining the model, showing the excellent generalization
ability of the proposed method.

The rest of the paper is organized as follows. Section II introduces the
related work of scene recognition and object knowledge for indoor scene
representation. Section III describes IOM that pretrained on ADE20K with 150
categories based on the scene parsing algorithm, and the BORM that conduct an
in-depth study of object relation in a Bayesian perspective. In Section IV, we
discuss the PlacesCNN model for scene recognition. Moreover, we combine the
proposed BORM and the PlacesCNN model as the CBORM for scene recognition.
Section V shows the experimental settings and results, and numerical
experiments have been conducted to verify the effectiveness of our proposed
method. Finally, the conclusions and future directions are pointed out in
Section VI.

Figure 2: BORM that based on the baseline IOM is shown in the diagram, where
blue and red arrow shows the flow of the IOM and BORM, respectively.

##  II Related Work

In this section, we review the research works related to our paper in two
aspects: scene recognition, and object knowledge from object detection or
scene parsing algorithm for indoor scene representation. We also discuss the
differences and connections between these related works and our method.

###  II-A Scene Recognition

Scene recognition has been an important research problem in robotics area for
decades, which can be utilized for topological map construction and mobile
robot localization [1][2][14]. Moreover, it has the potential to be applied
for robot to perform efficient recognition of functional areas [15],
identification of the person [16], and execute task accordingly. The early
methods, mainly focus on extraction of local features like color descriptors
[2][17]. To better recognize indoor images, the combination of local and
global features are utilized [3]. However, these methods only captures lower-
level features of the scene while the high-level semantic structures are
difficult to capture [18].

To utilize the high-level semantic information, some methods propose to
leverage the mid-level concepts. e.g., Zhou et al. [19][20] propose a CNN
based classifier to learn deep features for scene recognition on the Places
Dataset. Liao et al. [21] use deep learning with a multi-task training method
that incorporate both semantic segmentation and scene recognition tasks. Zhu
et al. [18] propose a discriminative multi-modal feature fusion framework for
scene recognition. However, these methods neglect the critical object
information for scene recognition.

To incorporate object information, Li et al. [22] represent images by using
objects appearing in them as object bank (OB) method. Brucker et al. [23]
counts the co-occurrence frequencies as potentials in conditional random field
for scene labeling. In DEDUCE [10], one hot object vector is used as
complementary information for scene recognition, where the object information
is separately considered. In context based Word Embeddings [5], there are
three streams for scene recognition, one stream is the scene parsing model
pretrained on ADE20K, the other is ResNet50 pretrained on reduced Places365
dataset, and a Word Vectors Module computes the content of two modules and
refines the results. Song et al. [24] considers spatial object-to-object
relations with the intermediate (object) representations. In Semantic-aware
method [25], the image representation and context information are combined,
where context information consists of scene objects and stuff, and their
relative locations. However, all of these methods have not considered the
probabilistic relation of the object pairs given the various scene. In this
work, we consider the object pair co-occurrences given various scene in a
Bayesian Perspective with proposed Bayesian object relation model (BORM). To
the best of our knowledge, in the scene recognition task, the object relation
modeled in a probabilistic perspective is less exploited.

###  II-B Object Knowledge for Indoor Scene Representation

Recently, object detection is a popular research area in computer vision,
thanks to the emerging of the large-scale labeled data and advanced GPUs. Many
excellent algorithms have been developed for object detection like the one-
stage object detector, YoloV3 [26], SSD [27] and two-stage object detector,
like Joint SSD [28], Faster R-CNN [29], Mask RCNN [30], Cascaded RCNN [31].
The one-stage detector has a higher speed while maintain the similar
performance compared with two-stage detector, therefore, the YoloV3 is adopted
as the part of object model in DEDUCE [10]. To incorporate the object
knowledge for scene understanding, the object detection algorithm is first
pretrained on the public available dataset. There are several mainstream
dataset for object detection. PASCAL VOC [32] contains 20 categories of
objects, which is very limited, and most labels are like car, bus, bicycle,
airplane, and all of these are not relevant to indoor scene representation. To
better represent indoor scene, DEDUCE use the YoloV3 pretrained on the MS COCO
dataset [33], which contains 80 object categories. However, the half of
objects in MS COCO are outdoor objects like giraffe, elephant, which are not
relevant to indoor scene representations.

Scene parsing algorithm is used to segment objects and stuff in the still
image, which has demonstrated surprisingly performance recently [34]. Zhou et
al. [35] propose a scene parsing algorithm to detect a wide range of object
and stuff in the pixel level with ADE20K Dataset, which contains 150 classes
of object knowledge in pixel level. To utilize the rich object knowledge of
the ADE20K Dataset, we adapt the scene parsing model pretrained on the ADE20K
dataset as the improved object model (IOM), which shows a significantly
improvements over the OM pretrained on the MS COCO dataset.

##  III Bayesian Object Relation Model

In Section III, we present the baseline model IOM that uses more object
categories for object representation and the BORM that models the object pair
relation, as shown in Fig. 2. Since the object model proposed in Deduce [10]
only contains information about few categories of objects pretrained on the MS
COCO dataset, which is very limited for indoor scene recognition because most
of the object categories are not relevant to indoor scene representation.
e.g., the object categories of elephant and giraffe from the super-category of
animal, airplane and bus from the super-category of vehicle, traffic light and
stop sign from the super-category of outdoor, and so on. In total, there are
half of the objects can be categorized as outdoor objects. We believe that if
the object model possesses the more rich and relevant object knowledge about
the scene, the better performance of scene recognition can be obtained.
Therefore, we utilize the IOM pretrained on the ADE20K dataset with rich and
relevant object categories as a baseline model for indoor scene
representation. To consider the probabilistic relation of object pairs, we
assume some object pairs are scene-specific, and some object pairs are scene-
common. To this end, we propose a novel BORM that highlights the scene-
specific object pairs while suppresses the scene-common object pairs from a
Bayesian probabilistic perspective.

###  III-A Improved Object Model (IOM)

Different from the basic OM that only have 80 objects information of
environment based on the YoloV3 pretrained on the MS COCO dataset, we present
an IOM based on the scene parsing algorithm pretrained on ADE20K [34] dataset
that convert the output the scene parsing algorithm to an object vector
$\mathbf{X_{iom-150}}$ with 150 dimensions where the 1 means the detected
objects, while 0 means the objects are not in the given image. Compared with
the OM, we now have a new scene representation of 150 dimension
$\mathbf{X_{iom-150}}$. The new scene representation $\mathbf{X_{iom-150}}$
will be fed to a two-layer fully connected network for classification with the
size of 32 and the number of scenes, respectively.

Figure 3: The BORM contains the discriminative value of object pairs. We have
conducted statistical analysis on the Places365-7 dataset. The top figure
shows the $p(scene|(bed,curtain))$, the probability distribution of object
pair (bed, lamp) over the seven indoor scenes, and the standard deviation is
0.30, which means the $(bed,curtain)$ is a scene-specific object pair for
indoor scene recognition. The bottom figure shows the $p(scene|(wall,floor))$,
the probability distribution of object pair $(wall,floor)$ among seven indoor
scenes, which are almost the same. The standard deviation is 0.01, which
indicates the $(wall,floor)$ is a scene-common object pair for indoor scene
recognition, because this object pair appears in every scene equally.

###  III-B Bayesian Co-occurrence Probability Analysis

Figure 4: The Bayesian object relation matrix of first 20 objects. e.g., the
$(bed,curtain)$ is the scene-specific object pair, in which will mostly
leading to the bedroom, while $(wall,floor)$ forms the scene-common object
pair because it is appear in everywhere with almost same probability.

Instead of using one hot object vector as the feature representations for the
scene, which lacks the probabilistic relation between different object pairs,
we propose a novel BORM method that measures the probabilistic relation of
object pairs in a Bayesian perspective. The scene recognition problem is
shaped by the fact that a few object pairs are scene-common, but most object
pairs are scene-specific. As observed in Fig. 3, the probability of scene-
specific object pair $(bed,curtain)$ and scene-common object pair
$(wall,floor)$ are presented.

The scene-specific object pairs indicate the object pairs that have a high
probability at a particular scene and have low probability at others. e.g.,
the object pairs like $(bed,curtain)$ tend to be scene-specific, which means
their probability $p(scene|(bed,curtain))$ can be quite distinctive among the
various scene categories and has the highest probability that appears in the
bedroom. Therefore, they have a large standard deviation (0.31). In contrast,
scene-common object pairs distributed at various scenes with a similar
probability. e.g., the common object pairs like $(wall,floor)$ frequently
appear in many different scenes, which means their joint conditional
probability $p(scene|(wall,floor))$ is quite similar across various scenes,
i.e., they have an extremely small standard deviation (0.01) compared with
those scene-specific object pairs.

Figure 5: The proposed combined Bayesian object relation model (CBORM)
contains two streams. The first stream with the red arrow is the proposed BORM
that utilizes the scene parsing algorithm for scene segmentation. The second
stream with the black arrow is the PlacesCNN model for feature extraction with
the ResNet backbone network. The output of the BORM will first be converted to
a discriminative object feature. Meanwhile, the results of the BORM and
feature extraction result of PlacesCNN will be concatenated as one combined
scene feature and fed to the two-layer FC network for scene classification.

We adopt a pipeline to estimate the posterior probability
$P(c_{j}|o_{h},o_{i})$. First, we use the Places365-7 and Places365-14 dataset
for learning the BORM statistically with only the training set, while testing
on the SUN RGB-D dataset, where the probability distribution is regarded
similar to the Places365-7 dataset.

Given a set of images $I_{c_{j}}$ from a scene category $c_{j}$, the
conditional probability of object $o_{i}$ appears on the scene $c_{j}$ is:

$\begin{split}P\left(o_{i}|c_{j}\right)=N_{o_{i}}/N_{I_{c_{j}}}\\\
i\in[1,N_{objs}],j\in[1,N_{scenes}]\end{split}$ (1)

where $N_{o_{i}}$ is the total number of i-th object $o_{i}$ appears in
$I_{c_{j}}$ and $N_{I_{c_{j}}}$ is total number of images of $I_{c_{j}}$.
Meanwhile, where $N_{objs}$, and $N_{scenes}$ represent the number of objects
in the pretrained model and number of scene categories we have in the dataset
respectively. Assume the statistical independence of each object, we obtain
the joint conditional probability $P\left(o_{h},o_{i}|c_{j}\right)$ of $o_{h}$
and $o_{i}$ appear in the scene $c_{j}$:

$\begin{split}P\left(o_{h},o_{i}|c_{j}\right)=P\left(o_{h}|c_{j}\right)P\left(o_{i}|c_{j}\right)\\\
h,i\in[1,N_{objs}],j\in[1,N_{scenes}]\end{split}$ (2)

The $P(c_{j}|o_{h},o_{i})$, the posterior probability of scene class $c_{j}$
given an object pair ($o_{h},o_{i}$), can be derived by the Bayes Rule and Law
of Total Probability.

$\begin{split}P\left(c_{j}|o_{h},o_{i}\right)&=\frac{P\left(o_{h},o_{i}|c_{j}\right)P\left(c_{j}\right)}{P(o_{h},o_{i})}\\\
&=\frac{P\left(o_{h},o_{i}|c_{j}\right)P\left(c_{j}\right)}{\sum_{j}P\left(o_{h},o_{i}|c_{j}\right)P\left(c_{j}\right)}\\\
\end{split}$ (3)

where the $P(c_{j})$ is the probability of scene $c_{j}$ in the dataset, and
$\sum_{j=1}^{N_{scenes}}P(c_{j})=1$. We construct the posterior probability
matrix $P(c_{j}|o,o)$ by calculating relation of each object pairs:

$\scriptsize
P\left(c_{j}|o,o\right)=\begin{bmatrix}P\left(c_{j}|o_{1},o_{1}\right)&\cdots&P\left(c_{j}|o_{1},o_{k}\right)&\cdots&P\left(c_{j}|o_{1},o_{n}\right)\\\
\vdots&&\vdots&&\vdots\\\
P\left(c_{j}|o_{j},o_{1}\right)&\cdots&P\left(c_{j}|o_{j},o_{k}\right)&\cdots&P\left(c_{j}|o_{j},o_{n}\right)\\\
\vdots&&&&\vdots\\\
P\left(c_{j}|o_{n},o_{1}\right)&\cdots&P\left(c_{j}|o_{n},o_{k}\right)&\cdots&P\left(c_{j}|o_{n},o_{n}\right)\end{bmatrix}$
(4)

First, the posterior probability $P(c_{j}|o_{h},o_{i})$ is calculated. To
calculate the discriminative value of the object pairs, the standard deviation
is applied to posterior probabilities among scene categories, denoted as
$std(P(c_{j}|o_{h},o_{i}))$. The discriminative value of the posterior
probabilities among scene categories is defined as $dis(o_{h},o_{i})$:

$\operatorname{dis}\left(o_{h},o_{i}\right)=std_{c\in
1,\ldots,N_{scenes}}(P(c_{j}|o_{h},o_{i}))$ (5)

The $dis(o_{h},o_{i})$ is the discriminative value for measuring the
discriminalibility of the object pair to the scene categories. Instead of
using object vector as the feature representations for the scene, which
dismiss the relationship between different objects, we first construct an
object relation matrix
$\mathbf{X_{orm}}=\mathbf{X_{iom}}\mathbf{X_{iom}^{\mathrm{T}}}$ for
representing the given scene. The value of $\mathbf{X_{orm}}$ will be replaced
by the discriminative value $dis(o_{h},o_{i})$ at the same position.
Therefore, the Bayesian object relation matrix is obtained by element-wise
matrix multiplication,
$\mathbf{X_{borm}}=\mathbf{X_{orm}}*\mathbf{dis(o_{h},o_{i})}$, with the size
of 150x150. After that, the matrix will be stretched to a one dimensional
vector with the size of 22500x1, and will be fed into the three-layer fully
connected network with the size of 8192, 2048, and number of scene classes.

As shown in Fig. 4, the discriminative matrix of the first 20 object pairs are
displayed. The $(bed,curtain)$ is a scene-specific object pair that mostly
appear in the bedroom, and they form a scene-specific object pair with a
discriminative value of 0.30. The $(wall,floor)$ is a common object pair
appear everywhere and have an extremely small discriminative value of 0.01,
which means they form a scene-common object pair.

##  IV CBORM Model

###  IV-A PlacesCNN Model

In order to obtain the scene representation, we use the PlacesCNN model [20]
with the base architecture ResNet [4] as a backbone network, which is
pretrained on the ImageNet [36] dataset. There are two versions of ResNet
according to the settings in DEDUCE [10] and Word2Vec [5]. ResNet18 is used
for Places365-7 and SUN RGB-D dataset, while ResNet50 is used for Places365-14
dataset. Specifically, we preserve the output of ResNet18 with 512 dimensions
or ResNet50 with 2048 dimensions, namely $F_{scene}$, the feature of the
PlacesCNN model for the scene representation.

###  IV-B Combined Model

The Bayesian object relation matrix of the BORM is with the size of 150x150,
which is first stretched to a vector of size 22500x1. The vector of BORM will
be fed into the two fully connected layers and an discriminative object
feature $F_{BORM}$ with 512 dimensions (for Places365-7) or 2048 (For
Places365-14) will be the feature representation of BORM.

Meanwhile, we combine the PlacesCNN model of the ResNet backbone with BORM, as
the combined Bayesian object relation model (CBORM).

###  IV-C Combined Classifier

For the Places365-7 dataset, there are two streams. To fair compare with the
object model [10], one stream is based on ResNet18 pretrained on the Places365
and finetuned on the Places365-7 dataset. The other stream is BORM. The
feature representations of ResNet18 and BORM will be concatenated and fed into
a two-layer FC network with a dimension of 512, and 7 respectively.

Similarly, for the Places365-14 dataset, there are two streams. To fair
compare with the word-embedding [5] model, while one stream is ResNet50, and
the other is BORM. The feature representations of ResNet50 and BORM will be
fed into a two-layer FC network with a dimension of 512, and 14 respectively.

##  V Experimental Results

###  V-A Experimental Settings

We evaluated the proposed models on reduced SUN RGB-D and Places365 dataset.
To be noticed, aim to investigate the generalizability of our model, we
evaluate our model pretrained on Places365-7 dataset on the SUN RGB-D dataset
without retraining it. We’ll introduce the implementation details and training
procedure and different experiment settings.

####  V-A1 Implementation Details

For the PlacesCNN model, ResNet18 or ResNet50 architecture is adopted in our
experiment for ablation study. The optimizer used is the Stochastic Gradient
Descent (SGD) with an initial learning rate of 0.01, the momentum of 0.9, and
the weight decay of 0.0001. We decrease the learning rate 10 times every 10
epoch, and every time when updating the learning rate, we reload the
parameters which have the best accuracy before this timestamp. The total
number of epoch during training is 40. To be noticed, we use the training sets
of Places365-7 and Places365-14 dataset for learning the BORM statistically,
while testing on the SUN RGB-D dataset, where the BORM is the same as the
Places365-7 dataset.

####  V-A2 Dataset Settings

TABLE I: Dataset split setting, where the number of training set and number of test set are listed below. Dataset | Training | Test
---|---|---
Places365-7 | 35000 | 701
Places365-14 | 75000 | 1500
SUN RGB-D | 35000(from Places365-7) | 2077

Places365 Dataset: In this paper, we use the reduced Places365 [20] dataset to
test our methods, since it is the most largest and challenging scene
classification dataset yet, and it contains broad categories in the indoor
environment. In the experiment, we only consider the indoor scene recognition.
There are two different settings on the reduced Places365 dataset. The one is
Places365 with 7 classes includes Corridor, Dinning Room, Kitchen, Living
Room, Bedroom, Office, and Bathroom, denoted as Places365-7. The test set
setting is the same as the official dataset and described in [10]. In
addition, we use the reduced Places365 with 14 indoor scenes in Home
environment includes Wet bar, Home theater, Balcony, Closet, Kitchen, Bedroom,
Playroom, Laundromat, Bathroom, Living Room, Home office, Dining room,
Staircase, and Garage denoted as Places365-14. The dataset splitting follows
the same setting as described in [5]. The dataset splition can be seen from
Table. I.

SUN RGB-D Dataset: SUN RGB-D dataset [37] is a challenging dataset for scene
understanding that contains not only RGB images but also depth information of
each image. It contains 3784 images collected by Kinect V2 and 1159 collected
by Intel RealSense. Moreover, it incorporates 1449 images from the NYUDepth V2
[38], and 554 images from the Berkeley B3DO Dataset [39], both captured by
Kinect V1. Finally, it takes 3389 manually selected distinguished frames
without significant motion blur from the SUN3D videos [40] captured by Asus
Xtion.

In our experiment, we mainly consider the indoor environment understanding.
Therefore, we use the reduced SUN RGB-D dataset includes Office, Kitchen,
Bedroom, Corridor, Bathroom, Living room, and Dining room, where the test set
split is the same as the official dataset. There are 3741 RGB images in total
for testing. And we test our model pretrained on the Place365-7 on SUN RGB-D
dataset without retraining.

###  V-B Experimental Results

####  V-B1 Effect of Object Knowledge

As illustrated in Fig. 6, a group of ablation studies have been conducted for
evaluating the effect of object knowledge to indoor scene recognition on the
Places365-14, Places365-7, and SUN RGB-D dataset, respectively. The x-axis
represent the number of object information IOM have about the indoor scene and
is added by 20 from 90 to 150 sequentially selected from the vector. Plus,
IOM-80 is the baseline accuracy. Obviously, as the number of object
information increases, the much better scene recognition accuracy is achieved,
e.g., on the Places365-14 dataset, the IOM-150 reaches 74.1% accuracy, which
is 10.0% higher than IOM-80. This improvement shows the number of object
information is proportional to scene recognition accuracy. Moreover, the
comparison experiments between the IOM and OM on three datasets, shows an
average of 20.5% improvements can be achieved. After analyzing the object
categories of OM pretrained on the MS COCO, we observed there are only half of
the object categories are related to indoor scenes. In contrast, the other
half is related to outdoor scenes. Therefore, the relevance of object
categories of object model with the scenes is essential for scene recognition,
e.g., the information of elephant and giraffe in OM will not be valuable for
indoor scene recognition. Similarly, the bus and train are not beneficial to
indoor scene recognition.

####  V-B2 Analysis of BORM

Figure 6: Ablation study of improved object model (IOM) with different number
of object knowledge ranging from 80 to 150 categories as shown in horizontal
axis. The vertical axis shows the accuracy on percentage.

We conduct experiments for BORM and IOM in the reduced Places365-7 dataset,
and results are displayed in Table III. The experiment results show the BORM
and IOM model has an advantage over the OM and yields an average accuracy of
83.1% and 82.4%, surpassing the OM model about 20% accuracy. The experiment
result proves that with more object knowledge about the surrounding
environment, the greater scene recognition accuracy can be reached. Then, we
test the model pretrained on the Places365-7 dataset on the SUN RGB-D test
set, and similar conclusion can be drawn. Moreover, the BORM outperforms the
IOM with 0.7% and 1.1% accuracy on the Places365-7 and SUN RGB-D dataset,
respectively, which validates the knowledge of the co-occurrences between
object pairs and their probabilistic relation forms an important indoor scene
representation.

Similarly, in the Table III, we have conducted experiments on the reduced
Places365-14 dataset, and experiment results show the IOM and BORM
tremendously improves the performance over OM with 27% accuracy. The results
suggest the effectiveness of BORM and IOM over the OM especially when the
number of scene classes of dataset is large.

####  V-B3 Performance Comparison

TABLE II: Comparison with the state-of-the-art methods on the reduced Places365-7 Dataset and SUN dataset of scene recognition accuracy Method | Config | Acc(Places365-7) | Acc(SUN)
---|---|---|---
PlacesCNN [20] | ResNet18 | 80.4 | 63.3
ResNet50 | 82.7 | 67.2
Deduce [10] |  $\Phi_{obj}$ (OM) | 62.6 | 53.6
$\Phi_{scene}$ | 87.3 | 66.8
$\Phi_{comb.}$ | 88.1 | 70.1
Ours | IOM | 82.4 | 68.1
BORM | 83.1 | 69.2
2-4 | CBORM | 90.1 | 72.1%

TABLE III: Comparison with the state-of-the-art methods on the reduced Places365-14 Dataset of scene recognition accuracy, the * indicates the re-implement of the method.  Method | Config | Acc
---|---|---
PlacesCNN [20] | ResNet18 | 76.0
ResNet50 | 80.0
Word2Vec [5] | ResNet50+Word2Vec | 83.7
*Deduce [10] |  $\Phi_{obj}$ (OM) | 47.0
Ours | IOM | 74.1
BORM | 74.9
2-3 | CBORM | 85.8

As shown in Table III, we conduct the ablation study of using only the BORM
model and the CBORM model. We found the CBORM yield an average accuracy of
90.1%, which greatly outperforms the ResNet18 and ResNet50 of PlacesCNN
baselines about 10% and 8% respectively. Meanwhile, CBORM outperforms the BORM
with 7% accuracy and 3% on the Places365-7 and SUN RGB-D dataset,
respectively.

In comparison with the state of the art, Table III shows that the CBORM
improves the scene recognition by 2.0% in the reduce Places365-7 dataset.
Also, as shown in Table III, the CBORM improve the recognition accuracy by
2.1% in the reduced Places365-14 dataset. Both results show the combined model
achieves comparable results to some recent approaches that use the word-
embedding method to extract the semantic meaning of the environment, or use
the combination of scene and object representations for better scene
understanding. Moreover, Table III shows the performance of our method over
the method in [10] with 2%, showing the excellent generalization ability of
CBORM over other methods on the Reduced SUN RGB-D dataset.

These results demonstrate that CBORM is successful in recognizing the scene
images with a competitive accuracy. This improved effectiveness of CBORM over
the state-of-the-art justifies our reasonable assumption that relation of
object pairs is an essential complementary information for indoor scene
recognition.

##  VI Conclusion and Future Work

In this paper, we aim to transfer object knowledge from humans to indoor scene
recognition. First, we propose the IOM that with rich and relevant object
categories for indoor scene representation. Besides, inspired by the nature of
scene-common and scene-specific object pairs, we establish a BORM that obtains
the probabilistic relations among object pairs given various scene, where the
probability of scene-specific object pairs will be enhanced and the
probability of scene-common object pairs will be suppressed. Moreover, we
utilize the PlacesCNN model with ResNet as a backbone network for
classification, which demonstrates a substantial result on the scene
understanding. Hence, we combine the PlacesCNN and proposed BORM as CBORM for
a more interpretable scene recognition algorithm, and experiment results show
our proposed method significantly outperforms the state-of-the-art methods.

In the future, we plan to integrate our algorithm to real robots like mobile
robots, and flying robots for the construction of the semantic map includes
the label of the scene and detailed semantic information of the environment.
The construction of a semantic map using the proposed scene recognition
algorithm would be useful for navigation in unknown places for robots and
humans because both the semantic label of the region and semantic meaning of
the environment are provided. Furthermore, our system could be applied to
autonomous robots and enabling them to assist humans in safety and rescue
missions inside a house or a building.

## References

  * [1] M. Liu, D. Scaramuzza, C. Pradalier, R. Siegwart, and Q. Chen, “Scene recognition with omnidirectional vision for topological map using lightweight adaptive descriptors,” in _2009 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2009, pp. 116–121.
  * [2] M. Liu and R. Siegwart, “Topological mapping and scene recognition with lightweight color descriptors for an omnidirectional camera,” _IEEE Transactions on Robotics_ , vol. 30, no. 2, pp. 310–324, 2013.
  * [3] A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 413–420.
  * [4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [5] B. X. Chen, R. Sahdev, D. Wu, X. Zhao, M. Papagelis, and J. K. Tsotsos, “Scene classification in indoor environments for robots using context based word embeddings,” in _2018 IEEE International Conference on Robotics and Automation (ICRA) Workshop_ , 2018.
  * [6] J. Wu, H. I. Christensen, and J. M. Rehg, “Visual place categorization: Problem, dataset, and algorithm,” in _2009 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2009, pp. 4763–4770.
  * [7] P. Espinace, T. Kollar, A. Soto, and N. Roy, “Indoor scene recognition through object detection,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 1406–1413.
  * [8] X. Song, S. Jiang, B. Wang, C. Chen, and G. Chen, “Image representations with spatial object-to-object relations for rgb-d scene recognition,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 525–537, 2019.
  * [9] C. Laranjeira, A. Lacerda, and E. R. Nascimento, “On modeling context from objects with a long short-term memory for indoor scene recognition,” in _2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_. IEEE, 2019, pp. 249–256.
  * [10] A. Pal, C. Nieto-Granda, and H. I. Christensen, “Deduce: Diverse scene detection methods in unseen challenging environments,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 4198–4204.
  * [11] H. Zeng, X. Song, G. Chen, and S. Jiang, “Learning scene attribute for scene recognition,” _IEEE Transactions on Multimedia_ , 2019.
  * [12] B. Miao, L. Zhou, A. Mian, T. L. Lam, and Y. Xu, “Object-to-scene: Learning to transfer object knowledge to indoor scene recognition,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021.
  * [13] X. Cheng, J. Lu, J. Feng, B. Yuan, and J. Zhou, “Scene recognition with objectness,” _Pattern Recognition_ , vol. 74, pp. 474–487, 2018.
  * [14] H.-Y. Lin, C.-W. Yao, K.-S. Cheng _et al._ , “Topological map construction and scene recognition for vehicle localization,” _Autonomous Robots_ , vol. 42, no. 1, pp. 65–81, 2018.
  * [15] C. Ye, Y. Yang, R. Mao, C. Fermüller, and Y. Aloimonos, “What can i do around here? deep functional scene understanding for cognitive robots,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 4604–4611.
  * [16] Z. Wang, Z. Wang, Y. Zheng, Y.-Y. Chuang, and S. Satoh, “Learning to reduce dual-level discrepancy for infrared-visible person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 618–626.
  * [17] K. Van De Sande, T. Gevers, and C. Snoek, “Evaluating color descriptors for object and scene recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 32, no. 9, pp. 1582–1596, 2009.
  * [18] H. Zhu, J.-B. Weibel, and S. Lu, “Discriminative multi-modal feature fusion for rgbd indoor scene recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2969–2976.
  * [19] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in _Advances in neural information processing systems_ , 2014, pp. 487–495.
  * [20] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * [21] Y. Liao, S. Kodagoda, Y. Wang, L. Shi, and Y. Liu, “Understand scene categories by objects: A semantic regularized scene classifier using convolutional neural networks,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 2318–2325.
  * [22] L.-J. Li, H. Su, Y. Lim, and L. Fei-Fei, “Object bank: An object-level image representation for high-level visual recognition,” _International journal of computer vision_ , vol. 107, no. 1, pp. 20–39, 2014.
  * [23] M. Brucker, M. Durner, R. Ambruş, Z. C. Márton, A. Wendt, P. Jensfelt, K. O. Arras, and R. Triebel, “Semantic labeling of indoor environments from 3d rgb maps,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1871–1878.
  * [24] X. Song, C. Chen, and S. Jiang, “Rgb-d scene recognition with object-to-object relation,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 600–608.
  * [25] A. López-Cifuentes, M. Escudero-Viñolo, J. Bescós, and Á. García-Martín, “Semantic-aware scene recognition,” _Pattern Recognition_ , vol. 102, p. 107256, 2020.
  * [26] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [27] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [28] C. Yi, L. Zhou, Z. Wang, Z. Sun, and C. Tan, “Long-range hand gesture recognition with joint ssd network,” in _2018 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , 2018, pp. 1959–1963.
  * [29] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [30] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [31] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6154–6162.
  * [32] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [33] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [34] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 633–641.
  * [35] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision_ , vol. 127, no. 3, pp. 302–321, 2019.
  * [36] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255.
  * [37] S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 567–576.
  * [38] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _European conference on computer vision_. Springer, 2012, pp. 746–760.
  * [39] A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, K. Saenko, and T. Darrell, “A category-level 3d object dataset: Putting the kinect to work,” in _Consumer depth cameras for computer vision_. Springer, 2013, pp. 141–165.
  * [40] J. Xiao, A. Owens, and A. Torralba, “Sun3d: A database of big spaces reconstructed using sfm and object labels,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 1625–1632.