# Multi-lingual Common Semantic Space Construction via Cluster-consistent Word
Embedding

Lifu Huang1, Kyunghyun Cho2, Boliang Zhang1, Heng Ji1, Kevin Knight3
1 Rensselaer Polytechnic Institute, 2 New York University
3 Information Sciences Institute, University of Southern California
1{huangl7, zhangb8<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We construct a multilingual common semantic space based on distributional
semantics, where words from multiple languages are projected into a shared
space to enable knowledge and resource transfer across languages. Beyond word
alignment, we introduce multiple cluster-level alignments and enforce the word
clusters to be consistently distributed across multiple languages. We exploit
three signals for clustering: (1) neighbor words in the monolingual word
embedding space; (2) character-level information; and (3) linguistic
properties (e.g., apposition, locative suffix) derived from linguistic
structure knowledge bases available for thousands of languages. We introduce a
new cluster-consistent correlational neural network to construct the common
semantic space by aligning words as well as clusters. Intrinsic evaluation on
monolingual and multilingual QVEC tasks shows our approach achieves
significantly higher correlation with linguistic features than state-of-the-
art multi-lingual embedding learning methods do. Using low-resource language
name tagging as a case study for extrinsic evaluation, our approach achieves
up to 24.5% absolute F-score gain over the state of the art.

##  1 Introduction

More than 3,000 languages have electronic record, e.g., at least a portion of
the Christian Bible had been translated into 2,508 different languages.
However, the training data for mainstream natural language processing (NLP)
tasks such as information extraction and machine translation is only available
for dozens of dominant languages. In this paper we aim to construct a
multilingual common semantic space where words in multiple languages are
mapped into a distributed, language-agnostic semantic continuous space, so
that resources and knowledge can be shared across languages.

Words can be clustered through explicit (e.g., sharing affixes of certain
linguistic functions) or implicit clues (e.g., sharing neighbors from
monolingual word embedding). We hypothesize that the distribution of such
clusters should be consistent across multiple languages. We achieve this
cluster-level consistency by aligning word clusters across languages. Based on
this intuition we propose to create clusters through three kinds of signals as
follows, without any extra human annotation effort. Then we aggregate the
embedding vectors of words in each cluster and ensure that the clusters (or
the words therein) are consistent across multiple languages.

Neighbor based clustering and alignment. We build our common space based on
correlational neural network (CorrNet) which is an extension of autoencoder
framework by enabling cross-lingual reconstruction. In contrast to previous
work Chandar et al. (2016); Rajendran et al. (2015), we extend CorrNet to
neighbor-consistent correlation network by using each word’s neighbors (the
nearest words within monolingual semantic space) to ensure that the cross-
lingual mapping from and to the common semantic space is locally smooth. For
instance, the neighboring words of China in English (Japan, India and Taiwan)
should be close to the neighboring words of Cina in Italian (Beijing, Korea,
Japan) in the common semantic space. In other words, we encourage the
consistency of neighborhoods across multiple languages.

Character based clustering and alignment. Many related languages share very
similar character set, and many words that refer to the same concept share
similar compositional characters or patterns, e.g., China (English), Kina
(Danish), and Cina (Italian).

Linguistic property based clustering and alignment. Many languages also share
linguistic properties, e.g., apposition, conjunction, and plural suffix
(English (-s / -es), Turkish (-lar / -ler), Somali (-o)). Linguists have
created a wide variety of linguistic property knowledge bases, which are
readily available for thousands of languages. For example, the CLDR (Unicode
Common Locale Data Repository) includes closed word classes and affixes
indicating various linguistic properties. We propose to take advantage of
these language-universal resources to create clusters, where the words within
one cluster share the same linguistic property, and build alignment between
clusters for common semantic space construction.

We evaluate our approach on monolingual and multilingual QVEC Tsvetkov et al.
(2015) tasks, as well as an extrinsic evaluation on name tagging for low-
resource languages. Experiments demonstrate that our framework is effective at
capturing linguistic properties and significantly outperforms state-of-the-art
multi-lingual embedding learning methods.

##  2 Related Work

Multilingual word embeddings have advanced many multilingual natural language
processing tasks, such as machine translation Zou et al. (2013); Mikolov et
al. (2013); Madhyastha and España-Bonet (2017), dependency parsing Guo et al.
(2015); Ammar et al. (2016a), and name tagging Zhang et al. ; Tsai and Roth
(2016). Using bilingual aligned words, previous methods project multiple
monolingual embeddings into a shared semantic space using linear mappings
Mikolov et al. (2013); Rothe et al. (2016); Zhang et al. (2016); MarcoBaroni
(2015); Xing et al. (2015) or canonical correlation analysis (CCA) Ammar et
al. (2016b); Faruqui and Dyer (2014); Lu et al. (2015). Compared with CCA,
which only optimizes the correlation for each individual pair of languages,
linear mapping based methods can jointly optimize all the languages in the
common semantic space. We focus on learning linear mappings to construct the
common semantic space and adopt correlational neural networks Chandar et al.
(2016); Rajendran et al. (2015) as the basic model. In contrast to previous
work which only exploited monolingual word semantics, we introduce multiple
cluster-level alignments.

Beyond word alignment, another branch of approaches for multilingual word
embeddings are based on parallel or comparable data, such as parallel
sentences AP Chandar et al. (2014); Gouws et al. (2015); Luong et al. (2015);
Hermann and Blunsom (2014); Schwenk et al. (2017), phrase translations Duong
et al. (2016) and comparable documents Vulic and Moens (2015). Moreover, to
reduce the need of bilingual alignment, several approaches have been designed
to learn cross-lingual embeddings based on a small seed dictionary Vulic and
Korhonen (2016); Zhang et al. (2016); Artetxe et al. (2017), or even with no
supervision Cao et al. (2016); Zhang et al. (2017b, a); Conneau et al. (2017).
However, such methods are still limited to bilingual word embedding learning
and remaining to be explored for common semantic space construction.

##  3 Approach

###  3.1 Overview

Figure 1: Architecture Overview. In each monolingual semantic space, the words
within solid rectangle denote a neighbor based cluster and the words within
dotted rectangle denote a linguistic property based cluster.

Figure 1 shows the overview of our neural architecture. We project all
monolingual word embeddings into a common semantic space based on word-level
as well as cluster-level alignments and learn the transformation functions.
First, on word-level, we build a neighborhood-consistent CorrNet to augment
word representations with neighbor based clusters and align them in the common
semantic space. In addition, we apply a language-independent convolutional
neural networks to compose character-level word representation and concatenate
it with word representation in the common semantic space. Finally, we
construct clusters based on linguistic properties, such as closed word classes
and affixes, and align them in the common semantic space. We jointly optimize
for all the alignments in the common semantic space for each pair of
languages.

###  3.2 Basic Model

We briefly describe the basic model for learning the common semantic space:
correlational neural networks (CorrNets) Chandar et al. (2016); Rajendran et
al. (2015). CorrNets have been widely adopted for learning multilingual or
multi-view representations. It combines the advantages of canonical
correlation analysis (CCA) and autoencoder (AE).

Given the bilingual aligned word pairs between two languages $l_{1}$ and
$l_{2}$, we first use their monolingual word embeddings to initialize each
word with a vector and obtain $M_{l_{1}}\in\mathbb{R}^{|V_{l_{1}}|\times
d_{l_{1}}}$ and $M_{l_{2}}\in\mathbb{R}^{|V_{l_{2}}|\times d_{l_{2}}}$, where
$V_{l_{1}}$ and $V_{l_{2}}$ are the bilingual dictionary of $l_{1}$ and
$l_{2}$. $V_{l_{1}}^{i}$ is the translation of $V_{l_{2}}^{i}$, and
$d_{l_{1}}$ and $d_{l_{2}}$ are the vector dimensionalities. Then for each
language we learn a linear projection function to project $M_{l_{1}}$ and
$M_{l_{2}}$ into the common semantic space:

$H_{l_{1}}=\sigma(M_{l_{1}}\cdot W_{l_{1}}+b_{l_{1}})\ ,$
$H_{l_{2}}=\sigma(M_{l_{2}}\cdot W_{l_{2}}+b_{l_{2}})\ ,$

where $H_{l_{1}}\in\mathbb{R}^{V_{l_{1}}\times h}$ and
$H_{l_{2}}\in\mathbb{R}^{V_{l_{2}}\times h}$ are the vector representations
for $V_{l_{1}}$ and $V_{l_{2}}$ in the common semantic space respectively. $h$
is the vector dimensionality in the shared semantic space.
$W_{l_{1}}\in\mathbb{R}^{V_{l_{1}}\times h}$ and
$W_{l_{2}}\in\mathbb{R}^{V_{l_{2}}\times h}$ are the transformation matrices,
and $b_{l_{1}}$ and $b_{l_{2}}$ are the bias vectors. $\sigma$ denotes Sigmoid
function.

After we project the monolingual embeddings into the common semantic space, we
further reconstruct $M_{l_{1}}$ and $M_{l_{2}}$ from $H_{l_{1}}$ and
$H_{l_{2}}$ separately:

$M^{{}^{\prime}}_{l_{1}}=\sigma(H_{l_{1}}\cdot
W^{\top}_{l_{1}}+b^{{}^{\prime}}_{l_{1}})\ ,$
$M^{*}_{l_{1}}=\sigma(H_{l_{2}}\cdot
W^{\top}_{l_{1}}+b^{{}^{\prime}}_{l_{1}})\ ,$
$M^{{}^{\prime}}_{l_{2}}=\sigma(H_{l_{2}}\cdot
W^{\top}_{l_{2}}+b^{{}^{\prime}}_{l_{2}})\ ,$
$M^{*}_{l_{2}}=\sigma(H_{l_{1}}\cdot
W^{\top}_{l_{2}}+b^{{}^{\prime}}_{l_{2}})\ ,$

where $b^{{}^{\prime}}_{l_{1}}$, $b^{{}^{\prime}}_{l_{2}}$ are the bias
vectors. $M^{{}^{\prime}}_{l_{1}}$ and $M^{{}^{\prime}}_{l_{2}}$ are the
monolingual reconstructions of $M_{l_{1}}$ and $M_{l_{2}}$ from the common
space, and $M^{*}_{l_{1}}$ and $M^{*}_{l_{2}}$ are cross-lingual
reconstructions. $W_{l_{1}}^{\top}$ and $W_{l_{2}}^{\top}$ are the transposes
of $W_{l_{1}}$ and $W_{l_{2}}$ respectively.

To learn the common semantic space, we minimize the distance between the
aligned word vectors as well as the loss of monolingual and cross-lingual
reconstruction:

$O_{W}=\sum_{\\{l_{i},l_{j}\\}\in
A}L(M^{{}^{\prime}}_{l_{i}},M_{l_{i}})+L(M^{*}_{l_{i}},M_{l_{i}})\\\
+L(M^{{}^{\prime}}_{l_{j}},M_{l_{j}})+L(M^{*}_{l_{j}},M_{l_{j}})+L(H_{l_{i}},H_{l_{j}})\
,$

where $l$ denotes any specific language that we want to project into the
common semantic space, $A$ denotes all bilingual dictionaries, and $L$ denotes
a similarity metric. In our work, we use cosine similarity as the similarity
metric.

###  3.3 Neighborhood-Consistent CorrNet

CorrNet can project multiple monolingual word embeddings into a common
semantic space using bilingual word alignment. However, the same concepts may
have different semantic bias in various languages. For example, the top five
nearest words of the concept “China” are: (Japan, India, Taiwan, Chinese,
Asia) in English, (Cosco, Shenzhen, Australian, Shanghai, manufacturing) in
Danish, and (Beijing, Korea, Japan, aluminum, copper) in Italian respectively.
The neighboring words can reflect the semantic meanings of each concept within
each semantic space. In order to ensure the consistency of the neighborhoods
within the common semantic space and make the cross-lingual mapping locally
smooth, we propose to augment monolingual word representation with its top-$N$
nearest neighboring words from the original monolingual semantic space.111We
set $N$ as 5 in our experiments.

Given the monolingual embeddings of the bilingual aligned words for two
languages $l_{1}$ and $l_{2}$, $M_{l_{1}}$ and $M_{l_{2}}$, for each word, we
extract the top-$N$ nearest neighbors and construct the neighborhood clusters.
Each cluster $t_{l}=\\{w_{1},w_{2},...,w_{|t_{l}|}\\}$ in language $l$ is
represented by

$c_{t_{l}}=\frac{1}{|t_{l}|}\sum_{w\in t_{l}}E_{w}\ ,$

where $E_{w}$ denotes the monolingual word embedding for $w$.

We obtain all the neighborhood cluster vector representations $C_{l_{1}}$,
$C_{l_{2}}$ for $l_{1}$ and $l_{2}$. We incorporate these neighborhood cluster
information into the common semantic space when projecting monolingual
embeddings:

$\displaystyle H_{l_{1}}=\sigma(M_{l_{1}}\cdot W_{l_{1}}+C_{l_{1}}\cdot
U_{l_{1}}+b_{l_{1}}),$ $\displaystyle H_{l_{2}}=\sigma(M_{l_{2}}\cdot
W_{l_{2}}+C_{l_{2}}\cdot U_{l_{2}}+b_{l_{2}}),$ (1)

Besides the monolingual and cross-lingual reconstructions for $M_{l_{1}}$ and
$M_{l_{2}}$ in CorrNets, we also add monolingual and cross-lingual
reconstructions for the neighborhood clusters:

$C^{{}^{\prime}}_{l_{1}}=\sigma(H_{l_{1}}\cdot
U^{\top}_{l_{1}}+b^{*}_{l_{1}})\ ,$ $C^{*}_{l_{1}}=\sigma(H_{l_{2}}\cdot
U^{\top}_{l_{1}}+b^{*}_{l_{1}})\ ,$
$C^{{}^{\prime}}_{l_{2}}=\sigma(H_{l_{2}}\cdot
U^{\top}_{l_{2}}+b^{*}_{l_{2}})\ ,$ $C^{*}_{l_{2}}=\sigma(H_{l_{1}}\cdot
U^{\top}_{l_{2}}+b^{*}_{l_{2}})\ ,$

In addition to optimizing the loss functions described in the Section 3.2, we
further optimize the monolingual and cross-lingual reconstruction for
neighborhood clusters:

$O_{N}=\sum_{\\{l_{i},l_{j}\\}\in
A}L(C^{{}^{\prime}}_{l_{i}},C_{l_{i}})+L(C^{*}_{l_{i}},C_{l_{i}})\\\
+L(C^{{}^{\prime}}_{l_{j}},C_{l_{j}})+L(C^{*}_{l_{j}},C_{l_{j}})\ ,$

###  3.4 Character-Level Word Alignment

Bilingual word alignment is not always enough to induce a common semantic
space, especially for low-resource languages. Although the words that refer to
the same concept are not exactly the same in multiple languages, they usually
share a set of similar characters, especially in related languages written in
the same script, such as Amharic and Tigrinya. For example, the same entity is
spelled slightly differently in three languages: Semsettin Gunaltay in
English, Şemsettin Günaltay in Turkish, and Semsetin Ganoltey in Somali.
Beyond word-level alignment, we introduce character-level alignment by
composing word representations from its compositional characters using
convolutional neural networks (CNN). For each language, we adopt a language-
independent CNN to generate character-level word representation.

#### Character Lookup Embeddings

Let $S_{l}$ be the character set for language $l$ and
$E_{S_{l}}\in\mathbb{R}^{|S_{l}|\times d}$ be the character lookup embeddings,
where $d$ is the dimensionality of each character embedding. Here, we use a
simple yet effective method to induce character embeddings from word
embeddings. For each character $c$, we average the embeddings of all words
which contain the character. The character embeddings will be further tuned by
the model.

#### Character-Level CNN Kim et al. (2016)

The input layer is a sequence of characters of length $k$ for each word. Each
character is represented by a d-dimensional lookup embedding. Thus each input
sequence is represented as a feature map of dimensionality $d\times k$.

The convolution layer is used to learn the representation for each sliding
$n$-gram characters. We make $p_{i}$ as the concatenated embeddings of $n$
continuous columns from the input matrix, where $n$ is the filter width. We
then apply the convolution weights $W\in\mathbb{R}^{d\times nd}$ to $p_{i}$
with a biased vector $b\in\mathbb{R}^{d}$ as follows:

$p^{{}^{\prime}}_{i}=\tanh(W\cdot p_{i}+b)$

All $n$-gram representations $p_{i}^{{}^{\prime}}$ are used to generate the
word representation $y$ by max-pooling.

In our experiments, we apply multiple filters with various widths to obtain
the representation for word $w_{i}^{l}$. The final character-level word
representation $\hat{w}_{i}^{l}$ is the concatenation of all word
representations with varying filter widths.

#### Cross-lingual Mapping

Given the bilingual aligned word pairs, we directly minimize the distance of
the character-level word representations in the common semantic space by:

$O_{char}=\sum_{\\{l_{i},l_{j}\\}\in
A}L(\hat{W}_{l_{i}}^{char},\hat{W}_{l_{j}}^{char})$

The final word representation of $w_{i}^{l}$ in the common semantic space is
the concatenation of character-level word presentation $\hat{w}_{i}^{l}$ and
projected word representation $h_{i}^{l}$.

Class Name | Words / Word Pairs
---|---
Colors | white, yellow, red, blue, green …
Weekdays | monday, tuesday, friday, sunday …
Months | january, february, march, april …
numbers | one, two, three, four, five …
pronouns | i, me, you, he, she, her, they …
prepositions | of, in, on, for, from, about …
conjunctions | but, and, so, or, when, while …
clothes | hat, shirt, pants, skirt, socks …
-like | (god, godlike), (bird, birdlike) …
-able | (accept, acceptable), (adopt, adoptable) …
micro- | (gram, microgram), (chip, microchip) …
auto- | (maker, automaker), (gas, autogas) …
Table 1: Examples of closed word classes and linguistic properties based
clusters

###  3.5 Linguistic Property Alignment

Linguists have made great efforts at building linguistic property knowledge
bases for thousands of languages in the world. These knowledge bases include a
large number of topological properties (phonological, lexical and grammatical)
which we will use to build a high-level alignment between words across
languages. We exploit the following resources:

  * •

CLDR (Unicode Common Locale Data
Repository)222http://cldr.unicode.org/index/charts which includes multilingual
gazetteers for months, weekdays, cardinal and ordinal numbers;

  * •

Wiktionary333https://en.wiktionary.org which is a multilingual, web-based
collaborative project to create an English content dictionary, includes word
and prefix/suffix dictionaries for 1,247 languages;

  * •

Panlex444http://panlex.org/ database which contains 1.1 billion pairwise
translations among 21 million expressions in about 10,000 language varieties.

Parameter Name | Value
---|---
Monolingual Word Embedding Size | 512
Multilingual Word Embedding Size | 512
# of Filters in Convolution Layer | 20
Filter Widths | 1, 2, 3
Batch Size | 500
Initial Learning Rate | 0.5
Optimizer | Adadelta
Table 2: Hyper-parameters. |  | 3 Languages | 12 Languages
---|---|---|---
|  | Monolingual | Multilingual | Monolingual | Multilingual
|  | QVEC | QVEC-CCA | QVEC | QVEC-CCA | QVEC | QVEC-CCA | QVEC | QVEC-CCA
MultiCluster | 10.8 | 9.1 | 63.6 | 45.8 | 10.4 | 9.3 | 62.7 | 44.5
MultiCCA | 10.8 | 8.5 | 63.8 | 43.9 | 10.8 | 8.5 | 63.9 | 43.7
MultiSkip | 7.8 | 7.3 | 57.3 | 36.2 | 8.4 | 7.2 | 59.1 | 36.5
MultiCross | - | - | - | - | 11.9 | 8.6 | 46.4 | 31.0
CorrNet | W | 14.8 | 11.3 | 63.6 | 43.4 | 14.7 | 13.2 | 63.8 | 43.9
W+N | 15.9 | 12.7 | 64.5 | 45.3 | 15.5 | 13.6 | 65.0 | 46.4
W+N+Ch | 15.2 | 12.1 | 66.3 | 44.5 | 14.8 | 12.9 | 67.2 | 47.3
W+N+L | 15.8 | 12.8 | 64.3 | 45.3 | 16.3 | 14.5 | 65.0 | 45.9
W+N+Ch+L | 15.5 | 12.7 | 66.5 | 46.3 | 14.9 | 13.1 | 67.3 | 47.2
Table 3: QVEC and QVEC-CCA scores. W: word alignment. N: neighbor based
clustering and alignment. Ch: character based clustering and alignment. L:
linguistic property based clustering and alignment.

We mainly exploit two types of linguistic properties to extract word clusters.
The first type is closed word classes, such as colors, weekdays, and months.
Table 1 shows some examples of the word clusters we automatically extracted
from CLDR and Wiktionary for English. The second type of word clusters are
generated based on morphological information, including affixes that indicate
various linguistic functions. These properties tend to be consistent across
many languages. For example, “-like” is a suffix denoting “similar to” in
English, while in Danish “-agtig” performs the same function. For each affix,
we extract a set of word pairs (basic word, extended word with affix) to
denote its semantics from each language.

We extract a set of word clusters from each language, and align the clusters
based on their functions defined in CLDR, Wiktionary and Panlex. For each
language $l$, each cluster $r^{l}_{i}\in R^{l}$ contains a set of words or
word-pairs sharing the same function. We use the average operation to obtain
an overall vector representation for each cluster $M_{l}^{R}$.555For each word
pair, we use the vector of the extend word minus the vector of the basic word
as the vector representation of the word pair. Then, we project the cluster-
level vectors into the shared semantic space and minimize the distance between
them:

$H_{l_{i}}^{R}=\sigma(M_{l_{i}}^{R}\cdot W_{l_{i}}+b^{R}_{l_{i}})\ ,$
$H_{l_{j}}^{R}=\sigma(M_{l_{j}}^{R}\cdot W_{l_{j}}+b^{R}_{l_{j}})\ ,$
$O_{R}=\sum_{\\{l_{i},l_{j}\\}\in A}L(H^{R}_{l_{i}},H^{R}_{l_{j}})\ ,$

where $W$ is the same as the $W$ used in Section 3.3 for each language. We
finally optimize the sum of the losses by finding the parameters
$\theta=\\{W_{l},b_{l},b^{{}^{\prime}}_{l},U_{l},b^{*}_{l},$
CNN${}_{l},b^{R}_{l}\\}$, where $l$ denotes a specific language:

$O_{\theta}=O_{W}+O_{N}+O_{char}+O_{R}$

##  4 Experiments

###  4.1 Experiment Setup

Previous work Ammar et al. (2016b); Duong et al. (2017) evaluated multilingual
word embeddings on a series of intrinsic (e.g., monolingual and cross-lingual
word similarity, word translation) and extrinsic (e.g., multilingual document
classification, multilingual dependency parsing) evaluation tasks. Compared
with previous work, we aim at incorporating more linguistic features into the
multilingual embeddings, which can be helpful for downstream NLP tasks. In
order to evaluate the quality of the multilingual embeddings, we use QVEC
Tsvetkov et al. (2015) tasks (details will be described in Section 4.2) as the
intrinsic evaluation platform. In addition, to demonstrate the effectiveness
of our common semantic space for knowledge transfer, especially for low-
resource scenarios, we adopt the low-resource language name tagging task for
extrinsic evaluation.

For fair comparison with state-of-the-art methods on building multi-lingual
embeddings Ammar et al. (2016b); Duong et al. (2017), we use the same
monolingual data and bilingual dictionaries as in their work. We build
multilingual word embeddings for 3 languages (English, Italian, Danish) and 12
languages (Bulgarian, Czech, Danish, German, Greek, English, Spanish, Finnish,
French, Hungarian, Italian, Swedish) respectively. The monolingual data for
each language is the combination of the Leipzig Corpora
Collection666http://wortschatz.uni-leipzig.de/en/download/ and
Europarl.777http://www.statmt.org/europarl/index.html The bilingual
dictionaries are the same as those used in Ammar et al.
(2016b).888http://128.2.220.95/multilingual/data/

For each task, we evaluate the performance of our common semantic space in
comparison with previously published multilingual word embeddings
(MultiCluster, MultiCCA, MultiSkip, and MultiCross). MultiCluster Ammar et al.
(2016b) groups multilingual words into clusters based on bilingual
dictionaries and forces all the words from various languages within one
cluster share the same embedding. MultiCCA Ammar et al. (2016b); Faruqui and
Dyer (2014) uses CCA to estimate linear projections for each pair of
languages. MultiSkip is an extension of the multilingual skip-gram model Luong
et al. (2015), which requires parallel data. MultiCross is an approach to
unify bilingual word embeddings into a shared semantic space using post hoc
linear transformations Duong et al. (2017).

Table 2 lists the hyper-parameters used in the experiments.

| QVEC | QVEC-CCA
---|---|---
| Monolingual | Multilingual | Monolingual | Multilingual
40,000 | multiCCA | 10.8 | 8.5 | 63.8 | 43.9
CorrNet W | 14.8 | 11.3 | 63.6 | 43.4
| CorrNet W+N+Ch+L | 15.5 | 12.7 | 66.5 | 46.3
10,000 | multiCCA | 9.8 | 6.5 | 63.6 | 42.3
CorrNet W | 14.8 | 11.3 | 63.4 | 43.0
| CorrNet W+N+Ch+L | 15.4 | 12.1 | 66.4 | 46.2
2,000 | multiCCA | 9.9 | 6.2 | 63.6 | 40.9
CorrNet W | 14.5 | 7.1 | 62.0 | 39.2
| CorrNet W+N+Ch+L | 14.7 | 11.7 | 66.6 | 45.5
Table 4: Results using bilingual lexicons with varying sizes (40,000, 10,000,
2,000) and three languages. CorrNet W+N+Ch+L is the proposed approach with all
the cluster types.

###  4.2 Intrinsic Evaluation: QVEC

In order to evaluate the quality of multilingual embeddings, especially on
linguistic aspect, we adopt QVEC Tsvetkov et al. (2015) as the intrinsic
evaluation measure. It evaluates the quality of word embeddings based on the
alignment of distributional word vectors to linguistic feature vectors
extracted from manually crafted lexical resources, e.g., SemCor Miller et al.
(1993).

$\text{QVEC}=\max_{\sum_{j}a_{ij}\leq
1}\sum_{i=1}^{D}\sum_{j=1}^{P}r(x_{i},s_{j})\times a_{ij}$

where $x\in\mathbb{R}^{D\times 1}$ denotes a distributional word vector and
$s\in\mathbb{R}^{P\times 1}$ denotes a linguistic word vector. $a_{ij}=1$ iff
$x_{i}$ is aligned to $s_{j}$, otherwise $a_{ij}=0$. $r(x_{i},s_{j})$ is the
Pearson’s correlation between $x_{i}$ and $s_{j}$. QVEC-CCA Ammar et al.
(2016b) is extended from QVEC by using CCA to measure the correlation between
the distributional matrix and the linguistic vector matrix, instead of
cumulative dimension-wise correlation.

Using QVEC and QVEC-CCA, we evaluate the quality of multilingual embeddings
for both monolingual (English) and multilingual (English, Danish, Italian)
settings.

As shown in Table 3, our approaches outperform previous approaches in all
cases. Specifically, by augmenting word representation with neighboring words
in the common semantic space as in Eq. (3.3), the performance for monolingual
QVEC and QVEC-CCA tasks is consistently improved. In addition, by aligning
character-level compositional representations and linguistic property based
clusters in the shared semantic space, the multilingual representation quality
is further improved.

| Amh | Tig | Uig | Tur | Eng
---|---|---|---|---|---
Train | 1,506 | 1,585 | 1,711 | 3,404 | 14,029
Dev | 167 | 176 | 190 | 378 | 3,250
Test | 711 | 440 | 476 | 1,604 | 3,453
Table 5: Data statistics (# of Sentences) for name tagging

###  4.3 Impact of Bilingual Dictionary Size

In order to show the impact of the size of bilingual lexicons, we use three
languages as a case study, and gradually reduce the size of the lexicons for
each pair of languages from 40,000 to 10,000 and to 2,000. Both MultiCluster
and MultiSkip by default take advantage of identical strings from any pair of
languages when they learn the multilingual embeddings. For fair comparison, we
thus use MultiCCA as a baseline. Table 4 shows the results. We observe that
both MultiCCA and CorrNet approaches are sensitive to the size of the
bilingual lexicons. Our approach on the other hand could maintain high
performance, even when the bilingual lexicons were reduced to 2,000.

###  4.4 Low-Resource Name Tagging

We evaluate the quality of multilingual embeddings on a downstream task by
using the embeddings as input features. Here, we use low-resource language
name tagging as a target task. We experiment with two sets of languages. The
first set Amh+Tig consists of Amharic and Tigrinya. Both languages share the
same Ge’ez script and descend from the proto-Semitic language family. The
other set Eng+Uig+Tur consists of one high-resource language (English), one
medium-resource language (Turkish) and one low-resource language (Uighur). It
also consists of two distinct language scripts: English and Turkish use Latin
script while Uighur uses Arabic script.

|  | Multilingual
---|---|---
| Mono- |  | CorrNet
| lingual | MultiCCA | W | W+N+Ch+L
Amh | 52.0 | 50.6 | 52.4 | 55.8
Tig | 78.2 | 78.4 | 77.9 | 77.6
Uig | 70.0 | 63.6 | 66.8 | 66.0
Tur | 73.9 | 65.3 | 72.4 | 75.6
Table 6: Name tagging result (F-score, %) using monolingual embedding and
multilingual embeddings.

We use LSTM-CRF architecture Huang et al. (2015); Lample et al. (2016); Ma and
Hovy (2016) for name tagging. Table 5 shows the statistics of training,
development, and test sets for each language released by Linguistic Data
Consortium (LDC).999The annotations are from: Amh (LDC2016E87), Tig
(LDC2017E27), Uig (LDC2016E70), Tur (LDC2014E115), Eng Tjong Kim Sang and De
Meulder (2003) For each language pair in each language set, we combine the
bilingual aligned words extracted from Wiktionary and extracted from
monolingual dictionaries based on identical strings.101010We extracted 23,781
pairs of words for Amh and Tig, 16,868 pairs for Eng and Tur, 3,353 pairs for
Eng and Uig, and 3,563 pairs for Tur and Uig. We evaluate the quality from
several aspects:

|  |  | CorrNet
---|---|---|---
Train | Test | MultiCCA | W | W+N+Ch+L
Amh | Tig | 15.5 | 28.3 | 31.7
Tig | Amh | 11.1 | 12.8 | 23.3
Eng | Uig | 8.4 | 16.9 | 15.4
Tur | Uig | 1.1 | 18.1 | 25.6
Eng+Tur | Uig | 8.0 | 20.3 | 20.6
Eng | Tur | 20.6 | 21.4 | 17.3
Uig | Tur | 10.4 | 10.1 | 17.7
Eng+Uig | Tur | 18.5 | 21.1 | 29.4
Table 7: Name tagging performance (F-score, %) when the tagger was trained on
a source language and tested on a target language. CorrNet W+N+Ch+L is the
proposed approach with all the cluster types.

#### Monolingual embedding quality evaluation

Table 6 shows the name tagging performance for each language using the
original monolingual embeddings and multilingual embeddings. For both Amharic
and Turkish, the multilingual embeddings learned from our approach
significantly improve over the monolingual embeddings, compared to MultiCCA.
In the case of Uighur, all the multilingual embeddings fail to outperform the
original monolingual embeddings. We conjecture that this is due to the use of
Arabic script in Uighur, which differs from Turkish and English.

#### Cross-lingual direct transfer

We further demonstrate the effectiveness of our multilingual embeddings on
direct knowledge transfer. In this setting, we train a name tagger on one or
two languages using multilingual embeddings and test it on a new language
without any annotated data. Table 7 shows the performance. For each testing
language, our approach achieves better performance than MultiCCA and CorrNet.
The closer that the languages are, such as Amharic and Tigrinya, and Turkish
and Uighur, the better performance could be achieved, even when they may have
distinct language scripts (e.g., Turkish and Uighur).

We however also notice that a larger extra annotation from another language
does not necessarily result in the improvement. For instance, the proposed
approach (CorrNet W+N+Ch+L) suffers from English annotated examples when
tested on Turkish. This suggests that we need to be careful and aware of
linguistic properties among different languages for transfer learning.

#### Mutual enhancement

We finally show the improvement by adding more cross-lingual annotated data
and using multilingual embeddings in Table 8. The multilingual embeddings
learned by our approach consistently outperforms MultiCCA. More specifically,
when there are not enough annotated examples, the performance could be
improved by incorporating annotated examples from other languages. This is
evident for Amharic, Tigrinya and Uighur.

|  |  | CorrNet
---|---|---|---
Train | Test | MultiCCA | W | W+N+Ch+L
Tig+Amh | Amh | 52.9 | 52.1 | 56.5
Amh+Tig | Tig | 78.0 | 78.1 | 78.7
Eng+Uig | Uig | 67.9 | 67.8 | 68.3
Tur+Uig | Uig | 67.7 | 67.5 | 68.8
Eng+Tur+Uig | Uig | 68.7 | 67.4 | 65.9
Uig-Tur | Tur | 65.9 | 69.2 | 72.8
Eng-Tur | Tur | 66.9 | 70.4 | 73.4
Eng+Uig+Tur | Tur | 67.5 | 68.5 | 72.9
Table 8: Name tagging performance (F-score, %) when the training set for the
tagger was enhanced by annotated examples in other languages. CorrNet W+N+Ch+L
is the proposed approach with all the cluster types.

##  5 Conclusions and Future Work

We construct a common semantic space for multiple languages based on a
cluster-consistent correlational neural network. It combines word-level
alignment and multi-level cluster alignment, including neighbor based
clusters, character-level compositional word representations, and linguistic
property based clusters induced from the readily available language-universal
linguistic knowledge bases. By introducing cluster consistency into
multilingual embedding learning, our approach achieved significantly higher
performance than state-of-the-art multilingual embedding learning methods
through both intrinsic and extrinsic evaluations. In the future, we will
further extend our approach to multi-lingual multi-media common semantic space
construction.

## References

  * Ammar et al. (2016a) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016a.  Many languages, one parser.  _arXiv preprint arXiv:1602.01595_.
  * Ammar et al. (2016b) Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016b.  Massively multilingual word embeddings.  _arXiv preprint arXiv:1602.01925_.
  * AP Chandar et al. (2014) Sarath AP Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014.  An autoencoder approach to learning bilingual word representations.  In _Proceedings of NIPS_.
  * Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017.  Learning bilingual word embeddings with (almost) no bilingual data.  In _Proceedings of ACL_.
  * Cao et al. (2016) Hailong Cao, Tiejun Zhao, Shu Zhang, and Yao Meng. 2016.  A distribution-based model to learn bilingual word embeddings.  In _Proceedings of COLING_.
  * Chandar et al. (2016) Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. 2016\.  Correlational neural networks.  _Neural computation_.
  * Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017.  Word translation without parallel data.  _arXiv preprint arXiv:1710.04087_.
  * Duong et al. (2016) Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016.  Learning crosslingual word embeddings without bilingual corpora.  _arXiv preprint arXiv:1606.09403_.
  * Duong et al. (2017) Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2017.  Multilingual training of crosslingual word embeddings.  In _Proceedings of EMNLP_.
  * Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014.  Improving vector space word representations using multilingual correlation.  In _Proceedings of EMNLP_.
  * Gouws et al. (2015) Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015.  Bilbowa: Fast bilingual distributed representations without word alignments.  In _Proceedings of ICML_.
  * Guo et al. (2015) Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015.  Cross-lingual dependency parsing based on distributed representations.  In _Proceedings of ACL_.
  * Hermann and Blunsom (2014) Karl Moritz Hermann and Phil Blunsom. 2014.  Multilingual models for compositional distributional semantics.  In _Proceedings of ACL_.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016.  Character-aware neural language models.  In _Proceedings of AAAI_.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  _arXiv preprint arXiv:1603.01360_.
  * Lu et al. (2015) Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015.  Deep multilingual correlation for improved word embeddings.  In _Proceedings of HLT-NAACL_.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D Manning. 2015.  Bilingual word representations with monolingual quality in mind.  In _Proceedings of HLT-NAACL_.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  _arXiv preprint arXiv:1603.01354_.
  * Madhyastha and España-Bonet (2017) Pranava Swaroop Madhyastha and Cristina España-Bonet. 2017.  Learning bilingual projections of embeddings for vocabulary expansion in machine translation.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_.
  * MarcoBaroni (2015) AngelikiLazaridou GeorgianaDinu MarcoBaroni. 2015.  Hubness and pollution: Delving into cross-space mapping for zero-shot learning.
  * Mikolov et al. (2013) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013.  Exploiting similarities among languages for machine translation.  _arXiv preprint arXiv:1309.4168_.
  * Miller et al. (1993) George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993.  A semantic concordance.  In _Proceedings of HLT_.
  * Rajendran et al. (2015) Janarthanan Rajendran, Mitesh M Khapra, Sarath Chandar, and Balaraman Ravindran. 2015.  Bridge correlational neural networks for multilingual multimodal representation learning.  _arXiv preprint arXiv:1510.03519_.
  * Rothe et al. (2016) Sascha Rothe, Sebastian Ebert, and Hinrich Schütze. 2016.  Ultradense word embeddings by orthogonal transformation.  _arXiv preprint arXiv:1602.07572_.
  * Schwenk et al. (2017) Holger Schwenk, Ke Tran, Orhan Firat, and Matthijs Douze. 2017.  Learning joint multilingual sentence representations with neural machine translation.  _arXiv preprint arXiv:1704.04154_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of NAACL-HLT_.
  * Tsai and Roth (2016) Chen-Tse Tsai and Dan Roth. 2016.  Cross-lingual wikification using multilingual embeddings.  In _Proceedings of HLT-NAACL_.
  * Tsvetkov et al. (2015) Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015\.  Evaluation of word vector representations by subspace alignment.  In _Proceedings of EMNLP_.
  * Vulic and Korhonen (2016) Ivan Vulic and Anna Korhonen. 2016.  On the role of seed lexicons in learning bilingual word embeddings.  In _Proceedings of ACL_.
  * Vulic and Moens (2015) Ivan Vulic and Marie-Francine Moens. 2015.  Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction.  In _Proceedings of ACL_.
  * Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015.  Normalized word embedding and orthogonal transform for bilingual word translation.  In _Proceedings of HLT-NAACL_.
  * (33) Boliang Zhang, Xiaoman Pan, Ying Lin, Tongtao Zhang, Kevin Blissett, Samia Kazemi, Spencer Whitehead, Lifu Huang, and Heng Ji.  Rpi blender tac-kbp2017 13 languages edl system.
  * Zhang et al. (2017a) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017a.  Adversarial training for unsupervised bilingual lexicon induction.  In _Proceedings of ACL_.
  * Zhang et al. (2017b) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017b.  Earth mover’s distance minimization for unsupervised bilingual lexicon induction.  In _Proceedings of EMNLP_.
  * Zhang et al. (2016) Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi S Jaakkola. 2016.  Ten pairs to tag-multilingual pos tagging via coarse mapping between embeddings.  In _Proceedings of HLT-NAACL_.
  * Zou et al. (2013) Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013.  Bilingual word embeddings for phrase-based machine translation.  In _Proceedings of EMNLP_.