#  AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages
with Adversarial Examples

Qianchu Liu1, Edoardo M. Ponti1 2, Diana McCarthy3, Ivan Vulić1, Anna
Korhonen1
1 3Language Technology Lab, TAL, University of Cambridge, UK
2 Mila/McGill University Montreal, Canada
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Capturing word meaning in context and distinguishing between correspondences
and variations across languages is key to building successful multilingual and
cross-lingual text representation models. However, existing multilingual
evaluation datasets that evaluate lexical semantics “in-context” have various
limitations, in particular 1) their language coverage is restricted to high-
resource languages and skewed in favor of only a few language families and
areas, 2) a design that makes the task solvable via superficial cues, which
results in artificially inflated (and sometimes super-human) performances of
pretrained encoders, and 3) no support for cross-lingual evaluation. In order
to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning
in Context), a wide-coverage cross-lingual and multilingual evaluation set; it
aims to faithfully assess the ability of state-of-the-art (SotA)
representation models to understand the identity of word meaning in cross-
lingual contexts for 14 language pairs. We conduct a series of experiments in
a wide range of setups and demonstrate the challenging nature of AM2iCo. The
results reveal that current SotA pretrained encoders substantially lag behind
human performance, and the largest gaps are observed for low-resource
languages and languages dissimilar to English.

##  1 Introduction

Pretrained language models (LMs) such as BERT Devlin et al. (2019) and XLM-R
Conneau et al. (2020) offer a natural way to distinguish different meanings of
words in context without performing explicit sense disambiguation. This
property of “meaning contextualization” is typically evaluated either via
standard entity linking Rao et al. (2013); Shen et al. (2014) and Word Sense
Disambiguation (WSD) tasks Navigli (2009); Moro et al. (2014); Raganato et al.
(2017) or, more recently, relying on the Word-in-Context (WiC) evaluation
paradigm Pilehvar and Camacho-Collados (2019); Raganato et al. (2020).

Although monolingual evaluation in English is still predominant, a need has
been recognized to construct similar resources for other languages to support
cross-lingual evaluation and model diagnostics. This includes multilingual and
cross-lingual WSD benchmarks (Navigli and Ponzetto, 2012; Navigli et al.,
2013; Scarlini et al., 2020; Barba et al., 2020, inter alia), cross-lingual
entity linking Tsai and Roth (2016); Raiman and Raiman (2018); Upadhyay et al.
(2018) and, most recently, multilingual WiC (termed XL-WiC) spanning 12
languages Raganato et al. (2020).

This most recent WiC evaluation approach is particularly attractive as 1) it
bypasses the dependence on modeling predefined ontologies (entity linking) and
explicit sense inventories (WSD), and 2) it is framed as a simple binary
classification task: for a target word $w$ appearing in two different contexts
$c_{1}$ and $c_{2}$, the system must decide whether $w$ conveys the same
meaning in both contexts, or not.

However, the current WiC evaluation still allows ample room for improvement:
1) current language coverage is limited, and biased towards resource-rich
Indo-European languages; 2) coverage of lexical concepts, due to their paucity
in language-specific WordNets, is also limited; 3) XL-WiC is a monolingual
resource available in different languages, i.e., it does not support cross-
lingual assessments. Further, 4) the current WiC datasets offer low human
upper bounds and inflated (even super-human) performance for some
languages.111In turn, this might give a false impression that some languages
in the task are ’solved’ by the current pretrained LMs. This is due to
superficial cues where 5) many examples in the current WiC datasets can be
resolved relying either on the target word alone without any context or on the
context alone, which eludes evaluation honing in on the interplay between
target words and their corresponding contexts.

In order to address these limitations and provide a more comprehensive
evaluation framework, we present AM2iCo (Adversarial and Multilingual Meaning
in Context), a novel multilingual and cross-lingual WiC task and resource. It
covers a typologically diverse set of 15 languages, see Table 2. Based on
Wikipedia in lieu of WordNet, AM2iCo covers a wider set of ambiguous words
which especially complements WiC on the long tail of entity names and adds
challenges in generalisation for a larger vocabulary than a restricted set of
common words. More importantly, the use of Wikipedia enables WiC evaluation on
low-resource languages (e.g., Basque, Georgian, Bengali, Kazakh). We also
improve the WiC resource design; it now 1) includes adversarial examples and
careful data extraction procedures to prevent the models from backing off to
superficial clues, 2) results in a more challenging benchmark with truer and
much wider gaps between current SotA pretrained encoders and human capability
(see §2.3), and 3) enables cross-lingual evaluation and analysis.

The ample and diverse data in AM2iCo enables a wide spectrum of experiments
and analyses in different scenarios. We evaluate SotA pretrained encoders,
multilingual BERT and XLM-R, both off-the-shelf using a metric-based approach
(i.e., without any task adaptation) and after task-specific fine-tuning. With
fine-tuned models, we investigate zero-shot cross-lingual transfer as well as
transfer from multiple source languages. In general, our results across these
diverse scenarios firmly indicate a large gap between human and system
performance across the board, which is even more prominent when dealing with
resource-poor languages and languages dissimilar to English.

In the hope that AM2iCo will be a challenging and valuable diagnostic and
evaluation asset for future work in multilingual and cross-lingual
representation learning, we release the data along with the full guidelines
at: https://github.com/cambridgeltl/AMiCo.

##  2 AM2iCo: Cross-Lingual Word-in-Context Evaluation

Task Definition. Each dataset in AM2iCo contains data for a standard binary
classification task spanning a set of $N$ examples $\hat{x}_{i}$,
$i=1,\ldots,N$ for a language pair. Each example $\hat{x}_{i}$ is in fact a
pair of items $\hat{x}_{i}=(x_{i,src},x_{i,trg})$, where the item $x_{i,src}$
is provided in the source language $L_{src}$ and the item $x_{i,trg}$ is in
the target language $L_{trg}$. The item $x_{i,src}$ in turn is another pair
$x_{i,src}=(w_{i,src},c_{i,src})$; it contains a target word $w_{i,src}$ from
$L_{src}$ and its (wider) context $c_{i,src}$ (also in $L_{src}$) in which
that word appears, as illustrated in Table 1; the same is valid for
$x_{i,trg}$. The classification task is then to judge whether the words
$w_{i,src}$ and $w_{i,trg}$ occurring in the respective contexts $c_{i,src}$
and $c_{i,trg}$ have the same sense/meaning (i.e., whether they refer to the
same entity/concept), or not.

Final Resource. The full AM2iCo resource comprises datasets for 14 language
pairs, where English is paired with 14 target languages. For brevity, in the
rest of the paper we refer to the dataset of each language pair simply with
the $L_{trg}$ language code (e.g., ZH instead of EN-ZH); languages and codes
are provided in Table 2.

As illustrative examples, we show a positive pair (label ‘T’) and a negative
pair (label ‘F’) from the ZH AM2iCo dataset in Table 1 (Examples 1 and 2). In
the positive example, both target words ‘Apollo’ and ‘ 阿波罗’ in their contexts
refer to the same concept: the Apollo spaceflight program. In the negative
example, the Chinese target word ‘ 阿波罗’ refers to the Apollo aircraft, but the
English target word ‘Apollo’ now refers to the Greek God.

no. |  English $x_{i,src}$ |  Chinese $x_{i,trg}$ | Label
---|---|---|---
1 |  Bill Kaysing ( July 31 , 1922 – April 21 , 2005 ) was an American writer who claimed that the six Apollo Moon landings between July 1969 and December 1972 were hoaxes , and so a founder of the Moon hoax movement . |  泰坦系列导弹的发射任务结束后，LC-16被移交给NASA用做双子座计划的航天员训练及 阿波罗  中飞船服务舱的静态试车。[…]
(After the launch of the Titan missiles, LC-16 was handed over to NASA for the training of the astronauts in the Gemini program and the static test run of the service module of the spacecraft in Apollo […]) | T
2 |  Nearer the house , screening the service wing from view , is a Roman triumphal arch , the “ Temple of Apollo ” , also known ( because of its former use a venue for cock fighting ) as “ Cockpit Arch ” , which holds a copy of the famed Apollo Belvedere . |  阿波罗-联盟测试计划中，美国的 阿波罗 航天器和苏联的联盟航天器在地球轨道中对接。…
(In the Apollo-Soyuz test plan, America’s Apollo spacecraft and the Soviet Union’s Soyuz spacecraft are docked in the Earth orbit […]) | F
3 |  Bill Kaysing ( July 31 , 1922 – April 21 , 2005 ) was an American writer who claimed that the six Apollo Moon landings between July 1969 and December 1972 were hoax es , and so a founder of the Moon hoax movement . |  泰坦系列导弹的发射任务结束后，LC-16被移交给 NASA 用做双子座计划的航天员训练及阿波罗中飞船服务舱的静态试车。[…]
(After the launch of the Titan missiles, LC-16 was handed over to NASA for the training of the astronauts in the Gemini program and the static test run of the service module of the spacecraft in Apollo […]) | F
Table 1: Positive (1), negative (2) and adversarial negative examples (3) from ZH AM2iCo. Target words are provided in boldface and with a gray background. Translations of the ZH items are provided in italic. |  DE |  RU |  JA |  ZH |  AR |  KO |  FI |  TR |  ID |  EU |  KA |  BN |  KK |  UR
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
train |  50,000 |  28,286 |  16,142 |  13,154 |  9,622 |  7,070 |  6,322 |  3,904 |  1,598 |  978 |  – |  – |  – |  –
dev |  500 |  500 |  500 |  500 |  500 |  500 |  500 |  500 |  500 |  500 |  500 |  332 |  276 |  108
test |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  1,000 |  700 |  400 |  400
Table 2: Data sizes for AM2iCo across 14 language pairs. We also provide
larger dev and test sets for DE and RU spanning 5,000 and 10,000 examples,
respectively. EN=English; DE=German; RU=Russian; JA=Japanese; ZH=Chinese;
AR=Arabic; KO=Korean; FI=Finnish; TR=Turkish; ID=Indonesian; EU=Basque;
KA=Georgian; BN=Bengali, KK=Kazakh; UR=Urdu.

In what follows we describe the creation of AM2iCo. We also demonstrate the
benefits of AM2iCo and its challenging nature.

###  2.1 Data Creation

We start from Wikipedia, a rich source of disambiguated contexts for multiple
languages. The availability of Wikipedia’s cross-lingual links provides a
direct way to identify cross-lingual concept correspondence. The items
$x_{i,src}$ and $x_{i,trg}$ are then extracted by taking the surrounding
(sentential) context of a hyperlinked word in a Wikipedia article. We balance
the context length by (i) discarding items which are longer than 100 words,
and (ii) adding preceding and following sentences to the context for sentences
shorter than 30 words.

Using the Wikipedia dumps of our 15 languages (see Table 2), we create
monolingual items $x$ for each language. We select only ambiguous target words
($w$-s), that is, words that link to at least two different Wikipedia pages.
To avoid rare words that are potentially unknown to non-experts, we then
retain only words that are among the top 200k words by frequency in each
respective Wikipedia.

For each word, we then create monolingual positive examples by pairing two
items (i.e., word-context pairs) $x_{i}$ and $x_{j}$ in which the same target
word $w$ is linked to the same Wikipedia page, signaling the same meaning. In
a similar fashion, monolingual negative examples are created by pairing two
items where the same target word $w$ is linked to two different Wikipedia
pages. We ensure that there is roughly an equal number of positive and
negative examples for each target word.

Now, each monolingual example (i.e., pair of items) $\hat{x}$ contains the
same word occurring in two different contexts. In order to create a cross-
lingual dataset, we leverage the Wikipedia cross-lingual links; we simply (i)
replace one of the two items from each English pair with an item in the target
language, and (ii) replace one of the two items from each target language pair
with an English item, where the cross-lingual replacements point to the same
Wikipedia page as indicated by the cross-lingual Wiki links. Through this
procedure, the final datasets cover a sufficient (and roughly comparable)
number of examples containing ambiguous words both in English and in
$L_{trg}$. We also rely on data selection heuristics that improve the final
data quality, discussed in §2.2 and §2.3.

Finally, in each cross-lingual dataset we reserve 1,000 examples for testing,
500 examples as dev data; the rest is used for training. The exception are 4
resource-poor languages, where all the data examples are divided between dev
and test. All data portions in all datasets are balanced. We ensure zero
overlap between train, dev, and test portions. The final AM2iCo statistics are
given in Table 2.

Human Validation. We employ human annotators to assess the quality of AM2iCo.
For each dataset, we recruit two annotators who each validate a random sample
of 100 examples, where 50 examples are shared between the two samples and are
used to compute inter-rater agreement.222The annotators were recruited via two
crowdsourcing platforms, Prolific and Proz, depending on target language
coverage. The annotators were native speakers of the target language, fluent
in English, and with an undergraduate degree.

###  2.2 Data Selection Heuristics

One critical requirement of AM2iCo is ensuring a high human upper bound. In
the initial data creation phase, we observed several sources of confusion
among human raters, typically related to some negative pairs being frequently
labeled as positive; we identified two causes of this discrepancy and then
mitigated it through data selection heuristics.

First, some common monosemous words still might get linked to multiple
different Wikipedia pages, thus creating confusing negative pairs. For
instance, some pronouns (e.g., ‘he’, ‘it’) and common nouns (e.g., ‘daughter’,
‘son’) may link to different entities as a result of coreference resolution.
However, truly ambiguous words are typically directly defined in Wikipedia
Disambiguation pages. Therefore, we only keep the negative pairs that link to
separate entries found in the Wikipedia disambiguation pages. The second issue
concerns concept granularity, as Wikipedia sometimes makes too fine-grained
distinctions between concepts: e.g., by setting up separate pages for a
country’s name in different time periods.333E.g., ‘China’ can be linked to the
page ‘Republic of China (1912–1949)’ and to the page ‘Empire of China
(1915–1916)’. We mitigate this issue by requiring that the negative pairs do
not share common Wikipedia categories or parent Wikipedia categories.

The application of these heuristics during data creation (see §2.1) yields a
substantial boost in human performance: e.g., the scores increase from 74% to
88% for ZH, and from 76% to 94% for DE.

###  2.3 Adversarial Examples

Another requirement is assessing to which extent models can grasp the meaning
of a target word based on the (complex) interaction with its context. However,
recently it was shown that SotA pretrained LMs exploit superficial cues while
solving language understanding tasks due to spurious correlations seeping into
the datasets (Gururangan et al., 2018; Niven and Kao, 2019). This hinders
generalizations beyond the particular datasets and makes the models brittle to
minor changes in the input space (Jia and Liang, 2017; Iyyer et al., 2018). As
verified later in §4, we found this to be the case also for the existing WiC
datasets: just considering the target word and neglecting the context (or vice
versa) is sufficient to achieve high performance.

To remedy this issue, we already ensured that models could not rely solely on
target words in §2.1 by including both positive and negative examples for each
ambiguous word in different contexts. Further, we now introduce adversarial
negative examples in AM2iCo to penalize models that rely only on context
without considering target words. To create such negative examples, we sample
a positive pair ${x}_{i}$ and instead of the original target word $w_{i}$, we
take another related word $\tilde{w}_{i}$ in the same context $c_{i}$ as the
new target word $w_{i}$.

We define the related word as a hyperlinked mention sharing the same parent
Wiki category as the original target word: e.g., in Table 1 we change the
target word ‘ 阿波罗’ (Apollo) from Example 1 into the related word ‘NASA’,
resulting in Example 3. Both words share a common Wiki parent category,
美国国家航空航天局 (NASA). The contexts of both examples deal with spaceships; hence,
only a fine-grained understanding of lexical differences between the target
words warrants the ability to recognize Apollo as identical to ‘ 阿波罗’ but
different from ‘NASA’. Overall, adversarial examples amount to roughly $1/4$
of our dataset.

###  2.4 Data Statistics and Language Coverage

We summarize the main properties of AM2iCo while comparing against previous
word-in-context datasets WiC and XL-WiC in Table 3. First, we emphasize the
accrued reliability of AM2iCo, as both human accuracy and inter-annotator
agreement444More detailed per-language scores are listed in Table 4. are
substantially higher than with WiC and XL-WiC (i.e., rising by ~10 points).
Second, for a comparable overall dataset size we increase the number of
examples and word types in resource-poor languages. If we consider their
median across languages, AM2iCo has 8,570 and 8,520, respectively, around four
times more than XL-WiC (1,676 and 1,201). XL-WiC is heavily skewed towards a
small number of languages, namely German and French, and provides large
datasets in those languages. In contrast, AM2iCo provides a more balanced
representation of its languages. Third, in AM2iCo we deliberately include
longer contexts. While the data in WiC and XL-WiC are derived from concise
dictionary examples, AM2iCo data reflect natural text where key information
may be spread across a much wider context window.

| WiC | XL-WiC | AM2iCo
---|---|---|---
examples (mean) | 7,466 | 14,510 | 13,074
examples (median) | 7,466 | 1,676 | 8,570
word types (mean) | 4,130 | 7,255 | 9,868
word types (median) | 4,130 | 1,201 | 8,520
context length | 17 | 22.7 | 53.5
human accuracy | 80 | 81.8 | 90.6
human agreement | 80 | \- 555There is only one annotator for most languages in XL-WiC. Therefore no agreement score can be computed. | 88.4
languages | 1 | 12 | 15
language families | 1 | 5 | 10
Table 3: Comparison of the most salient data statistics of AM2iCo versus WiC
and XL-WiC.

Our selection of languages is guided by the recent initiatives to cover a
typologically diverse language sample Ponti et al. (2020a). In particular,
AM2iCo covers 15 languages, more than XL-WiC (12 languages). Diversity can be
measured along multiple axes, such as family, geographic areas, and scripts
(Ponti et al., 2019). AM2iCo includes 10 language families, namely: Afro-
Asiatic (1 language), Austronesian (1), Basque (1), Indo-European (5), Japonic
(1), Kartvelian (1), Koreanic (1), Sino-Tibetan (1), Turkic (2), Uralic (1).
This provides a more balanced sample of the cross-lingual variation compared
to XL-WiC (5 families). Regarding geography, in addition to the areas covered
by XL-WiC (mostly Europe and Eastern Asia), we also represent South-East Asia
(with ID), the Middle East (TR), the Caucasus (KA), the Indian subcontinent
(UR and BN), as well as central Asia (KK). Finally, AM2iCo also introduces
scripts that were absent in other datasets, namely the Georgian alphabet and
the Bengali script (a Northern Indian abugida), for a total of 8 distinct
scripts.

##  3 Experimental Method

We now establish a series of baselines on AM2iCo to measure the gap between
current SotA models and human performance.

Pretrained Encoders. Multilingual contextualized representations
$\mathbf{e}_{i}\in\mathbb{R}^{d}$ for each target word are obtained via base
variants of cased multilingual BERT (mBERT, Devlin et al., 2019) and
XLM-R666Due to limited resources, we leave experiments with large variants of
the models for future research.  Conneau et al. (2020), available in the
HuggingFace repository Wolf et al. (2020).

Classification. Given two contextualized representations $\mathbf{e}_{i,src}$
and $\mathbf{e}_{i,trg}$ for a pair of target words, two setups to make
prediction are considered: the first, metric-based, is a non-parametric setup.
In particular, we follow Pilehvar and Camacho-Collados (2019) and score the
distance $\delta$ between the representations via cosine similarity. A
threshold $t$ from the development set is set via grid search across $0.02$
intervals in the $[0,1]$ interval. Therefore, if
$\delta(\mathbf{e}_{i,src},\mathbf{e}_{i,trg})\geq t$ the pair is classified
as negative, and positive otherwise. On the other hand, the fine-tuning setup
is parametric: following Raganato et al. (2020), we train a logistic
regression classifier that takes the concatenation of the contextualized
representations $[\mathbf{e}_{i,src}\oplus\mathbf{e}_{i,trg}]$ as input.777The
first WordPiece is always used if a word is segmented into multiple
WordPieces. The entire model (both the encoder and the classifier) is then
fine-tuned to minimize the cross-entropy loss of the training set examples
with Adam Kingma and Ba (2015). We perform grid search for the learning rate
in $[5e-6,1e-5,3e-5]$, and train for 20 epochs selecting the checkpoint with
the best performance on the dev set.

Cross-lingual Transfer. In addition to supervised learning, we also carry out
cross-lingual transfer experiments where data splits may belong to different
language pairs. The goal is transferring knowledge from a source language pair
$\ell_{s}$ to a target language pair $\ell_{t}$. To simulate different
scenarios of data paucity, in the fine-tuning setup we consider: 1) zero-shot
transfer, where train and development sets belong to $\ell_{s}$ and the test
set to $\ell_{t}$; 2) zero-shot + TLD888TLD stands for “Target Language
Development (set)” transfer, which is similar except for the dev set given in
$\ell_{t}$; 3) on top of zero-shot + TLD, we provide a small amount of
training examples in $\ell_{t}$, which we denote as few-shot transfer; 4)
finally, in the joint multilingual setup, we train a single model on the
concatenation of the train sets for all language pairs and select the hyper-
parameters on the development set of $\ell_{t}$.

##  4 Results and Discussion

|  | DE | RU | JA | ZH | AR | KO | FI | TR | ID | EU | KA | BN | KK | UR
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Mtr | mBERT | 67.1 | 65.0 | 62.3 | 65.8 | 63.9 | 62.1 | 61.7 | 57.1 | 66.3 | 64.1 | 60.4 | 60.0 | 59.2 | 58.8
XLM-R | 65.0 | 63.1 | 56.7 | 56.7 | 58.4 | 57.5 | 64.1 | 62.4 | 65.7 | 62.9 | 58.3 | 56.2 | 58.0 | 55.5
Ft | mBERT | 80.0 | 75.1 | 73.9 | 71.0 | 67.4 | 68.2 | 71.6 | 69.3 | 64.6 | 62.2 | – | – | – | –
XLM-R | 77.4 | 76.1 | 75.9 | 68.9 | 65.9 | 65.3 | 68.4 | 64.4 | 54.6 | 55.8 | – | – | – | –
Hum | accuracy | 93.5 | 89.5 | 93.0 | 87.5 | 93.5 | 93.5 | 90.5 | 90.5 | 91.5 | 92.5 | 90.0 | 89.5 | 85.5 | 88.0
agreement | 90.0 | 78.0 | 90.0 | 94.0 | 100.0 | 92.0 | 88.0 | 96.0 | 92.0 | 84.0 | 94.0 | 80.0 | 80.0 | 80.0
Table 4: Accuracy of mBERT and XLM-R on AM2iCo in a supervised learning
setting. We report metric-based classification (Mtr) results, as well as the
scores in the fine-tuning setup (Ft). The third group of rows (Hum) displays
human performance, in terms of both accuracy and inter-rater agreement.

Metric-based vs Fine-tuning. We report the results for the supervised learning
setting (where all data splits belong to the same language pair) in Table 4.
The metric-based approach achieves consistent scores across all languages,
fluctuating within the range $[57.1,67.1]$ for mBERT and $[55.5,65.0]$ for
XLM-R. This indicates that the pretrained encoder alone already contains some
relevant linguistic knowledge, to a certain degree. In comparison, fine-tuning
yields more unequal results, being more data-hungry. In particular, it
performs worse than the metric-based approach on languages with small training
data size (e.g. ID and EU in Table 4), whereas it surpasses the metric-based
approach on languages with abundant examples (e.g. DE and RU). For an in-depth
analysis of the effect of train data size on fine-tuning, cf. Appendix A.

XLM-R vs mBERT. Table 4 also reveals that XLM-R is more sensitive to train
data size than mBERT, often falling behind in both the metric-based and fine-
tuning setups, especially for resource-poorer languages. These findings are in
line with what Vulić et al. (2020) report for Multi-SimLex, which are grounded
on lexical semantics similarly to AM2iCo. However, they contradict the
received wisdom from experiments in other multilingual sentence-level tasks
(Ponti et al., 2020a; Conneau et al., 2020), where XLM-R outperforms mBERT in
cross-lingual transfer. While the exact causes go beyond the scope of this
paper, we speculate that the two encoders excel in separate aspects of
semantics, the lexical level and the sentence level.

Zero-shot Transfer. The results are presented in Table 5. We select the
training data of each of the five languages with most data (DE, RU, JA, ZH,
AR) in turn for source-language fine-tuning. Subsequently, we report the
average prediction performance across all remaining 9 target languages.

| Zero-shot | +TLD
---|---|---
$\ell_{s}$ | mBERT | XLM-R | mBERT | XLM-R
DE (all) | 71.2 | 72.0 | 71.5 | 71.7
RU (all) | 71.1 | 69.8 | 71.0 | 69.9
JA (all) | 68.1 | 61.9 | 68.6 | 63.2
ZH (all) | 66.2 | 60.3 | 66.6 | 62.1
AR (all) | 67.7 | 61.8 | 67.1 | 62.1
DE (10k) | 65.4 | 62.4 | 65.9 | 62.5
RU (10k) | 66.4 | 64.7 | 66.1 | 64.6
JA (10k) | 67.5 | 61.3 | 67.2 | 61.7
ZH (10k) | 66.0 | 62.8 | 65.8 | 62.1
AR (10k) | 67.7 | 61.8 | 67.1 | 62.1
Table 5: Zero-shot transfer from 5 high-resource source languages to the
remaining 9 languages in AM2iCo. The parentheses contain (approximate) train
data sizes.

First, we note that the TLD variant for hyperparameter selection does not
yield gains. Second, the best choice of source language appears to be German
across the board, achieving an average score of 71.2 with mBERT and 72.0 with
XLM-R. Nevertheless, this is simply due to its ample number of examples (50k).
In fact, controlling for this variable by equalizing the total size of each
train split to 10k, see the bottom half of Table 5, all source languages
perform comparably. Breaking down the average results into individual
languages in Table 6 (top section), however, reveals an even more intricate
picture. In particular, the best source language for KK is RU and for JA is
ZH, rather than DE. This can be explained by the fact that these pairs share
their scripts, Cyrillic and Kanji / Hanzi, respectively, at least in part.
This indicates that a resource-leaner but related language might sometimes be
a more effective option as source language than a resource-rich one. It is
also noteworthy that zero-shot transfer from DE outperforms supervised
learning in most languages, except for those both resource-rich and distant
(JA, ZH and AR).

$\ell_{s}$ | model | DE | RU | JA | ZH | AR | KO | FI | TR | ID | EU | KA | BN | KK | UR |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot Transfer |
DE | mBERT | - | 75.1 | 71.1 | 68.9 | 64.8 | 71.1 | 78 | 75.7 | 75.4 | 74.2 | 70.2 | 68 | 62.8 | 68.5 |
| XLM-R | - | 77.5 | 70.1 | 67.7 | 67.4 | 70.2 | 77.2 | 76.6 | 74.8 | 70.7 | 71.9 | 70.6 | 66.5 | 67 |
RU | mBERT | 74.4 | - | 71.7 | 68.4 | 65.1 | 71 | 74.8 | 72.6 | 74.5 | 71.2 | 69.3 | 70.3 | 67 | 67.8 |
| XLM-R | 72.4 | - | 68.2 | 64.4 | 69.5 | 72.3 | 73.7 | 71.2 | 69.7 | 68.8 | 69.6 | 68.3 | 69 | 66.7 |
JA | mBERT | 70.9 | 70.7 | - | 68.9 | 63.2 | 72.5 | 70.9 | 72.5 | 70.1 | 67.7 | 65.3 | 67.9 | 65.8 | 65 |
| XLM-R | 64.7 | 65.7 | - | 70.4 | 63.2 | 67.1 | 66.5 | 64.9 | 62.5 | 62.5 | 61.7 | 66.7 | 57.8 | 59.3 |
ZH | mBERT | 71.5 | 69.2 | 68.7 | - | 61.6 | 70.6 | 68.6 | 70.8 | 66.9 | 68 | 64.9 | 65.6 | 61.5 | 62.8 |
| XLM-R | 66.5 | 67.1 | 72.1 | - | 62.6 | 62.9 | 66.2 | 63.8 | 63.4 | 61.6 | 59.8 | 61.1 | 59.8 | 60.3 |
AR | mBERT | 69.2 | 67.6 | 68.5 | 66.3 | - | 67.1 | 68.6 | 69.5 | 68.2 | 67.6 | 66.9 | 68.1 | 63.8 | 63.8 |
| XLM-R | 62 | 63 | 62.2 | 62.0 | - | 61.9 | 62.2 | 63.6 | 63.9 | 61.3 | 62.1 | 64.9 | 60.8 | 58.3 |
Joint Multilingual Learning |
all | mBERT | 80.4 | 82.1 | 78.2 | 75.2 | 73.3 | 75.8 | 81.2 | 80.6 | 78.4 | 75.9 | 76.5 | 76 | 71.3 | 73.3 |
| XLM-R | 79.4 | 80.9 | 79.4 | 76.1 | 73.6 | 76 | 81.2 | 80.5 | 77.9 | 74.2 | 77.7 | 73 | 74.5 | 72.5 |
Table 6: Results for zero-shot transfer from a single source (top section) and
joint transfer from multiple sources (bottom section) in AM2iCo. The best
scores for each setup are in bold. Figure 1: Performance in two $L_{trg}$ RU
and JA) across different amounts of training examples under different
settings: supervised learning (examples directly from $L_{trg}$) few-shot
transfer (examples from DE $L_{src}$ plus $L_{trg}$ examples), and zero-shot
transfer from DE.

Few-shot Transfer. To study the differences between training on $\ell_{s}$ and
$\ell_{t}$ with controlled train data size, we plot the model performance on
two target languages (RU and JA) as a function of the amount of available
examples across different transfer conditions in Figure 1. Comparing
supervised learning (based on target language data) with zero-shot learning
(based on DE data), it emerges how the former is always superior if the number
of examples is the same. However, zero-shot learning may eventually surpass
the peak performance of supervised learning by taking advantage of a larger
pool of examples: this is the case in RU, but not in JA. This illustrates a
trade-off between quality (in-domain but possibly scarce data) and quantity
(abundant but possibly out-of-domain data).

Few-shot learning combines the desirable properties of both approaches. After
pre-training a model on DE, it can be adapted on a small amount of target-
language examples. Performance continues to grow with more shots; with as few
as 1k JA examples it is comparable to supervised learning on 15k examples.
Few-shot learning thus not only achieves the highest scores, but also
leverages costly target-language data in a sample-efficient fashion.

Joint Multilingual Learning. The results for this setting (see §3) are shown
in the bottom section of Table 6. We observe a substantial boost in
performance across all the languages compared to both zero-shot transfer from
any individual language and supervised learning (cf. Table 4), including high-
resource languages such as DE and RU. Low-resource languages enjoy the most
copious gains: with an mBERT encoder, UR improves by 4.8 points, KK by 4.3, BN
by 5.7, and KA by 6.3. However, this is still insufficient to equalize
performances across the board, as the latter group of languages continues to
lag behind: between DE and UR remains a gap of 7.2 points. We speculate that
the reason behind this asymmetry is the fact that in addition to being
resource-poor, UR, KK, BN, and KA are also typologically distant from
languages where most of the examples are concentrated. Overall, these findings
suggest that leveraging multiple sources is better than a single one by virtue
of the transfer capabilities of massively multilingual encoders, as previously
demonstrated (Wu and Dredze, 2019; Ponti et al., 2020b).

Adversarial Examples. Finally, we investigate whether the inclusion of
adversarial examples (see §2.3) makes AM2iCo less likely to be solved by
relying on superficial clues. In Table 7, we compare the performance of mBERT
trained on the full input (which we label Full) with two adversarial baselines
We implement the Tgt variant by inputting only the target word to the
classification model and the Ctx variant by replacing the target word with a
‘[MASK]’ token. We perform analysis across AM2iCo, WiC and XL-WiC. For better
diagnosis and comparability, we select languages with sufficient and
comparable training data from AM2iCo and XL-WiC: we select ZH for AM2iCo, and
DE for AM2iCo and XL-WiC.

In previous datasets, at least one of the adversarial baselines reaches
performance close to the full model: in WiC (EN), Ctx has a gap of only 2
points. In XL-WiC (DE), Tgt is only 1 point away from Full. This would also be
the case in AM2iCo were it not for the extra adversarial examples (rows +A):
by virtue of this change, the distance between Full and the best adversarial
baseline is 6.4 points in DE and 5.0 in ZH. Therefore, it is safe to conclude
that a higher score on AM2iCo better reflects a deep semantic understanding by
the model. Moreover, the last column of Table 7 also includes human accuracy
as a reference. While the best baseline in XL-WiC even surpasses the human
upper bound, the addition of adversarial examples in AM2iCo combined with
higher human accuracy drastically increases the gap between the two: 13.5
points in DE and 16.5 in ZH. Overall, this results in a much more challenging
evaluation benchmark.

Dataset | $\ell$ |  Full |  Ctx |  Tgt |  Hum
---|---|---|---|---|---
WiC | EN |  67.05 |  65.05 |  55.2 |  80.0
XL-WiC | DE |  81.0 |  75.0 |  80.0 |  74.0
AM2iCo-A | DE |  84.2 |  83.6 |  50.0 |  93.0
AM2iCo +A | DE |  80.0 |  73.6 |  66.9 |  93.5
AM2iCo-A | ZH |  79.8 |  78.2 |  49.0 |  86.0
AM2iCo +A | ZH |  71.0 |  66.0 |  61.6 |  87.5
Table 7: The impact of adversarial examples. +A indicates the presence of
adversarial examples, -A their absence. Ctx represents training a model on the
context only, Tgt on the target word only, and full on the whole input. Hum
stands for human accuracy.

##  5 Related Work

Cross-Lingual Evaluation of Word Meaning in Context. Going beyond readily
available sense inventories required for WSD-style evaluations, the
comprehensive benchmarks for evaluating word meaning in context cross-
lingually are still few and far between. XL-WiC Raganato et al. (2020) extends
the original English WiC framework of Pilehvar and Camacho-Collados (2019) to
12 other languages, but supports only monolingual evaluation, and suffers from
issues such as small gaps between human and system performance. A currently
ongoing SemEval-2021 shared task (MCL-WiC) does focus on cross-lingual WiC,
but covers only five high-resource languages from three language families
(English, French, Chinese, Arabic, Russian). Both XL-WiC and MCL-WiC mainly
focus on common words and do not include less frequent concepts (e.g., named
entities). Further, their language coverage and data availability are heavily
skewed towards Indo-European languages.999We note that, since details
regarding MCL-WiC have not been released yet, we cannot provide other direct
comparisons to this dataset in our work.

Besides WiC-style evaluation resources, there are several other datasets
designed to evaluate cross-lingual context-aware lexical representations.
Bilingual Contextual Word Similarity (BCWS) Chi and Chen (2018) challenges a
model to predict graded similarity of cross-lingual word pairs given
sentential context, one in each language. In the Bilingual Token-level Sense
Retrieval (BTSR) task Liu et al. (2019), given a query word in a source
language context, a system must retrieve a meaning-equivalent target language
word within a target language context.101010BTSR could be seen as a
contextualized version of the standard bilingual lexicon induction task
(Mikolov et al., 2013; Søgaard et al., 2018; Ruder et al., 2019, inter alia).
However, both BCWS and BTSR are again very restricted in terms of language
coverage: BCWS covers only one language pair (EN-ZH), while BTSR contains two
pairs (EN-ZH/ES). Further, they provide only test data: therefore, they can
merely be used as general intrinsic probes for pretrained models, but cannot
support fine-tuning experiments and cannot fully expose the relevance of
information available in pretrained models for downstream applications. This
might be problematic as intrinsic tasks in general do not necessarily
correlate well with downstream performance Chiu et al. (2016); Glavaš et al.
(2019).

AM2iCo vs. Entity Linking. Our work is related to the standard entity linking
(EL) task Rao et al. (2013); Cornolti et al. (2013); Shen et al. (2014)
similarly to how the original WiC (based on WordNet knowledge) is related to
WSD. EL systems must map entities in context to a predefined knowledge base
(KB). While WSD relies on the WordNet sense inventory, the EL task focuses on
KBs such as Wikipedia and DBPedia. For instance, when each entity mention is
mapped to a unique Wikipedia page, this procedure is termed wikification
Mihalcea and Csomai (2007). The cross-lingual wikification task Ji et al.
(2015); Tsai and Roth (2016) grounds multilingual mentions to the EN Wikipedia
pages. Similar to WSD, the evaluation of an EL task is tied to a specific KB.
Therefore, it faces similar limitations of WSD in terms of restricting
meanings and their distinctions to those predefined in the inventory. In
comparison, AM2iCo leverages Wikipedia only as a convenient resource for
extracting the examples, similar to how the original WiC work leverages
WordNet. AM2iCo itself is then framed on natural text, without requiring the
modeling of the KBs. Also, in comparison with EL, AM2iCo focuses on higher
data quality and a more challenging evaluation of complex word-context
interactions, which is achieved by a carefully-designed data extraction and
filtering procedure.

##  6 Conclusion

We presented AM2iCo, a large-scale and challenging multilingual benchmark for
evaluating word meaning in context (WiC) across languages. AM2iCo is
constructed by leveraging multilingual Wikipedias, and subsequently validated
by humans. It covers 15 typologically diverse languages and a vocabulary
substantially larger than all previous WiC datasets. Thus, it provides more
comprehensive and reliable quality estimates for multilingual encoders.
Moreover, AM2iCo includes adversarial examples whose solution requires genuine
lexical understanding, as opposed to relying on spurious correlations from
partial input. Finally, AM2iCo offers the possibility to perform cross-lingual
evaluation, pairing context between different languages, and with coverage of
a wide range of language families, involving several scripts, we are able to
explore the impact of language relatedness on model performance by performing
transfer learning from multiple source languages. We established a series of
baselines on AM2iCo, based on SotA multilingual models, revealing that the
task is far from being ‘solved’ even with large training data. All models
struggle especially when transferring to distant and resource-lean target
languages. We hope that AM2iCo will guide and foster further research on
effective representation learning across different languages.

## Acknowledgments

We acknowledge Peterhouse College at University of Cambridge for funding
Qianchu Liu’s PhD research. The work was also supported by the ERC
Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909)
awarded to Anna Korhonen. We also appreciate many helpful discussions and
feedback from our colleagues in the Language Technology Lab.

## References

  * Barba et al. (2020) Edoardo Barba, Luigi Procopio, Niccolò Campolungo, Tommaso Pasini, and Roberto Navigli. 2020.  MuLaN: Multilingual label propagation for word sense disambiguation.  In _Proceedings of IJCAI 2020_ , pages 3837–3844.
  * Chi and Chen (2018) Ta-Chung Chi and Yun-Nung Chen. 2018.  CLUSE: Cross-lingual unsupervised sense embeddings.  In _Proceedings of EMNLP 2018_ , pages 271–281.
  * Chiu et al. (2016) Billy Chiu, Anna Korhonen, and Sampo Pyysalo. 2016.  Intrinsic evaluation of word vectors fails to predict extrinsic performance.  In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP_ , pages 1–6.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of ACL 2020_ , pages 8440–8451.
  * Cornolti et al. (2013) Marco Cornolti, Paolo Ferragina, and Massimiliano Ciaramita. 2013.  A framework for benchmarking entity-annotation systems.  In _Proceedings of WWW 2013_ , pages 249–260.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT 2019_ , pages 4171–4186.
  * Glavaš et al. (2019) Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019.  How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions.  In _Proceedings of ACL 2019_ , pages 710–721.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of NAACL-HLT 2018_ , pages 107–112.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of NAACL-HLT 2018_ , pages 1875–1885.
  * Ji et al. (2015) Heng Ji, Joel Nothman, Ben Hachey, and Radu Florian. 2015.  Overview of TAC-KBP2015 tri-lingual entity discovery and linking.  In _Proceedings of TAC 2015_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of EMNLP 2017_ , pages 2021–2031.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _Proceedings of ICLR 2015_.
  * Liu et al. (2019) Qianchu Liu, Diana McCarthy, Ivan Vulić, and Anna Korhonen. 2019.  Investigating cross-lingual alignment methods for contextualized embeddings with token-level evaluation.  In _Proceedings of CoNLL 2019_ , pages 33–43.
  * Mihalcea and Csomai (2007) Rada Mihalcea and Andras Csomai. 2007.  Wikify! Linking documents to encyclopedic knowledge.  In _Proceedings of CIKM 2007_ , pages 233–242.
  * Mikolov et al. (2013) Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013.  Exploiting similarities among languages for machine translation.  _arXiv preprint, CoRR_ , abs/1309.4168.
  * Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014.  Entity linking meets word sense disambiguation: a unified approach.  _Transactions of the ACL_ , 2:231–244.
  * Navigli (2009) Roberto Navigli. 2009.  Word sense disambiguation: A survey.  _ACM Computing Surveys_ , 41(2):1–69.
  * Navigli et al. (2013) Roberto Navigli, David Jurgens, and Daniele Vannella. 2013.  Semeval-2013 task 12: Multilingual word sense disambiguation.  In _Proceedings of SEMEVAL 2013_ , pages 222–231.
  * Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012.  Joining forces pays off: Multilingual joint word sense disambiguation.  In _Proceedings of EMNLP-CoNLL 2012_ , pages 1399–1410.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of ACL 2019_ , pages 4658–4664.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of NAACL-HLT 2019_ , pages 1267–1273.
  * Ponti et al. (2020a) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020a.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of EMNLP 2020_ , pages 2362–2376.
  * Ponti et al. (2019) Edoardo Maria Ponti, Helen O’horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019.  Modeling language variation and universals: A survey on typological linguistics for natural language processing.  _Computational Linguistics_ , 45(3):559–601.
  * Ponti et al. (2020b) Edoardo Maria Ponti, Ivan Vulić, Ryan Cotterell, Marinela Parovic, Roi Reichart, and Anna Korhonen. 2020b.  Parameter space factorization for zero-shot learning across tasks and languages.  _Transactions of the ACL_.
  * Raganato et al. (2017) Alessandro Raganato, José Camacho-Collados, and Roberto Navigli. 2017.  Word sense disambiguation: A unified evaluation framework and empirical comparison.  In _Proceedings of EACL 2017_ , pages 99–110.
  * Raganato et al. (2020) Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020.  XL-WiC: A multilingual benchmark for evaluating semantic contextualization.  pages 7193–7206.
  * Raiman and Raiman (2018) Jonathan Raiman and Olivier Raiman. 2018.  DeepType: Multilingual entity linking by neural type system evolution.  In _Proceedings of AAAI 2018_ , pages 5406–5413.
  * Rao et al. (2013) Delip Rao, Paul McNamee, and Mark Dredze. 2013.  Entity linking: Finding extracted entities in a knowledge base.  In _Multi-source, multilingual information extraction and summarization_ , pages 93–115.
  * Ruder et al. (2019) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019.  A survey of cross-lingual embedding models.  _Journal of Artificial Intelligence Research_ , 65:569–631.
  * Scarlini et al. (2020) Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020.  Sense-annotated corpora for word sense disambiguation in multiple languages and domains.  In _Proceedings of LREC 2020_ , pages 5905–5911.
  * Shen et al. (2014) Wei Shen, Jianyong Wang, and Jiawei Han. 2014.  Entity linking with a knowledge base: Issues, techniques, and solutions.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(2):443–460.
  * Søgaard et al. (2018) Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018.  On the limitations of unsupervised bilingual dictionary induction.  In _Proceedings of ACL 2018_ , pages 778–788.
  * Tsai and Roth (2016) Chen-Tse Tsai and Dan Roth. 2016.  Cross-lingual wikification using multilingual embeddings.  In _Proceedings of NAACL-HLT 2016_ , pages 589–598.
  * Upadhyay et al. (2018) Shyam Upadhyay, Nitish Gupta, and Dan Roth. 2018.  Joint multilingual supervision for cross-lingual entity linking.  In _Proceedings of EMNLP 2018_ , pages 2486–2495.
  * Vulić et al. (2020) Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, et al. 2020\.  Multi-simlex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity.  _Computational Linguistics_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of EMNLP 2020: System Demonstrations_ , pages 38–45.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of EMNLP-IJCNLP 2019_ , pages 833–844.

##  Appendix A Effect of Data Size on Fine-Tuning

To further investigate the effect of train data size on fine-tuning, we
perform an in-depth analysis on some selected languages (DE, RU and JA). Note
that we use the larger dev and test sets for DE and RU for this experiment. We
study how performance changes as we vary the number of training examples from
500 to the full set. Based on the results plotted in Figure 2, we observe that
while fine-tuning starts lower than the metric-based baseline, it grows
steadily and begins to take the lead from around 2,500 train examples.

(a) mBERT (b) XLM-R

Figure 2: Effect of changes of train data size for fine-tuning performance in
DE, RU and JA from AM2iCo. X axis is in the log scale.


#  J-CHAT: Japanese Large-scale Spoken Dialogue Corpus
for Spoken Dialogue Language Modeling

Wataru Nakata1∗ Kentaro Seki1 Hitomi Yanaka1
Yuki Saito1 Shinnosuke Takamichi1,2 Hiroshi Saruwatari1
1 The University of Tokyo, Japan 2 Keio University, Japan
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Spoken dialogue plays a crucial role in human-AI interactions, necessitating
dialogue-oriented spoken language models (SLMs). To develop versatile SLMs,
large-scale and diverse speech datasets are essential. Additionally, to ensure
hiqh-quality speech generation, the data must be spontaneous like in-wild data
and must be acoustically clean with noise removed. Despite the critical need,
no open-source corpus meeting all these criteria has been available. This
study addresses this gap by constructing and releasing a large-scale spoken
dialogue corpus, named Japanese Corpus for Human-AI Talks (J-CHAT), which is
publicly accessible. Furthermore, this paper presents a language-independent
method for corpus construction and describes experiments on dialogue
generation using SLMs trained on J-CHAT. Experimental results indicate that
the collected data from multiple domains by our method improve the naturalness
and meaningfulness of dialogue generation.

J-CHAT: Japanese Large-scale Spoken Dialogue Corpus
for Spoken Dialogue Language Modeling

Wataru Nakata1∗ Kentaro Seki1 ††thanks:  Equal contribution. Hitomi Yanaka1
Yuki Saito1 Shinnosuke Takamichi1,2 Hiroshi Saruwatari1 1 The University of
Tokyo, Japan 2 Keio University, Japan<EMAIL_ADDRESS>

##  1 Introduction

Spoken dialogue systems have garnered significant attention for their role in
enabling human-AI communication Mitsui et al. (2023). Spoken language holds
rich expressiveness through tone, intonation, and non-verbal vocalizations
such as laughter and screams, highlighting the importance of spoken dialogue
systems. To leverage the expressive capabilities of spoken language in spoken
dialogue systems, it is essential to develop spoken language models (SLMs)
that model the linguistic structures of spoken dialogue. In other words, SLMs
are crucial for facilitating smooth and effective human-AI communication.
Generative spoken language modeling (GSLM, Lakhotia et al., 2021) using HuBERT
Hsu et al. (2021) as a speech feature extractor is a well-known example of
SLMs.

Training SLMs requires a large-scale speech corpus. Open-source corpora are
crucial because they lower research barriers and ensure reproducible results.
Despite their importance, the development of spoken dialogue corpora has been
limited. Current methods for constructing spoken dialogue corpora Cieri et al.
(2004); Saito et al. (2022) are labor-intensive making it challenging to scale
the corpora. Furthermore, large-scale spoken dialogue corpora have primarily
been constructed for English Cieri et al. (2004), with limited progress in
other languages, such as Japanese.

Therefore, we construct and release a new open-source corpus named “Japanese
Corpus for Human-AI Talks” (J-CHAT)111https://huggingface.co/datasets/sarulab-
speech/J-CHAT. To collect large-scale speech data, we propose a fully
automated thus scalable method for corpus construction, including data
collection from the web. Gathering in-the-wild data ensures the collection of
spontaneous speech covering a wide range of conversational topics.
Additionally, we use noise removal techniques to make the data acoustically
clean. Our contributions include:

  * •

Constructing and releasing a novel, open-source, and large-scale speech
corpus.

  * •

Providing a spontaneous and clean speech corpus, thereby facilitating the
generation of natural dialogues.

  * •

Developing a corpus construction method that is independent of the language.

Table 1:  Comparison of speech corpora related to this study, sorted by size.

Corpus name | Size(hours) | Open-source | Dialogue | Spontaneous | Clean speech
---|---|---|---|---|---
STUDIES (Saito et al., 2022) | $8.2$ |  |  |  |

& ✅ ✅ DailyTalk (Lee et al., 2023) $20$ ✅ ✅ ✅ ✅ Fisher (Cieri et al., 2004)
$2k$ ✅ ✅ ✅ GigaSpeech (Chen et al., 2021) $33k$ ✅ \- J-CHAT (This study) $69k$
✅ ✅ ✅ ✅

##  2 Related Work

###  2.1 Spoken Language Models

Large-scale SLMs have been reported to enhance the performance of various
downstream tasks. Training these models requires corpora that are
significantly larger than traditional ones. For instance, research using 100k
hours of audio for dialogue generation Borsos et al. (2023), 680k hours for
speech recognition Radford et al. (2023), and 4.5M hours for speech
translation Barrault et al. (2023) has been reported. These studies
demonstrate that increasing the amount of data leads to performance
improvements, motivating us to construct large-scale speech corpora.

However, these studies primarily focus on model architecture improvements, and
the datasets used in the experiments are often not publicly available, with
little to no detail on their construction methods. One research problem that
suffers from a lack of data is dialogue spoken language modeling Nguyen et al.
(2023). In a previous study, 2000 hours of dialogue data were used, drawing
from one of the largest available dialogue speech data resources. Despite this
substantial dataset, the researchers reported that its size was insufficient
for effective dialogue spoken language modeling. By proposing and releasing a
method for constructing a large-scale corpus specifically for the dialogue
domain, we aim to facilitate discussions and advancements in corpus
construction techniques.

###  2.2 Existing Speech Corpora

Training SLMs for dialogues requires large-scale, spontaneous and acoustically
clean dialogue speech data. Table 1 shows corpora related to this study.
STUDIES corpus Saito et al. (2022) and DailyTalk corpus Lee et al. (2023) are
open-source dialogue corpora designed for multi-turn conversations, but they
are very small in size because they are manually recorded spoken dialogues.
Fisher corpus Cieri et al. (2004) is more extensive but is not open-source.
GigaSpeech corpus Chen et al. (2021) is a large-scale open-source corpus, but
it is designed for speech recognition and segmented into individual
utterances, making it unsuitable for dialogue modeling.

##  3 Corpus Construction Methodology

Figure 1:  Corpus construction methodology proposed in this work.

Figure 1 shows the overall flow of the corpus construction process. To
construct a target-language (in our case, Japanese) spoken dialogue corpus, we
collected data from the internet, excluded inappropriate content from the
linguistic, dialogue, and acoustic perspectives by identifying the spoken
language, extracting dialogue utterances, and removing background noise.

###  3.1 Data Collection

Since previous research Seki et al. (2023) has demonstrated that a diverse
range of speech can be collected from YouTube, we used it as one of our data
sources. We searched YouTube with randomly extracted keywords from Wikipedia
page names, as in the previous research Takamichi et al. (2021), resulting in
approximately 600k audio files totaling around 180k hours of audio data.

YouTube contains not only dialogue-dominant videos but also non-speech videos
like music and monologue videos like game commentaries. This resulted in a low
proportion of dialogue data, making it challenging to secure a sufficient
amount of data. To address this, we also collected data from podcasts to
expand the scale of our dataset. Podcasts are speech platforms, making it
efficient to gather speech data from them. Furthermore,
PodcastIndex222https://podcastindex.org/ provides extensive metadata,
including labels that indicates which language the content is. We retrieved
RSS feed URLs of all podcast stations labeled as Japanese from podcast Index.
Subsequently, we searched and downloaded for any audio URLs listed on the
collected RSS feeds. As a result, we obtained approximately 880k audio files
totaling around 140k hours of audio data.

###  3.2 Data Selection

####  3.2.1 Extracting Japanese Speech Data

For filtering out non-Japanese data, we used Whisper’s language identification
model Radford et al. (2023), with the probability $p$ of being Japanese. Then,
we extracted data only if satisfying $p$ exceed $0.8$. This process extracted
$55.7\%$ of the data from YouTube and $84.7\%$ of the data from podcasts.

####  3.2.2 Extracting Dialogue Speech Data

From this Japanese speech dataset, we specifically extracted dialogue speech
data. This requires detecting dialogue segments within the audio data. Related
to this, speaker diarization (SD) is studied to identify who is speaking and
when. SD analyzes the audio data to detect speech segments and links segments
spoken by the same speaker, outputting pairs of speech segments and speaker
IDs. In this study, we used publicly available pre-trained SD models
(PyAnnote, Plaquet and Bredin, 2023) to obtain speech segments with speaker
IDs.

Based on the speech segments (hereafter referred to as turns), we split the
audio data into separate dialogues at gaps of 5 seconds or more. Next, we
filtered out dialogues where a single speaker’s turns account for more than
$80\%$ of the time, ensuring that only valid conversations are selected. This
is to consider dialogues dominated by a single speaker as monologues. Here, we
allowed dialogues with two or more distinct speaker IDs as valid types of
conversations.

Through the above process, we obtained pairs of dialogue speech data and their
labels (each turn’s duration and speaker ID). The proportion of Japanese
speech data containing dialogues was $41.9\%$ for the YouTube data and
approximately $45.0\%$ for the podcast data.

###  3.3 Data Cleansing

Podcasts often include background music (BGM), which acts as noise for speech
generation models, so it needs to be removed. Techniques for extracting speech
from data mixed with BGM are studied in the fields of speech enhancement and
source separation, and recently, machine learning models have achieved high
performance in this area. We applied a pre-trained speech enhancement model
(Demucs Rouard et al., 2023) for data cleansing to all the collected audio and
obtained J-CHAT.

##  4 Corpus Analysis

###  4.1 Dataset Size

Table 2: Corpus statistics by its subsets, YouTube and Podcast. # means “number of”. feature | YouTube | Podcast | Total
---|---|---|---
total duration[hr] | 11,001 | 57,891 | 68,892
# dialogue | 1,013,488 | 3,924,009 | 4,937,497
mean duration [s] | 39.07 | 53.11 | 50.23
mean # turns | 7.58 | 10.68 | 10.10
mean # speakers | 3.23 | 3.12 | 3.14

Table 2 shows the statistical analysis of J-CHAT. J-CHAT consists of a total
of 69k hours of Japanese speech data, with the YouTube subset accounting for
11k hours and the podcast subset for 58k hours.

A notable difference between the subsets is that the average duration of
dialogues in the podcast subset is approximately 1.4 times longer, which is
attributed to a higher number of turns per dialogue. On the other hand, the
average number of speakers per dialogue is nearly the same for both subsets.

###  4.2 Phonetic Diversity

Figure 2:  Distribution of HuBERT Hsu et al. (2021) features extracted from
J-CHAT (ours), STUDIES (simulated dialogue), and JNV (non-verbal expression).

To confirm the phonetic diversity of J-CHAT, we analyzed the distribution of
HuBERT Hsu et al. (2021) features. HuBERT is a speech encoding model similar
to BERT Devlin et al. (2019) which captures phonetic information from speech
Hsu et al. (2021). We randomly sampled 1000 frame-wise HuBERT features from
each subset of J-CHAT, as well as from STUDIES Saito et al. (2022) and JNV Xin
et al. (2024b). For each sampled point, we compute HuBERT features by randomly
selecting 5-second intervals. These features were then reduced in
dimensionality using t-SNE Van der Maaten and Hinton (2008), as illustrated in
Figure 2. The distributions for STUDIES and JNV, which correspond to text-
aligned speech and non-verbal expressions respectively, were found to be
limited to specific regions. In contrast, the subsets of J-CHAT encompassed
both regions, illustrating the broad phonetic diversity present in J-CHAT.

##  5 Experiments

To validate that J-CHAT corpus is a large-scale corpus suitable for training
dialogue-oriented SLMs, we trained and evaluated the performance of dialogue
generative spoken language model (dGSLM, Nguyen et al., 2023). dGSLM follows a
framework proposed by Lakhotia et al., 2021 which consists of three distinct
modules: speech-to-unit, unit language model, and unit-to-speech (vocoder).

Our experiments include three models and resynthesis samples for comparison: a
resynthesized J-CHAT test subset utilizing speech-to-unit and vocoder
(resynth), dGSLM on the YouTube subset of J-CHAT (dGSLM-YouTube), dGSLM
trained on the podcast subset (dGSLM-podcast), and dGSLM trained on all
subsets of J-CHAT (dGSLM-J-CHAT). Trained models, generated samples and
training data are available1.

###  5.1 Experimental Conditions

We split the dialogues in the J-CHAT corpus into two channels by swapping the
output channel of speech according to the turn-taking event. Subsequently we
randomly split the each subset of J-CHAT corpus into train/valid/test sets.
Then, we performed discretization of speech using k-means clustering on
HuBERT-extracted features using train set from both subsets. The number of
clusters for k-means was set to 1000.

For the vocoder used for speech generation from the discretized speech, we
used HiFi-GAN Kong et al. (2020) conditioned with speaker information from
XVector Snyder et al. (2018), as used in previous work Kharitonov et al.
(2022). Please refer to appendix A for more details.

###  5.2 Evaluation

For evaluation, we performed subjective listening tests on the mean opinion
score (MOS) on the naturalness and meaningfulness of dialogues following
previous work Nguyen et al. (2023). For each subjective listening test, we
recruited 60 Japanese native speakers, and each listener evaluated 8 samples.
For sample generation used in the evaluation, we used the first 5 seconds of
the test set derived from J-CHAT to prompt the model, then performed inference
to predict the next 25 seconds of the dialogue based on those initial 5
seconds. For the sampling method, we used the beam search with a beam size of
5. When synthesizing speech from discritized speech, we conditioned the
vocoder with JVS001 (male) and JVS002 (female) speaker from the JVS corpus
Takamichi et al. (2020). This resulted in the male speaker’s voice being heard
from the first audio channel and the female speaker’s voice from the second
channel.

###  5.3 Result

Table 3: Result for MOS tests with their 95% confidence intervals. Model | Naturalness | Meaningfulness
---|---|---
resynth | 2.55 $\pm$ 0.18 | 2.48 $\pm$ 0.18
dGSLM-Youtube | 1.44 $\pm$ 0.13 | 1.56 $\pm$ 0.14
dGSLM-podcast | 1.44 $\pm$ 0.13 | 1.52 $\pm$ 0.13
dGSLM-J-CHAT | 2.28 $\pm$ 0.19 | 2.18 $\pm$ 0.19

Table 3 shows the results of the subjective evaluation. From these results, it
can be seen that dGSLM-J-CHAT achieves the best performance among the dGSLM
generated samples in both naturalness and meaningfulness. This suggests that
J-CHAT is a useful corpus for constructing generative dialogue language
models. We can also see that there is no statistical significance between
dGSLM-YouTube and dGSLM-podcast, despite dGSLM-podcast being trained on a
dataset approximately four times larger in number of dialogues. This indicates
that simply scaling up the dataset is not enough to enhance dGSLM performance.
However, there is a significant difference between resynth and dGSLM-J-CHAT in
terms of both naturalness and meaningfulness. The trained model occasionally
produces sensible words, but the generated dialogue often lacks coherence.
This might be improved with better modeling or by increasing the dataset size.
For a detailed explanation on the statistical significance of each result,
refer to the appendix B.2.

##  6 Conclusion

We have demonstrated a scalable dataset construction method. The experimental
results showed that collecting from multiple domains, such as YouTube and
podcasts, is beneficial in training dialogue language models. We hope that
this corpus will contribute to the advancement of research in dialogue speech
models.

## Limitations

In this study, we believe that obtaining data from multiple domains, such as
YouTube and podcasts, allows us to gather a wide range of speech. However, we
also recognize that there are limitations to the range that can be covered.
For example, OGVC corpus Arimoto et al. (2012) reports a method to collect a
laughter corpus from the internet, but found that the amount of screaming was
limited because the internet data is typically collected under controlled
conditions. Similarly, podcast dialogues are intended for listeners, so
informal expressions used among close friends are likely to be limited.

## Ethics Statement

We recognize that speech synthesis technology carries the risk of being
misused for unauthorized generation of others’ voices, commonly referred to as
voice cloning. To prevent such misuse, we converted the collected speech into
latent variables known as HuBERT features, which lack speaker identity
information. Although HuBERT does not completely eliminate speaker
information, it obscures it sufficiently to make the restoration of speaker
identity challenging, thus serving as an effective countermeasure against
voice cloning. In our experiments, we synthesized speech using the speaker
identity of individuals who had consented to their voice being used for
synthesis at the time of recording, ensuring that no unauthorized voice
replication was performed.

On the subjective evaluation conducted on Section 5.2, participants are paid
with adequate amount of money and they were instructed with the purpose of the
research and how the collected data will be used. Please refer to the Section
B for the details on the subjective evaluation.

## Acknowledgements

This work was supported by AIST KAKUSEI project (FY2023), JST Moonshot
JPMJMS2011 and JST FOREST JPMJFR226V.

## References

  * Arimoto et al. (2012) Yoshiko Arimoto, Hiromi Kawatsu, Sumio Ohno, and Hitoshi Iida. 2012.  Naturalistic emotional speech collection paradigm with online game and its psychological and acoustical assessment.  _Acoustical science and technology_ , 33(6):359–369.
  * Barrault et al. (2023) Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. 2023.  Seamless: Multilingual expressive and streaming speech translation.  _arXiv preprint arXiv:2312.05187_.
  * Borsos et al. (2023) Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023.  Soundstorm: Efficient parallel audio generation.  _Preprint_ , arXiv:2305.09636.
  * Chen et al. (2021) Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021.  GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio.  In _Proc. Interspeech 2021_ , pages 3670–3674.
  * Cieri et al. (2004) Christopher Cieri, David Miller, and Kevin Walker. 2004.  The Fisher Corpus: a resource for the next generations of speech-to-text.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Kharitonov et al. (2022) Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2022.  Text-free prosody-aware generative spoken language modeling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8666–8681, Dublin, Ireland. Association for Computational Linguistics.
  * Kong et al. (2020) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020.  HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17022–17033. Curran Associates, Inc.
  * Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354.
  * Lee et al. (2023) Keon Lee, Kyumin Park, and Daeyoung Kim. 2023.  DailyTalk: Spoken dialogue dataset for conversational text-to-speech.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Mitsui et al. (2023) Kentaro Mitsui, Yukiya Hono, and Kei Sawada. 2023.  Towards human-like spoken dialogue generation between ai agents from written dialogue.  _Preprint_ , arXiv:2310.01088.
  * Nguyen et al. (2023) Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023.  Generative spoken dialogue language modeling.  _Transactions of the Association for Computational Linguistics_ , 11:250–266.
  * Plaquet and Bredin (2023) Alexis Plaquet and Hervé Bredin. 2023.  Powerset multi-class cross entropy loss for neural speaker diarization.  In _Proc. INTERSPEECH 2023_ , pages 3222–3226.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 28492–28518.
  * Rouard et al. (2023) Simon Rouard, Francisco Massa, and Alexandre Défossez. 2023.  Hybrid transformers for music source separation.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Saito et al. (2022) Yuki Saito, Yuto Nishimura, Shinnosuke Takamichi, Kentaro Tachibana, and Hiroshi Saruwatari. 2022.  STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agent.  In _Proc. Interspeech 2022_ , pages 5155–5159.
  * Sawada et al. (2024) Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024.  Release of pre-trained models for the Japanese language.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 13898–13905, Torino, Italia. ELRA and ICCL.
  * Seki et al. (2023) Kentaro Seki, Shinnosuke Takamichi, Takaaki Saeki, and Hiroshi Saruwatari. 2023.  Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 1–5.
  * Snyder et al. (2018) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018.  X-Vectors: Robust dnn embeddings for speaker recognition.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5329–5333.
  * Takamichi et al. (2021) Shinnosuke Takamichi, Ludwig Kürzinger, Takaaki Saeki, Sayaka Shiota, and Shinji Watanabe. 2021.  JTubeSpeech: corpus of japanese speech collected from youtube for speech recognition and speaker verification.  _Preprint_ , arXiv:2112.09323.
  * Takamichi et al. (2020) Shinnosuke Takamichi, Ryosuke Sonobe, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari. 2020.  JSUT and JVS: Free japanese voice corpora for accelerating speech synthesis research.  _Acoustical Science and Technology_ , 41(5):761–768.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(86):2579–2605.
  * Xin et al. (2024a) Detai Xin, Junfeng Jiang, Shinnosuke Takamichi, Yuki Saito, Akiko Aizawa, and Hiroshi Saruwatari. 2024a.  JVNV: A corpus of japanese emotional speech with verbal content and nonverbal expressions.  _IEEE Access_ , 12:19752–19764.
  * Xin et al. (2024b) Detai Xin, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2024b.  JNV corpus: A corpus of japanese nonverbal vocalizations with diverse phrases and emotions.  _Speech Communication_ , 156:103004.

##  Appendix A dGSLM Training Details

For the implementation of dGSLM, we used
fairseq333https://github.com/facebookresearch/fairseq repository by Meta which
includes official implementation of the model. This repository is licensed as
MIT.

The number of train/valid/test is 1,003,397/9,991/100 dialogues for YouTube
subset and for podcast subset it is 3,885,007/38,902/100 dialogues.

For HuBERT model used in the speech-to-unit, we used the pretrained Japanese
HuBERT model Sawada et al. (2024)444https://huggingface.co/rinna/japanese-
hubert-base. For k-means clustering, we used the original implementation of
the dGSLM except for the number of clusters which is set to 1,000.

The dGSLM model was configured as in the previous research Nguyen et al.
(2023). The number of model parameters in each dGSLM model is 23,638,529.

For training dGSLM model, we used 32 NVIDIA V100 GPUs with the learning rate
of $2\times 10^{-4}$. The training was performed for 100,000 steps. The total
duration of training was around 48 hours. The batch size was 36,864 tokens and
the the maximum number of tokens per sample are set to 3,000. This is
equivalent to the 1 minute of dialogue. For other hyper-parameters regarding
the training, we followed the original paper Nguyen et al. (2023).

For the XVector conditioned vocoder, we used the pretrained XVector
model555https://huggingface.co/speechbrain/spkrec-xvect-voxceleb. The vocoder
was trained with the JVS Takamichi et al. (2020) and JVNV Xin et al. (2024a)
corpus. These two corpora include Japanese reading-style, studio-quality
speech without/with non-verbal expression, respectively. Training of vocoder
took 2 days with 4 NVIDIA V100 GPUs.

##  Appendix B Subjective Evaluation of the dGSLM Outputs

###  B.1 Participant Details

Participants of the each subjective evaluation were recruited on
https://lancers.jp. They are paid with 75 Japanese yen upon the completion of
the test. Given that the minimum wage of Japan is 1,004 Japanese yen and the
test took approximately 4 minutes, the payment is adequate.

Before the participation, the participants were instructed with the research
purpose and how the collected data will be used in this research. The data
collection protocol is approved by our institution’s ethics review board.

Figure 3: Screenshot of instruction given to the participants on the
naturalness MOS evaluation. Figure 4: Screenshot of instruction given to the
participants on the meaningfulness MOS evaluation.

Figures 3 and 4 show the screenshots of the instructions given to the
participants for the naturalness and meaningfulness MOS evaluations,
respectively. The English translation of the instruction for the naturalness
MOS evaluation is as follows:

> Listen to the audio and rate the naturalness of the sound on a scale of 1 to
> 5 (very unnatural (1) to very natural (5)). Please conduct the evaluation
> using headphones in a quiet environment. Please note that some audio samples
> may have degraded sound quality, so please be careful with the volume level.
> Do not use your browser’s “back” or “reload” buttons during the test.

The English translation of the instruction for the meaningfulness MOS
evaluation is as follows:

> Listen to the audio and rate the meaningfulness of the dialogue on a 5-point
> scale from very meaningless (1) to very meaningful (5). Please evaluate in a
> quiet environment using headphones. Please note that some of the audio
> samples may have degraded sound quality, so please be careful with the
> volume. Please do not use your browser’s “back” or “reload” buttons during
> the test.

###  B.2 P-values from the Subjective Evaluation

Figure 5: P-values from the subjective evaluation on the naturalness of the
dGSLM output samples Figure 6: P-values from the subjective evaluation on the
meaningfulness of the dGSLM output samples

Figure 5 and Figure 6 shows the p-values from the subjective evaluation on the
dGSLM output conducted on Section 5.2. Specifically, we performed two sided
T-test on the MOS score.