# Bilingual Corpus Mining and Multistage Fine-Tuning for Improving Machine
Translation of Lecture Transcripts

Haiyue Song1,2 Raj Dabre1 Chenhui Chu2 Atsushi Fujita1 Sadao Kurohashi3
1NICT / Kyoto, Japan 2Kyoto University / Kyoto, Japan 3NII / Tokyo, Japan
{haiyue_song, raj.dabre<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Lecture transcript translation helps learners understand online courses,
however, building a high-quality lecture machine translation system lacks
publicly available parallel corpora. To address this, we examine a framework
for parallel corpus mining, which provides a quick and effective way to mine a
parallel corpus from publicly available lectures on Coursera. To create the
parallel corpora, we propose a dynamic programming based sentence alignment
algorithm which leverages the cosine similarity of machine-translated
sentences. The sentence alignment $F_{1}$ score reaches 96%, which is higher
than using the BERTScore, LASER, or sentBERT methods. For both
English–Japanese and English–Chinese lecture translations, we extracted
parallel corpora of approximately 50,000 lines and created development and
test sets through manual filtering for benchmarking translation performance.
Through machine translation experiments, we show that the mined corpora
enhance the quality of lecture transcript translation when used in conjunction
with out-of-domain parallel corpora via multistage fine-tuning. Furthermore,
this study also suggests guidelines for gathering and cleaning corpora, mining
parallel sentences, cleaning noise in the mined data, and creating high-
quality evaluation splits. For the sake of reproducibility, we have released
the corpora as well as the code to create them. The dataset is available at
https://github.com/shyyhs/CourseraParallelCorpusMining.

Bilingual Corpus Mining and Multistage Fine-Tuning for Improving Machine
Translation of Lecture Transcripts

Haiyue Song1,2 Raj Dabre1 Chenhui Chu2 Atsushi Fujita1 Sadao Kurohashi3 1NICT
/ Kyoto, Japan 2Kyoto University / Kyoto, Japan 3NII / Tokyo, Japan
{haiyue_song, raj.dabre<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Massive open online courses (MOOCs)111http://mooc.org have proliferated,
enabling people to attend lectures regardless of their geographical location.
Typically, such lectures are taught by professors from various universities
and are made available through video recordings. The lectures are usually
taught in one particular language, and the video is accompanied by
transcripts. The transcripts are typically made by volunteers or an automatic
high-accuracy speech recognition system. To distribute knowledge to a more
significant number of people speaking other languages, these transcripts are
then translated into other languages.

Figure 1: Overview of the corpus creation framework. In-domain parallel
documents are crawled from the Coursera website. The machine translation model
trained on out-of-domain data maps data to the same space during sentence
alignment.

A high-quality machine translation (MT) system is desirable to translate
lecture transcripts efficiently since manual transcript translation is a time-
consuming task given that there is an enormous volume of online lectures in
different languages. For lecture transcripts translation, there is sufficient
work for lectures in European languages but none for Japanese and Chinese
languages. The TraMOOC project Kordoni et al. (2015) aims to improve the
accessibility of online lectures in European languages through MT. They focus
on collecting translations of lecture transcripts and constructing MT systems
for 11 European languages. However, the number of parallel resources involving
other languages, such as Chinese and Japanese, is still relatively low
compared to European languages.

Both translating from English to other languages and from other languages to
English are urgent and essential. Given that most lectures are spoken in
English, and non-English speakers worldwide greatly outnumber English
speakers, translating English transcripts to other languages can help
knowledge distribution. Meanwhile, lectures are taught in languages other than
English as well. For instance, several universities in Japan and China offer
online courses taught mainly in Japanese and Chinese, respectively. It is also
essential to enable non-Japanese and non-Chinese speakers to participate in
these courses by translating lecture transcripts into other languages,
especially English.

Subtitle222Lecture transcripts are also known as subtitles, and in this paper,
we use these terms interchangeably. translation falls simultaneously under the
spoken language domain and the scientific domain, wherein there are several
differences between both domains. Firstly, compared with spoken language
domain data such as transcripts of TED talks (Cettolo et al., 2012),
university lectures are devoted mainly to educational purposes, and there are
more terminologies in the lectures. Secondly, compared with scientific domain
data in the ASPEC corpus Nakazawa et al. (2016), lecture transcripts contain
many spoken features. The subtle difference in the domain may hinder
translation quality unless appropriate data reflecting the properties of both
domains are used for developing lecture translation systems.

Obtaining a parallel corpus is the first step in training the MT system, and
applying automatic methods to obtain high-quality parallel data is more
desirable than manual methods. Manual methods such as hiring professional
translators or employing crowdsourcing platforms are economically infeasible,
especially for universities and other non-profit organizations, because the
desired size of the MT dataset is usually large, often ranging from tens of
thousands to hundreds of thousands of examples. There are many automatic
methods for extracting parallel sentences from roughly parallel documents
(also called comparable corpora) (Tiedemann, 2012).333cf. Comparable corpora,
such as Wikipedia, contain pairs of documents comprising the contents on the
same topic. However, their parallelism is not necessarily guaranteed, and the
corresponding sentences are not necessarily in the same order (Chu et al.,
2015). In particular, MT-based approaches are quite desirable because of their
simplicity and the possibility of using existing translation models to extract
additional parallel data. However, if there exists no in-domain MT model,
using an MT system trained on data from another domain can yield unreliable
translations, which can lead to extracted parallel data of low quality. In
addition to using parallel corpora, leveraging in-domain monolingual corpora
through back-translation (Sennrich et al., 2016c) is also an effective way to
improve the performance of the MT system. In this work, we focus on
constructing parallel corpora not only because they usually bring larger
improvement (Edunov et al., 2018), but also because we can build a test set to
benchmark the lecture domain MT systems.

We propose a framework shown in Figure 1 to create high-quality English to
Japanese (Chinese) and Japanese (Chinese) to English MT systems for lecture
transcripts translation. It contains: 1) crawling and cleaning parallel
documents444Parallel documents are also known as comparable documents which
are documents containing the same information in different languages., 2)
extracting in-domain parallel sentences using an initial NMT system trained on
out-of-domain corpora and proposed alignment algorithm, and 3) leveraging both
the lecture domain corpora and out-of-domain corpora to improve the MT system
for lecture transcripts. Our contributions are as follows:

First, we crawl and clean parallel lecture transcript document pairs from
Coursera.555https://www.coursera.org We download the transcripts in all
available languages for each course in Coursera and clean noisy data.

Second, we propose a novel automatic alignment method based on cosine
similarities of sentence vector representations by the word2vec model and
dynamic programming (DP) algorithms to extract parallel sentences from roughly
aligned transcript document pairs. The proposed alignment algorithm achieved a
higher $F_{1}$ score than methods using the BLEU-score-based method, LASER, or
sentBERT. Tested on four manually created document pairs, we show that the
proposed method achieved a 96% $F_{1}$ score. In contrast, the BLEU-based DP
method, BERTScore based DP method, LASER embedding based greedy method, and
LASER embedding based DP method achieved 93%, 89%, 75%, and 90% $F_{1}$
scores, respectively.

Third, with the alignment algorithm, we created two corpora from lectures in
Coursera. The English–Japanese parallel corpus contains 50,543 lines in the
train set, 2,068 lines in the test set, and 555 lines in the development set.
We do manual filtering for the test and development sets to ensure quality. In
contrast to our previous paper Song et al. (2020), we created a high-quality
English–Chinese parallel corpus of 40,074 lines with a manually checked test
set with 2,009 sentences and a development set with 865 sentences. We have
made the data publicly available so that other researchers can use it to
benchmark their MT
systems.666https://github.com/shyyhs/CourseraParallelCorpusMining

Finally, we demonstrate the effectiveness of the proposed corpora through MT
experiments, showing that the datasets are especially useful when combined
with larger out-of-domain corpora using domain adaptation techniques.
Specifically, we explored the potential of multiple datasets for educational
lecture translation. Following the curriculum learning paradigm, we modified
the commonly used two-stage fine-tuning method to a multistage fine-tuning
strategy. Compared with the fine-tuning method Zoph et al. (2016), the
multistage fine-tuning model achieved up to 4.1 BLEU score improvements on
English and Japanese (Chinese) lecture transcripts MT settings. The multistage
method gave the best BLEU scores for all the translation directions.

##  2 Related Work

###  2.1 Educational Domain Corpora

The TraMOOC project (Kordoni et al., 2015, 2016) aims to provide access to
multilingual (mainly European languages) transcripts of online courses using
MT and provides parallel corpora to train the MT system. The AMARA platform
(Jansen et al., 2014) aims at European languages, such as German, Polish, and
Russian. However, there are also many Japanese MOOCs such as The Japan MOOC
777https://gacco.org and Chinese MOOCs such as
CNMOOC888https://www.cnmooc.org/home/index.mooc that contain numerous courses
in Japanese and Chinese. To translate them into English and vice versa,
creating a Japanese (Chinese)–English parallel dataset is the first step.

The domain difference is another issue. There are spoken language corpora such
as TED talks (Cettolo et al.,
2012).999https://wit3.fbk.eu/mt.php?release=2017-01-ted-test and
OpenTranscripts (Tiedemann, 2012)101010https://www.opentranscripts.org.
Moreover, there is ASPEC dataset which contains English–Japanese parallel data
from scientific papers. However, the first two are not in the scientific
domain and ASPEC is not in the spoken language domain. There are yet no
corpora for Japanese (Chinese)–English that falls in both the scientific and
spoken language domains.

Therefore, we collected educational domain data from the Coursera website
which is a prevalent platform for MOOCs and many lectures have multilingual
transcripts. Additionally, they are created by professional and non-
professional high-level human translators, ensuring the high-quality of data.

###  2.2 Automatic Parallel Sentences Extraction

Automatic sentence alignment can extract parallel sentences that are orders of
magnitude larger than those obtained by manual translation. If the crawled
data in the source and target languages is sentence-level, then the neural MT
based method (Chen et al., 2020) can achieve good performance, pretrained
model (Zhang et al., 2020a) can filter noisy data, and the LASER tool for
bitext mining (Chaudhary et al.,
2019)111111https://github.com/facebookresearch/LASER with greedy algorithm
also works well. If the crawled data is document-level, MT-based (Sennrich and
Volk, 2010, 2011; Liu et al., 2018) and similarity-based methods (Bouamor and
Sajjad, 2018; Wang and Neubig, 2019; Zhang et al., 2020b) can give more
accurate sentence alignments. The transcripts are crawled from Coursera. They
are in the document format where each document contains the transcript of one
language from one course. Additionally, lines in the document are in the time
order. Therefore, we use this feature and propose our algorithm using dynamic
programming for alignment (Utsuro et al., 1994). Previous work Vecalign
(Thompson and Koehn, 2019) uses LASER embedding and DP algorithm.

They rely on the similarity of source and target sentence embeddings that are
automatically mapped into the multilingual embedding space. However, we show
that translating data to the same language by an NMT system and utilizing
similarities in the monolingual embedding space gives a more robust alignment
performance.

###  2.3 Domain Adaptation

Neural machine translation (NMT) systems trained on large parallel corpus
provide a high-quality translation (Sutskever et al., 2014; Bahdanau et al.,
2015; Vaswani et al., 2017; Zoph et al., 2016; Koehn and Knowles, 2017).
However, we can only construct small-scale datasets for lecture transcript
translation due to the scarcity of raw data.

Transfer learning (Chu et al., 2017a; Luong and Manning, 2015; Sennrich et
al., 2016a; Zoph et al., 2016; Chu et al., 2017b; Zhang et al., 2019; Hu et
al., 2019; Zeng et al., 2019) is widely used to obtain better translation
quality for low-resource domains. The two-stage fine-tuning algorithm is a
special case of curriculum learning (Bengio et al., 2009) whose basic idea is
to train a model using data ranging from unrelated to related in the learning
stages. It is inspired by the human learning process and was first explored in
the machine learning field, showing that curriculum learning results in better
generalization and faster convergence during training. It has been verified
and widely applied to tasks in different fields, including object
classification (Gong et al., 2016), text-to-text machine translation (Liu et
al., 2020; Zhou et al., 2020), and end-to-end speech translation (Wang et al.,
2020).

In NMT, we usually use two types of datasets during fine-tuning: larger out-
of-domain datasets and smaller in-domain datasets. The model trained on the
out-of-domain dataset obtains a general knowledge of translation. Through
fine-tuning in-domain data as presented, it also acquires domain-specific
knowledge and, thus, performs better in the domain of interest.

There are more than two datasets in our settings, therefore we propose to
modify the two-stage fine-tuning (Zoph et al., 2016) and mixed fine-tuning Chu
et al. (2017b) methods to a multistage fine-tuning method which makes the task
transition smoother and prevents knowledge forgetting of the previous stages.

##  3 Corpus Construction Pipeline

This section describes the pipeline to construct the lecture transcripts
corpus, including 1) crawling and cleaning the parallel lecture transcripts
documents from Coursera as described in Section 3.1, 2) extracting parallel
sentences from parallel documents in Section 2, and 3) creating high-quality
evaluation sets manually in Section 3.3.

###  3.1 Crawling and Cleaning Parallel Transcripts Documents

We discuss how to obtain and clean the raw parallel lecture transcripts
documents in this subsection.

Crawling. We obtain raw lecture transcripts documents from Coursera by
crawling. First, we scrape the list of names of available courses on the
Coursera website. Then, we use the tool Coursera-
dl121212https://github.com/coursera-dl/coursera-dl to download the transcripts
in all available languages for each course in the list. Each transcript
document contains sentences in the time order, and the multilingual documents
for one course are roughly aligned.

Data Cleaning. We applied the following 5-step procedure to clean the raw
data:

  1. 1.

Normalizing Text Encoding: Text in the documents is converted into UTF-8, and
variants of character encoding are normalized to the
NFKC131313https://unicode.org/reports/tr15 format.

  2. 2.

Detecting Language Mismatch: Some transcript document is in a different
language than mentioned in its filename. Thus, it is necessary to detect and
exclude such mismatches. We use handwritten rules for the target languages
English, Japanese, and Chinese considering their special character types. The
automatic language detection tool
langdetect141414https://github.com/Mimino666/langdetect (Raffel et al., 2020)
is also an option.

  3. 3.

Splitting Lines into Sentences: Some lines contain more than one sentences. We
use punctuation marks as the clue to segment such lines into multiple
sentences. We discard files containing no punctuation marks because there is
no reliable way to deal with them.

  4. 4.

Removing Meta Tokens: Tokens indicating meta-information, such as “[Music]”
and “$<<$,” are removed to reduce noise.

  5. 5.

Eliminating Imbalanced Document Pairs: Some document pairs are imbalanced in
terms of size: one side has twice or more sentences than the other. These
document pairs are usually noisy and we eliminate them.

###  3.2 Parallel Sentences Extraction

Figure 2: The proposed sentence alignment flowchart. An initial MT system
translates one document into the language of the other document. Sentences are
converted into embeddings in the same vector space and dynamic programming is
applied to extract 1–to–1 and many–to–many alignments.

We extract parallel sentences from parallel documents as illustrated in Figure
2. We first apply an initial MT system to align the languages of the two
documents. Secondly, we convert sentences to embeddings in the same vector
space. Finally, we apply dynamic programming (Utsuro et al., 1994) to extract
only 1–to–1 case but also many–to–many alignments.

Initial MT System. Given a document pair in two different languages, we
translate one of them into the other language using an MT system (Sennrich and
Volk, 2010). To train such a system, we can leverage any existing parallel
data in related or even distant domains such as scientific paper domain corpus
ASPEC (Nakazawa et al., 2016) and spoken language domain corpus TED talks,
(Cettolo et al., 2012) or news domain corpus News Commentary (Tiedemann,
2012).

Similarity Measure. One key component in the DP algorithm is the scoring
function, i.e., similarity measure. Given two sentences (to be precise, two
sequences, each of which can contain more than one sentence) in different
languages, the function returns a real number indicating how similar they are.

Existing methods, such as those in (Sennrich and Volk, 2010), used sentence-
level BLEU scores of machine-translated sentences in the source language
against the sentences in the target language as their similarity score,
formally:

$\mathit{Sim}_{\mathit{BLEU}}(s_{i},s^{\prime}_{j})=\mathit{BLEU}(\mathit{MT}(s_{i}),s^{\prime}_{j}),$
(1)

where $s_{i}$ and $s^{\prime}_{j}$ are two sentences in different languages.
However, due to the lack of in-domain data, MT systems can give only
translations of low quality, and thus the BLEU scores can be misleading,
especially for low-resource situations, such as lecture transcripts
translation.

An alternative method is to directly compute the embedding cosine similarity
of a given sentence pair (Bouamor and Sajjad, 2018), relying on pre-trained
multilingual word embeddings such as word2vec, LASER, or sentBERT to represent
sentences in different languages with the same vector space through element-
wise addition of word embeddings (Mikolov et al., 2013). However, cross-
lingual embeddings are often inaccurate for distant language pairs, especially
if they have been pre-trained on data from another domain.

Taking inspiration from both these approaches, we employ MT combined with a
cosine similarity of sentence embeddings to measure the similarity of two
sentences in different languages, formulated as follows:

$\displaystyle\begin{split}\mathit{Sim}_{\mathit{EMB}}(s_{i},s^{\prime}_{j})=\cos\left(\mathit{emb}(\mathit{MT}(s_{i})),\mathit{emb}(s^{\prime}_{j})\right)\end{split}$
(2)

We first translate each sentence in the source language document into the
target language. We then calculate the sentence embeddings through function
$\mathit{emb}(\cdot)$, which averages the embeddings of words in that sentence
by pre-trained word vectors,151515http://vectors.nlpl.eu/repository as in the
work by (Mikolov et al., 2013). In practice, we take the average of
$Sim_{EMB}(s_{i},s^{\prime}_{j})$ and $Sim_{EMB}(s^{\prime}_{j},s_{i})$ to
improve the robustness.

DP Algorithm The pseudocode of DP is shown in Algorithm 1, where
$sentences_{1}$ and $sentences_{2}$ represent the lists of sentences in the
given two respective documents. $Order$ represents the maximum number of
sentences we can combine into one. And $similarity(\cdot,\cdot)$ is a function
that returns a score showing the similarity of two lines in different
languages. The DP algorithm can process not only 1–to–1 alignment, but also
many–to–many alignments.

In our scenario, DP is effective because of the monotonic nature of the
transcripts, as the corresponding sentences in each pair of documents are
roughly in the same order. This is based on the fact that the sentences in
lecture transcripts of different languages are in the same order as the
professor’s speech in the course. Consequently, comparing all pairs of
sentences between document pairs is unnecessary (Abdelali et al., 2014).

Input : $\mathit{sentences_{1}}$, $\mathit{sentences_{2}}$, $\mathit{Order}$,
$\mathit{similarity()}$

Output : $\mathit{path}$

1 foreach  _$\mathit{i}\in\mathit{range(0,len(sentences_{1})-1)}$_ do

2 foreach  _$\mathit{j}\in\mathit{range(0,len(sentences_{2})-1)}$_ do

3 $\mathit{f[i][j]=-1}$;

4 $\mathit{path[i][j]=(0,0)}$;

5 foreach  _$\mathit{\delta_{i}}\in\mathit{range(0,Order)}$_ do

6 foreach  _$\mathit{\delta_{j}}\in\mathit{range(0,Order)}$_ do

7 $\mathit{chunk_{1}=sentences_{1}[i-\delta_{i}+1]+...+sentences_{1}[i]}$;

8 $\mathit{chunk_{2}=sentences_{2}[j-\delta_{j}+1]+...+sentences_{2}[j]}$;

9 $\mathit{similarity\\_score=similarity(chunk_{1},chunk_{2})}$;

10 $\mathit{score=similarity\\_score+f[i-\delta_{i}][j-\delta_{j}]}$;

11 if  _$\mathit{score >f[i][j]}$_ then

12 $\mathit{f[i][j]=score}$;

13 $\mathit{path[i][j]=(\delta_{i},\delta_{j})}$;

14

15

16

17

18

19$\mathit{return\ path}$;

Algorithm 1 Pseudocode for the sentence alignment algorithm. We can recover
alignment results from $path$.

###  3.3 High-quality Evaluation Sets Creation

A high-quality test set is required to benchmark the performance of
educational lecture translation, and a high-quality development set can be
useful for tuning MT systems. We create test and development sets by manually
filtering the aligned sentence pairs obtained in the previous step.

We first select the document pairs that have a high similarity score under the
automatic evaluation similarity criteria. Specifically, we sort all document
pairs in descending order of the average similarity of aligned sentence pairs
within.

We then subject these sorted and sentence-aligned pairs to human evaluation to
obtain well-aligned test and development sets. As presented in Algorithm 2,
the target volume of each set ($\mathit{volume}$) and document-level
comparability ($\mathit{ratio}$) are the two parameters. In line $2$, while
the number of clean sentence pairs is fewer than required, we manually check
another document pair. For each document pair, we manually check all sentence
pairs and only add the correct ones, as shown in lines $7$ to $10$. We use the
remaining sentence-aligned document pairs for training. Our test, development,
and train sets are all constructed at the document level; thus our corpora
could be used to evaluate document-level translation in the future, if
required.

Input : $\mathit{DocPairs}$, $\mathit{volume}$, $\mathit{ratio}$

Output : $\mathit{DocPairs}$, $\mathit{SentencePairs}$

1 $\mathit{SentencePairs}\leftarrow\\{\\}$;

2 while  _$|\mathit{SentencePairs}| <\mathit{volume}$_ do

3 $\mathit{DocPair}\leftarrow$ pickBestDocPair($\mathit{DocPairs}$);

4
$\mathit{DocPairs}\leftarrow\mathit{DocPairs}\backslash\\{\mathit{DocPair}\\}$;

5 $\mathit{CandidatePairs}\leftarrow$ getAlignments($\mathit{DocPair}$);

6 $\mathit{Correct}\leftarrow$ {};

7 foreach  _$\mathit{Pair}\in\mathit{CandidatePairs}$_ do

8 $\mathit{Judge}\leftarrow$ manualEvaluation($\mathit{Pair}$);

9 if  _$\mathit{Judge}$ == good_ then

10 $\mathit{Correct}\leftarrow\mathit{Correct}\cup\\{\mathit{Pair}\\}$;

11

12

13 if  _$|\mathit{Correct}| >|\mathit{CandidatePairs}|*\mathit{ratio}$_ then

14
$\mathit{SentencePairs}\leftarrow\mathit{SentencePairs}\cup\mathit{Correct}$;

15

16

Algorithm 2 Document-aware sentence filtering

##  4 Corpus Construction Experiments

We report on the application of our framework to create English–Japanese and
English–Chinese Coursera corpus, focusing on settings, performance, and
analysis, including 1) cleaning documents, 2) initial MT system creation, 3)
sentence alignment results, 4) Coursera corpus creation, and 5) corpus
analysis.

###  4.1 Document Cleaning Experiments

For the sentence segmentation settings, we first segment paragraphs with a
full-stop (“.”), exclamation (“!”), and question marks (“?”) in Latin encoding
and their full-width counterparts in UTF-8 followed by a space or the end of
the line. We then tokenize sentences, using Juman++ (Tolmachev et al.,
2018)161616https://github.com/ku-nlp/jumanpp for Japanese, NLTK
tool171717https://www.nltk.org for English and
StanfordCoreNLP181818https://stanfordnlp.github.io/CoreNLP/ for Chinese.

For the language detection settings, we calculate the number of English and
Japanese characters to judge whether the given document is in Japanese or
English. More specifically, we define a set of characters, EnglishChar, with
“a” to “z” and “A” to “Z,” and another set of characters, JapaneseChar, with
Hiragana and Katakana characters. Compared with the langdetect tool (Raffel et
al., 2020), on an evaluation set of 100 manually annotated samples, the rule-
based method classified the language of all documents correctly and the
langdetect tool correctly classified 99 of them. For English–Chinese, we rely
on the langdetect tool to filter out data in other languages.

###  4.2 Initial MT System Creation

We use TED Talks corpus (Cettolo et al., 2012) and ASPEC corpus (Nakazawa et
al., 2016)191919We selected the best 1.0 million sentence pairs according to
the pre-computed score in the provided corpus for each sentence pair. to build
the English–Japanese MT system, use TED talks and News Commentary Tiedemann
(2012) to build the English–Chinese MT system. Table 1 gives the statistics of
these three corpora.

To train the MT models, we leverage the smaller TED Talks corpus and the
larger ASPEC or News Commentary corpus through fine-tuning and mixed fine-
tuning approaches Chu et al. (2017b). When performing mixed fine-tuning on the
concatenation of both corpora, the TED Ja–En and TED Zh–En corpora were
oversampled to match the size of the ASPEC Ja–En and News Commentary Zh–En
corpora, respectively. NMT models are trained using tensor2tensor framework
with its default hyperparameters as shown in Section 5.2.

We evaluate these MT models through a test set from the TED dataset through
the BLEU score Papineni et al. (2002), since we do not have a test set for the
target domain yet, i.e., Coursera. Table 2 gives the results where “A,” “T,”
and “N,” stand for ASPEC, TED, and News Commentary, “AT” or “NT” stands for
the balanced mixture of two datasets. “$\rightarrow$” means fine-tuning on the
right-hand side data.

We found that the mixed fine-tuning strategy yields the best results for both
Japanese$\rightarrow$English and Chinese$\rightarrow$English translations. We,
therefore, use these models during the sentence alignment procedure.

Table 1: Number of sentence pairs in each English–Japanese and English–Chinese
corpus to train initial MT systems.

Split Dataset | English–Japanese | English–Chinese
---|---|---
ASPEC | TED | News | TED
Train | 1.0M | 223k | 321k | 215k
Development | 1,790 | 1,354 | - | 1,261
Test | 1,812 | 1,194 | - | 1,064

Table 2: BLEU scores of Ja$\rightarrow$En and Zh$\rightarrow$En translation
directions on TED test set. “A” represents the ASPEC train set, “N” represents
the News Commentary train set, and “T” represents the TED train set.
“$\rightarrow$” indicates fine-tuning. The best result is marked bold.
Japanese$\rightarrow$English
---
Training path | BLEU score
A | 4.1
T | 12.2
AT | 14.6
A $\rightarrow$ T | 13.9
A $\rightarrow$ AT | 15.0
Chinese$\rightarrow$English
Training path | BLEU
N | 11.1
N $\rightarrow$ NT | 21.5

###  4.3 Sentence Alignment Experiments

We compare our proposed sentence alignment methods with the BLEU-score based
method (Sennrich and Volk, 2010), BERTScore (Zhang et al., 2020b) based
method, LASER universal embedding based methods (Chaudhary et al., 2019;
Thompson and Koehn, 2019), and sentBERT embedding (Reimers and Gurevych, 2020)
based method. We measure the alignment $F_{1}$ score on four manually
annotated document pairs (2 English–Japanese and 2 English–Chinese) with a
total of 426 lines.

####  4.3.1 Sentence Alignment Methods

We try the combinations of these variables in the alignment methods: 1)
embedding methods including word2vec, LASER, sentBERT, BERT, or w/o embedding
for BLEU-score based method, 2) similarity measurement: cosine similarity, L2
distance, BERTScore, or BLEU-score, and 3) alignment algorithms including
greedy, minimum-cost flow, and DP.

Regarding the embedding methods, word2vec models we used are from NLPL word
embedding repository.202020http://vectors.nlpl.eu/repository, ID 35, 40, 53
for Chinese, English, and Japanese, respectively We convert words in one
sentence into embeddings and average the word embeddings as sentence
embeddings. We used RoBERTa-large212121https://github.com/Tiiiger/bert_score
for BERTScore. The LASER model we used is a multilingual BiLSTM
model222222dl.fbaipublicfiles.com/laser/models/bilstm.93langs.2018-12-26.pt
and the sentBERT model we used is a multilingual MPNet
model.232323https://huggingface.co/sentence-transformers/all-mpnet-base-v2 The
multilingual sentence-to-vector models can convert sentences in different
languages into embeddings. Note that the vector spaces are aligned across
languages for LASER but not aligned for sentBERT.

Concerning the similarity measurement, for the BLEU score which is the higher
the better, we use the NLTK toolkit to calculate sentence BLEU score with the
default setting. BERTScore, the higher the better, leverages the pre-trained
contextual embeddings from a BERT-like model and matches words in candidate
and reference sentences by cosine similarity as defined in (Zhang et al.,
2020b) and we use the $F_{1}$ score of BERTScore. We translate sentences in
Japanese and Chinese into English before applying BERTScore.

In the case of cosine similarity, the higher the better, is defined as:

$\cos({\bf v},{\bf v^{\prime}})=\frac{\sum_{i=1}^{D}{{\bf v}_{i}{\bf
v^{\prime}}_{i}}}{\sqrt{\sum_{i=1}^{D}{({\bf
v}_{i})^{2}}}\sqrt{\sum_{i=1}^{D}{({\bf v^{\prime}}_{i})^{2}}}}$ (3)

where ${\bf v}$ and ${\bf v^{\prime}}$ are two sentence embeddings from
$Document_{1}$ and $Document_{2}$ with dimension $D$ and ${\bf v}_{i}$ or
${\bf v^{\prime}}_{i}$ is the $i_{th}$ dimension of embedding ${\bf v}$ or
${\bf v^{\prime}}$.

Finally, L2 distance, the lower the better, we define it as below. To ensure
consistency in the analogy with other measurements, in which higher values are
better, we take the negative of L2 distance in practice.

$Distance_{L2}\left({\bf v},{\bf
v}^{\prime}\right)=\sqrt{\sum_{i=1}^{D}\left({\bf v}_{i}-{\bf
v}^{\prime}_{i}\right)^{2}}$ (4)

Besides the DP algorithms as shown in Algorithm 1, we also try two other
algorithms: greedy and minimum-cost flow. The greedy algorithm is simple and
implemented as follows: for each sentence in the source language $s_{i}$, we
find a sentence in the target language $s^{\prime}_{j}$ that maximizes
$Similarity(s_{i},s^{\prime}_{j})$. The set $S$ of matched pairs $(i,j)$ for
all the extracted sentences in the source language $s_{i}$ with a
corresponding sentence in the target language $s^{\prime}_{j}$ can be
represented as follows:

$S=\\{(i,\mathop{\rm
arg~{}max}\limits_{j}[Similarity(s_{i},s^{\prime}_{j})]\\}_{i=1}^{N}$ (5)

Note that using the greedy algorithm, it is possible that for two pairs
$(s_{i},s^{\prime}_{j})$ and $(s_{k},s^{\prime}_{l})$ with $s_{i}\neq s_{k}$
and $s^{\prime}_{j}=s^{\prime}_{l}$. This method is suitable for mining
parallel sentences from large-scale web corpora, e.g. BUCC bitext mining task
using
LASER.242424https://github.com/facebookresearch/LASER/blob/main/tasks/bucc

An improved version of the greedy algorithm is using the minimum cost maximum
flow algorithm252525https://en.wikipedia.org/wiki/Minimum-cost_flow_problem to
achieve 1–to–1 global best match, where the algorithm gives maximum total
similarity scores allowing only 1–to–1 match. For every two pairs
$(s_{i},s^{\prime}_{j})$ and $(s_{k},s^{\prime}_{l})$ in $S$, $s_{i}\neq
s_{k}$, $s^{\prime}_{j}\neq s^{\prime}_{l}$ always holds. The graph is
constructed as follows; an example is shown in Figure 3:

  * •

from the source point $A$ to the all sentences in the source language $s_{i}$,
add edges with $(capability,cost)=(1,0)$

  * •

from all sentences in the target language $s^{\prime}_{j}$ to the target point
$B$, add edges with $(capability,cost)=(1,0)$

  * •

from all sentences in the source language $s_{i}$ to all sentences in the
target language $s^{\prime}_{j}$, add edges with
$(capability,cost)=(1,-similarity(s_{i},s^{\prime}_{j}))$

We then send an infinite amount of flow from source vertex $A$ to sink vertex
$B$ and obtain the residual network. We select sentence pair
$(s_{i},s^{\prime}_{j})$ if the remaining capability of edge
$(s_{i},s^{\prime}_{j})$ is zero. By the property of this algorithm, it is
more suitable for extracting parallel sentences from two comparable documents
instead of corpora mining from the web.

Figure 3: The graph for the minimum-cost maxflow algorithm. A represents the source and B represents the sink, $s_{i}$ are the points for sentences in the source language and $s^{\prime}_{j}$ are the points for sentences in the target language. Each edge is with $capability/cost$ information. Table 3: The statistical information of $2$ English–Japanese and $2$ English–Chinese annotated document pairs for sentence alignment evaluation. #Doc | #En lines | #Ja lines | #Zh lines
---|---|---|---
1 | 75 | 91 | -
2 | 52 | 63 | -
3 | 34 | - | 22
4 | 47 | - | 42
Total | 208 | 154 | 64
Table 4: Average $F_{1}$ scores of different sentence alignment methods. Tested on four manually annotated document pairs. The best result is marked bold. We also show $F_{1}$ scores of two English–Japanese document pairs $F_{1}^{En-Ja}$ and $F_{1}$ scores of two English–Chinese document pairs $F_{1}^{En-Zh}$ to the footer of each $F_{1}$ score. Model | With MT | Similarity | Algorithm
---|---|---|---
Greedy | Min-cost flow | DP
Baseline: |  |  |  |  |
- | Yes | BLEU | $0.74_{0.76/0.72}$ | $0.82_{0.79/0.85}$ | $0.93_{0.98/0.89}$
RoBERTa | Yes | BERTScore | $0.70_{0.74/0.66}$ | $0.81_{0.79/0.84}$ | $0.89_{0.97/0.82}$
LASER | No | Cosine | $0.75_{0.77/0.73}$ | $0.82_{0.78/0.85}$ | $0.90_{0.98/0.82}$
LASER | No | Distance | $0.75_{0.77/0.73}$ | $0.82_{0.78/0.85}$ | $0.83_{0.91/0.76}$
sentBERT | No | Cosine | $0.12_{0.08/0.16}$ | $0.18_{0.10/0.25}$ | $0.52_{0.41/0.64}$
sentBERT | No | Distance | $0.12_{0.08/0.16}$ | $0.18_{0.10/0.25}$ | $0.25_{0.18/0.32}$
Proposed: |  |  |  |  |
word2vec | Yes | Cosine | $0.65_{0.73/0.57}$ | $0.79_{0.76/0.82}$ | $\boldsymbol{0.96}_{0.97/0.95}$

####  4.3.2 Sentence Alignment Benchmark

We manually annotated gold alignments of four document pairs and the
statistical information is shown in Table 3. There are 2 English–Japanese and
2 English–Chinese document pairs containing a total of 426 sentences of 208
English sentences, 154 Japanese sentences, and 64 Chinese sentences. In
detail, they contain 162 ($84\%$) 1–to–1 alignment cases, 8 ($4\%$) many–to–1
cases, 21 ($11\%$) 1–to–many cases, and 1 ($1\%$) many–to–many alignment
cases, which requires the ability to combine multiple lines into one.

####  4.3.3 Sentence Alignment Performance

We measure the performance of alignment methods through macro $F_{1}$ score on
four document pairs. For each document pair, the number of manually annotated
alignment pairs is defined as $Total\\_Pairs$, the number of extracted
alignment pairs is defined as $Extracted\\_Pairs$, and the number of correctly
extracted pairs is defined as $Correct\\_Pairs$. We define $Precision$ as
$\frac{Correct\\_Pairs}{Extracted\\_Pairs}$, $Recall$ as
$\frac{Correct\\_Pairs}{Total\\_Pairs}$, and $F_{1}$ score as
$\frac{2*Precision*Recall}{Precision+Recall}$.

The results of the combinations of embeddings, algorithms, and similarity
measurements are shown in Table 4. Our proposed method using word2vec as
embedding method, cosine similarity as similarity measurement, and DP
algorithm achieved the best macro average $F_{1}$ score of 96% on four
document pairs, which is higher than the performance of the BLEU-based method
(Sennrich and Volk, 2010), BERTScore-based method (Zhang et al., 2020b),
original LASER method with greedy algorithm (Chaudhary et al., 2019) for BUCC
task that is only 75% and even higher than the LASER with DP (Thompson and
Koehn, 2019) method that is 90%. Additionally, we can observe the following:

  * •

In our situation where there are no crossing alignments, the DP algorithm
achieved much better results than greedy, or minimum-cost flow, because it can
process the 1–to–many, many–to–1 and many–to–many sentence alignment cases.

  * •

MT-based methods, including word2vec, BLEU-based and BERTScore, are better
than multilingual embedding based methods, including the LASER and sentBERT
based methods. In the English–Japanese and English–Chinese scenarios where
relatively good initial MT systems exist, it is better to first translate the
sentences in the source language into sentences in the target language and
measure the similarity using the two sentences in the same language. The
performance of the sentBERT model is low because the vector spaces are not
aligned across languages.

  * •

Using cosine similarity gives slightly better performance than the pairwise
distance that is used in the LASER method.

Table 5: The size of English–Japanese Coursera parallel dataset.

|  # of document pairs |  # of aligned lines |  # of deleted lines
---|---|---|---
Test | 50 | 2,068 | 140
Development | 16 | 555 | 88
Train | 818 | 50,543 | -

Table 6: The size of English–Chinese Coursera parallel dataset.

|  # of document pairs |  # of aligned lines |  # of deleted lines
---|---|---|---
Test | 90 | 2,009 | 83
Development | 34 | 865 | 43
Train | 997 | 40,074 | -

###  4.4 Coursera Dataset Creation

We introduce the created corpora and detailed settings in this section. With
the proposed alignment algorithm, we obtained a total of 53,394 pairs of
sentences from 884 document pairs for English–Japanese and a total of 43,074
pairs of sentences from 1,121 document pairs for English–Chinese.

Test and development sets are created through manually filtering262626The
checker is a native Chinese speaker, has the N1 certification (the highest
level) of the Japanese Language Proficiency Test, and got 99 points in TOEFL
iBT. as introduced in Section 3.3. We set 2,000 and 500 sentences for
English–Japanese and 2,000 and 850 for English–Chinese as the target
$\mathit{volume}$ for the test and development sets, respectively, and set
$\mathit{ratio}=0.50$. As shown in Table 6, for English–Japanese, a total of
2,851 sentence pairs drawn from 66 documents were manually judged and
approximately 8.0% of them ((140+88)/2,851) were filtered out. We did not
perform manual filtering for the train set due to its large scale.

As a result, the train, development, and test sets in the English–Japanese
corpus contain 50,543, 555, and 2,086 parallel sentences, respectively. For
English–Chinese, with the same method, the train, development, and test sets
contain 40,074, 865, and 2,009 parallel sentences, respectively.

###  4.5 Corpus Analysis

We show the sentence length analysis and similarity with other corpora of our
created Coursera corpora.

The sentence length information is presented in Table 8 and Table 8, with the
tokenization methods introduced in Section 4.1. For the English–Japanese
Coursera corpus, the average sentence length is between ASPEC and TED and
relatively closer to TED. For the English–Chinese Coursera corpus, the average
sentence length is between that of News Commentary and TED corpora and is
closer to News Commentary.

We measure the similarity with other corpora using the language model (LM).
Specifically, we trained 4-gram LMs on the lower-cased version of the English
side of each train set. We then computed the per-token log-likelihood of train
sets with each of these LMs. As shown in Table 10, the three English–Japanese
datasets are visibly distant from each other. Nevertheless, TED seems
relatively more exploitable than ASPEC for helping to translate Coursera
datasets, presumably because they comprise a spoken language, unlike ASPEC. As
given by Table 10, when tested on the LM trained on the Coursera corpus, the
TED corpus achieves a higher likelihood than that of the News commentary
corpus, suggesting that the TED corpus is more similar to the Coursera corpus
than the News commentary corpus.

Table 7: Sentence length of English–Japanese train set.

Dataset | English | Japanese
---|---|---
Mean / Median / s.d. | Mean / Median / s.d.
ASPEC | 25.4 / 23 / 11.4 | 27.5 / 20 / 12.0
TED | 20.4 / 17 / 13.9 | 19.8 / 16 / 14.1
Coursera | 21.1 / 19 / 11.1 | 22.2 / 20 / 11.8

Table 8: Sentence length of English–Chinese train set.

Dataset | English | Chinese
---|---|---
Mean / Median / s.d. | Mean / Median / s.d.
News commentary | 26.0 / 25 / 12.7 | 24.7 / 23 / 12.2
TED | 20.5 / 17 / 15.3 | 18.9 / 15 / 14.5
Coursera | 25.7 / 22 / 25.7 | 24.2 / 20 / 22.2

Table 9: Average per-token log-likelihood of En–Ja datasets.

LM Corpus | ASPEC | TED | Coursera
---|---|---|---
ASPEC | -1.157 | -3.543 | -3.364
TED | -3.468 | -1.158 | -2.570
Coursera | -3.482 | -2.800 | -0.790

Table 10: Average per-token log-likelihood of En–Zh datasets.

LM Corpus | News commentary | TED | Coursera
---|---|---|---
News commentary | -1.001 | -2.516 | -2.635
TED | -2.775 | -1.055 | -2.375
Coursera | -2.976 | -2.542 | -0.824

##  5 MT Experiments

In this section, we show the effectiveness of the in-domain parallel corpora
created by the proposed method through MT experiments. Specifically, we apply
the created lecture domain parallel corpora to creating an MT system,
combining it with larger-scale out-of-domain corpora. To leverage small in-
domain and large out-of-domain datasets efficiently, we modify the two-stage
fine-tuning method to a multistage fine-tuning method.

###  5.1 Datasets

We leveraged both the in-domain Coursera corpora and out-of-domain corpora in
the English–Japanese and English–Chinese MT experiments. The information of
the corpora is shown in Tables 1, 6, and 6.

We used these abbreviations for the datasets: in the English–Japanese MT
experiments, “A” stands for ASPEC train set of 1.0 million lines, “T” stands
for TED Talks train set of 0.2 million lines, and “C” for Coursera train set
of 50 thousand lines. In the English–Chinese MT experiments, we refer to the
News Commentary train set with 0.3 million lines as “N,” TED train set with
0.2 million lines as “T,” and the Coursera train set with 40 thousand lines as
“C.”

When performing dataset combinations, we always oversample the smaller ones to
match the size of the largest ones. We denote the concatenated corpus by a
concatenation of the letters representing them: e.g., “AT” for the mixture of
ASPEC data with five times oversampled TED data, and “ATC” for the
concatenation of ASPEC with five times oversampled TED data and 25 times
oversampled Coursera data. We use $\rightarrow$$D$ to represent fine-tuning
the model on the dataset $D$, e.g. “A”$\rightarrow$“AT” means that we first
train the model on ASPEC and then fine-tune it on the mixture of ASPEC and TED
with oversampling.

All the datasets are tokenized into subwords using Byte-Pair-Encoding (BPE)
(Sennrich et al., 2016b) method. For English–Japanese experiments, we created
a shared Japanese and English subword vocabulary from the ASPEC and TED train
sets with oversampling, with 32,000 merge operations. For English–Chinese
experiments, we created a shared Chinese and English subword vocabulary from
the News Commentary and TED train sets with oversampling, with 32,000 merge
operations. These vocabularies are used for all experiments.

###  5.2 Model and Training Settings

We used the tensor2tensor framework (Vaswani et al.,
2018)272727https://github.com/tensorflow/tensor2tensor, version 1.14.0. with “
_transformer_big_ ” architecture and default settings for this architecture,
such as $dropout=0.2$, $attention\ dropout=0.1$, $optimizer=adam$ with
$beta1=0.9$, $beta2=0.997$.

During training, we used eight Tesla V100 32GB GPUs with a batch size of 4,096
subword tokens in all the experiments. We used early stopping on token-level
BLEU score computed on the development set: the training process stops if the
score shows no gain larger than 0.1 for 10,000 steps. When fine-tuning the
model on a different dataset, we always resumed the training process from the
last checkpoint in the previous stage.

During inference, we used the average of the last ten checkpoints and decoded
the test sets with a beam size of 4 and a length penalty $\alpha$ of 0.6
consistently across all the models. We calculate the BLEU scores for the
decoded text and target text using the SacreBLEU
tool.282828https://github.com/mjpost/sacreBLEU

Table 11: BLEU of English–Japanese translation for all domain adaptation methods on Coursera test set. The training path shows the datasets used in each fine-tuning stage. The best score in each direction is marked bold. Method | Training path | Japanese$\rightarrow$English | English$\rightarrow$Japanese
---|---|---|---
Baseline: |  |  |
Out-of-domain data | A | 14.1 | 10.4
Out-of-domain data | T | 16.7 | 8.6
In-domain data only | C | 6.8 | 7.7
Mixed out-of-domain data | AT | 24.9 | 13.2
Mixed data | ATC | 25.6 | 17.3
Two-stage fine-tuning | ATC$\rightarrow$C | 23.8 | 18.1
Mixed fine-tuning | A$\rightarrow$ATC | 27.8 | 17.9
Proposed: |  |  |
Multistage fine-tuning | A$\rightarrow$AT$\rightarrow$ATC | 27.9 | 18.5
Table 12: BLEU of English–Chinese translation for all domain adaptation methods on Coursera test set. The training path shows the datasets used in each fine-tuning stage. The best score in each direction is marked bold. Method | Training path | Chinese$\rightarrow$English | English$\rightarrow$Chinese
---|---|---|---
Baseline: |  |  |
Out-of-domain data | N | 18.3 | 13.3
Out-of-domain data | T | 15.5 | 11.6
In-domain data only | C | 14.8 | 14.5
Mixed out-of-domain data | NT | 21.6 | 18.4
Mixed data | NTC | 28.2 | 26.6
Two-stage fine-tuning | NTC$\rightarrow$C | 27.0 | 27.1
Mixed fine-tuning | N$\rightarrow$NTC | 29.5 | 26.8
Proposed: |  |  |
Multistage fine-tuning | N$\rightarrow$NT$\rightarrow$NTC | 29.5 | 27.1
Table 13: BLEU scores for all the multistage training options examined in our
Japanese–English MT experiment. Models A1–A16 and B2–B16 represent all 31
($=2^{5}-1$) sub-paths of the
A$\rightarrow$AT$\rightarrow$ATC$\rightarrow$TC$\rightarrow$C flow. Bold
indicates the initial training, and red, blue, and gray cells indicate
inflation, deflation, and replacement of the training data, respectively.

ID | Training schedule | Ja$\rightarrow$En | En$\rightarrow$Ja |  | ID | Training schedule | Ja$\rightarrow$En | En$\rightarrow$Ja
---|---|---|---|---|---|---|---|---
A1 | A |  |  |  |  | 14.1 | 10.4 |  |  |  |  |  |  |  |
A2 | A | AT |  |  |  | 26.1 | 14.0 |  | B2 | AT |  |  |  | 24.9 | 13.2
A3 | A | AT | ATC |  |  | 27.9 | 18.5 |  | B3 | AT | ATC |  |  | 27.4 | 18.1
A4 | A | AT | ATC | TC |  | 27.2 | 17.5 |  | B4 | AT | ATC | TC |  | 26.3 | 17.3
A5 | A | AT | ATC | TC | C | 25.9 | 18.6 |  | B5 | AT | ATC | TC | C | 25.0 | 18.1
A6 | A | AT | ATC |  | C | 25.8 | 18.8 |  | B6 | AT | ATC |  | C | 25.4 | 18.3
A7 | A | AT |  | TC |  | 27.8 | 18.4 |  | B7 | AT |  | TC |  | 26.8 | 17.9
A8 | A | AT |  | TC | C | 25.7 | 18.3 |  | B8 | AT |  | TC | C | 24.9 | 17.7
A9 | A | AT |  |  | C | 25.3 | 18.2 |  | B9 | AT |  |  | C | 24.8 | 18.3
A10 | A |  | ATC |  |  | 27.8 | 17.9 |  | B10 |  | ATC |  |  | 25.6 | 17.3
A11 | A |  | ATC | TC |  | 27.1 | 17.4 |  | B11 |  | ATC | TC |  | 24.4 | 16.8
A12 | A |  | ATC | TC | C | 25.5 | 18.4 |  | B12 |  | ATC | TC | C | 23.4 | 17.6
A13 | A |  | ATC |  | C | 26.3 | 18.2 |  | B13 |  | ATC |  | C | 23.8 | 18.1
A14 | A |  |  | TC |  | 27.3 | 18.0 |  | B14 |  |  | TC |  | 18.7 | 13.3
A15 | A |  |  | TC | C | 25.1 | 18.1 |  | B15 |  |  | TC | C | 18.8 | 14.0
A16 | A |  |  |  | C | 23.2 | 17.5 |  | B16 |  |  |  | C | 6.8 | 7.7
C3 | A | AT | T |  |  | 24.1 | 12.1 |  | D3 | AT | T |  |  | 22.6 | 11.5
C4 | A | AT | T | TC |  | 25.9 | 17.4 |  | D4 | AT | T | TC |  | 25.5 | 16.5
C5 | A | AT | T | TC | C | 25.3 | 18.1 |  | D5 | AT | T | TC | C | 24.4 | 17.3
C6 | A | AT | T |  | C | 24.6 | 17.5 |  | D6 | AT | T |  | C | 24.0 | 18.1
C10 | A |  | T |  |  | 24.0 | 11.9 |  | D10 |  | T |  |  | 16.7 | 8.6
C11 | A |  | T | TC |  | 26.2 | 17.1 |  | D11 |  | T | TC |  | 21.4 | 14.1
C12 | A |  | T | TC | C | 24.9 | 17.7 |  | D12 |  | T | TC | C | 21.1 | 15.9
C13 | A |  | T |  | C | 24.3 | 17.7 |  | D13 |  | T |  | C | 20.7 | 16.0
E14 | A |  |  | AC |  | 24.8 | 18.1 |  | F14 |  |  | AC |  | 19.5 | 15.2
E15 | A |  |  | AC | C | 23.7 | 17.9 |  | F15 |  |  | AC | C | 19.4 | 15.4

###  5.3 Domain Adaptation Methods

We describe domain adaptation methods used in the experiments when combining
the smaller in-domain Coursera corpus with larger out-of-domain corpora.

The simplest baseline is to mix multiple datasets (with oversampling) without
fine-tuning. However, the performance of NMT is extremely sensitive to the
domain of the dataset and to the order in which datasets are included in the
training stages. As such, instead of mixing all the datasets, it is common to
divide training into multiple stages where each stage uses data from different
domains to maximize the impact of the domain-specific training data.

Two-stage fine-tuning (Zoph et al., 2016) is a commonly used domain adaptation
technique. Suppose there are $N$ datasets where $N-1$ of them are out-of-
domain datasets and one of them is the in-domain dataset. MT model is trained
on the mixture of $N$ datasets (or $N-1$ out-of-domain datasets) until
convergence in the first stage and continuously trained on the in-domain
dataset until convergence. Mixed fine-tuning Chu et al. (2017b) fine-tunes the
mixture of $N$ datasets in the last step. (Imankulova et al., 2019) and (Dabre
et al., 2019) showed that training in multiple stages where each stage
contains different proportions of various types of training data leads to the
best results. Thus, in our case where there are more than two datasets, we
modify the two-stage fine-tuning to multistage fine-tuning.

To make the transition between fine-tuning stages more smooth, we modify the
two-stage fine-tuning to a multistage fine-tuning method. We first sort all
the datasets into a list by the similarity with the in-domain dataset in terms
of average per-token log-likelihood from the smallest to the largest given by
the in-domain LM as described in Section 4.5. For English–Japanese the sorted
datasets are (A, T, C) and for English–Chinese the sorted datasets are (N, T,
C). There are $N$ stages, in the $i_{th}$ stage, the first $i$ datasets in the
list are combined to train or fine-tune the model.

###  5.4 Experimental Results

Main Results. We illustrate the effectiveness of the proposed Coursera corpus
with the multistage fine-tuning domain adaptation method through the MT
results in Tables 11 and 12 for English–Japanese and English–Chinese
translations, respectively. We observed that:

Firstly, leveraging the proposed in-domain dataset (Mixed data) yields huge
performance improvements compared with using two out-of-domain datasets (Mixed
out-of-domain data). For example, in the English$\rightarrow$Japanese
direction, the BLEU score improvement is $4.1$ points, and in the
Chinese$\rightarrow$English direction, the BLEU score improved from $21.6$ to
$28.2$, achieving an improvement of $6.6$ points. Note that because our
dataset itself is small, it does not work well on its own due to the
overfitting problem.

Secondly, Multistage fine-tuning shows more robust if not better performance
than Mixed data, two-stage fine-tuning (Zoph et al., 2016) or mixed fine-
tuning (Chu et al., 2017b). For English–Japanese MT experiments, multistage
fine-tuning shows 0.1 and 0.4 BLEU score improvements compared with previous
methods on Ja$\rightarrow$En and En$\rightarrow$Ja directions, respectively.
For English–Chinese MT experiments, multistage fine-tuning also achieved the
best results in both translation directions.

More detailed results are shown in Table 13, which summarizes the BLEU scores
of all the MT systems trained up to five training stages for the
Japanese–English tasks. The training schedule with all the stages, i.e.,
A$\rightarrow$AT$\rightarrow$ATC$\rightarrow$ TC$\rightarrow$C (A5) did not
achieve the best results in either translation direction. For the
Ja$\rightarrow$En task, an intermediate model,
A$\rightarrow$AT$\rightarrow$ATC (A3), achieved the best BLEU score with a
gain of more than 20 points over the model trained only on the in-domain
parallel data (B16). For the reverse direction, that is, the En$\rightarrow$Ja
task, the schedule A$\rightarrow$AT$\rightarrow$ATC$\rightarrow$C (A6)
achieved the best BLEU score with a 11.1 point BLEU gain. Whenever additional
training data were introduced (marked red in Table 13), the BLEU scores were
improved.292929Compare the pairs (A1, A2), (A2, A3), (A1, A10), (B2, B3), (C3,
C4), (C10, C11), (D3, D4), (D10, D11), and (A1, E14). This is mostly in line
with the observations of (Chu et al., 2017b); starting from out-of-domain data
only and ending with a mixture of out-of- and in-domain data gives the best
results for in-domain translation. As shown in Table 10, A is more dissimilar
to C, and T more similar to C. As such, it seems reasonable to gradually
introduce the in-domain data by relying on related domain data for
intermediate training steps. According to (Dabre et al., 2019), the final
stage of fine-tuning on C should provide the best translation quality.
However, in our setting, this holds true only for some cases in the
En$\rightarrow$Ja task, suggesting the necessity of hyper-parameter tuning for
fine-tuning on deflated training data (marked blue in Table 13). In contrast,
training a model directly in one stage on ATC (B10) yielded significantly
lower results than the multistage results.

##  6 Conclusion

In this paper, we proposed a framework to create corpora for the lecture
transcript translation. We proposed a parallel sentence extraction algorithm
that extracts parallel sentences from parallel lecture transcripts, which is a
DP algorithm based on machine translation and cosine similarity over sentence
embeddings. Experimental results show a higher $F_{1}$ score compared with
using BLEU-score similarity measurement or using the RoBERTa, LASER, or
sentBERT embeddings.

With the sentence extraction algorithm, we created an English–Japanese
Coursera corpus with 50,543, 555, and 2,068 lines of train, development, and
test sets, respectively. We also created an English–Chinese Coursera corpus
with 40,074, 865, and 2,009 lines of train, development, and test sets,
respectively. The quality of the test and development sets is ensured through
manual filtering so that they can be used to reliably benchmark translation
performance and tune MT models.

We illustrated the effectiveness of the proposed Coursera corpora through MT
experiments. For English–Japanese and English–Chinese lecture translation,
leveraging Coursera corpora brings large translation performance improvement
measured by BLEU scores. Additionally, we modified the two-stage fine-tuning
algorithm for multiple datasets and called it multistage fine-tuning.
Experimental results showed more robust if not better performance compared
with previous domain adaptation methods.

Our future work aims to explore the method of multilingualism by constructing
a multilingual corpus with a multilingual sentence alignment algorithm. To
maximize the use of the corpora, we focus on adaptation techniques for low-
resource domains.

##  7 Acknowledgment

Part of this work was conducted under the program “Research and Development of
Enhanced Multilingual and Multipurpose Speech Translation System” of the
Ministry of Internal Affairs and Communications (MIC), Japan. This work was
also supported by JSPS KAKENHI Grant Number JP23H03454 and JSPS Research
Fellow for Young Scientists (DC1).

## References

  * Abdelali et al. (2014) Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014.  The AMARA Corpus: Building Parallel Language Resources for the Educational Domain.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC)_ , pages 1856–1862, Reykjavik, Iceland. European Language Resources Association.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural Machine Translation by Jointly Learning to Align and Translate.  In _Proceedings of the Third International Conference on Learning Representations (ICLR)_ , San Diego, USA.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009.  Curriculum Learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , pages 41–48. ACM.
  * Bouamor and Sajjad (2018) Houda Bouamor and Hassan Sajjad. 2018.  H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings.  In _Proceedings of the Eleventh Workshop on Building and Using Comparable Corpora_ , pages 43–47, Miyazaki Japan. European Language Resources Association.
  * Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012.  Wit3: Web Inventory of Transcribed and Translated Talks.  In _Proceedings of the 16th Conference of European Association for Machine Translation_ , pages 261–268, Trento, Italy. European Association for Machine Translation.
  * Chaudhary et al. (2019) Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019.  Low-resource corpus filtering using multilingual sentence embeddings.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 261–266, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2020) Pinzhen Chen, Nikolay Bogoychev, Kenneth Heafield, and Faheem Kirefu. 2020.  Parallel sentence mining by constrained decoding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1672–1678, Online. Association for Computational Linguistics.
  * Chu et al. (2017a) Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017a.  An empirical comparison of domain adaptation methods for neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 385–391, Vancouver, Canada. Association for Computational Linguistics.
  * Chu et al. (2017b) Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017b.  An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 385–391, Vancouver, Canada. Association for Computational Linguistics.
  * Chu et al. (2015) Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2015.  Integrated Parallel Sentence and Fragment Extraction from Comparable Corpora: A Case Study on Chinese–Japanese Wikipedia.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ , 15(2):10:1–10:22.
  * Dabre et al. (2019) Raj Dabre, Atsushi Fujita, and Chenhui Chu. 2019.  Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1410–1416, Hong Kong, China. Association for Computational Linguistics.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Gong et al. (2016) Chen Gong, Dacheng Tao, Stephen J Maybank, Wei Liu, Guoliang Kang, and Jie Yang. 2016.  Multi-modal Curriculum Learning for Semi-supervised Image Classification.  _IEEE Transactions on Image Processing_ , 25(7):3249–3260.
  * Hu et al. (2019) Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019.  Domain adaptation of neural machine translation by lexicon induction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2989–3001, Florence, Italy. Association for Computational Linguistics.
  * Imankulova et al. (2019) Aizhan Imankulova, Raj Dabre, Atsushi Fujita, and Kenji Imamura. 2019.  Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation.  In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_ , pages 128–139, Dublin, Ireland. European Association for Machine Translation.
  * Jansen et al. (2014) Dean Jansen, Aleli Alcala, and Francisco Guzman. 2014.  Amara: A Sustainable, Global Solution for Accessibility, Powered by Communities of Volunteers.  In _Universal Access in Human-Computer Interaction: Design for All and Accessibility Practice_ , pages 401–411. Springer International Publishing, Cham.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six Challenges for Neural Machine Translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Kordoni et al. (2015) Valia Kordoni, Kostadin Cholakov, Markus Egg, Andy Way, Lexi Birch, Katia Kermanidis, Vilelmini Sosoni, Dimitrios Tsoumakos, Antal van den Bosch, Iris Hendrickx, Michael Papadopoulos, Panayota Georgakopoulou, Maria Gialama, Menno van Zaanen, Ioana Buliga, Mitja Jermol, and Davor Orlic. 2015.  TraMOOC: Translation for Massive Open Online Courses.  In _Proceedings of the 18th Annual Conference of the European Association for Machine Translation_ , page 217, Antalya, Turkey. European Association for Machine Translation.
  * Kordoni et al. (2016) Valia Kordoni, Antal van den Bosch, Katia Lida Kermanidis, Vilelmini Sosoni, Kostadin Cholakov, Iris Hendrickx, Matthias Huck, and Andy Way. 2016.  Enhancing Access to Online Education: Quality Machine Translation of MOOC Content.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC)_ , pages 16–22, Portorož, Slovenia. European Language Resources Association.
  * Liu et al. (2018) Siyou Liu, Longyue Wang, and Chao-Hong Liu. 2018.  Chinese-Portuguese Machine Translation: A Study on Building Parallel Corpora from Comparable Texts.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC)_ , Miyazaki, Japan. European Language Resources Association.
  * Liu et al. (2020) Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020.  Norm-Based Curriculum Learning for Neural Machine Translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 427–436, Online. Association for Computational Linguistics.
  * Luong and Manning (2015) Minh-Thang Luong and Christopher D. Manning. 2015.  Stanford Neural Machine Translation Systems for Spoken Language Domains.  In _Proceedings of the Twelfth International Workshop on Spoken Language Translation_ , pages 76–79, Da Nang, Vietnam.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_ , pages 3111–3119. Curran Associates, Inc.
  * Nakazawa et al. (2016) Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016.  ASPEC: Asian Scientific Paper Excerpt Corpus.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC)_ , pages 2204–2208, Portorož, Slovenia. European Language Resources Association.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 311–318, Philadelphia, USA. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1):67.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving Neural Machine Translation Models with Monolingual Data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural Machine Translation of Rare Words with Subword Units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016c) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich and Volk (2010) Rico Sennrich and Martin Volk. 2010.  MT-based Sentence Alignment for OCR-Generated Parallel Texts.  In _Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas (AMTA)_ , Denver, USA. Association for Machine Translation in the Americas.
  * Sennrich and Volk (2011) Rico Sennrich and Martin Volk. 2011.  Iterative, MT-based Sentence Alignment of Parallel Texts.  In _Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011)_ , pages 175–182, Riga, Latvia. Northern European Association for Language Technology.
  * Song et al. (2020) Haiyue Song, Raj Dabre, Atsushi Fujita, and Sadao Kurohashi. 2020.  Coursera Corpus Mining and Multistage Fine-Tuning for Improving Lectures Translation.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3640–3649, Marseille, France. European Language Resources Association.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to Sequence Learning with Neural Networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 3104–3112. Curran Associates, Inc.
  * Thompson and Koehn (2019) Brian Thompson and Philipp Koehn. 2019.  Vecalign: Improved sentence alignment in linear time and space.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1342–1348, Hong Kong, China. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association.
  * Tolmachev et al. (2018) Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. 2018.  Juman++: A Morphological Analysis Toolkit for Scriptio Continua.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 54–59, Brussels, Belgium. Association for Computational Linguistics.
  * Utsuro et al. (1994) Takehito Utsuro, Hiroshi Ikeda, Masaya Yamane, Yuji Matsumoto, and Makoto Nagao. 1994.  Bilingual text, matching using bilingual dictionary and statistics.  In _COLING 1994 Volume 2: The 15th International Conference on Computational Linguistics_.
  * Vaswani et al. (2018) Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018.  Tensor2Tensor for Neural Machine Translation.  In _Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers)_ , pages 193–199, Boston, USA. Association for Machine Translation in the Americas.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2020) Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020.  Curriculum Pre-training for End-to-End Speech Translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3728–3738, Online. Association for Computational Linguistics.
  * Wang and Neubig (2019) Xinyi Wang and Graham Neubig. 2019.  Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5823–5828, Florence, Italy. Association for Computational Linguistics.
  * Zeng et al. (2019) Jiali Zeng, Yang Liu, Jinsong Su, Yubing Ge, Yaojie Lu, Yongjing Yin, and Jiebo Luo. 2019.  Iterative dual domain adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 845–855, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2020a) Boliang Zhang, Ajay Nagesh, and Kevin Knight. 2020a.  Parallel corpus filtering via pre-trained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8545–8554, Online. Association for Computational Linguistics.
  * Zhang et al. (2020b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2019) Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019.  Curriculum learning for domain adaptation in neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1903–1915, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhou et al. (2020) Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020.  Uncertainty-Aware Curriculum Learning for Neural Machine Translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6934–6944, Online. Association for Computational Linguistics.
  * Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.  Transfer Learning for Low-Resource Neural Machine Translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1568–1575, Austin, USA. Association for Computational Linguistics.


# Spatial Sequence Attention Network for Schizophrenia Classification from
Structural Brain MR Images

###### Abstract

Schizophrenia is a debilitating, chronic mental disorder that significantly
impacts an individual’s cognitive abilities, behavior, and social
interactions. It is characterized by subtle morphological changes in the
brain, particularly in the gray matter. These changes are often imperceptible
through manual observation, demanding an automated approach to diagnosis. This
study introduces a deep learning methodology for the classification of
individuals with Schizophrenia. We achieve this by implementing a diversified
attention mechanism known as Spatial Sequence Attention (SSA) which is
designed to extract and emphasize significant feature representations from
structural MRI (sMRI). Initially, we employ the transfer learning paradigm by
leveraging pre-trained DenseNet to extract initial feature maps from the final
convolutional block which contains morphological alterations associated with
Schizophrenia. These features are further processed by the proposed SSA to
capture and emphasize intricate spatial interactions and relationships across
volumes within the brain. Our experimental studies conducted on a clinical
dataset have revealed that the proposed attention mechanism outperforms the
existing Squeeze & Excitation Network for Schizophrenia classification.

Index Terms—  Schizophrenia (SZ), structural MRI (sMRI), Transfer Learning,
Spatial Sequence Attention (SSA).

##  1 Introduction

As per the World Health Organization (WHO), about 24 million people globally
grapple with schizophrenia (SZ), a condition characterized by enduring
hallucinations and disruptive behavior that profoundly affects their daily
lives [1]. The symptoms of SZ can often overlap with those of other
psychiatric disorders, making both diagnosis and treatment more challenging.
Brain MRI data has proven valuable in uncovering consistent structural
alterations in the brains of SZ patients, including reductions in hippocampal
and thalamic volumes, along with increased globus pallidus volumes. However,
these structural changes, while observable, provide only limited insights into
the disorder [2].

Advancements have been made in SZ classification in automated brain MRI
analysis. Notably, researchers have explored methods to differentiate SZ from
other psychiatric disorders and healthy controls based on gray matter density,
employing support vector machine (SVM) models [3]. Additionally, some studies
have used extended Granger causality to extract features from brain MRIs and
subsequently selected features using Kendall’s tau rank correlation
coefficients, followed by SVM-based classification [4] [5]. Recently, deep
learning, with the convolutional neural network (CNN) architecture, has shown
significant promise in the field of representation learning from images, even
when dealing with 3D structural MRI (sMRI) scans. Prominent works have
employed 3D CNNs for feature extraction to capture intricate spatial and
morphological information in SZ patients [6]. For instance, Hu et al. utilized
a 3D CNN and pre-trained 2D CNN models to detect SZ using 3D sMRI data [7].
However, most of these works tried to use the holistic features learned by
CNNs without using the attention mechanism to emphasize important features,
which might yield performance improvement.

The Squeeze and Excitation Network (SENet) is a neural network architecture
with attention, designed to improve feature extraction and model relationships
between channels in convolutional neural networks (CNNs) [8]. SENet
dynamically recalibrates channel importance in feature maps based on spatial
and channel-wise dependencies, enhancing the network’s ability to capture fine
details and improve performance in computer vision tasks, including medical
image classification [9]. However, SENet faces limitations in its ability to
adapt effectively to higher-dimensional data and localization aspects [10]. To
tackle this challenge, we develop a novel attention mechanism called Spatial
Sequence Attention Network (SSANet). The SSANet’s gating mechanism can
mitigate and outperform challenges by selectively emphasizing SZ-specific
information at spatial locations, learning to assign higher weights to
relevant regions and focusing on crucial local structures in feature maps. Our
experiments provide evidence supporting the superiority of our approach over
SENet, especially in situations with constrained training data.

Fig. 1: The proposed Spatial Sequence Attention Network (SSANet) architecture
extracts morphological features from 3D T1-weighted MRI images (121x145x121
voxels) using a pre-trained 3D DenseNet121 model. The Spatial Sequence
Attention (SSA) module then captures and accentuates complex spatial
interactions and relationships within the brain volumes. Finally, a Feed
Forward Neural Network (FFNN) classifies the images as schizophrenia or
healthy.

##  2 Methods

This study introduces the Spatial Sequence Attention Network (SSANet), a deep
learning architecture for automated SZ classification from 3D T1-weighted MRI
images with limited training data. Figure 1 illustrates the SSANet
architecture. In the subsequent sections, we provide detailed technical
insights into each phase of the proposed methodology.

###  2.1 Morphological Representation Learning

Over recent years, Convolutional Neural Networks (CNNs) have become prominent
for learning representations from visual data. They employ a series of
convolution operations to extract visual features by sliding a filter over the
provided input image, capturing patterns and shapes through weighted summation
of nearby pixel values, given the sufficient amount of training data [11]. The
availability of training data is scarce in the medical domain. To address
this, Transfer Learning was widely employed to extract representation from a
limited dataset using pre-trained models. Following the paradigm, we leverage
a pre-trained DenseNet, comprising 121 layers, to extract morphological
features, augmenting its capabilities through fine-tuning specifically on gray
matter density sMRIs.

In Convolutional Neural Networks, the input from the previous layer $n-1$ is
passed through a non-linear transformation denoted as $H_{n}$ to generate the
input for the current layer $n$, represented as $X^{(n)}=H_{n}(X^{(n-1)})$.
This direct back-to-front gradient propagation can lead to the issue of
vanishing gradients in the network. Addressing this, Huang et. al. presented
the DenseNet with the dense block architecture. The $n$-th layer establishes
connections with all preceding $L$ layers, including the $n-1$-th layer,
thereby facilitating improved gradient flow across the network [12]. Most of
the previous works focus on extracting penultimate fully connected layer
features. Apart from this, we utilize the convolutional feature maps. These
features retain spatial and volumetric information essential for
discriminating SZ. This type of transfer learning method proves advantageous
and adept at meeting the criteria for extracting morphological features [13].

$X_{i}=DenseNet121(I_{i})\quad\forall i\in\\{1,2,...N\\}.$ (1)

From the equation 1, the input gray matter density image for subject $i$ is
denoted by $I_{i}$. It is a $(121\times 145\times 121)$ voxel image that is an
input to a pre-trained 3D DenseNet121 model. The model first applies a 3D
convolution with a $(7\times 7\times 7)$ kernel and pooling to the image.
Then, it performs varying dense layers within each dense block. These layers
include two 3D convolutions with different kernel sizes: $(1\times 1\times 1)$
and $(3\times 3\times 3)$, which reduce the feature map size and improve
computation. Three transition layers (ReLU, batch normalization, and dimension
reduction pooling) connect the dense blocks. This produces a $(7\times 9\times
7\times 1024)$ output vector $X_{i}$ for each subject $i$, which represents
the morphological features of the subject’s brain [14].

###  2.2 Spatial Sequence Attention

The Spatial Sequence Attention [10] is a special mechanism designed to capture
spatial and volumetric dependencies in morphological feature maps $X_{i}$. It
consists of three main components - a 3D convolutional (Conv3D) layer with 64
filters of a $(3\times 3\times 3)$ kernel, a ConvLSTM layer with 64 filters
having $(3\times 3\times 3)$ contextual window, and a Conv3D layer with the
same number of filters and kernel size as the input feature maps. Each
component has its own significance in improving the provided feature maps.
First Conv3D layer learns holistic features from the input. ConvLSTM is a
variant of the LSTM (Long Short-Term Memory) with convolutional operations
integrated into the cell [15]. It uses tanh and sigmoid activation functions
to control the flow of information through the network and capture spatial
information through sparse connections between the input and state
transitions. This allows the model to learn intricate spatial interactions and
relationships across volume in morphological feature maps. The significance of
the current feature map is determined by considering both the current and past
feature maps in the input volumetric sequence. The other Conv3D projects the
output back to the original feature space.

High-level morphological feature representations obtained from the first
Conv3D layer of SSA are processed through input gate ($I_{t}$), forget gate
($F_{t}$), and output gate ($O_{t}$), resulting in the evolution of the cell
state ($\mathbf{C}_{t}$) and hidden state ($\mathbf{H}_{t}$) at a specific
time step $t$. The future state of a specific feature map is determined by
considering the inputs and previous states of neighboring feature maps within
the receptive field. Convolutional operations in the transitions from input to
state and from state to state facilitate this process. The ConvLSTM cell
operates based on a set of fundamental equations involving convolution ($*$)
and element-wise product ($\odot$). Resultants are attention-based feature
representations that are subsequently passed as input to a regularized fully
connected neural network to evaluate the presence of SZ in a given sMRI image.

$I_{t}=\sigma(W_{xi}*\mathbf{X}_{t}+W_{hi}*\mathbf{H}_{t-1}+W_{ci}*\mathbf{C}_{t-1}+\mathbf{b}_{i})$
(2)
$F_{t}=\sigma(W_{xf}*\mathbf{X}_{t}+W_{hf}*\mathbf{H}_{t-1}+W_{cf}*\mathbf{C}_{t-1}+\mathbf{b}_{f})$
(3)
$\mathbf{C}_{t}=F_{t}\odot\mathbf{C}_{t}+I_{t}\odot\tanh(W_{xc}*\mathbf{X}_{t}+W_{hc}*\mathbf{H}_{t-1}+\mathbf{b}_{c})$
(4)
$O_{t}=\sigma(W_{xo}*\mathbf{X}_{t}+W_{ho}*\mathbf{H}_{t-1}+W_{co}*\mathbf{C}_{t-1}+\mathbf{b}_{o})\mathbf{H}_{t}$
(5)

The above set of equations represents crucial steps in processing
morphological feature maps while capturing spatial dependencies. The input
gate ($I_{t}$) assesses the importance of each input feature map in sequence
($\mathbf{X}_{t}$), for inclusion in the cell state ($\mathbf{C}_{t}$). The
forget gate ($F_{t}$) determines the extent to which features from the
previous cell state ($\mathbf{C}_{t-1}$) should be retained. The cell state
($\mathbf{C}_{t}$) is then updated by selectively integrating features from
the input gate and forgetting unwanted features. Subsequently, the output gate
($O_{t}$) regulates the exposure of the updated cell state as the output
($\mathbf{H}_{t}$). The hidden state ($\mathbf{H}_{t}$) is derived by applying
the output gate to the hyperbolic tangent of the updated cell state, ensuring
that the ConvLSTM captures and utilizes both short and long-term dependencies
within sequential data, making it particularly effective for tasks involving
spatial and volumetric information. The weight matrices
($W_{xi},W_{hi},W_{ci},\ldots$) and bias vectors
($\mathbf{b}_{i},\mathbf{b}_{f},\mathbf{b}_{c},\ldots$) associated with each
gate are learned during the training process, enabling the ConvLSTM to adapt
to the specific characteristics of the data it processes.

###  2.3 Regularized Feed Forward Neural Network

The Feed Forward Neural Network (FFNN) comprises two dense layers, with 512
and 256 neurons, employing the Gaussian Error Linear Unit (GELU) activation
function, and includes a softmax layer for class labeling. To mitigate
overfitting, each dense layer is accompanied by a dropout layer set at a rate
of 0.5. Furthermore, weights in the fully connected layers are subjected to
$L_{1}$ regularization, while biases undergo $L_{1}L_{2}$ regularization,
providing additional protection against overfitting during the training
process. These three components are stacked and trained end-to-end through
gradient centralization to make predictions, collectively referred to as the
Spatial Sequence Attention Network. The combination of induced dropout,
parameter regularization, and gradient centralization serves to prevent
overfitting, allowing the model to generate meaningful predictions with
reasonable accuracy. However, it is worth noting that the model’s performance
can vary depending on the pre-trained convolutional base used for
morphological feature extraction.

##  3 Experiments

In this section, we start by presenting the details of the clinical dataset
employed in our experiments. Subsequently, we describe the experimental setup,
evaluation metrics, and conclude with both quantitative and qualitative
analyses of the results.

###  3.1 Dataset

To validate the effectiveness of the proposed Spatial Sequence Attention
network, we utilized a portion of the FBIRN dataset, which comprises both
chronic SZ patients and control individuals [16]. This dataset encompasses
various data modalities, including structural MRI (sMRI), functional MRI
(fMRI), single-nucleotide polymorphisms (SNP), and patient behavior
information. However, we used just sMRI for our experiments to validate the
effectiveness of the attention mechanism on morphological features. In total,
this dataset consists of 186 participants, with 82 individuals diagnosed with
SZ and 104 healthy subjects. Each MRI scan is downsampled by dimensions (121,
145, 121) for efficient processing.

During experiments on this dataset, various hyperparameter values were
explored, including learning rates (0.005 to 0.05), dropout rates (0.1 to
0.5), and regularization rates (0.001 to 0.1). Selected values to optimize
model performance: initial learning rate (0.0001), batch size (32), training
epochs (100), with L2 weight regularization (0.005) and L1_L2 bias
regularization (0.005).

###  3.2 Quantitative Evaluation

For quantitative evaluation, we employ 5-fold cross-validation for SZ
classification on the FBIRN dataset. We use standard classification metrics
such as Accuracy (Acc), Precision (Prec), Recall, and $F_{1}$ Score to report
the performance of conducted experiments. As a baseline model for our
attention mechanism, we incorporate the existing Squeeze and Excitation
Attention (SENet) into the feature extraction process of pre-trained
DenseNet121 for the classification of SZ based on morphological MRI features.
Subsequently, we aim to assess the effectiveness of the proposed Spatial
Sequence Attention (SSANet) in terms of the feature extraction process of pre-
trained DenseNet121. Additionally, we provide a comparative analysis with
recent studies in the field that have utilized the same dataset and
experimental configuration [14] [17], presenting the results in Table 1.

Table 1: Comparison of Five-fold Cross-validation Metrics (in %) for SZ Classification Method | Acc | Prec | Recall | $F_{1}$
---|---|---|---|---
DenseNet [12] | 70.00 | - | - | -
XGBoost [17] | 71.51 | - | - | -
SENet | 72.80 | 77.84 | 68.06 | 66.79
|  $\pm$ 0.05 |  $\pm$ 0.06 |  $\pm$ 0.08 |  $\pm$ 0.09
SSANet (Ours) | 75.12 | 77.22 | 72.76 | 71.91
|  $\pm$ 0.06 |  $\pm$ 0.04 |  $\pm$ 0.08 |  $\pm$ 0.09

Looking at Table 1, it is evident that our proposed SSANet outperformed not
only the Squeeze & Excitation Network (SENet) but also the fine-tuned DenseNet
[12] and DenseNet features combined with XGBoost [17] in terms of accuracy.
SENet, on the other hand, delivered competitive results. SSANet achieved
higher accuracy and more balanced performance metrics. Additionally, the
standard deviations accompanying our results highlight the stability of
SSANet’s performance. These findings underscore the efficacy of our proposed
model in enhancing the SZ classification based on morphological sMRI features
and emphasize its potential as a valuable addition to the field of medical
image analysis and diagnostics.

###  3.3 Qualitative Evaluation

The qualitative evaluation involves visualizing attention heatmaps for axial,
coronal, and sagittal slices. The superior performance of our proposed method
for SZ classification is evident from the attention heatmaps from Figure 2,
which can identify brain regions (frontal lobe, and temporal lobe) for
attention and capture the gray matter density loss in SZ subjects. These
attention maps contribute to enhanced discriminatory features, showcasing the
effectiveness of our approach in capturing nuanced abnormalities associated
with schizophrenia.

Fig. 2: Visualization of (A) Attention Heatmaps; (B) Gray Matter Density Maps
of a SZ (Top) and a normal (Bottom) subject; SSANet correctly gives attention
around the disease impacted areas such as frontal lobe and temporal lobe. Note
the decreased gray matter density in the SZ subject compared with the normal
subject in those regions.

##  4 Conclusion

In this paper, we introduced the Spatial Sequence Attention Network (SSANet),
a novel deep-learning model addressing automated SZ classification with
limited training data. SSANet successfully captures intricate spatial
interactions and relationships across volumes within the brain, yielding
improved SZ classification performance compared with the Squeeze & Excitation
Network (SENet) and DenseNet without attention mechanism. This highlights its
potential for enhancing schizophrenia diagnosis despite limited training data
and complex spatial dependencies. As a future work, our model’s
interpretability and the clinical relevance of its learned features will be
further investigated. Additionally, we will explore aspects of our proposed
method, emphasizing multi-modal data integration, interpretability, ethical
considerations, and generalization across demographics. These directions hold
the potential for enhancing diagnostic accuracy and ensuring equitable
healthcare.

## References

  * [1] Tyrone D Cannon, Isabelle M Rosso, J Megginson Hollister, Carrie E Bearden, Laura E Sanchez, and Trevor Hadley,  “A prospective cohort study of genetic and perinatal influences in the etiology of schizophrenia,”  Schizophrenia bulletin, vol. 26, no. 2, pp. 351–366, 2000.
  * [2] Unn Kristin Haukvik, Cecilie Bhandari Hartberg, and Ingrid Agartz,  “Schizophrenia–what does structural mri show?,”  Tidsskrift for Den norske legeforening, 2013.
  * [3] Hugo G Schnack, Mireille Nieuwenhuis, Neeltje EM van Haren, Lucija Abramovic, Thomas W Scheewe, Rachel M Brouwer, Hilleke E Hulshoff Pol, and René S Kahn,  “Can structural mri aid in clinical classification? a machine learning study in two independent samples of patients with schizophrenia, bipolar disorder and healthy subjects,”  Neuroimage, vol. 84, pp. 299–306, 2014.
  * [4] Axel Wismüller and M Ali Vosoughi,  “Classification of schizophrenia from functional mri using large-scale extended granger causality,”  in Medical Imaging 2021: Computer-Aided Diagnosis. SPIE, 2021, vol. 11597, pp. 337–350.
  * [5] Axel Wismüller, Ali Vosoughi, Akhil Kasturi, and Nathan Hadjiyski,  “Large-scale augmented granger causality (lsagc) for discovery of causal brain connectivity networks in schizophrenia patients using functional mri neuroimaging,”  in Medical Imaging 2023: Biomedical Applications in Molecular, Structural, and Functional Imaging. SPIE, 2023, vol. 12468, pp. 278–286.
  * [6] Delaram Sadeghi, Afshin Shoeibi, Navid Ghassemi, Parisa Moridian, Ali Khadem, Roohallah Alizadehsani, Mohammad Teshnehlab, Juan M Gorriz, Fahime Khozeimeh, Yu-Dong Zhang, et al.,  “An overview of artificial intelligence techniques for diagnosis of schizophrenia based on magnetic resonance imaging modalities: Methods, challenges, and future works,”  Computers in Biology and Medicine, vol. 146, pp. 105554, 2022.
  * [7] Mengjiao Hu, Xing Qian, Siwei Liu, Amelia Jialing Koh, Kang Sim, Xudong Jiang, Cuntai Guan, and Juan Helen Zhou,  “Structural and diffusion mri based schizophrenia classification using 2d pretrained and 3d naive convolutional neural networks,”  Schizophrenia research, vol. 243, pp. 330–341, 2022.
  * [8] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [9] Jyostna Devi Bodapati, Shaik Nagur Shareef, Veeranjaneyulu Naralasetti, and Nirupama Bhat Mundukur,  “Msenet: Multi-modal squeeze-and-excitation network for brain tumor severity prediction,”  International Journal of Pattern Recognition and Artificial Intelligence, vol. 35, no. 07, pp. 2157005, 2021.
  * [10] Nagur Shareef Shaik and Teja Krishna Cherukuri,  “Hinge attention network: A joint model for diabetic retinopathy severity grading,”  Applied Intelligence, vol. 52, no. 13, pp. 15105–15121, 2022.
  * [11] Yann LeCun, Yoshua Bengio, et al.,  “Convolutional networks for images, speech, and time series,”  The handbook of brain theory and neural networks, vol. 3361, no. 10, pp. 1995, 1995.
  * [12] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger,  “Densely connected convolutional networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  * [13] Jyostna Devi Bodapati, Nagur Shareef Shaik, and Veeranjaneyulu Naralasetti,  “Deep convolution feature aggregation: an application to diabetic retinopathy severity level prediction,”  Signal, Image and Video Processing, vol. 15, pp. 923–930, 2021.
  * [14] H. Yu, T. Florian, V. Calhoun, and D. Ye,  “Deep learning from imaging genetics for schizophrenia classification,”  in 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022, pp. 3291–3295.
  * [15] Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO,  “Convolutional lstm network: A machine learning approach for precipitation nowcasting,”  in Advances in Neural Information Processing Systems. 2015, vol. 28, pp. 802–810, Curran Associates, Inc.
  * [16] David B Keator, Theo GM van Erp, Jessica A Turner, Gary H Glover, Bryon A Mueller, Thomas T Liu, James T Voyvodic, Jerod Rasmussen, Vince D Calhoun, Hyo Jong Lee, et al.,  “The function biomedical informatics research network data repository,”  Neuroimage, vol. 124, pp. 1074–1079, 2016.
  * [17] Ayush Kanyal, Srinivas Kandula, Vince Calhoun, and Dong Hye Ye,  “Multi-modal deep learning on imaging genetics for schizophrenia classification,”  in 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2023, pp. 1–5.