11institutetext: College of Computer Science, Inner Mongolia University,
Hohhot, China
National & Local Joint Engineering Research Center of Intelligent Information
Processing Technology for Mongolian, Hohhot, China
Inner Mongolia Key Laboratory of Mongolian Information Processing Technology,
Hohhot, China
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>{csfeilong,
<EMAIL_ADDRESS>
Muhan Na et al.

# A Deep Investigation of RNN and Self-attention for the Cyrillic-Traditional
Mongolian Bidirectional Conversion††thanks: Rui Liu is the corresponding
author. This research is funded by the National Key Research and Development
Program of China (No.2018YFE0122900), China National Natural Science
Foundation (No.62066033), the High-level Talents Introduction Project of Inner
Mongolia University (No. 10000-22311201/002) and the Young Scientists Fund of
the National Natural Science Foundation of China (No. 62206136), Applied
Technology Research and Development Program of Inner Mongolia Autonomous
Region (No.2019GG372, 2020GG0046, 2021GG0158, 2020PT002), Young science and
technology talents cultivation project of Inner Mongolia University
(No.21221505), The Research Program of The National Social Science Fund of
China (No.18XYY030).

Muhan Na  0000-0002-8104-1749 Rui Liu ${}^{(\textrm{{\char 0\relax}})}$
Feilong Bao  0000-0001-7312-1629 Guanglai Gao

###### Abstract

Cyrillic and Traditional Mongolian are the two main members of the Mongolian
writing system. The Cyrillic-Traditional Mongolian Bidirectional Conversion
(CTMBC) task includes two conversion processes, including Cyrillic Mongolian
to Traditional Mongolian (C2T) and Traditional Mongolian to Cyrillic Mongolian
conversions (T2C). Previous researchers adopted the traditional joint sequence
model, since the CTMBC task is a natural Sequence-to-Sequence (Seq2Seq)
modeling problem. Recent studies have shown that Recurrent Neural Network
(RNN) and Self-attention (or Transformer) based encoder-decoder models have
shown significant improvement in machine translation tasks between some major
languages, such as Mandarin, English, French, etc. However, an open problem
remains as to whether the CTMBC quality can be improved by utilizing the RNN
and Transformer models. To answer this question, this paper investigates the
utility of these two powerful techniques for CTMBC task combined with
agglutinative characteristics of Mongolian language. We build the encoder-
decoder based CTMBC model based on RNN and Transformer respectively and
compare the different network configurations deeply. The experimental results
show that both RNN and Transformer models outperform the traditional joint
sequence model, where the Transformer achieves the best performance. Compared
with the joint sequence baseline, the word error rate (WER) of the Transformer
for C2T and T2C decreased by 5.72% and 5.06% respectively.

###### Keywords:

Cyrillic Mongolian Traditional Mongolian Bidirectional Conversion Recurrent
Neural Network (RNN) Self-attention.

##  1 Introduction

Mongolian language belongs to the language group of the Altaic language family
and is both the most widely spoken and most-known member of the Mongolic
language family [11]. The number of speakers across all its dialects may be
5.2 million, including the vast majority of the residents of Mongolia and many
of the ethnic Mongol residents of the Inner Mongolia Autonomous Region of the
People’s Republic of China. In Mongolia, the Mongolian character is currently
written in Cyrillic Mongolian script [14]. In Inner Mongolia of China, the
language is written in the Traditional Mongolian script [11]. The Cyrillic-
Traditional Mongolian Bidirectional Conversion (CTMBC) [9] consists of
Cyrillic Mongolian to Traditional Mongolian conversion (C2T) and Traditional
Mongolian to Cyrillic Mongolian conversion (T2C). Therefore, the CTMBC
facilitates the language communication between the compatriots of both
countries and has great importance to the scientific, economic, and cultural
fields of both countries.

The traditional method mainly focuses on the rule-based approach and the
statistical model, such as the joint sequence model. Specifically, Bao Sarina
et al. [22, 19] focused on the nouns and case suffixes translation in the
CTMBC task and proposed a hybrid translation method that includes bilingual
dictionary, rules and N-gram language model. Gao et al. [17] also proposed a
hybrid method based on dictionaries and rules. Feilong et al. [9] first
adopted joint sequence model for the CTMBC task. In 2017, Feilong et al. [8]
further proposed a hybrid method based on a combination of rule and joint
sequence model. It adopts the rule-based approach to convert the words in the
vocabulary, and uses joint sequence model to convert the out-of-vocabulary
(OOV) words.

However, the conversion methods based on rule or statistical model perform
some shortcomings. 1) The rule-based approach is unavailable when facing OOV
words and loanwords [8]: Mongolian words are made up of stems and suffixes.
Different types of suffixes are added to the same stem to form different
words, which leads to a huge vocabulary; Furthermore, the Mongolian language
contains many loanwords that do not follow the Mongolian word-formation rules
[11]; 2) The statistical model performs poor generalization ability and
limited modeling capability: the joint sequence model holds shallow
architecture and does not have strong nonlinear modeling ability [23, 13].

We note that the CTMBC task can be interpreted as a standard machine
translation task [9] between two languages. In other words, it is also a
sequence-to-sequence (Seq2Seq) [23] modeling problem. The “encoder-decoder”
structure [18, 23, 13] have been successfully applied to various Seq2Seq
tasks, including neural machine translation [23, 7, 24], speech synthesis [6,
5, 2, 4, 3], and the grapheme-to-phoneme (G2P) conversion [10, 30, 28], etc.
Recurrent neural networks (RNN) and self-attention based “encoder-decoder”
models and pre-training language model [15, 29] have lately received attention
from the community [13, 24, 12].

For the CTMBC task, we don‘t have enough data to train a large-scale pre-
training model. Meanwhile, the RNN and Self-attention based encoder-decoder
models have great performance in Seq2Seq modeling tasks. However, an open
problem remains as to whether the CTMBC performance can be improved by
utilizing the RNN and Transformer models. To answer this question, this paper
investigates the utility of these two powerful techniques without large-scale
pre-training model for CTMBC task combined with agglutinative characteristics
of Mongolian language. In this study, we validate the RNN and self-attention
in the CTMBC task respectively. 1) We deep study the agglutinative
characteristics of the Mongolian language, including the Traditional and
Cyrillic Mongolian; 2) We build RNN and self-attention based CTMBC models
according to the analyzed agglutinative characteristics; 3) To identify the
optimal network configurations, we investigate and compare the RNN and self-
attention models with various configurations. The experimental results show
that both RNN and self-attention models outperform the traditional joint
sequence model, where the self-attention model achieves the best performance.

The main contributions of this paper include, 1) We conduct a deep
investigation of RNN and self-attention for the CTMBC task; 2) We combine the
agglutinative characteristics of Mongolian language with RNN and self-
attention models to achieve outstanding performance. To our best knowledge,
this is the first deep investigation of the recent powerful deep learning
models, including RNN and self-attention models, for the CTMBC task.

##  2 Task Challenges

###  2.1 Data Sparseness

For machine translation tasks, a large amount of aligned sentence-level data
is required. Although this paper treats CTMBC as a word-level machine
translation task, there is currently no high-quality and large-scale training
data for model training due to the low-resource nature of Mongolian language.

###  2.2 Agglutinative Characteristics

For the Mongolian written form, there are two styles including Cyrillic
Mongolian and Traditional Mongolian scripts. The similarity and difference
between them in terms of agglutinative Characteristics can be summarized as
follows.

####  2.2.1 Similarity

The similarity between Traditional and Cyrillic Mongolian scripts are
summarized as word formation, pronunciation, and grammar rules.

$\bullet$ Word formation: Cyrillic Mongolian and Traditional Mongolian is an
agglutinative language in which a new word is created by joining multiple
suffixes to a word stem.

$\bullet$ Pronunciation: Cyrillic Mongolian can always find at least one
Traditional Mongolian with the same pronunciation.

$\bullet$ Grammar rules: Cyrillic Mongolian retains most of the grammatical
features of Traditional Mongolian. They are consistent in grammar.

####  2.2.2 Difference

The difference between Traditional and Cyrillic Mongolian scripts are
summarized as symbol systems, morphological rules and the correlation between
pronunciation and spelling.

$\bullet$ Different symbol systems: Cyrillic script has 13 vowels, 20
consonants, 1 hardened character, and 1 softened character [8]. Traditional
script has only 8 vowels and 27 consonants [8]. Cyrillic script performs case-
sensitive and follows the rule of capitalization of the initial letter,
Traditional Mongolian is not so. Traditional Mongolian have inconsistencies
between code and presentation form. To avoid this phenomenon, in this work, we
transform the Traditional Mongolian characters into their corresponding Latin
transcriptions [20].

$\bullet$ Different morphological rules: The morphological rules of Cyrillic
Mongolian has 66 categories [25, 16], while Traditional Mongolian just has 4
categories [11]. Such difference lead to the fact that the characters of the
two kind of Mongolian words can not correspond one by one. For example,
Traditional Mongolian word “” (Latin: bariyasv) has 8 characters, and its
corresponding Cyrillic Mongolian word “” has 6 characters.

$\bullet$ Different correlation between pronunciation and spelling: The
pronunciation and spelling of Cyrillic Mongolian are one-to-one
correspondence, Traditional Mongolian is otherwise, occur a one-to-many
relationship. In Traditional Mongolian, the vowels or consonants may be
dropped, added, or changed when reading. As a result, one Cyrillic Mongolian
word may correspond to more than one Traditional Mongolian word . For example,
Cyrillic Mongolian for both “” (Latin: dala, means: expand) and “” (Latin:
dalv, means: shoulder) is “”.

Mongolian words are constructed by successively concatenating suffixes to
stems, which results in the word forms in Mongolian being very large and
having a lot of out-of-vocabulary words. Meanwhile, different morphological
rules make it difficult for the model to learn the mapping relationship
between two Mongolian characters.

The above data sparseness and agglutinative characteristics problems bring
huge challenge for our CTMBC task. The agglutinative characteristics of
Traditional and Cyrillic Mongolian scripts are the knowledge we must master
for the CTMBC task. In the next section, we will fully explore the above
language knowledge to complete the C2T and T2C conversion.

##  3 RNN and Self-attention based Encoder-Decoder Frameworks for C2T and T2C

In this section, we will introduce the overall Encoder-Decoder frameworks for
C2T and T2C, at first. Then we will explain their workflows in detail
respectively. Last and not least, the backbone of the encoder-decoder
framework, including RNN and self-attention, will be introduced.

Figure 1: The overall frameworks of (a) C2T conversion model, (b) T2C
conversion model. Our task is similar to G2P task [28]. In the process of T2C
conversions, our input is Traditional Mongolian word and output is Cyrillic
Mongolian word, while vice versa in C2T conversion.

###  3.1 Overall Framework

As shown in Fig. 1, the overall frameworks of C2T and T2C are illustrated in
the upper and bottom panels respectively. We follow G2P [28] workflow to build
our C2T and T2C frameworks, in which the input and output both are word-level.
As mentioned in Section. 2.2, there are some similarities and differences
between Cyrillic and traditional Mongolian scripts in terms of agglutinative
Characteristics. Therefore, we make necessary processing for Mongolian scripts
and design the workflows for C2T and T2C specifically.

###  3.2 Workflow of C2T

As shown in Fig. 1(a), C2T conversion consists of 3 parts, including “Text
Pre-processing”, “Encoder-Decoder” and “Post-processing”. Text Pre-processing
takes the Cyrillic Mongolian word as input to output the character sequence.
Then the encoder reads the character sequence to generate the high-level
hidden representation, which is then fed to the decoder to predict the Latin
transcriptions of Traditional Mongolian characters. At last, Post-processing
module converts the Latin transcription to the Traditional Mongolian scripts
as the final conversion results.

####  3.2.1 Text Pre-processing

Given that $C$ is the input Cyrillic Mongolian word, Text Per-processing can
output the character sequence $\textbf{c}=c_{1},c_{2},c_{3},c_{4},...,c_{n}$
($n$ means the sequence length or character number). Words and sub-words also
are two possible training units. However, the complex word or sub-word
generation pipeline will bring some noise and bring about side effects. In
other words, incorrect tokenization or separation results will affect model
performance.

####  3.2.2 Encoder-Decoder

The encoder summarizes the input sequence, Cyrillic Mongolian characters, into
a set of vectors while the decoder conditions the encoded input sequence, and
generates the output sequence, Traditional Mongolian Latin transcription
characters, one token at a time. Let $\textbf{o}=o_{1},o_{2},...,o_{m}$ be the
sequence of $m$ symbols in The encoder is simply a function of the following
form: $x=Encoder(c_{1},c_{2},...,c_{n})$, in which
$\textbf{x}=x_{1},x_{2},...,x_{m}$ is a list of fixed size vectors.

The decoder is often trained to predict the next word $o_{t}$ ($t\in[1,m]$)
given all the previously predicted words $o_{1},o_{2},...,o_{t-1}$. In other
words, the decoder defines a probability over the conversion o by decomposing
the joint probability into the ordered conditionals:
$P\left(o\middle|x\right)=\prod_{t=1}^{M}P\left(o_{t}\middle|o_{1},o_{2},...,o_{t-1};x_{1},x_{2},...,x_{n}\right)$.

####  3.2.3 Post-processing

The output of decoder is character-level Latin transcription. Therefore, Post-
processing aims to restore the Latin transcription to Traditional Mongolian
word $T$.

###  3.3 Workflow of T2C

As shown in Fig. 1(b), different from the C2T conversion process, T2C
conversion consists of the following 2 parts, including “Text Pre-processing”
and “Encoder-Decoder”. Text Pre-processing module aims to convert Traditional
Mongolian word to their character-level Latin transcription. The Encoder
processes the character-level Latin transcription and outputs the high-level
representation. Decoder module predicts the Cyrillic Mongolian as the result.

####  3.3.1 Text Pre-processing

There are two steps in the Text Pre-processing of T2C. First, convert the
Traditional Mongolian into Latin transcription. Second, divide word-level
Latin transcription into character-level Latin sequence. As mentioned in
Section 2.2, in this work, we transform the Traditional Mongolian characters
to their corresponding Latin transcription for model training. Similar with
Section 3.2.1, we also divide Latin transcription into Latin characters.

####  3.3.2 Encoder-Decoder

In the T2C task, the encoder-decoder module is the same as that, as described
in Section 3.2.2 in the C2T task. The only difference is that the input of T2C
task is Traditional Mongolian Latin transcription characters and the output is
Cyrillic Mongolian characters.

###  3.4 Backbone of Encoder-Decoder Framework

In this section, we use C2T as an example to introduce the backbone of
Encoder-Decoder Framework. To validate the RNN and Self-attention for the
CTMBC task, we employ attention based RNN and self-attention model as the
backbone of Encoder-Decoder framework. We will introduce the details next.

####  3.4.1 RNN

Specifically, RNN-based encoder is implemented with a multi-layer
bidirectional LSTM (BiLSTM), which transforms the input Cyrillic Mongolian
sequence $c$ into a high-level representation
$h=\left\\{h_{n}\right\\}_{n=1}^{N}$. The RNN-based decoder takes $h$ and all
previously predicted outputs $o_{1:m-1}$ as input, producing the probability
distribution of the token $o_{m}$. Specifically, at each decoder time step
$t$, the posterior distribution of the predicted output is generated from the
cascade of decoder states st and context vector $a_{t}$. The $a_{t}$ is the
context information produced by the attention module based on the hidden state
of the encoder and decoder.

####  3.4.2 Self-attention

Unlike the RNN-based encoder-decoder framework, the self-attention based
encoder-decoder framework, that is Transformer, replaces the RNN modules with
the pure self-attention mechanism. Specifically, Transformer encoder consists
of $N$ identical Transformer blocks [27]. Each block consists of two sub-
layers, including the multi-head self-attention mechanism and the fully
connected feed-forward network. Residual connection and normalization are
added to each sub-layers.

Transformer decoder also consists of $N$ identical Transformer blocks [27],
which include masked multi-head self-attention, multi-head self-attention and
the fully connected feed-forward network. Residual connection and
normalization are also added to each sub-layers. In addition, different from
the RNN based model, the Transformer encoder and decoder take a position
encoding [27] as an additional input.

##  4 Experiments and Results

###  4.1 Datasets

We report the experiments on a Cyrillic-Traditional Mongolian mapping
dictionary, denoted as “Mon_data_63668”. The Mon_data_63668 dataset includes
63668 word pairs which collected from the New Mongolian-Chinese Dictionary
[31]. Note that 58436 pairs were randomly selected as the training set, and
the remaining 5232 word pairs were used as the model testing set.

###  4.2 Evaluation Setup

####  4.2.1 Comparative study

We implement three frameworks for C2T and T2C in a comparative study. RNN and
self-attention based encoder-decoder models are studied for the first time for
CTMBC, while Joint sequence model is the baseline and re-implementation of
[9].

$\bullet$ Joint Sequence Model (“Joint” for short) (Baseline): The idea of the
joint sequence model [10] is to represent the relationship between an input
sequence and an output sequence in terms of a common sequence of joint units
composed of input and output symbols. N-gram language model is used to predict
character during decoding. Note that we will seek the optimal model
configuration by adjusting the value of $N$ in N-gram. We infer the model
parameters through the expectation maximization and trim the evidence to avoid
over-fitting. We also discount evidence for smoothing.

$\bullet$ RNN based Encoder-Decoder (New) We implement two architectures,
including RNN based Encoder-Decoder without attention (“RNN” for short) and
the attention based RNN Encoder-Decoder as mentioned in Section 3.4.1
(“RNN+ATT” for short), for comparison. Note that we set the RNN hidden size to
512 and 1024, and set the hidden layer to 1, 2, and 4 to explore the optimal
model configuration.

For RNN training, the input of encoder is 128-dimensional character sequence
and the output of decoder is mongolinan characters. The parameters of RNN are
set as following: batch size = 32, epochs = 100, learning rate = 0.0005. We
decrease the learning rate every 20 epochs by a factor of 0.9. The loss
function is using the cross-entropy(CE) criterion. We conduct experiments on
an NVIDIA GPU (Tesla P40).

$\bullet$ Self-attention based Encoder-Decoder (“Transformer” for short)
(New): “Transformer-Tiny” model [26] to build our self-attention based
Encoder-Decoder model since it holds a smaller vocabulary than “Transformer-
base” [26] model and therefore is more suitable for our task. Note that we set
the attention head to 2 and 4, and set the hidden layer to 1, 2, 4, and 6 to
explore the optimal parameter combination.

All the comparative experiments are conducted in the Tensor2Tensor [26]. We
set embedding dimensional = 128, train steps = 100000, batch size = 4096,
learning rate = 0.2 and the learning rate warm-up [21] steps to 8000. All word
embeddings are initialized randomly and then updated with the whole model. We
conduct experiments on an NVIDIA GPU (Tesla P40).

####  4.2.2 Metrics

The result is performed in terms of word error rate (WER) and character error
rate (CER). The formulas of WER and CER are as follows:
$WER=1-\frac{N_{corrent}}{N_{total}}$,
$CER=\frac{N_{ins}+N_{del}+N_{sub}}{N_{charatertotal}}$. $N_{corrent}$ is the
number of correctly predicted Mongolian words; $N_{total}$ is size of test
set; $N_{ins}$ is the number of character insertion errors; $N_{del}$ is the
number of character deletion errors; $N_{sub}$ is the number of character
substitution errors; $N_{charatertotal}$ is the number of word characters.

Following [10, 30] in the case of multiple references Mongolian, the variant
with the smallest edit distance is used. Similarly, if there are multiple
references of Mongolian for a word, a word error occurs only if the predicted
Mongolian doesn’t match any of the references.

Figure 2: The WER and CER results of C2T. The first panel is results of Joint
sequence model and “N” is delegates N of N-gram. The second panel and third
panel is results of RNN based model and RNN+ATT based model, “U” represents
BiLSTM hidden units and “L” represents BiLSTM hidden layer. The fourth panel
is results of Transformer model under different parameters, “S” represents the
layer size of Transformer and “H” represents the attention head number. Figure
3: The WER and CER results of T2C.

###  4.3 Performance Comparison for C2T and T2C

Fig. 2 shows the WER and CER results of all models under different parameters
for C2T. We will analysis the results next.

####  4.3.1 Joint vs. (RNN & RNN+ATT)

As shown in the first panel of Fig. 2, by increasing $N$ from 1 to 10, we can
find that the WER and CER are basically decreasing gradually. “N=9” for
optimal performance, with with WER=22.63% and CER=4.2%.

From the second panel of Fig. 2, the 1-layer BiLSTM with 1024 hidden units
obtains the best performance, which can reach 4.56% on CER and 22.46% on WER.
The optimal result, RNN(1024U_1L), is almost the same as “N=9”. We also fix
hidden layer number of BiLSTM and adjust the hidden unit size to observe the
performance. It can be found that the performance of the model can be improved
by increasing the hidden unit size. For example, RNN(512U_1L) outperforms the
RNN(1024U_1L) model. Stacking more hidden layers sometimes does not lead to
significant improvements. For example, RNN(1024U_1L) beats the RNN(1024U_4L)
model.

In the third panel of Fig. 2, we can find that the 2-layer BiLSTM with 1024
hidden units, RNN+ATT(1024U_2L), obtains the best performance, which can reach
3.72% in CER and 19.51% in WER. Compared with the optimal results of RNN,
RNN(1024U_1L) model, the WER of RNN+ATT(1024U_2L) is reduced by 2.95%. We
believe that adding the attention mechanism to the RNN based encoder-decoder
model can effectively capture the internal information of input Mongolian
character sequence better and improve the model performance.

T2C has similar results as C2T. As shown in Fig. 3, RNN(1024U_L1) model
outperforms the “N=8” model, RNN+ATT(1024U_L2) model beat the “N=8” model,
which consists of C2T.

####  4.3.2 Joint vs. Transformer

The fourth panel of Fig. 2 show the WER and CER of Transformer model under
different parameters.

The results show that the Transformer framework achieves best performance when
layer size is 6 and attention head is 4. “6S_4H” achieves 3.15% on CER and
16.92% on WER and gains 5.72% relative reduction compared to the “N=9” model
in terms of WER. We also find that increasing the attention head number can
effectively improve the model performance. For example, the “6S_4H” and the
“4S_4H” outperforms the “6S_2H” and the “4S_2H” respectively.

The result of T2C is similar to C2T. In Fig. 3, “6S_4H” model outperforms the
“N=8” model, which has identical conclusion to C2T.

####  4.3.3 (RNN & RNN+ATT) vs. Transformer

Comparing the optimal results of RNN and Transformer, Transformer achieves
better performance.

For example, the “6S_4H” achieves 16.92% WER and outperforms RNN(1024U_1L)
with 5.54% WER. The self-attention mechanism has stronger ability to capture
long-term dependencies within input sequence, therefore brings better
performance. Comparing RNN+ATT(1024U_2L) and “6S_4H”, “6S_4H” beat
RNN+ATT(1024U_2L) with 2.95% WER. The reason is that the self-attention
mechanism can capture global context information, it is helpful for model
prediction.

The results of T2C resemble C2T. As shown in Fig. 3, “6S_4H” model beat the
RNN(1024U_1L) and RNN+ATT(1024U_2L), which is consistent with C2T.

###  4.4 Case study

To analyze the translated words produced by the various model, we use two
examples extracted from test set to compare them in Fig. 4. The wrong
translated characters are highlighted in blue.

Figure 4: The example of translated words produced by various models. The
Latin characters not correctly translated are highlighted in blue. The long
vowels and suffix are highlighted in red.

As shown in Fig. 4(a), we take a Cyrillic Mongolian word “” as an example to
show the C2T conversion results.The Latin transcriptions is “alagavbtvr”. Note
that there is vowel substitution phenomenon in the pronunciation of this word.
The traditionally Mongolian word is pronounced with the fifth vowel “a”
becoming “v” and the ninth vowel “v” becoming “a”. The fifth row shows the
conversion result of Joint model. We can find that there are 4 wrong
characters in the results, which performs poor performance, while RNN and
RNN+ATT models just have 2 and 1 wrong characters respectively. We further
observe that the translated word output from Transformer model exactly the
same as the target Latin transcriptions “alagavbtvr” and contains no errors.

Similar with Fig. 4(a), as shown in the Fig. 4(b), we also take a traditional
Mongolian word “” as an example to show the T2C conversion results. The Latin
transcriptions is “arbatv-yin”. We observe the Joint, RNN, and RNN+ATT models
all have some wrong conversion characters, while Transformer model contains no
errors. Note that the Transformer improves the conversion accuracy of long
vowels and suffix that are highlighted in red.

The case study results show that the RNN and Transformer are stronger than
Joint model and Transformer achieves the best performance, which is consistent
with our previous results.

##  5 Conclusion

In this paper, we introduce RNN and self-attention based models into the
Cyrillic-Traditional Mongolian Bidirectional Conversion (CTMBC) task. This is
the first deep investigation of the recent powerful deep learning models for
the CTMBC task. Compared with the joint sequence model baseline, RNN based
models and self-attention model gains significant improvements. Note that
self-attention model achieves the best performance at both T2C and C2T tasks.
In addition, we also compared the parameter setting of each model, such as
hidden layer number, hidden unit size, attention head number, etc., in detail
to determine the optimal model configuration. Note that this paper mainly
studies the CTMBC task at word-level, which similar with G2P. However, word
level conversion is plagued by frequent polyphonic words and lack of
contextual information, which may limit model performance. In the future work,
we will focus on CTMBC task at sentence-level to make full use of context
information for more accurate conversion performance.

## References

  * [1]
  * [2] Liu, R., Bao, F., Gao, G., Zhang, H., Wang, Y.: Improving mongolian phrase break prediction by using syllable and morphological embeddings with bilstm model. In: Interspeech. pp. 57–61 (2018)
  * [3] Liu, R., Sisman, B., Bao, F., Gao, G., Li, H.: Modeling prosodic phrasing with multi-task learning in tacotron-based tts. IEEE Signal Processing Letters 27, 1470–1474 (2020)
  * [4] Liu, R., Sisman, B., Bao, F., Yang, J., Gao, G., Li, H.: Exploiting morphological and phonological features to improve prosodic phrasing for mongolian speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 274–285 (2020)
  * [5] Liu, R., Sisman, B., Li, J., Bao, F., Gao, G., Li, H.: Teacher-student training for robust tacotron-based tts. In: ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 6274–6278. IEEE (2020)
  * [6] Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al.: Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 (2017)
  * [7] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. Computer Science (2014)
  * [8] Bao, F., Gao, G., Wang, H., Lu, m.: Combining of rules and statistics for cyrillic mongolian to traditional mongolian conversion. journal of chinese information procession 31(3), 156-162 (2017)
  * [9] Bao, F., Gao, G., Yan, X., Wei, H.: Research on conversion approach between traditional mongolian and cyrillic mongolian. Computer Engineering and Applications pp. 206–211 (2014)
  * [10] Bisani, M., Ney, H.: Joint-sequence models for grapheme-to-phoneme conversion. Speech communication 50(5), 434–451 (2008)
  * [11] Chinggaltai: A grammar of the Mongolian language. Inner Mongolia Peoples Publishing House, Hohhot (1991)
  * [12] Cho, K., Van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. Computer Science (2014)
  * [13] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. Computer Science (2014)
  * [14] Chuma, S.: A comparative study of Mongolian and Cyrillic orthography. Inner Mongolia Education Press, Hohhot (2010)
  * [15] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. pp. 4171–4186. Association for Computational Linguistics (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://aclanthology.org/N19-1423
  * [16] Dulamragchaa, U., Chadraabal, S., Ivanov, B., Baatarkhuu, M.: Mongolian language morphology and its database structure. In: 2017 International Conference on Green Informatics (ICGI). pp. 282–285. IEEE (2017)
  * [17] Gao, H., Ma, x.: Automatic system on cyrillic mongolian webpage conversion to traditional mongolian script. Journal of Inner Mongolia University for Nationalities 18(5), 17–18 (2012)
  * [18] Kalchbrenner, N., Blunsom, P.: Recurrent continuous translation models. In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1700–1709 (2013)
  * [19] Li, H., Sarina, B.: The study of comparison and conversion about traditional mongolian and cyrillic mongolian. In: 2011 4th International Conference on Intelligent Networks and Intel-ligent Systems. pp. 199–202. IEEE (2011)
  * [20] Lu, M., Bao, F., Gao, G.: Language model for mongolian polyphone proofreading. In: Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pp. 461–471. Springer (2017)
  * [21] Popel, M., Bojar, O.: Training tips for the transformer model. arXiv preprint arXiv:1804.00247 (2018)
  * [22] Sarina, B.: The research on conversion of noun and its case from Classic Mongolian into Cyrillic Mongolian. Master’s thesis, Inner Mongolia University (2009)
  * [23] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014)
  * [24] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
  * [25] Uganbater.D: Research on Cyrillic and Mongolian script’s morphlolgy and conversion system. Ph.D. thesis, Inner Mongolia University (2014)
  * [26] Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A.N., Gouws, S., Jones, L., Kaiser, Ł., Kalchbrenner, N., Parmar, N., et al.: Tensor2tensor for neural machine translation. arXiv preprint arXiv:1803.07416 (2018)
  * [27] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [28] Wang, Y., Bao, F., Zhang, H., Gao, G.: Joint alignment learning-attention based model for grapheme-to-phoneme conversion. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7788–7792. IEEE (2021)
  * [29] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019)
  * [30] Yao, K., Zweig, G.: Sequence-to-sequence neural net models for grapheme-to-phoneme conversion. computer science (2015)
  * [31] Zhang, Z.: New Mongolian Chinese Dictionary. Commercial Press, Beijing (2011)