# MTet: Multi-domain Translation for English and Vietnamese

Chinh Ngo∗, Trieu H. Trinh∗, Long Phan∗, Hieu Tran∗,
Tai Dang, Hieu Nguyen, Minh Nguyen and Minh-Thang Luong
VietAI Research

###### Abstract

We introduce MTet, the largest publicly available parallel corpus for English-
Vietnamese translation. MTet consists of 4.2M high-quality training sentence
pairs and a multi-domain test set refined by the Vietnamese research
community. Combining with previous works on English-Vietnamese translation, we
grow the existing parallel dataset to 6.2M sentence pairs. We also release the
first pretrained model EnViT5 for English and Vietnamese languages. Combining
both resources, our model significantly outperforms previous state-of-the-art
results by up to 2 points in translation BLEU score, while being 1.6 times
smaller.

**footnotetext: The first four authors contributed equally to this work

##  1 Introduction

Machine Translation is an impactful subdomain of Natural Language Processing
that directly benefits the world’s interconnected regions and nations,
especially so for fast-developing economies such as Vietnam Baum (2020).
Neural machine translation, however, is hindered for many pairs of languages
due to their scarce availability. The literature tackling this problem
consists mainly of regularization and data augmentation methods Provilkov et
al. (2019); Nguyen and Salazar (2019a); Clark et al. (2018). Recently a more
data-centric view with more successful results arises: directly growing the
small existing datasets (Fan et al., 2020; Ngo and Trinh, 2021; Cruz and
Cheng, 2021) and better pretraining methodologies to extract value from large
corpora (Liu et al., 2020; Lample and Conneau, 2019; Song et al., 2019).

In this work, we introduce EnViT5, the first pretrained Transformer-based
encoder-decoder model for English-Vietnamese, and MTet - Multi-domain
Translation for English-VieTnamese, the largest high-quality multi-domain
corpus for English-Vietnamese translation of size 4.2M. Notably, MTet also
focuses on highly technical, impactful yet mostly neglected domains due to
their expensive-to-obtain nature such as law and biomedical bitexts. We also
introduce a test set of four distinctively different domains, refined and
cross-checked by human experts through a data crowdsourcing platform. Our
final model, pretrained on EnViT5 and finetuned on MTet + phoMT Doan et al.
(2021a) outperforms previous results by a significant margin of up to 2 points
in BLEU score. Finally, we perform experiments to confirm that with the same
amount of training data, a multi-domain training set results in a better test
performance as shown in Section 6, further supporting the multi-domain nature
of MTet.

##  2 Related Works

In recent years, research works focusing on improving Machine Translation
Systems for Low-Resource Languages have received a lot of attention from both
academia and the industry Chen et al. (2019); Shen et al. (2019); Gu et al.
(2018); Nasir and Mchechesi (2022). Prior works include collecting more
parallel translation data Thu et al. (2016); Bañón et al. (2020); Sánchez-
Cartagena et al. , training large multilingual models Fan et al. (2020); Liu
et al. (2020), and utilizing data augmentation or regularization techniques He
et al. (2019); Edunov et al. (2018); Provilkov et al. (2019). Previous works
from ParaCrawl Bañón et al. (2020) and BiCleaner Sánchez-Cartagena et al.
focused on mass crawling parallel translation data for many low-resource
language pairs. Yet, previous work Doan et al. (2021b) shows that crawling at
scale still has limitation and affect downstream translation performance. We
also compare our high-quality MTet with other crawling at-scale datasets in
Section 3.

Encouraging results have also been achieved in low-resource English-Vietnamese
translation. The most popular and well-adopted translation dataset for
English-Vietnamese is IWSLT15 Cettolo et al. (2015b), which consists of 133K
text pairs collected from TED talk transcripts. Some studies Provilkov et al.
(2020); Xu et al. (2019); Nguyen and Salazar (2019b) show decent improvements
through different regularization techniques. Recently, PhoMT Doan et al.
(2021b) and VLSP2020 Ha et al. (2020) released larger parallel datasets of
size 3M and 4M text pairs, extracted from publicly available resources for the
English-Vietnamese translation. mBART model trained on PhoMT sets the current
state-of-the-art results

Table 1: Results on PhoMT English-Vietnamese Translation Test Set

Model | # Params | Pretrained | Finetuned | En-Vi | Vi-En
---|---|---|---|---|---
Dataset | # pairs
M2M100 | 1.2B | - | CCMatrix + CCAligned | 7.5B | 35.83 | 31.15
Google Translate | - | - | - | 39.86 | 35.76
Bing Translator | - | - | - | 40.37 | 35.74
Transformer-base | 65M | - | PhoMT | 3M | 42.12 | 37.19
Transformer-big | 213M | - | PhoMT | 3M | 42.94 | 37.83
mBART$\dagger$ | 448M | CC25 | PhoMT | 3M | 43.46 | 39.78
EnViT5-base | 275M | CC100 | MTet | 4.2M | 43.87 |  39.57
MTet + PhoMT | 6.2M | 45.47 | 40.57

  * •

Notes: The best scores are in bold and second best scores are underlined.
($\dagger$) mBART trained on PhoMT train set are published work Doan et al.
(2021b) that previously achieved state-of-the-art results on English-
Vietnamese translation.

##  3 MTet: a Machine Translation dataset in English and Vietnamese

In this section, we describe in details our MTet - Multidomain Translation for
English-vieTnamese dataset. We curated a total of 4.2M training examples111Our
work started and progress concurrently to PhoMT, therefore a significant chunk
of our data is overlapped. After deduplication, 3M new training examples are
contributed on top of PhoMT existing training set.. Based on the curation
methodology, we divide this data into four types.

#### Combining existing sources

This includes sources from the Open Parallel corPUS Tiedemann (2012), spanning
across different domains such as educational videos Abdelali et al. (2014),
software user interface (GNOME, KDE4, Ubuntu), COVID-related news articles
(ELRC), religious texts Christodouloupoulos and Steedman (2015), subtitles
(Tatoeba), Wikipedia Wołk and Marasek (2014), TED Talks Reimers and Gurevych
(2020). Together with the original IWSLT’15 Cettolo et al. (2015a) training
set, the total dataset reaches 1.2M training examples. We train a base
Transformer on this data, denoted $bT_{A}$, to aid the collection of other
data sources described below.

#### Scoring and filtering

Another large source from OPUS is OpenSubtitles Lison and Tiedemann (2016) and
CCAlign-envi El-Kishky et al. (2020) of sizes 3.5M and 9.3M respectively. For
OpenSubtitles, manual inspection showed inaccurate translations similar to the
previous observations in Doan et al. (2021b). Including CCAlign-envi as-is
will significantly reduce the model test performance in test set (Appendix C).
For this reason, we make use of $bT_{A}$ to score each bitext by computing the
loss of all text pairs and select the best 700K training examples using cross-
validation on the tst2013 test set222https://github.com/stefan-it/nmt-en-vi.
CCAlign-envi, on the other hand, is entirely discarded through the same
process.

#### Dynamic Programming style alignment

Another large source of parallel data but trickier to extract comes from
weakly-aligned books and articles Ladhak et al. (2020). This includes many
mismatches at sentence and paragraph levels due to versioning, translator
formatting, extra headers and page footers information. We propose a dynamic-
programming style alignment algorithm detailed in Algorithm 1, a simplified
version of BleuAlign (Sennrich and Volk, 2011), to filter and align sentences
between each pair of documents, maximizing the total BLEU score after
alignment. In total, we collected 900K training examples from 300 bilingual
books and news articles.

#### Manual crawl and clean

For this source, we focus on more technical and high-impact domains, this
include law documents and biomedical scientific articles. We manually crawl
and clean across 20 different websites of public biomedical journals and law
document libraries, treating them individually due to their significantly
different formatting. We also manually crawl and clean some other available
websites that are more straightforward to process, as detailed in Appendix D.
Overall, this source contributed another 1.2M training examples.

#### Data crowdsourcing for MTet multi-domain test set

We utilize dataset.vn to distribute 4K test examples held out from the
collected data to 13 human experts to further refine its content. These
domains include biomedical, religion, law, and news.

Overall, we collected 4.2M training examples across all sources. After
combining MTet with PhoMT and IWSLT’15, we grew the existing training set from
3M to 6M training examples. Compared to the existing data sources, this
dataset is both larger and much more diverse, with the inclusion of technical,
impactful, yet so far mostly neglected domains such as law and biomedical
data.

##  4 EnViT5

###  4.1 Model

EnViT5 is a Text-to-Text Transfer Transformer model follows the encoder-
decoder architecture proposed by Vaswani et al. (2017) and the T5 framework
proposed by Raffel et al. (2019). The original works of T5 proposed five
different configurations in model size: small, base, large, 3B, and 11B. For
the practical purpose of the study, we adapt the base architecture for EnViT5
and save the bigger models for future works.

We train EnViT5 models from scratch with the input and output length of 1024
tokens and batch size of 256. For the self-supervised learning objectives, we
use the span-corruption objective with a corruption rate of 15%.

###  4.2 Pretraining data

We use the CC100 Dataset (Monolingual Datasets from Web Crawl Data) Wenzek et
al. (2020) for pre-training the model. The corpus contains monolingual data
for over 100 languages. The corpus was constructed using the pipeline provided
by Wenzek et al. (2020) through processing January-December 2018 Commoncrawl
snapshots. Following the discussion regarding the importance of long context
sequences during pretraining for T5 models from previous works Phan et al.
(2022), we process and filter for 80GB of long sequence (fit in 1024-length
embedding) for each language.

Table 2: BLEU scores of Transformerbase on MTet Multi-Domain Test Set Dataset | En-Vi | Vi-En
---|---|---
Law | Religion | News | Medical | Law | Religion | News | Medical
300K Ted-talk | 16.43 |  20.55 |  27.74 |  14.68 | 10.92 |  18.54 |  20.50 | 7.61
300K Law |  20.6 | 5.2 | 13.07 | 14.035 |  19.15 | 4.97 | 11.275 |  12.535
Multi-domain | 22.07 | 34.77 | 34.77 | 28.76 | 20.45 | 32.21 | 28.66 | 22.4

##  5 Benchmarking EnViT5 and MTet

###  5.1 Experimental settings

To develop our analysis, we conduct experiments to verify the quality of our
MTet dataset and our pretrained bilingual model EnViT5 on both English-to-
Vietnamese and Vietnamese-to-English translation. We are interested in the
final performance of EnViT5 trained on MTet and PhoMT and aim to demonstrate
the best results for both research communities and industry applications.

We compare EnViT5 against well-known engines and baseline models: Google
Translate, Bing Translator, Transformer-base, Transformer-large Vaswani et al.
(2017), and mBART Doan et al. (2021b). All our models are trained for 30
epochs with a batch size of 256. We use SacreBLEU Post (2018) to compute the
case-sensitive BLEU score on the PhoMT test set Doan et al. (2021b).

###  5.2 Results

Table 1 presents BLEU scores of our models on both translation directions. A
first takeaway is that the large finetuned English-Vietnamese translation
dataset accounts for the significant improvement of both En-Vi and Vi-En
translations. Both Transformer models Vaswani et al. (2017) and EnViT5 models
Raffel et al. (2019) without self-supervised learning steps still achieve
notable results on translations compared to current famous translation models
from Google Translate and Bing Translator.

Our EnViT5base model when training on a combination of MTet and the released
PhoMT achieves state-of-the-art results on low-resource English-Vietnamese
translation (45.47 and 40.57 for En-Vi and Vi-En respectively). EnViT5 models
outperform current existing multilingual models mBART and M2M100 while being
significantly smaller in parameter size (275M parameters compared to 448M and
1.2B). This allows our models not only be able to scale in academia but also
very promising for industry and community applications.

##  6 Evaluating multi-domain training data

In this section, we investigate the importance of multi-domain in training
data for a Machine Translation. Since each domain tends to be different in
textual structure and style, the ability to generalize across domains will
makes translation models more practical in real-world applications.

For fair comparison between different domains, pretraining is not used. We
start from Transformerbase Vaswani et al. (2017) and compare the following
three training data on our multi-domain test set described in Section 3: (1)
300k Multi-Domain sentence pairs, (2) 300K Ted-talk sentence pairs, and (3)
300K Law sentence pairs.

Besides TED Talk and Law, other domains do not have enough data to fairly take
part in our comparison. The result of this experiment is shown in Table 2.
There is a significant increase in BLEU scores across all domains when the
model is trained on a Multi-domain training set. Surprisingly, training on
Multi-domain data gives better performance on the Law domain than training on
the pure Law parallel training dataset itself. This result indicates that
multi-domain data during supervised training does indeed lead to better test
set performance.

##  7 A time budget comparison of self-supervised and supervised data

In this experiment, we first start with IWSLT’15 of 133K training examples and
follow two separate processes to improve test performance on top of this
initial data point: (1) we pretrain the model on an amount of non-aligned
bilingual texts described in section 4.2 before further fine-tuning it on the
IWSLT’15 training set for one epoch; (2) we simply grow the IWSLT’15 training
set by an amount of high-quality parallel text before training for one epoch
from random weights.

Figure 1: Improvement on 133k bitexts

In both methods, we measure the improvement in BLEU score at various amounts
of additional data. Following this, we are able to measure the amount of
training wall time needed to achieve the target BLEU score. This time is also
directly proportional to the added amount of data.

As reported in Figure 1, we first confirmed that BLEU score on the test set
steadily improved as both types of data grows, albeit at vastly different
rates. BLEU scores improvement from pretraining quickly diminishes, eventually
hitting a wall. After this point, it becomes infeasible to reach further
target BLEU scores by pure pretraining, a 1.5X increase in pretraining data
does not lead to any meaningful improvement. At a target BLEU score of 34, we
found that it took close to 1000X the amount of data and 2000X training wall
time for pretraining to reach the same performance as supervised training.

##  8 Conclusion

In this work, we released a state-of-the-art pretrained Transformer model and
the largest multi-domain parallel dataset for English-Vietnamese translation.
Namely, MTet consists of 4.2M high-quality training sentence pairs collected
using various methods across multiple domains of data. Combining with phoMT,
the total training data grow to 6.2M sentence pairs, currently the largest
publicly available dataset. Further, we released EnviT5, the first pretrained
model for English and Vietnamese languages. Finetuning EnviT5 on MTet, we
obtained state-of-the-art results with improvements up to 2 points in BLEU
score for English-Vietnamese Translation and 1 BLEU score in Vietnamese-
English translation. With much better test results, our model is also 1.6
times smaller than previous translation models with much faster inference
time.

##  9 Limitations

Although we conjecture that behaviors observed in our work will exhibit
similarly in other low-resource language pairs, there are legitimate reasons
to believe different languages might behave differently due to their own
unique morphology. Generalizing our work to other pairs requires nontrivial
effort and we leave this for future investigation.

##  10 Acknowledgements

We would like to thank the Google TPU Research Cloud (TRC) program, Soonson
Kwon (Google ML Ecosystem programs Lead) and Ba Ngoc Nguyen (Google Developer
Experts in ML) for their support. This project also receives generous support
from Cohost.ai, specifically Mr. Kim Cuong Pham, and dataset.vn, for managing
and labeling our multi-domain test data. We appreciate the effort of
volunteers who refine the multi-domain test dataset: Hàn Thọ Hoà, Trần Viết
Đình, Dương Ngọc Doanh, Nguyễn Bùi Thiên Anh, Nguyễn Công Khanh, Triệu Khắc
Đức, Trần Thị Lành, Hoàng An, Hữu Doanh, Ngọc Khánh, Trọng Văn, Huỳnh Anh, Thu
Huyền.

## References

  * Abdelali et al. (2014) Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014.  The AMARA corpus: Building parallel language resources for the educational domain.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 1856–1862, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Baum (2020) Anja Baum. 2020.  Vietnam’s Development Success Story and the Unfinished SDG Agenda.  _IMF Working Papers_ , 20(31):1–31.
  * Cettolo et al. (2015a) M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and Marcello Federico. 2015a.  The IWSLT 2015 Evaluation Campaign.  In _Proceedings of the International Workshop on Spoken Language Translation_.
  * Cettolo et al. (2015b) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015b.  The IWSLT 2015 evaluation campaign.  In _Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign_ , pages 2–14, Da Nang, Vietnam.
  * Chen et al. (2019) Peng-Jen Chen, Jiajun Shen, Matt Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, and Marc’Aurelio Ranzato. 2019.  Facebook ai’s WAT19 myanmar-english translation task submission.  _CoRR_ , abs/1910.06848.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the bible in 100 languages.  _Language resources and evaluation_ , 49(2):375–395.
  * Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018.  Semi-supervised sequence modeling with cross-view training.  _CoRR_ , abs/1809.08370.
  * Cruz and Cheng (2021) Jan Christian Blaise Cruz and Charibeth Cheng. 2021.  Improving large-scale language models and resources for filipino.  _arXiv preprint arXiv:2111.06053_.
  * Doan et al. (2021a) Long Doan, Linh The Nguyen, Nguyen Luong Tran, Thai Hoang, and Dat Quoc Nguyen. 2021a.  PhoMT: A high-quality and large-scale benchmark dataset for Vietnamese-English machine translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4495–4503, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Doan et al. (2021b) Long Doan, Linh The Nguyen, Nguyen Luong Tran, Thai Hoang, and Dat Quoc Nguyen. 2021b.  Phomt: A high-quality and large-scale benchmark dataset for vietnamese-english machine translation.  _CoRR_ , abs/2110.12199.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  _CoRR_ , abs/1808.09381.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_ , pages 5960–5969, Online. Association for Computational Linguistics.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.  _CoRR_ , abs/2010.11125.
  * Gu et al. (2018) Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018.  Universal neural machine translation for extremely low resource languages.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 344–354, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ha et al. (2020) Thanh-Le Ha, Van-Khanh Tran, and Kim-Anh Nguyen. 2020.  Goals, Challenges and Findings of the VLSP 2020 English-Vietnamese News Translation Shared Task.  In _Proceedings of the 7th International Workshop on Vietnamese Language and Speech Processing - VLSP 2020_.
  * He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019.  Revisiting self-training for neural sequence generation.  _CoRR_ , abs/1909.13788.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization.  _arXiv preprint arXiv:2010.03093_.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _CoRR_ , abs/1901.07291.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation_ , pages 923–929.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Nasir and Mchechesi (2022) Muhammad Umair Nasir and Innocent Amos Mchechesi. 2022.  Geographical distance is the new hyperparameter: A case study of finding the optimal pre-trained language for english-isizulu machine translation.
  * Ngo and Trinh (2021) Chinh Ngo and Trieu H. Trinh. 2021.  Styled augmented translation (sat).  _https://github.com/vietai/SAT_.
  * Nguyen and Salazar (2019a) Toan Q. Nguyen and Julian Salazar. 2019a.  Transformers without tears: Improving the normalization of self-attention.  _CoRR_ , abs/1910.05895.
  * Nguyen and Salazar (2019b) Toan Q. Nguyen and Julian Salazar. 2019b.  Transformers without tears: Improving the normalization of self-attention.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Phan et al. (2022) Long Phan, Hieu Tran, Hieu Nguyen, and Trieu H. Trinh. 2022.  Vit5: Pretrained text-to-text transformer for vietnamese language generation.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  _CoRR_ , abs/1804.08771.
  * Provilkov et al. (2019) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019.  Bpe-dropout: Simple and effective subword regularization.  _CoRR_ , abs/1910.13267.
  * Provilkov et al. (2020) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020.  BPE-dropout: Simple and effective subword regularization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1882–1892, Online. Association for Computational Linguistics.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * (32) Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez-Sánchez.  Prompsit’s submission to wmt 2018 parallel corpus filtering shared task.  In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_ , Brussels, Belgium. Association for Computational Linguistics.
  * Sennrich and Volk (2011) Rico Sennrich and Martin Volk. 2011.  Iterative, mt-based sentence alignment of parallel texts.  In _NODALIDA_.
  * Shen et al. (2019) Jiajun Shen, Peng-Jen Chen, Matt Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2019.  The source-target domain mismatch problem in machine translation.  _CoRR_ , abs/1909.13151.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: masked sequence to sequence pre-training for language generation.  _CoRR_ , abs/1905.02450.
  * Thu et al. (2016) Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016.  Introducing the Asian language treebank (ALT).  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1574–1578, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Wołk and Marasek (2014) Krzysztof Wołk and Krzysztof Marasek. 2014.  Building subject-aligned comparable corpora and mining it for truly parallel sentence pairs.  _Procedia Technology_ , 18:126–132.
  * Xu et al. (2019) Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019.  _Understanding and Improving Layer Normalization_. Curran Associates Inc., Red Hook, NY, USA.

##  Appendix A Dataset Statistics

The data distribution of our MTet dataset is described in Figure 2.

Figure 2: Training data distribution across multiple domains

Data:

($l_{e}$, $l_{v}$) : a weakly-aligned pair of documents.

$l_{e}=$ ordered list of $N$ English sentences.

$l_{v}=$ ordered list of $M$ Vietnamese sentences.

$t_{src\rightarrow dst}:$ translation model from $src$ to $dst$.

Result:

$p=$ ordered list of aligned text pairs $(e\in l_{e},v\in l_{v})$

that maximizes $\sum_{(e,v)\in p}\texttt{s}(e,v)$, where

$\texttt{s}(e,v)=\texttt{BLEU}(e,t_{vi\rightarrow
en}(v))+\texttt{BLEU}(t_{en\rightarrow vi}(e),v)$

Initialize table dp[0 .. $M$, 0 .. $N$] with $0$s;

for  _$m=1\rightarrow M$_ do

for  _$n=1\rightarrow N$_ do

dp[$m$, $n$] = max(

dp[$m-1$, $n$]

dp[$m$, $n-1$]

dp[$m-1$, $n-1$] + s($l_{e}[m]$, $l_{v}[n]$)

);

end for

end for

$m=M$;

$n=N$;

$p=[]$;

while  _$m >1,n>1$_ do

if  _case 1_ then

$m$ = $m-1$

else if  _case 2_ then

$n$ = $n-1$

else

add pair ($l_{e}[m],l_{v}[n]$) to $p$;

$m$ = $m-1$;

$n$ = $n-1$;

end while

return _p_ ;

Algorithm 1 Alignment algorithm for weakly-aligned pairs of documents. The
algorithm strips away a portion of sentences in each document and matches the
remaining sentences into pairs, aiming to maximize the total BLEU score with
respect to a given translation model.

##  Appendix B Data collection time

We record human time as the time spent developing different code bases for
crawlers, inspecting manually, cleaning of different data sources, aggregating
website sources, and converting files to appropriate text format. Machine time
is execution time for long-running jobs such as crawling and rendering
millions of websites, batch downloading files, preprocessing large volumes of
texts, running inference for millions of sentences on Transformer models, and
computing BLEU scores between billions of pairs of sentences. The recorded
time is shown in Figure 3

Figure 3: Time required to 4.2M bitexts, color-coded for four tiers of data
sources (1) combine existing open-sourced corpora, (2) score and filter noisy
sources, (3) DP alignment from weakly-aligned documents, and (4) manual crawl
and clean. With comparable outputs, the time invested is vastly different
between them. The most expensive approach is manual crawl and clean, while the
most scalable is DP alignment.

##  Appendix C Quality of existing BiText Mining Datasets

MultiCCAligned El-Kishky et al. (2020) massively crawled the Web and aligned
bilingual texts using the auto-metric of embedding-based document similarity.
This results in 9.3M English-Vietnamese text pairs - the largest collection
available to the public at the moment333The MultiCCAligned paper reported
12.4M pairs, we detected and removed duplicates, which accounted for nearly
one quarter of their released data.. However, auto-metric-based alignment
produces data of lower quality than our carefully hand-curated collection,
many pairs in MultiCCAligned are themselves low-quality machine translated.
Training on MultiCCAligned, therefore, gives a much lower BLEU score, while
incorporating MultiCCAligned into our own data slightly decreases our result.

Figure 4: Performance comparison between parallel datasets

##  Appendix D Data sources for Manual Crawl and Clean

Medical

  * •

https://yhoctphcm.ump.edu.vn

  * •

http://jmp.huemed-univ.edu.vn

  * •

http://tonghoiyhoc.vn

  * •

http://hoinhikhoavn.com

  * •

http://hoiyhoctphcm.org.vn

  * •

https://jns.vn

  * •

https://jprp.vn

  * •

http://hocvienquany.edu.vn

  * •

https://sinhlyhoc.com.vn

  * •

https://tapchinghiencuuyhoc.vn

  * •

http://tapchi.vienbongquocgia.vn

  * •

http://vienduoclieu.org.vn

  * •

https://vjpm.vn/index.php

  * •

http://vjfc.nifc.gov.vn

  * •

https://vjs.ac.vn

  * •

http://vutm.edu.vn

  * •

https://jcmhch.com

  * •

http://www.yhth.vn

  * •

https://sj.ctu.edu.vn

  * •

https://radiology.com.vn

  * •

https://vjol.info.vn

  * •

http://www.vjph.vn

Others websites

  * •

https://vietanhsongngu.com

  * •

https://baosongngu.com

  * •

https://sachsongngu.top

  * •

https://tvpl.vn

  * •

http://vbpl.vn

  * •

http://automation.net

  * •

http://tapchixaydungbxd.vn

  * •

https://duytan.edu.vn

  * •

https://tapchikhcn.haui.edu.vn

  * •

https://tapchivatuyentap.tlu.edu.vn

  * •

http://tapchimoitruong.vn

  * •

https://translations.launchpad.net

  * •

https://translationproject.org

  * •

https://issuu.com/

  * •

https://lyricstranslate.com

  * •

https://www.wikihow.com

  * •

https://d2l.aivivn.com

Youtube Channels

  * •

Khan Academy

  * •

Ted Ed

  * •

Asap Science

  * •

Crash courses

  * •

GCP Grey

  * •

Veritasium

  * •

Vsauce