[1]Equal contribution. Correspondence to .
[2]Work done while interning on the Translate team at Google.

Neural machine translation (NMT) has progressed rapidly over the past several years, and modern models are able to achieve relatively high quality using only monolingual text data, an approach dubbed Unsupervised Machine Translation (UNMT). However, these models still struggle in a variety of ways, including aspects of translation that for a human are the easiest—for instance, correctly translating common nouns. This work explores a cheap and abundant resource to combat this problem: bilingual lexica (BiLexs). We test the efficacy of bilingual lexica in a real-world set-up, on 200-language translation models trained on web-crawled text. We present several findings:  (1) using lexical data augmentation, we demonstrate sizable performance gains for unsupervised translation; (2) we compare several families of data augmentation, demonstrating that they yield similar improvements, and can be combined for even greater improvements; (3) we demonstrate the importance of carefully curated lexica over larger, noisier ones, especially with larger models; and (4) we compare the efficacy of multilingual lexicon data versus human-translated parallel data. Finally, we open-source [<https://github.com/google-research/url-nlp/tree/main/gatitos>], a new multilingual lexicon for 26 low-resource languages, which had the highest performance among lexica in our experiments.

§ INTRODUCTION

Neural machine translation (NMT) has emerged as the dominant way of training machine translation models <cit.>, where translation is modeled as a sequence-to-sequence task to be learned by neural networks <cit.>. Massively multilingual machine translation (MMMT) refers to the concept of training a single machine translation model on many languages and language pairs using a shared set of parameters, and has also seen success in recent years <cit.>. Training these models typically relies on large-scale parallel corpora mined from the web <cit.>.

However, beyond the traditional technique of training NMT models with human-translated parallel texts, a number of other strategies have shown success recently, especially on lower-resource languages. One of these techniques is self-supervised training using monolingual corpora <cit.>. With this approach, NMT models are pretrained or jointly trained on a self-supervised task with monolingual data, such as the MASS <cit.> or BART <cit.> tasks, as well as the usual neural machine translation task. This training regime can aid the model in performing zero-shot translation <cit.>, in cases where a language has monolingual data but no parallel data. Moreover, both the self-supervised task and the supervised MT task can be modeled as neural sequence-to-sequence (Seq2Seq) problems, meaning a single Seq2Seq model can be used for training on both tasks.

Another technique that has shown success in training MMMT models is bitext mining <cit.>, in which parallel or nearly-parallel sentence pairs are mined from sets of monolingual corpora using a cross-lingual sentence encoder that has been trained to generate language-agnostic embeddings of sentences in different languages. This approach has proven useful in expanding MT models to hundreds more languages and thousands of language pairs <cit.>.

Other techniques that have proven useful for low-resource MT include back-translation <cit.> and the incorporation of language models into MT training <cit.>. There has also been extensive work on training completely unsupervised MT systems using monolingual corpora only <cit.>. For example, <cit.> uses a combination of denoising autoencoding with pretrained cross-lingual embeddings and on-the-fly back-translation to achieve reasonable MT performance with zero parallel data.

In our work, we supplement the approach that combines supervised and self-supervised training with multilingual lexica. The motivation for using this resource is as follows. Despite the successes of the approach combining supervised and self-supervised training, cross-lingual vocabulary alignment is still highly imperfect in these models, especially for low-resource and unsupervised languages (see <cit.> for examples of some common failure modes). That is, training on all languages using a shared set of parameters is insufficient to induce perfect cross-lingual vocabulary alignment. And although multilingual lexica generally do not contain sentence-level parallel information, they can greatly extend cross-lingual vocabulary coverage at the word or phrase level by providing examples of word or phrase translations. (Of course, we are not the first to experiment with using multilingual lexica to improve NMT performance, or multilingual NLP applications more generally; Section <ref> gives more details.)

In our approach, we experiment with augmenting monolingual and parallel data with translations from multilingual lexica. Using the publicly available massively multilingual lexicon Panlex <cit.>, we demonstrate that this added lexical data leads to small but significant gains over a baseline model on average, even for high-resource languages; and with smaller but carefully curated bilingual lexica, the gains are substantially larger. In both cases, the gains are most significant for unsupervised and low-resource languages. Our contributions are as follows:

* We provide a thorough comparison of several lexicon-based data augmentation variants for MT, all of which are simple, generalizable, and easy to implement;

* We test these approaches “in the wild”, i.e. on in a highly multilingual, web-mined data regime such as production systems tend to use, with hundreds of languages and billions of monolingual and parallel sentences;

* We explore the effects of lexical data quality and quantity;

* We demonstrate the efficacy of bilingual lexicon-based approaches as models scale in size;

* We open-source the high-quality multilingual  lexicon for low-resource languages.

The tl;dr of this paper is that bilingual lexica help low-resource and zero-shot NMT in almost all cases, and that most training-time augmentation methods have similar efficacy, and can be combined to be more effective. When scaling up to larger and more expressive models, these methods retain their efficacy, but the quality of the translated bilingual lexica becomes more important than the sheer quantity of lexical data points used for data augmentation. For instance, small, high-quality lexica like  show about 5x larger  improvement than larger, noisier lexica like Panlex.

§ RELATED WORK

A number of works have looked at using multilingual lexicon data augmentation for NMT and other NLP tasks. The first class of augmentations that we experiment with is “codeswitching,” where words in the source sentence are swapped out for their dictionary translations to create mixed-language sentences. This approach has been used for a range of multilingual NLP tasks, including MT <cit.>. Many of these, however, only look at codeswitching between the source and target languages, e.g. substituting source words with dictionary translations into the target language, or word-for-word BiLex translations of the target to make synthetic back-translated data <cit.>. <cit.> experiment with codeswitching on NLI, sentiment classification, document classification, dialogue state tracking, and spoken language understanding, <cit.> looks at codeswitching embeddings for language modeling, and <cit.> experiment on NER, POS tagging, and dependency parsing. Another similar work is <cit.>, in which the MLM task is modified such that instead of predicting masked source tokens in the source language, the authors provided language embeddings to cue the model to predict the masked tokens in a different language instead. Codeswitching augmentations go by a variety of different names, e.g. “dictionary denoising” <cit.>, “Random Aligned Substitution” <cit.>, or “code-mixing”. In our paper, we will stick to the term “codeswitching,” though we will try to point out where an identical or similar approach has been tried under a different name.

The second class of augmentations we experiment with involves prepending lexicon translations to source sentences as additional cross-lingual signal, as instead of swapping out words in the source sentence. This approach has been tried as well for enhancing MT performance, e.g. in <cit.>, and for similar tasks like language modeling <cit.>. One potential advantage this approach has over the codeswitching method is that it can be applied at inference time as well: multilingual lexicon entries can be prepended to sentence queries to steer the model toward more accurate word translations. Outside of NMT models, this lexical prompting approach has also been applied to translation with LLMs: <cit.> provide LLMs with dictionary translations of some of the source sentence words, which the model can use to cover gaps in its vocabulary coverage (although the authors do not experiment with truly low-resource languages). With the rise in popularity of LLMs for MT and other tasks, this is an exciting area for further research.

§ THE PROBLEM WITH UNMT

Various automatic metrics like  and  have demonstrated that Unsupervised Machine Translation can produce results of relatively high quality. However, such metrics hide the fact that UNMT approaches tend to make very specific and unusual mistakes. <cit.> demonstrate that unsupervised models are remarkably good at aspects of translation like fluency, but frequently make mistakes by confusing distributionally similar words. When training on parallel data, it is straightforward enough to learn a mapping from a token in one language to another; when training on monolingual data only, this mapping can only be inferred from considerable amounts of context, necessitating a very fine-grained representation of its meaning, and consequently large amounts of monolingual data. This issue is especially noticeable with nouns that fall into the same semantic category, like animal names. For instance, <cit.> demonstrate how their models, despite getting good  scores, translate `lion' as `hyena', `snake', `rabbit', and `seizures', depending on the language.

The interesting thing about these errors is that in some sense they are the easiest to fix. Any student of a language with a pocket dictionary handy could correct many of these mistakes.
And even a traditional machine translation system – especially a phrase-based one – would rarely if ever mistranslate “cat” as “dog”. This error mode is the primary motivation for incorporating bilingual lexica into translation systems. For this reason, this paper focuses on unsupervised and slightly supervised (low-resource) languages.

§ TRAINING DATA

Our models were trained entirely on web-mined, sentence-level text data.

§.§ Monolingual data

The models used in this paper build off the existing data used by <cit.>, which in turn uses the specialized low-resource data crawling techniques described in <cit.>. However, to make rapid training and development possible, we subsampled the monolingual data for the 100 highest-resource languages to 10% of its original size. In sum, this totaled about 4B sentences, or about 80B tokens. For the large model experiments in section <ref> of the Appendix, we used the full (not subsampled) data, totalling 27B sentences (540B tokens). All 208 languages in our experiments had monolingual data.

§.§ Parallel data

The parallel data is also the same as that from <cit.>, which itself is a slightly extended version of the corpus described in <cit.>. The present paper extends it slightly further with the addition of small amounts of parallel data for some lower-resource languages (on the order of 1,000 - 5,000 website-template tokens). The parallel data tend to be much noisier than the monolingual data, and misalignments, boilerplate, and nonlinguistic content are common. All parallel data are sampled to 10% of their original size, resulting in 9B parallel sentences (162M tokens) into English and the same number out of English, as well as 700K non-English-centric sentence-pairs. The large models in Section <ref> use the full dataset, which is ten times larger than this.

§.§ Multilingual lexica

The  dataset is a new dataset open-sourced in this paper. It consists of 4000 short English segments translated into a 26 very low-resource languages. The English source text is a mixture of words from a variety of sources, including frequent tokens in the English language, words for numbers, months, days of the week, Swadesh words, names of the languages themselves (including the endonym), and a few short sentences. The tokens were manually reviewed by the authors of this paper to ensure they looked reasonable[Exactly two tokens snuck into the dataset that are definitively odd, namely `simp' and `sh']. As the name implies, this dataset is mostly very short, consisting of 93% single tokens. There are also some short sentences, though only 23 entries have over 5 tokens. We hope this dataset will complement existing publicly available multilingual lexicons like MUSE <cit.>

§.§ Panlex

Panlex <cit.> is a free, open-access massive online database consisting of word and phrase translations for $5000+$ languages, sourced from $2500$ individual dictionaries. Panlex contains $\approx 1.3$B translations across all language pairs. For our experiments, we use a subset of the Panlex database covering 177 languages and containing $66$M word pairs. Languages were chosen largely by the availability of eval sets; details given in Appendix <ref>.

§ EVALUATION

We use two translation evaluation sets:  <cit.>, an open-sourced evaluation set consisting of 2009 English Wikipedia sentences translated by humans into 200 languages, and  (Google AuTOmatic NTL Eval Set), an in-house evaluation set of 1,200 English sentences translated into various languages, for comparison with <cit.>. We use the SacreBLEU <cit.> implementation of  [signature ] for our evaluation metric. Higher-quality, embedding-based metrics like Bleurt <cit.> are not available for these languages, and token-based metrics like  are especially questionable on highly-inflecting languages, as explored in <cit.> Furthermore,  has generally been found to correlate better with human judgement than , even for high-resource languages <cit.>

For this study, we only evaluate on English-centric directions. The reason for this is that, although both evaluation sets are multi-way parallel, they are both also English-original. Therefore, using either of them for non-English-centric evaluation is not especially meaningful – and specifically, such an evaluation set cannot measure any improvement that a direct model would have over a pivoting approach (first translating to English, and then to the target). For example, any sentence with “you” in the English side erases the formality and gender distinctions that either of the other languages may have, so if a direct model is able to correctly preserve formality, the eval set can't measure this.

Furthermore, this study places more weight on the  direction than the  direction. The main reason for this is that the  direction is generally an easier direction for models to learn (since they see so much more English text), so the  direction is usually the limiting reagent when it comes to model quality; as a result we care more about improving this direction. Similarly, the smaller models trained in this study will lag larger models much more on the  direction, so our results in this direction are not as meaningful.

§ MODEL

For our experiments we use a Transformer Big encoder-decoder model <cit.> with approximately $475$M parameters. We train each model for 400K steps on 64 TPU v2 chips. Our models assign a 40% weight to the translation task, and a 60% weight to the MASS task. For models augmented with a monolingual data augmentation, we split the 60% weight on the MASS task into a 30% weight on the augmented data and a 30% weight on the non-augmented data. Parallel data augmentations were done in an analogous way.  For the raw-token-pair augmentation, we add in this task with a 5% weight and shrink the other weights accordingly. We use a task-specific token for each of the six tasks the models may see, namely translation, MASS, GlowupMono, GlowupParallel, CodeswitchMono, and CodeswitchParallel.

§ METHODS

In this paper, we divide our augmentation approaches into two classes: “codeswitching” approaches, which involve substituting source sentence words for their dictionary translations, and “,” (Guiding Lexical Output With Understandable Prompts) which entails prepending dictionary translations of source words to source sentences. The main difference between these approaches is whether dictionary translations are substituted for source text (in the case of codeswitching) or added to the sentence (in the case of ). As a third augmentation, we experiment with training on raw lexicon token pairs directly, treating them like any other parallel data.

The novelty of our contribution lies not so much in any one of our methods, but rather in (1) the application of these methods to unsupervised Machine Translation; (2) the number of methods we apply in controlled experiments to discover the best augmentation, (3) the scale of our experiments, in terms of number of languages, data quantity, and model capacity; and (4) the application of these methods to “in the wild” web-crawled data. While a variety of papers (e.g. <cit.>) have explored specific augmentations on particular language pairs, we believe our paper is the first to undertake a rigorous comparison of different augmentation strategies across hundreds of languages in a real-world setting.

§.§ Codeswitching

In our “codeswitching” augmentation strategy, words in the source sentence are swapped out for their dictionary translations to create mixed-language sentences. We experiment with this augmentation on both monolingual and parallel data. The details of this method are described below.

§.§.§ Multilingual codeswitching autoencoding (MCA)

Our multilingual codeswitching autoencoding (MCA) approach is similar to the “dictionary denoising” objective in <cit.>. Let $D$ represent a multilingual lexicon containing word or phrase translation pairs for many languages. Given a source sentence $x = (x_{1}, x_{2}, . . ., x_{n})$ from monolingual corpus $X_{mono}$, we substitute each token in $x$ for its dictionary translation with probability $p_{tr} = 0.4$.  (More implementation details can be seen in Appendix Section <ref>)

Note that <cit.> also apply additional noise to $x$ on top of codeswitching, along the lines of (m)BART <cit.>. For simplicity and so we can better examine the effects of lexicon information in isolation, we do not do this. Furthermore, while MCA indeed creates “noisy” codeswitched sentences, we note that word translations provide a meaningful cross-lingual signal for the model in a way that other noising functions (e.g. deletion, random word substitution) do not, as these augmentations corrupt the source sentence without adding additional useful information. Although models have been shown to learn from these denoising tasks, we use only lexical augmentation because it more directly furthers our ends of improving cross-lingual vocabulary alignment across many languages.

It should also be noted that because we are choosing dictionary translations at random without attempting to do any sort of word sense disambiguation beforehand, some of the translations will actually express the wrong sense of the source word (e.g. translating the verb “fool” as “sot” in French, even though the correct translation should be “tromper”). We do not try to avoid this phenomenon because we believe this is an acceptable type of source-side noise, and may actually help our models learn to perform word sense disambiguation implicitly.

§.§.§ Codeswitching MT

Our codeswitching MT task is essentially the same approach as described in Section <ref>, except it applies to parallel rather than monolingual data. Given a source sentence $x$ from parallel corpus $X_{parallel}$, we perform the identical procedure described in section Section <ref> to obtain multilingual codeswitched sentence $x'$. We then train the model on the translation task using sentence pairs $(x', y)$, where $(x, y)$ is a sentence pair in $X_{parallel}$. This method is effectively identical to the Random Aligned Substitution method proposed in <cit.>. As with MCA, we use $p_{tr} = 0.4$ and apply the augmentation on half the available parallel data.

Examples of the monolingual (top) and parallel (bottom) codeswitching augmentation strategies. In both cases, random tokens in the source sentence are replaced with their translations in a random language from a multilingual lexicon. Color coding is used to indicate which source words have been swapped for their dictionary translation. The different colors are used simply to point out the fact that the words are from codeswitched words in each sentence come from different languages.

§.§ Lexical prompting (GLOWUP)

The second class of lexical augmentations we experiment with is lexical prompting, which we call , short for Guiding Lexical Outputs With Understandable Prompts. The difference between these approaches and the codeswitching approaches described in Section <ref> is that instead of substituting source sentence words for their dictionary translations, we prepend those translations to the source sentences. In particular, we add $(src, transl)$ pairs to the beginning of source sentence, for some random fraction of the translatable words in that sentence. These hints can then be used to help the model guess the translation or the denoised sentence, depending on the flavor, as described below.

The  task has the advantage that it can be used at inference time, can be used without retraining a model, and may be simpler to implement. However, it does result in longer and less balanced sequence lengths, which can pose problems for decoding.

§.§.§ MASS with monolingual GLOWUP

Our  augmentation on monolingual data involves prepending lexical prompts to source sentences and then applying MASS on top of those sentences, after which the model attempts to reconstruct the original source sentence (including the prompts). Given multilingual lexicon $D$ and source sentence $x$, we sample some number of translatable tokens in $x$ as in the codeswitching augmentation. However, instead of substituting these tokens for their translations, we instead prepend these tokens and their translations to $x$. These $\{src, tgt\}$ translation pairs then form a “ prompt” that is separated from the source sentence by indicator tokens. Unlike the codeswitching augmentation, we sample from a uniform distribution over $[0, k]$, where $k$ is the number of translatable tokens in $x$, when deciding how many tokens to translate. The reasons for this are described in Section <ref>.

Finally, we apply MASS to mask random subsequences of $x'$, possibly including the  prompt itself. We then train the model to reconstruct $x$. As with MCA, we apply the augmentation on half the monolingual data and train the rest on vanilla MASS.

§.§.§ -MT

Our  augmentation on parallel data, which we call -MT, is effectively the same as the monolingual variant of the task, but without the MASS element. For a given sentence pair $(x, y)$ in the training corpus, the prompting is performed on the source sentence $x$, essentially to give it hints about how to produce $y$. The model is then trained on the translation task using $(x', y)$, with the task token  instead of

One notable advantage that the -MT augmentation has over the codeswitching MT augmentation is that the  variant may be applied at inference time as well. That is, given an unseen sentence, we can append source words and their translations by using a multilingual lexicon at inference time. For this reason, we opt to add word translations in the target language only, unlike the other augmentations discussed where we have added translations in multiple, random languages per sentence.

Examples of the monolingual (top) and parallel (bottom)  augmentation strategies. In both cases, random tokens in the source sentence are prepended to the source sentence, along with their translations in a random language from a multilingual lexicon. As in Figure <ref>, differential color coding is done to draw attention to dictionary translations in different languages.

§ EXPERIMENTS

§.§ Training regimes

In our experiments, we train models with various combinations of the augmentations outlined above, as well as a baseline (with no data augmentation of any kind) and a model where we simply provide word pairs from the lexicon as additional parallel data. The details of our training regimes are discussed below.

§.§.§ Baseline

We first train a baseline model with no data augmentation using the monolingual and parallel data described in Sections <ref> and <ref>, respectively. This model is essentially a smaller version of the model trained in <cit.>. Using the model setup described in Section <ref>, we train the model on the MT task using all available parallel data and on the MASS task using all available monolingual data. Note that because this model is smaller and uses substantially less data than the one in <cit.>, the performance is predictably not as high. We use this smaller model for the purposes of running experiments more quickly and efficiently, although we understand the relative gains of our approaches may not be identical with those of a larger model. For the same reasons, we also do not perform back-translation as done in <cit.>. Larger models, which have $1.6$B parameters and are trained on substantially more parallel and monolingual data, are explored in Section <ref>.

§.§.§ Token-pair-only model (GatiPanlexTokenPairs)

In addition to the baseline model, we also experiment with the extremely simple approach of providing raw word pairs from multilingual lexica to the model as additional parallel data. That is, given a dictionary entry $s$ and its translation $t$, we provide the model with a “sentence” pair of the form $($ $s, t)$. We use all $66$M token pairs from the Panlex subset described in Section <ref>, which covers $177$ of the $208$ languages present in the monolingual and parallel data, as well as all 400K token pairs from the  dataset described in section <ref>, covering 26 languages. We call this token-pair baseline GatiPanlexTokenPairs.

§.§.§ Single augmentation models

We also train models on each of the augmentations described in Section <ref>. As noted in that section, we only augment half the relevant data (monolingual, parallel, or both) before training each of these models, leaving the other half to be trained identically to the baseline (i.e. joint training on the MT task and MASS). Our naming conventions for our models are as follows:

* CodeswitchMono is augmented with Monolingual Codeswitching (Section <ref>)

* CodeswitchParallel is augmented with Parallel Codeswitching (Section <ref>)

* GlowupMono is augmented with Monolingual  (Section <ref>)

* GlowupParallel is augmented with Parallel  (Section <ref>)

§.§.§ Dual augmentation models

We also train some models on a combination of data augmentation approaches to see whether a hybrid strategy may work better than any single augmentation. We train the following additional models:

* CodeswitchMonoParallel and GlowupMonoParallel are identical to the CodeswitchMono and GlowupMono models, but also add the corresponding parallel augmentations.

* CodeswitchMonoParallelGatiPanlex and GlowupMonoParallelGatiPanlex take these models and further add the raw token pair objective (see Section <ref>).

We leave experimentation with a hybrid codeswitch- approach (e.g. CodeswitchMonoGlowupMono) for future work.

§ RESULTS

We evaluate all our models on the  dataset <cit.>, which contains English-aligned parallel sentences for $200$ languages. In Appendix section <ref> we also report scores on the in-house  eval set, which is the eval set used by <cit.>.

In analyzing our results across languages, we use the following resourcedness classifications:

* High-Resource Languages (HRLs): $>2$B total training tokens in parallel data

* Medium-Resource Languages (MRLs): $360$M to $2$B training tokens in parallel data

* Low-Resource Languages (LRLs): $1$ to $360$M training tokens in parallel data

* Unsupervised Languages (URLs): no parallel data

The results relative to the baseline, in $\Delta$, are summarized in Tables <ref> () and <ref> (). A few trends jump out. Firstly, all models trained with only monolingual data augmentations see consistent performance gains over the baseline. Conversely, models with only parallel data augmentations show performance degradations. Predictably, models mixing monolingual and parallel data augmentations fare in-between those poles.

These general trends are the same between  and  directions, though the gains are generally lower in the  direction. As noted in Section <ref>, this is expected, and this direction is less of a priority for translation improvements. Results on the  eval set, in Appendix Section <ref>, show the same trends, though the performance gains tend to be larger for all augmentations, with CodeSwitchMonoGatiPanlex gaining $+2.3$  for URLs. Interestingly, though the Codeswitch augmentation seems to have higher overall performance on unsupervised languages in the  direction,  usually has the edge for higher-resource languages and the  direction.

Despite these trends seeming robust across models, the effect sizes are relatively small, maxing out at about $+1.5$ average $\Delta\chrf{}$ gain.  However, the picture changes dramatically when only looking at the subset of languages that has the higher-quality  training lexicons. For these 26 languages, every augmentation, even the parallel ones, have large performance gains. The winning augmentations for URLs remain CodeSwitchMonoGatiPanlex and GlowupMonoGatiPanlex, the former having an average gain of $+7.0 \chrf{}$ on and $+8.0 \chrf{}$ on , and the latter having $+5.6 \chrf{}$ on  and $+9.5 \chrf{}$ on .

Finally, although it was not the first place model in any category, the GatiPanlexTokenPairs has large gains in all directions over baseline, and only falls short of the more complex augmentations by a small margin. Furthermore, when used in conjunction with either Glowup or Codeswitch, it further improves performance across all categories. This may be one of the most useful long-term findings of this study: raw token pairs perform roughly on par with all the other fancy augmentations!

§.§ Scaling up: bigger models, more data

Sometimes, results on smaller models do not transfer to larger models. To this end, we train larger transformer models with $1.6$B parameters using $10\times$ the parallel data and $10\times$ the high-resource monolingual data that was used for training the smaller models.

We trained three large models: a baseline, a token-pair model, and a token-pairs + CodeswitchMono model. The  scores for these models can be seen in Table <ref>. There are a number of obvious differences between these results and the results on the smaller ($475$M parameter) models trained with $\approx \frac{1}{10}$th the data. First, the positive impact of the data augmentations is smaller in all categories, indicating that the gains previously seen from augmentations are partially washed out in the larger-data, larger model regime. On  , both models are very close to baseline—within the realm of noise. On  , they see consistent small gains of around $+0.5$ , much smaller than previously. For both eval sets in the  direction, there are consistent small losses.

Is this the Bitter Lesson <cit.> getting us again? Perhaps—but the picture is less bleak than it first appears. When we look at the subset of the languages where we have a higher assurance that the bilingual lexica are higher-quality—namely, those that use  bilingual lexica—we still see consistent wins, even in the  direction. For these 26 languages, all models see consistent gains, and as before, the gains are bigger on unsupervised languages, and adding in the Codeswitch augmentation gives a consistent leg up on the unsupervised languages.

Overall, the takeaway from these experiments is that one has to ensure that the data is of high quality when applying lexical data augmentation at such a large scale. While we saw substantial improvements for many languages, these were balanced out by losses for other languages (especially those with only Panlex, but not , data).

§.§ Glowup Decoding

In principle, one of the advantages of the GlowupParallel approach is that lexica can be used at inference time. Therefore, we experimented with decoding the eval sets not with the translation task ID, but with the Glowup task ID, along with the relevant lookups from the lexica. Unfortunately, these decodes failed impressively, with performance degrading the more prompts that were included. Model decodes often had long sequences of control tokens. Further work should not disregard this direction; indeed, a variant of this likely has particular promise in the world of foundation models. The current approach likely just needs some tweaks to eliminate this sort of out-of-domain decoding errors we were seeing, but we leave an investigation of this hypothesis for future work.

§.§ Oracles: what happens if we use trusted parallel text?

Parallel text is much more costly to produce than bilingual lexica, but also contains many more useful signals, including examples of word usage in context. But how much more helpful is it, really, than bilingual lexica? The answer seems to be “much more helpful”.

To measure this, we trained a model with a variety of public parallel datasources for a few of our lowest-resource languages. The datasets we used were HornMT <cit.>, SALT (Sunbird AI Language Translation) dataset <cit.>, FFR: Fon-French Neural Machine Translation <cit.>, Tatoeba <cit.>, The Makerere MT Corpus: English to Luganda parallel corpus <cit.>, Commonvoice <cit.>, Kencorpus: Kenyan Languages Corpus <cit.>, Chuvash-Russian parallel corpus<cit.>, Abkhaz Corpus <cit.>, Bashkir Corpus<cit.>, Sprotin Faroese Corpus<cit.>, Jojajovai Guarani-Spanish Parallel Corpus <cit.>, and the NLLB Seed data <cit.>. We only used datasets with a license permitting commercial use, and therefore avoided other excellent datasets such as AmericasNLP <cit.>. Note that these are high-quality, trusted datasets, prepared by community members – a very different resource than the web-mined parallel data that the model is otherwise trained on.

To keep the experiment more controlled, we compared performance only on languages that had token-pair data from the  datasource, since this empirically seems to be of higher quality. Note that, although these languages are technically not zero-resource (unsupervised) in the baseline, they only had a few thousand short parallel sentences from a translation memory, so the findings here are similar to as if they had been on true 0-resource languages.

The results are in Table <ref>. We compare four models: the standard baseline and GatiPanlexTokenPairs model, as well as the “Parallel” model (which adds the external parallel translation task with a 5% weight) and the “Parallel + GatiPanlexTokenPairs” model, which uses both the token-pairs and the parallel data, with a combined weight of 5%. On both eval sets, we see that using Bilex  yields a gain of around +3.5 , but using true parallel yields a much larger gain of about +10 . Using the bilingual lexica on top of the parallel data yields a further gain of about +0.5 , demonstrating that, though many of the gains have been washed out by the true parallel data, there are still modest gains to be had from bilex training.

Between these two resources, trusted parallel data is clearly more effective. However, the number of tokens (counting only the target language) is also much greater for the parallel data. On the extreme end, Tsonga (`ts') has 300 times more tokens in the parallel data than in the bilingual lexicon. In this light, it is impressive that bilex alone adds around +4 , and the parallel data only adds another +7 on top of that. Future work should look at quality gains conditioned on the number of tokens between parallel data and bilingual lexica.

§.§ How many token pairs do I really need?

We examine the relationship between the number of lexical token pairs provided during training and MT performance (). First, we perform regressions using $\Delta\chrf{}$ over baseline as the outcome variable and three predictor variables: (1) number of Panlex entries, (2) number of  entries, and (3) number of monolingual sentences. We include monolingual sentences in the regression to control for it as a confound. To eliminate parallel data quantity as a confound, we only perform this analysis on URLs.

The regression results are given in Table <ref>. As expected, both the number of Panlex word pairs for a given language (No. Panlex) and the number of  word pairs (No. ) have a positive $\beta$ coefficient. However, note that the $\beta$ for No.  is $\approx 3\times$ as large as the $\beta$ for No. Panlex on , and $\approx 39\times$ as large for . This aligns with our observation that the  dictionaries are more efficient for improving MT than Panlex, probably due to their higher quality.

The most practical question we can seek to answer is, If I can spend $X on translating tokens, how much quality increase can I expect? To investigate this “bang for buck” question in a more controlled way, we observe the effects of the  dataset in isolation, without Panlex. We train an “GatiTokenPairs” model, which is identical to the “GatiPanlexTokenPairs” model, except the token-pair task has only  data. Thus, this tells us specifically what gains we can expect if we are to get 4,500 tokens' worth of a bilingual lexicon per language.

The results are in Table <ref>, reported in delta versus the baseline model for both  and the  eval set in the  direction. The improvement for unsupervised languages is around +5.0  for both eval sets; the improvement for languages with some parallel data is less but still noticeable, hovering around +2 . The largest improvement is in Goan Konkani (+11.0 ), with Mizo, Ilocano, and Bambara close on its heels with gains of around +8 . Only Maithili, which has interesting properties a a close dialect of Hindustani, sees a loss on both eval sets. The gains are not obviously related to the total number of tokens per language.

As an aside, it is heartening that  and  seem to agree very nicely, despite their different domains (Wikipedia versus web + question-answers).

§ CONCLUSIONS

In this paper we explore the ways that that augmenting training data with bilingual lexicon information can improve the performance of machine translation models on low-resource and unsupervised languages, and open-source the  dataset, which leads to average gains of about +8  alone on unsupervised languages. We perform extensive experimentation with three main types of lexical augmentation: codeswitching, lexical prompting, and raw token-pair training. The results show that applying any of these augmentations to monolingual data yields substantial improvements, and that they can be combined for greater effect. The leader (by a small margin) is the combination of CodeswitchMono and raw token-pair training. These results hold when scaling up model and data size, but in the settings with more data and larger models, the quality of the bilingual lexica plays a relatively bigger role, and augmentation with the noisier Panlex begins to lag in quality behind the much smaller, yet higher-quality,  dataset.

Future work will likely want to focus on prompting foundation models with bilingual lexica. Large Language Models show promise on machine translation for high-resource languages <cit.>, but their capabilities on low-resource languages have yet to be thoroughly explored. Additionally, a more thorough investigation of the trade-off between cost and quality for tiny datasets can be explored: if one only has X dollars, or X dedicated volunteers with k total hours, should they spend their time translating text, making monolingual text, or making bilingual lexica?

§ ACKNOWLEDGEMENTS

We would like to thank Aditi Chaudhary, Ankur Bapna, Daan van Esch, and Machel Reid for their comments and suggestions on this project.

§ RESULTS ON

§.§   Eval set

The main paper reports the scores on the more widely-used  eval set; this section reports on the other dataset that we evaluate our models on. This is an in-house test set for the $1000$ Languages Initiative, which we call  (Google AuTOmatic NTL Eval Set). The dataset contains $63$ languages, most of which are unsupervised or low-resource (although there are a small number of MRLs).

§.§.§ Summary

The evaluation results on  for the  direction are summarized in Table <ref>. The trends are mostly the same as what we saw for the   evaluation. Once again, the models trained with augmentation on monolingual data but not parallel data generally yielded improvements (the sole exception being GlowupMono, which decreased performance on URLs by $-0.2$  relative to the baseline), and the parallel data augmentations didn't do as well. CodeswitchMonoGatiPanlex emerges, even more definitively than on , as the best model, winning all categories except MRLs (on which CodeswitchMono performs best).

However, there are some minor differences between this evaluation and the  one when we look more closely. First, although CodeswitchParallel again does quite poorly, GlowupParallel doesn't do too badly. Still, it is far from being on par with CodeswitchMonoGatiPanlex or even just GatiPanlexTokenPairs. GlowupMonoParallel doesn't do too poorly either, actually beating out GlowupMono on URLs by $1.3$ , but it doesn't help nearly as much as CodeswitchMonoGatiPanlex (or, for that matter, the two augmentations from which this hybrid model is trained). Overall, this part of the evaluation still provides a compelling case for CodeswitchMonoGatiPanlex as the single most effective augmentation method for URLs and LRLs.

The evaluation results for the  direction are given in Table <ref>. The single most important takeaway from this part of the analysis is the same as it was for the  evaluation: the plain GatiPanlexTokenPairs model helps URLs the most in this direction, with a $\Delta \chrf{}$ of $+1.1$ over the baseline. Yet again, the improvements are smaller in this direction than for . The only other thing that stands out about this part of the evaluation is that the GlowupMono augmentation doesn't seem to be as helpful according to this test set as for the  set. Although GlowupMonoParallel and GlowupMonoGatiPanlex do reasonably well, their improvements are significantly smaller than the improvement from using GatiPanlexTokenPairs alone, and the GlowupMono augmentation by itself actually results in losses on URLs. So taking the  and  results together, it seems that adding raw token pairs as additional parallel data is the best way, out of the techniques we tried, to improve performance in the  direction for very low-resource languages.

§  EFFECTS ON DISTRIBUTIONALLY SIMILAR NOUN MISTRANSLATION

Part of the motivation for using bilingual lexicons for unsupervised translation was to see whether we could repair the common error mode of mis-translating distributionally similar nouns. <cit.> note that this error mode is particularly common for two categories of nouns: animals and colors.

To measure improvement on this phenomenon, we define the token hit-rate as the following: for some set of desired tokens $D$, let $R_D$ be the subset of the eval set such that each reference contains at least one token in $D$. The hit-rate is then the percentage of times in $R_D$ that the model correctly generated one of the desired tokens in $D$. For instance, if the desired tokens are “kitten” and “puma”,  $R_D$ is the set of references containing one of these words, e.g. “The kitten lies” and “A Puma eats hot chip”. If the model produces “kitten lie on floor” and “Crocodile charge they phone” from the corresponding sources, it has a hit-rate of 50%, since it correctly got “kitten”, but not “puma”.

Table <ref> looks at the token hit-rate for the models BaselineBig and CodeswitchMonoGatiPanlexBig, for the categories of animals occurring in  (bear, bee, bird, butterfly, cat, chicken, deer, dog, elephant, fish, frog, goat, horse, insect, lion, monkey, parrot, pig, rabbit, sheep, snail, snake, tiger, turkey, turtle), animals NOT appearing in  (ant, antelope, buffalo, cheetah, crocodile, dolphin, dormouse, gorilla, jellyfish, koala, leopard, moose, mosquito, newt, ocelot, otter, reindeer, robin, scorpion, shark, sloth, spider, springbok, tortoise, velociraptor), colors (black, white, red, blue, yellow, green, purple, orange, grey), and numbers (one, two, three, four, five, six, seven, eight, nine, ten, hundred, million). All numbers and colors appear in . Numbers are included as a weak control, since the model will tend to make fewer mistakes on them, though such UNMT-style mistakes do occur.

As expected, the -augmented model performs better on these tokens. Two things are worth noting. First, the model improves noticeably on the complementary distribution—words that do not appear in the lexicon training data—but unsurprisingly improves more on the words that are present in . Second, the improvements are not as large as expected: why is it not now getting 100% accuracy? Digging into the outputs, it seems that this is mainly due to the high baseline of (a) undertranslation; (b) hallucination; and (c) copying, as we expect from a model trained without various other tricks like back-translation (see Section <ref> for an analysis of common error types). This point is underscored by the models' imperfect performance on the “easy” class of numbers.

Number of word pairs available in Panlex for each of $4750$ BCP-47 languages (log scale).

§ BIGGEST WINNERS

We also look at the top $5$ languages that were the biggest gainers over the baseline for each model. In some cases these may represent remarkable successes of a particular approach—though in other cases they may represent noisy outliers, as is to be expected when evaluating 200 languages.

The biggest winners for each model in the  direction for the  evaluation set are given in Table <ref>.

There are seven languages that gained at least $5$  over the baseline on at least one model trained with data augmentation. These languages are:

* Bhojpuri (bho): up to $+14.5$

* Ilocano (ilo): up to $+9.1$

* Serbian (sr): up to $+8.3$

* Bambara (bm): up to $+8.1$

* Tibetan (bo): up to $+8.0$

* Nuer (nus): up to $+6.8$

* Mizo (lus): up to $+6.2$

Unsurprisingly, most of these languages are unsupervised or low-resource, except for Serbian which is medium-resource in our dataset. Of the seven languages listed above, we use Panlex data for Ilocano, Serbian, Bambara, Tibetan, Nuer, and Mizo, and there is  data for Bhojpuri, Ilocano, Bambara, and Mizo. As will be discussed in Section <ref>, the  bilingual lexica are clearly a very useful resource for MT, although evidently Panlex alone can help as well. Another interesting finding is that Nuer, which has no English-aligned entries in Panlex but $\approx 20$K non-English-aligned entries, still sees large improvements when translating from English. This is evidence that lexicon data can improve performance even in the zero-shot case, where e.g. the model learns better vocabulary alignment between English and Nuer despite not receiving explicit alignment information during training. In Section <ref>, we look at the relationship between the number of lexical data points for a language and the  improvement, which provides some insight (albeit not perfect clarity) into why these particular languages did well.

Table <ref> shows the top $5$ biggest winners per model for the  direction. Clearly there is a lot of overlap with the  direction, although there are some differences. Also, note again that the magnitude of improvement in this direction is smaller, likely because the baseline performance is higher and there is less improvement to be made simply by better aligning vocabulary cross-linguistically. Some of the biggest winners in this direction that weren't already discussed for the  direction are:

* Tsonga (ts): up to $+3.1$

* Guarani (gn): up to $+2.8$

* Bashkir (ba): up to $+2.5$

* Minangkabau (min): up to $+2.5$

The biggest winners on  in the  direction are given in Table <ref>. Though there is some overlap with the biggest winners on the  dataset (e.g. Ilocano, Bambara, Mizo, Bhojpuri), a number of different languages perform well too, some of which simply aren't included in the  set. The languages which gain $> 5.0$  on this part of the evaluation are:

* Adyghe (ady): up to $+14.1$

* Kedah Malay (meo): up to $+12.6$

* Goan Konkani (gom): up to $+11.6$

* Bhojpuri (bho): up to $+10.4$

* Ilocano (ilo): up to $+9.8$

* Avar (av): up to $+9.5$

* Bambara (bm): up to $+9.0$

* Mizo (lus): up to $+8.9$

* Madurese (mad): up to $+6.6$

* Assamese (as): up to $+6.3$

* Pattani Malay (mfa): up to $+5.7$

* Kalaallisut (kl): up to $+5.4$

* Tibetan (bo): up to $+5.2$

The biggest winners in the  direction are given in Table <ref>. Many of the biggest winners overlap with the  direction, but some of the languages that haven't yet been mentioned are:

* Manipuri (mni{-Mtei}): up to $+7.7$

* Dogri (doi): up to $+5.4$

* Dhivehi (dv): up to $+3.4$

* Tigrinya (ti): up to $+3.0$

§.§ Big model winners

The biggest winners for the $1.6$B parameter models are given in Tables <ref>, <ref>, <ref>, and <ref>.

§.§ Empirical study of the quality of Panlex

One way to judge the quality of a dataset is to review it manually, as in <cit.>; another is to see the empirical effects on model quality of training on it. As a byproduct of using Panlex for this project, can we judge the quality of Panlex for different languages?

To reduce noise, we average the scores on the three main uses of the bilexes,  namely the TokenPairs model, the Glowup model, and the Codeswitch model. We average the  and  scores.  We then compare those scores to the baseline model for both  and . For the purposes of this analysis, we treat any absolute delta of under 0.3  to be noise.  The result is displayed in Tables <ref> and <ref>.

One would like to say that the upper left-hand corner represents languages with unequivocally high-quality lexical data, and the lower right-hand corner represents languages with poor quality lexical data. Alas, however, this picture is rather muddied when we scale up to larger models, as we see that many languages jump from one bucket to another. Nonetheless, we do see the trend that  languages tend to cluster to the upper left-hand corner in both cases, and that Shan (`shn`) and Latin (`la`) do poorly in all cases, and should likely be avoided by practitioners.

Teasing out the confound of the mixed  and Panlex data: For the 26  languages, it is harder to trust the previous analysis. However, we can compare the scores of these languages between the GatiPanlexTokenPairs model and the GatiTokenPairs model. The second of these models is trained on a strict subset of the data that the first is. If a language performs better with this subset of the data, we can presume that the Panlex data was on average lower quality; if a language performs better on the superset, the Panlex data might still be lower quality, but its quantity at least makes up for performance to some degree. The languages that do over +0.3  better on the subset data are , suggesting that those may have poorer-quality Panlex data, with the largest difference being  at +2.7 ; those that do better on the superset are , and , suggesting that Panlex still adds useful signal there.

The picture that begins to come together is that Panlex often has some useful signal, but also contains considerable amounts of noise. For a less expressive model that is already not able to reach very high quality, some noise in the lexicons does not hurt, and Panlex can help the model get off the ground for the lowest-resource languages. But for a stronger baseline model that produces higher-quality translations on average, this noise can actively harm performance. Therefore, more carefully curated bilingual lexica, like , will tend to will yield higher quality results when used for model training with bigger models, as evinced in Table <ref>.

§ DOES LEXICAL AUGMENTATION FIX COMMON MT MISTAKES?

In evaluating the “big” models with $1.6$B parameters, we wished to see whether our preferred lexical data augmentation methods (GatiPanlexTokenPairs or CodeswitchMonoGatiPanlex) reduced several common types of MT errors. The errors we looked at were (1) null output, or the “question mark phenomenon,” where the model simply outputs some unrelated symbol (such as question marks) instead of actual text; (2) copying, where the model copies some or all of the source sentence in its prediction; and (3) repetition, where the model erroneously repeats the same word or phrase many times. There are other error types we could look at, like hallucination, but we stick with these three basic types for this paper. More precise definitions of these errors are given below.

The results of this analysis are given in Table <ref>. For each error type, we computed the percentage of sentences that were affected by dividing the number of affected sentences by the total number of sentences in the eval set.  has $806248$ sentence pairs across all languages and  has $309887$.

Null output (question mark phenomenon)
The first error type occurs when the model outputs only “??', or some other arbitary character, as its prediction. Instances of this likely indicate catastrophic effects of out-of-domain phenomena for 0-shot translation.

Another common error is copying, where the model's prediction is close or identical to the source sentence. In measuring this phenomenon, we said that any prediction with $> 85\%$ character-level similarity to the source sentence was considered a copy. To measure character-level similarity, we took the multiset intersection of the character frequencies in the source and the character frequencies in the prediction, and then divided the size of the intersection by the number of characters in the source.

The last common error type we examined was repetition. To count these mistakes, we divided the total number of tokens in a sentence by the number of unique tokens. If the ratio was $> 3$, we counted the prediction as an instance of erroneous repetition.

§.§ Comparing sampling strategies for translating tokens

As one recalls from Section <ref>, the Codeswitch augmentation works as follows: Let $D$ represent a multilingual lexicon containing word or phrase translation pairs for many languages. Given a source sentence $x = (x_{1}, x_{2}, . . ., x_{n})$ from monolingual corpus $X_{mono}$, we substitute each token in $x$ for its dictionary translation with probability $p_{tr}$.

However, there is an issue with this formulation. Because the lexica we use do not have exhaustive coverage across languages, it is often the case that simply looping over $x$ and attempting to translate each token with probability $p_{tr}$ would result in translating a fraction of $x$ that is significantly less than $p_{tr}$. So in order to approximate this desired fraction $p_{tr}$ as closely as possible, we first count how many tokens in $x$ have dictionary translations. Let this number be $k$. We then compute the adjusted probability $\tilde p_{tr} = \max(\frac{np_{tr}}{k}, 1)$,  and sample from amongst the words in $x$ with translations with probability $\tilde p_{tr}$, to obtain the codeswitched sentence $x'$. When substituting a source word for its translation, we choose a translation uniformly at random from all available translations in all languages. Because of this, it is usually the case that $x$ is codeswitched into many languages. Finally, we train the model to reconstruct the monolingual sentence $x$ from $x'$ using the same sequence-to-sequence model and loss function as for the MT task.

In our experiments we use use $p_{tr} = 0.4$. We apply MCA to all $208$ languages in our corpus, but augment only half the available monolingual data and train the remaining half with MASS <cit.>, as done in the baseline training regime <cit.>. We prepend a task token, , to the codeswitched sentences to cue the model to perform the MCA task, as well as language () and script () tokens. The  and script  tags are used in all models, including the baseline.

Since this augmentation samples each token with some probability, the number of tokens translated in a given sentence follows a binomial distribution. The Glowup augmentation, however, samples a number of tokens to translate uniformly at random from all possible translatable tokens. So one has a binomial distribution over N tokens sampled, and the other has a uniform distribution—does this make a difference?

To test this we trained a version of the CodeswitchMono model using uniform sampling. The average  of the CodeswitchMonoUniform model was 0.1 to 0.2 higher on all four of the ( ) x ( ) directions. We conclude that this may have a slight benefit, but the difference is within the realm of noise, and does not affect the conclusions elsewhere in this paper.

Unsupervised Machine Translation models often confuse words that occur in similar contexts. A frequent example of this mistake is to translate one animal into another animal from a similar semantic category. For example, UNMT models frequently mis-translate “cat” (e.g. Odis on the left) as “dog”, (e.g. Sandy on the right). We hope that multilingual lexica will help fix this sort of mistake, which is easy for a human to correct with a dictionary.

§ RELATIONSHIP BETWEEN NUMBER OF TOKENS AND MT PERFORMANCE

We also graph the relationship between number of lexical word pairs and $\Delta \chrf{}$ in Figures <ref> () and <ref> () for URLs only. The results for  and  are combined in these plots. In both directions, we observe a moderate positive relationship between the number of lexical word pairs for a given language in the augmented data and the $\Delta \chrf{}$ over the baseline.

Number of lexicon word pairs in augmented data vs. $\Delta \chrf{}$ over baseline for unsupervised languages in the  direction. Results for  and  are combined here.

Number of lexicon word pairs in augmented data vs. $\Delta \chrf{}$ over baseline for unsupervised languages in the  direction. Results for  and  are combined here.

§ LANGUAGES

§.§ Rationale for Language Choice

Although this project is aligned with the 1000-language initiative from <cit.>, we wanted to use smaller models for more rapid iteration, and as a result, commensurately smaller data and number of languages to fit comfortably in the model. Therefore, we chose to work with about 200 languages.

With this in mind, we also wanted to choose specifically those languages whose performance we could measure. Therefore, our approach was as follows:

* Include all languages with supervised (parallel) data, for maximal cross-lingual transfer

* Include all languages that have non-zero data and a  eval set

* Include all languages that have non-zero data and a  eval set

§.§ Complete Language data

The following table gives a list of the languages used in our experiments, along with some linguistic and resource-related statistics. The numbers for data resources (i.e. Mono, Parallel, Panlex, and ) refer to the amount of data actually used in our experiments, not necessarily the total amount of data available. For example, we subsampled the parallel and high-resource monolingual data we had available by a factor of $10$.

§ FULL RESULTS

The full results on  for the various models we trained are available in Tables <ref> and <ref>. Model abbreviations are clarified in Table <ref>. Scores from the NLLB model are included as reference, though keep in mind that the smaller research models in this paper will naturally have lower quality; even the “Big” models are 30x smaller, and not optimized with back-translation and so on.


[1]

# Dynamical Formation of Graphene and Graphane Nanoscrolls

M. L. Pereira Júnior  L. A. Ribeiro Júnior<EMAIL_ADDRESS>D. S. Galvão  J.
M. De Sousa  Institute of Physics, University of Brasília, 70910-900,
Brasília, Brazil  Applied Physics Department, University of Campinas,
Campinas, São Paulo, Brazil  Center for Computing in Engineering and Sciences,
University of Campinas, Campinas, São Paulo, Brazil.  Federal Institute of
Education, Science and Technology of Piauí, São Raimundo Nonato, Piauí,
Brazil.

###### Abstract

Carbon nanoscrolls (CNSs) are nanomaterials with geometry resembling graphene
layers rolled up into a spiral (papyrus-like) form. Effects of hydrogenation
and temperature on the self-scrolling process of two nanoribbons interacting
with a carbon nanotube (CNT) have been studied by molecular dynamics
simulations for three configurations: (1) graphene/graphene/CNT; (2)
graphene/graphane/CNT, and (3) graphane/graphane/CNT. Graphane refers to a
fully hydrogenated graphene nanoribbon. Nanoscroll formation is observed for
configuration (1) and (2) for temperatures 300-1000 and 300-600 K,
respectively, while nanoribbons wrap CNT without nanoscroll formation for
configuration (3).

###### keywords:

Reactive Molecular Dynamics Graphene Nanoribbons Graphane Nanoribbons Carbon
Nanoscrolls Self-Scrolling

##  1 Introduction

After isolation of a single layer of graphite (Graphene) [1], obtained by
mechanical exfoliation, there is a renewed interest in obtaining new 2D
materials [2, 3]. From a topological point of view, graphene layers can be
considered as the basis to generate other structures, such as carbon nanotubes
[4] and nanoscrolls [5]. Carbon nanotubes are understood as graphene layers
rolled up into cylinders, while nanoscrolls are graphene layers rolled up into
a papyrus-like form. There are many theoretical and experimental works
investigating nanoscrolls of different materials [6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16].

In contrast to carbon nanotubes (CNTs), carbon nanoscrolls (CNSs) have open
ends, which allows an easily radial expansion (tunable interlayer distances)
explored in different applications [17, 18]. The most significant barrier in
their usage has been the limited synthesis of high-quality structures. It
leads to a controlling inability of some crucial aspects, among them, the
number of scrolled layers [19, 20, 21]. To overcome this barrier, new
alternative routes in obtaining CNSs that allow higher control over the final
structure to broaden their applications are desirable [22, 23, 24]. CNSs
exhibit distinct electronic and mechanical behaviors when compared with
graphene and CNTs [25, 26]. The CNSs nucleus presents vibrational properties
that can be useful in developing nano-actuators and electromechanical devices
with excellent performance as cathodes in lithium-sulfur batteries [27, 28,
29]. It is worth mentioning that Density Functional Theory (DFT) calculations
were used to address the synthetic growth concept of carbon-based materials
[30].

Recently, molecular dynamics simulations have been used to investigate other
possible synthetic routes [17, 13, 31]. Results showed that CNT triggers the
self-scrolling process of graphene nanoribbons. As a result, the layer
spontaneously rolls itself around the CNT to lower the surface energy, a
process governed by van der Waals interactions. The advantage of this method
is that it imposes dry, non-chemical, and room-temperature conditions [17, 13,
31]. An experimental study was conducted by Xie et. al. to obtain stable CNSs
focused on offering higher control over the final product [32]. In their
method, graphite is mechanically exfoliated, deposited over SiO2 substrate,
and then immersed in a solution of water and isopropyl alcohol. After resting
for a few minutes, the system spontaneously formed CNSs [32]. Another
possibility is to use functionalized structures such as graphene oxides and
hydrogenated ones [29, 33].

Herein, we used fully-atomistic reactive (ReaxFF) molecular dynamics (MD)
simulations to investigate the effects of hydrogenation on the self-scrolling
mechanism of graphene and graphane nanoribbons when triggered by CNTs. Our
results show that it is possible to obtain stable CNT-wrapped CNSs from pure
and hydrogenated graphane nanoribbons.

##  2 Methodology

In the present work, we used fully atomistic (ReaxFF) molecular dynamics
simulations for C/H/O [34], as implemented in LAMMPS code [35]. The initial
configurations of our model systems are presented in Figure 1, where we shown
three atomic models identified as Complex-A (graphene/graphene/CNT, Complex-B
(graphene/graphane/CNT), and Complex-C (graphane/graphane/ CNT). The two
graphene-based nanoribbons have a length and width of 150 Å and 21 Å,
respectively. The CNT used to trigger the self-scrolling mechanism is a
conventional zigzag CNT $(0,22)$ with a diameter of 17.21 Å and length of 100
Å. The graphene nanoribbon and CNT have 1296 and 2112 carbon atoms,
respectively. The initial distance between the nanoribbons and the CNT wall is
5.41 Å.In the initial assembly, the nanoribbons are placed into contact with
the nanotubes as shown in Figures 1(d-f). The nanoribbons were placed on
opposite sides regarding the CNT center.

Figure 1: Model systems used to simulate the formation of CNT-wrapped CNSs.
Illustration of the atomic configuration for (a) graphene and (b) graphane
nanoribbons. The dashed lines highlight the structural units. (c) Diagrammatic
representation for the initial arrangement of the system involving two
nanoribbons and a CNT, where $d_{1}=150$ Å (nanoribbon length), $d_{2}=17.21$
Å (CNT diameter), and $d_{3}=28.03$ Å (initial distance between the
nanoribbons). Initial configurations for the systems studied here: (d)
graphene/graphene/CNT (Complex–A), (e) graphene/graphane/CNT (Complex–B), (f)
and graphane/graphane/CNT (Complex–C). In the color scheme, the gray and
yellow spheres represent carbon and hydrogen atoms, respectively.

The composite systems (Complexes –A, –B, and –C) were equilibrated/thermalized
within an NVT ensemble at $300$, $600$, and $1000$ K using the Nosé-Hoover
thermostat [36]. We considered 1000 K as the uppermost temperature limit,
envisioned carbon-based optoelectronic applications. Scroll structures are
resilient to high temperatures (up to 4000K), as recently demonstrated in
references [14, 37]. The system equilibration was initially performed to
eliminate any residual stress during 100 ps. The time step for the simulations
was $0.1$ fs. The MD snapshots and trajectories were obtained by using the
free visualization and analysis program VMD [38].

In the present work, we performed 10 MD runs (with different seeds) for each
temperature. The random seeds defined distinct initial sets of atomic
velocities in the equilibration of the systems. All the MD runs yielded the
same products for each complex. In this sense, the results reported in the
manuscript are the most representative outcomes obtained for all the
simulations.

##  3 Results

We begin our discussion by presenting the dynamical behavior of the self-
scrolling process for all complexes studied here (see Figure 1). In Figure 2
we show a sequence of representative MD snapshots (in lines) for Complex-A,
h-B, and -C from top to bottom, respectively, for simulations at $300$ K. In
the very first stage of the simulation (about $2.5$ ps), the nanoribbons are
in contact with the CNT surface, and the self-scrolling process takes place.
This trend is realized in the simulations for all complexes, as shown in
Figure 2. Such behavior also occurred for the self-scrolling mechanism of a
single graphene [39] and graphane (see Supplementary Material) nanoribbons in
contact with a CNT.

Figure 2: Representative MD snapshots for the spontaneous self-scrolling
process at 300 K, for the complexes presented in Figure 1.

At $12.5$ ps of simulation, the nanoribbons start to curl around the CNT.
Subsequently, for Complexes A and B, the overlap between the nanoribbons
occurs. The graphane nanoribbons in Complex-C do not form a scroll. Instead,
it simply wraps the CNT by covering its surface and is temperature independent
(for the temperature values considered here, up to $1000$ K), creating a lock-
like mechanism. This behavior is attributed to stereo hydrogen-hydrogen
interactions when the graphane layers move to create an opposing force to the
sliding. This trend is similar to observed for other hydrocarbon structures,
as the lander molecule, where this lock-like mechanism is present [40].

Differences in the complexes dynamics become clear at $37.5$ ps. At this
stage, in Complexes A and B, the nanoribbons are almost completely wrapped, as
a consequence of the interlayer interaction presented by graphene and graphane
[20, 41, 42]. Graphene/graphane bilayer (Complex-B) presents an interlayer
interaction energy smaller than graphene/graphene one (Complex-A) [43]. Even
so, the self-scrolling process for both complexes takes the same time yielding
a completely scrolled configuration at $50$ ps. For Complex-C, the graphane
nanoribbons wrap the entire CNT wall at $130$ ps. In all cases, the
nanoribbons showed discontinuous wrinkles and corrugations during the
spontaneous self-scrolling process. Other reactive MD simulations performed
within the scope of the AIREBO interatomic potential [44, 26] also pointed for
such instabilities. The Supplemental Material includes the videos for the
dynamics of the complexes A-C. It is worth mentioning that the degree of
instability for the nanoribbons was more pronounced for temperatures higher
than $300$ K.

The Supplementary Material presents the case in which a single graphane
nanoribbon is interacting with a CNT. The main conclusions on the scrolling
process hold, and the system evolved to a single graphane-based nanoscroll.
Isolated graphane nanoscrolls were already investigated by molecular dynamics
simulations [42], and the results are consistent with the ones presented here.

It is important to stress that depending on the area of the nanoribbon, the
most stable configuration is not planar but a scrolled one. This trend is
related to the size of the overlapped region that enables van der Waals gains
to overcome the elastic cost to scroll the structures [9]. Even for small
graphene nanobelts, the self-scrolling can be a self-sustained process [45].

Figure 3: Total energy time evolution as function of simulation time for the
Complex-A at 10 K. The insets are MD snapshots extracted from the simulation
trajectory at $t_{1}=2.5$, $t_{2}=20$, $t_{3}=40$, $t_{4}=65$, and $t_{2}=80$
ps, respectively.

In Figure 3 we present the time evolution for the total energy for the
Complex-A at $10$ K at five different stages. During the dynamical process for
the CNSs formation, there is a linear downhill trend that ends at $65$ ps,
which represents a completely formed scroll. From that moment on, the total
energy stabilizes, and a flat region persists (neglecting thermal
fluctuations, scroll “breathing” - see video) until the end of the simulation.
For all the temperature regimes simulated here ($300$, $600$, and $1000$ K),
the time evolution of the total energy shows a similar trend. An overall
picture of the CNSs formation is presented in Figure 4. The existence of an
energy plateau is the signature of the scroll formation. At high temperatures,
the CNSs total energies are larger due to the higher degree of atomic thermal
vibrations, which tends to difficult the spontaneous scrolling process. Even
so, one can realize a well-defined plateau in all the cases, standing for the
scroll’s formation.

Figure 4: Total energy time evolution as a function of the simulation time for
all complexes at 300, 600, and 1000 K. The red, blue, and green data points
refer to the total energy values during the simulation time for the Complex-A,
Complex-B, and Complex-C, respectively. The lines were obtained from the data
interpolation.

##  4 Conclusion

Through fully-atomistic reactive (ReaxFF) MD simulations, we have studied the
hydrogenation effects on the self-scrolling process of graphene and graphane
(hydrogenated graphene) nanoribbons interacting with a carbon nanotube (CNT).
We have considered different configurations: (1) graphene/graphene/CNT; (2)
graphene/graphane/CNT, (3) graphane/graphane/CNT; and different temperatures,
$300$, $600$, and $1000$ K. Our results show that the scroll formation is
possible for configuration (1) and (2), for all temperatures. Configuration
(3) does not yield scrolls. The scroll formation is related to the existence
of an energy plateau in the total energy as a function of the simulation time.
The approach used here to assemble the nanoribbons and CNT is prototypical and
extensible to other species of 2D materials beyond graphene. The usage of
nanotubes for triggering the scrolling processes of sheets and nanoribbons was
investigated for carbon and boron nitrides systems [46, 47, 48]. The tribology
of carbon-based nanostructures has been an object of research in the last
years [49, 50]. In this sense, the present study can stimulate further
investigations along these lines.

## Acknowledgements

The authors gratefully acknowledge the financial support from Brazilian
research agencies CNPq, FAPESP, and FAP-DF. M.L.P.J gratefully acknowledges
the financial support from CAPES grant 88882.383674/2019-01. J.M.S. and D.S.G.
thanks the Center for Computing in Engineering and Sciences at Unicamp for
financial support through the FAPESP/CEPID Grants #2013/08293-7 and
#2018/11352-7. L.A.R.J acknowledges the financial support from a Brazilian
Research Council FAP-DF and CNPq grants $00193.0000248/2019-32$ and
$302236/2018-0$, respectively. L.A.R.J and J.M.S acknowledge CENAPAD-SP for
providing the computational facilities. L.A.R.J. gratefully acknowledges the
financial support from IFD/UnB (Edital $01/2020$) grant
$23106.090790/2020-86$. The authors acknowledge the National Laboratory for
Scientific Computing (LNCC/MCTI, Brazil) for providing HPC resources of the
SDumont supercomputer, which have contributed to the research results reported
within this paper. URL: http://sdumont.lncc.br.

## References

  * [1] Kostya S Novoselov, Andre K Geim, Sergei V Morozov, D Jiang, Y_ Zhang, Sergey V Dubonos, Irina V Grigorieva, and Alexandr A Firsov.  Electric field effect in atomically thin carbon films.  science, 306(5696):666–669, 2004.
  * [2] Yuanbo Zhang, Yan-Wen Tan, Horst L Stormer, and Philip Kim.  Experimental observation of the quantum hall effect and berry’s phase in graphene.  nature, 438(7065):201–204, 2005.
  * [3] Alexander A Balandin, Suchismita Ghosh, Wenzhong Bao, Irene Calizo, Desalegne Teweldebrhan, Feng Miao, and Chun Ning Lau.  Superior thermal conductivity of single-layer graphene.  Nano letters, 8(3):902–907, 2008.
  * [4] Sumio Iijima and Toshinari Ichihashi.  Single-shell carbon nanotubes of 1-nm diameter.  nature, 363(6430):603–605, 1993.
  * [5] Roger Bacon.  Growth, structure, and properties of graphite whiskers.  Journal of Applied Physics, 31(2):283–290, 1960.
  * [6] David Tománek.  Mesoscopic origami with graphite: scrolls, nanotubes, peapods.  Physica B: Condensed Matter, 323(1-4):86–89, 2002.
  * [7] M Lisa.  Viculis a chemical route to carbon nanoscrolls viculis, julia j. mack, richard b. kaner.  Science., 299(5611):1361, 2003.
  * [8] Lisa M Viculis, Julia J Mack, and Richard B Kaner.  A chemical route to carbon nanoscrolls.  Science, 299(5611):1361–1361, 2003.
  * [9] Scheila F Braga, Vitor R Coluci, Sergio B Legoas, Ronaldo Giro, Douglas S Galvão, and Ray H Baughman.  Structure and dynamics of carbon nanoscrolls.  Nano letters, 4(5):881–884, 2004.
  * [10] VR Coluci, SF Braga, RH Baughman, and DS Galvao.  Prediction of the hydrogen storage capacity of carbon nanoscrolls.  Physical Review B, 75(12):125404, 2007.
  * [11] Xinghua Shi, Yuan Cheng, Nicola M Pugno, and Huajian Gao.  Tunable water channels with carbon nanoscrolls.  Small, 6(6):739–744, 2010.
  * [12] Taewoo Uhm, Jeonghyeon Na, Jae-Ung Lee, Hyeonsik Cheong, Sang Wook Lee, Eleanor EB Campbell, and Sung Ho Jhang.  Structural configurations and raman spectra of carbon nanoscrolls.  Nanotechnology, 31(31):315707, 2020.
  * [13] Eric Perim, Ricardo Paupitz, and Douglas S Galvão.  Controlled route to the fabrication of carbon and boron nitride nanoscrolls: A molecular dynamics investigation.  Journal of Applied Physics, 113(5):054306, 2013.
  * [14] Marcelo Lopes Pereira Junior, Wiliam Ferreira da Cunha, Douglas Soares Galvão, and Luiz Antonio Ribeiro Junior.  A reactive molecular dynamics study on the mechanical properties of a recently synthesized amorphous carbon monolayer converted into a nanotube/nanoscroll.  Physical Chemistry Chemical Physics, 23(15):9089–9095, 2021.
  * [15] Raphael M Tromer, Levi C Felix, Luiz A Ribeiro, and Douglas S Galvao.  Optoelectronic properties of amorphous carbon-based nanotube and nanoscroll.  Physica E: Low-dimensional Systems and Nanostructures, 130:114683, 2021.
  * [16] Eric Perim, Leonardo Dantas Machado, and Douglas Soares Galvao.  A brief review on syntheses, structures, and applications of nanoscrolls.  Frontiers in Materials, 1:31, 2014.
  * [17] Zhao Zhang and Teng Li.  Carbon nanotube initiated formation of carbon nanoscrolls.  Applied physics letters, 97(8):081909, 2010.
  * [18] Xinghua Shi, Nicola M Pugno, and Huajian Gao.  Mechanics of carbon nanoscrolls: a review.  Acta Mechanica Solida Sinica, 23(6):484–497, 2010.
  * [19] Pingwei Liu, Zhong Jin, Georgios Katsukis, Lee William Drahushuk, Steven Shimizu, Chih-Jen Shih, Eric D Wetzel, Joshua K Taggart-Scarff, Bo Qing, Krystyn J Van Vliet, et al.  Layered and scrolled nanocomposites with aligned semi-infinite graphene inclusions at the platelet limit.  Science, 353(6297):364–367, 2016.
  * [20] AV Savin, EA Korznikova, IP Lobzenko, Yu A Baimova, and SV Dmitriev.  Symmetric scrolled packings of multilayered carbon nanoribbons.  Physics of the Solid State, 58(6):1278–1284, 2016.
  * [21] AV Savin, EA Korznikova, SV Dmitriev, and EG Soboleva.  Graphene nanoribbon winding around carbon nanotube.  Computational Materials Science, 135:99–108, 2017.
  * [22] Yoshiyuki Kuroda, Kazuyuki Ito, Kenichi Itabashi, and Kazuyuki Kuroda.  One-step exfoliation of kaolinites and their transformation into nanoscrolls.  Langmuir, 27(5):2028–2035, 2011.
  * [23] Tomohiro Tojo, Kazunori Fujisawa, Hiroyuki Muramatsu, Takuya Hayashi, Yoong Ahm Kim, Morinobu Endo, Mauricio Terrones, and Mildred S Dresselhaus.  Controlled interlayer spacing of scrolled reduced graphene nanotubes by thermal annealing.  Rsc Advances, 3(13):4161–4166, 2013.
  * [24] Young-Eun Shin, Young Jin Sa, Seungyoung Park, Jiwon Lee, Kyung-Hee Shin, Sang Hoon Joo, and Hyunhyub Ko.  An ice-templated, ph-tunable self-assembly route to hierarchically porous graphene nanoscroll networks.  Nanoscale, 6(16):9734–9741, 2014.
  * [25] Yu Chen, Jing Lu, and Zhengxiang Gao.  Structural and electronic study of nanoscrolls rolled up by a single graphene sheet.  The Journal of Physical Chemistry C, 111(4):1625–1630, 2007.
  * [26] HY Song, SF Geng, MR An, and XW Zha.  Atomic simulation of the formation and mechanical behavior of carbon nanoscrolls.  Journal of Applied Physics, 113(16):164305, 2013.
  * [27] Xinghua Shi, Nicola M Pugno, Yuan Cheng, and Huajian Gao.  Gigahertz breathing oscillators based on carbon nanoscrolls.  Applied Physics Letters, 95(16):163113, 2009.
  * [28] Xinghua Shi, Yuan Cheng, Nicola M Pugno, and Huajian Gao.  A translational nanoactuator based on carbon nanoscrolls on substrates.  Applied Physics Letters, 96(5):053115, 2010.
  * [29] Pengjian Zuo, Wei Zhang, Junfu Hua, Yulin Ma, Chunyu Du, Xinqun Cheng, Yunzhi Gao, and Geping Yin.  A novel one-dimensional reduced graphene oxide/sulfur nanoscroll material and its application in lithium sulfur batteries.  Electrochimica Acta, 222:1861–1869, 2016.
  * [30] RRQ Freitas, R Rivelino, F de B Mota, Gueorgui Kostov Gueorguiev, and CMC de Castilho.  Energy barrier reduction for the double proton-transfer reaction in guanine–cytosine dna base pair on a gold surface.  The Journal of Physical Chemistry C, 119(27):15735–15741, 2015\.
  * [31] Danhui Zhang and Houbo Yang.  Formation of carbon nanoscrolls from graphene sheet: A molecular dynamics study.  Journal of Molecular Structure, 1125:282–287, 2016.
  * [32] Xu Xie, Long Ju, Xiaofeng Feng, Yinghui Sun, Ruifeng Zhou, Kai Liu, Shoushan Fan, Qunqing Li, and Kaili Jiang.  Controlled fabrication of high-quality carbon nanoscrolls from monolayer graphene.  Nano letters, 9(7):2565–2570, 2009.
  * [33] Carlo A Amadei, Itai Y Stein, Gregory J Silverberg, Brian L Wardle, and Chad D Vecitis.  Fabrication and morphology tuning of graphene oxide nanoscrolls.  Nanoscale, 8(12):6783–6791, 2016.
  * [34] Chowdhury Ashraf and Adri CT Van Duin.  Extension of the reaxff combustion force field toward syngas combustion and initial oxidation kinetics.  The Journal of Physical Chemistry A, 121(5):1051–1068, 2017.
  * [35] Steve Plimpton.  Fast parallel algorithms for short-range molecular dynamics.  Journal of computational physics, 117(1):1–19, 1995.
  * [36] William G Hoover.  Canonical dynamics: equilibrium phase-space distributions.  Physical review A, 31(3):1695, 1985.
  * [37] Donggu Lee, Junmo Kim, Gyeongtae Park, Hyeong Woo Bae, Myungchan An, and Jun Young Kim.  Enhanced operating temperature stability of organic solar cells with metal oxide hole extraction layer.  Polymers, 12(4):992, 2020.
  * [38] William Humphrey, Andrew Dalke, Klaus Schulten, et al.  Vmd: visual molecular dynamics.  Journal of molecular graphics, 14(1):33–38, 1996.
  * [39] Dan Xia, Qingzhong Xue, Jie Xie, Huijuan Chen, Cheng Lv, Flemming Besenbacher, and Mingdong Dong.  Fabrication of carbon nanoscrolls from monolayer graphene.  Small, 6(18):2010–2019, 2010.
  * [40] Roberto Otero, Frauke Hümmelink, Fernando Sato, Sergio B Legoas, Peter Thostrup, Erik Lægsgaard, Ivan Stensgaard, Douglas S Galvão, and Flemming Besenbacher.  Lock-and-key effect in the surface diffusion of large organic molecules probed by stm.  Nature materials, 3(11):779–782, 2004.
  * [41] Emilio M Pérez and Nazario Martín.  $\pi$–$\pi$ interactions in carbon nanostructures.  Chemical Society Reviews, 44(18):6425–6433, 2015.
  * [42] Alexander V Savin, Ruslan A Sakovich, and Mikhail A Mazo.  Using spiral chain models for study of nanoscroll structures.  Physical Review B, 97(16):165436, 2018.
  * [43] Deivasigamani Umadevi and G. Narahari Sastry.  Graphane versus graphene: a computational investigation of the interaction of nucleobases, aminoacids, heterocycles, small molecules (co2, h2o, nh3, ch4, h2), metal ions and onium ions.  Phys. Chem. Chem. Phys., 17:30260–30269, 2015.
  * [44] Jie Huang and Chee How Wong.  Mechanical properties of bundled carbon nanoscroll.  Mechanics of Materials, 87:1–10, 2015.
  * [45] BVC Martins and DS Galvao.  Curved graphene nanoribbons: structure and dynamics of carbon nanobelts.  Nanotechnology, 21(7):075710, 2010.
  * [46] Yunfang Li.  Boron-nitride nanotube triggered self-assembly of hexagonal boron-nitride nanostructure.  Physical Chemistry Chemical Physics, 16(38):20689–20696, 2014.
  * [47] Eric Perim and Douglas S Galvao.  Novel nanoscroll structures from carbon nitride layers.  ChemPhysChem, 15(11):2367–2371, 2014.
  * [48] Andrei I Siahlo, Nikolai A Poklonski, Alexander V Lebedev, Irina V Lebedeva, Andrey M Popov, Sergey A Vyrko, Andrey A Knizhnik, and Yurii E Lozovik.  Structure and energetics of carbon, hexagonal boron nitride, and carbon/hexagonal boron nitride single-layer and bilayer nanoscrolls.  Physical Review Materials, 2(3):036001, 2018.
  * [49] Wanlin Guo, Wenyu Zhong, Yitao Dai, and Sunan Li.  Coupled defect-size effects on interlayer friction in multiwalled carbon nanotubes.  Physical Review B, 72(7):075409, 2005.
  * [50] IV Lebedeva, AA Knizhnik, AM Popov, Yu E Lozovik, and BV Potapkin.  Dissipation and fluctuations in nanoelectromechanical systems based on carbon nanotubes.  Nanotechnology, 20(10):105202, 2009.