significantly reduces both accuracy and fluency errors when compared with an
RNN-based model.

###  5.3 Introduction

A new era of high-quality translations has been heralded with the advent of
neural machine translation (NMT). Given that large datasets are a prerequisite
for high-quality NMT, these improvements are not always evident in the
translation of low-resource languages. In the context of such languages, which
suffer from a sparsity of data, alternative approaches must be adopted.

Developing applications and models to address the challenges of low-resource
language technology is an important part of this research. This technology
incorporates new methods, which reduce the impact that data scarcity has on
the digital engagement of low-resource languages. One approach is to use a
mechanism that helps NMT systems to learn from unlabelled data using dual-
learning [46, 3].

Out-of-the-box NMT systems, trained on English$\leftrightarrow$Irish
(EN$\leftrightarrow$GA) data, have been shown to achieve a lower translation
quality compared with using a tailored SMT system [37]. It is in this context
that further research is required in the development of NMT for low-resource
languages, and the Irish language in particular.

Most research on the choice of subword models has focused on high-resource
languages [36, 43]. Translation, by its nature, requires an open vocabulary
and the use of subword models aims to address the fixed-vocabulary problem
associated with NMT. Rare and unknown words are encoded as sequences of
subword units. By adapting the original BPE algorithm [41], the use of BPE
subword models can improve translation performance [103, 60]. In the context
of developing models for EN$\leftrightarrow$GA translation, there were no
clear recommendations on the choice of subword model types. Character-based
models were briefly explored due to their simplicity and reduced memory
requirements. However, they were not considered suitable given that most
single characters do not carry meaning in the English and Irish languages.
Therefore, one of the objectives of our research is to identify which type of
subword model performs best in this low-resource scenario.

An important goal of this study is to extend our previous work [65] by
providing a human evaluation and comparison of English to Irish
(EN$\rightarrow$GA) machine translation (MT) on systems that use either a
baseline RNN architecture or a subword-model optimised Transformer model.

This paper describes the context in which our research was conducted and
provides a background of the types of available architecture in Section 5.4. A
detailed overview of our approach is outlined in Section 5.5, where we provide
details of the data and parameters used in our NMT systems. The empirical
results, using both automatic metrics and a human evaluation, are presented in
Section 5.6. Finally, our findings are discussed in Section 5.8 and the
possibilities for future work are outlined in Section 5.9.

###  5.4 Background

Native speakers of low-resource languages are often excluded from useful
content since, more often than not, online content is not available to them in
their language of choice. This digital divide experienced by second-language
speakers has been well-documented in the research literature [78, 4].

Research on MT in low-resource scenarios seeks to directly address this
challenge of exclusion via pivot languages [72], and indirectly, via domain
adaptation of models [42]. Consequently, research efforts focusing on NMT [10,
28] have resulted in a state-of-the-art (SOTA) performance being attained for
multiple language pairs [20, 21]. The Irish language is a primary example of a
low-resource language that will benefit from this research. NMT involving
Transformer model development will improve performance in specific domains of
low-resource languages.

####  5.4.1 Hyperparameter Optimisation

Hyperparameters are employed to customise machine learning models such as
translation models. It has been shown that machine learning performance may be
improved through hyperparameter optimisation (HPO) rather than just using
default settings [98]. The principal methods of HPO are grid search [81] and
random search [16].

##### RNN

The tasks of natural language processing (NLP), speech recognition and MT are
often performed by RNNs. This architecture enables previous outputs to be used
as inputs while having hidden states. In the context of MT, such neural
networks were ideal due to their ability to process inputs of any length.
Furthermore, the model sizes do not necessarily increase with the input size.
Commonly used variants of RNN include bidirectional (BRNN) and deep (DRNN)
architectures. However, the problem of vanishing gradients coupled with the
development of attention-based algorithms often leads to Transformer models
performing better than RNNs.

##### Transformer

The greatest improvements have been demonstrated when either the RNN or the
CNN architecture is abandoned completely and replaced with an attention
mechanism creating a much simpler and faster architecture known as
Transformer. Experiments in MT tasks show such models are better in quality
due to greater parallelisation while requiring significantly less time to
train [114].

Transformer models use attention to focus on previously generated tokens. The
approach allows for models to develop a long memory, which is particularly
useful in the domain of language translation. Performance improvements to both
RNN and CNN approaches may be achieved through the introduction of such
attention layers in the translation architecture.

####  5.4.2 SentencePiece

Designed for NMT, SentencePiece is a language-independent subword tokenizer
that provides an open-source C++ and a Python implementation for subword
units. An attractive feature of the tokenizer is that SentencePiece directly
trains subword models from raw sentences [61].

####  5.4.3 Human Evaluation

Human evaluation, within NLP and MT, is a topic of growing importance, which
often has a dedicated research track or workshop at major conferences [14].
This focus has resulted in many publications in the area of human evaluation
that relate to MT [113, 26] and it has particularly benefited the evaluation
of low-resource languages [13, 50].

The best practice for the human evaluation of MT has been published in the
form of a series of recommendations [70]. As part of our research, we adopted
these recommendations, which are in line with similar human evaluation studies
of EN$\leftrightarrow$GA MT at the ADAPT centre [38]. Specifically, these
recommendations encourage the use of professional translators, evaluation at
the document level and assessments of both fluency and accuracy. Original
source texts were also used in the training and test data.

These recommendations have been complemented by a fine-grained human analysis,
which uses both a Scalar Quality Metric (SQM) and Multidimensional Quality
Metrics (MQM).

###  5.5 Proposed Approach

Considerable performance improvements have been achieved using the HPO of RNN
models in low-resource settings. One of the key research questions, evaluated
as part of this study, is to identify the extent to which such optimisation
techniques may be applied to low-resource Transformer models. Evaluations
included modifying the number of attention heads, changing the number of
layers and experimenting with regularisation techniques such as dropout and
label smoothing. Most importantly, the choice of subword model type and
vocabulary size is evaluated. Furthermore, previous research focuses on using
an automatic evaluation of performance, whereas we propose combining a human
evaluation approach with automatic metrics.

To test the effectiveness of our approach, optimisation was carried out on an
EN$\leftrightarrow$GA parallel dataset: a general corpus of 52k lines from the
Directorate General for Translation (DGT). With DGT, the test set used 1.3k
lines and the development set comprised 2.6k lines. All experiments involved
concatenating source and target corpora to create a shared vocabulary and a
shared SentencePiece subword model. The adopted approach is illustrated in
Figure 5.1.

####  5.5.1 Architecture Tuning

It is difficult and costly to tune systems using a conventional grid search
approach given the long training times associated with NMT. Therefore, we
adopted a random search approach in the HPO of our Transformer models.

Using smaller and fewer layers with low-resource datasets has previously been
shown to improve performance [5]. Furthermore, the use of shallow Transformer
models has been demonstrated to improve the translation performance of low-
resource NMT [17]. Guided by these findings, configurations were tested, which
varied the number of neurons in each layer and modified the number of layers
used in the Transformer architecture.

Varying degrees of dropout were applied to Transformer models to evaluate the
impact of regularisation. Configurations using smaller (0.1) and larger values
(0.3) were applied to the output of each feed-forward layer.

####  5.5.2 Subword Models

Incorporating a word segmentation approach, such as BPE, is now standard
practice when developing NMT models. Subword models are particularly
beneficial for low-resource languages since rare words are often a problem. In
the context of EN$\rightarrow$GA translation, there is no clear agreement as
to what constitutes the best approach. Consequently, subword regularisation
techniques involving BPE and unigram models were evaluated as part of this
study to determine the optimal parameters for maximising translation
performance. BPE models with varying vocabulary sizes of 4k, 8k, 16k and 32k
were evaluated.

Figure 5.1: Proposed approach to evaluate the baseline architectures of RNN and Transformer models. Using a random search approach, the values outlined in Table 5.1 were tested to determine the optimal hyperparameters. Short cycles of 5k training steps were applied to test a range of values for each parameter. Once an optimal value was identified within the sampled range, it was locked in for tests on subsequent parameters. A fine-grained human evaluation was conducted on the output from the DGT dataset and its results were compared with an automatic evaluation. Table 5.1: Transformer HPO using a random search approach. The optimal hyperparameters are highlighted in bold. The best-performing model used two attention heads and was trained on a 55k DGT corpus. Hyperparameter | Values
---|---
Learning rate |  0.1, 0.01, 0.001, 2
Batch size |  1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers |  5, 6
Feed-forward dimension | 2048
Embedding dimension |  128, 256, 512
Label smoothing |  0.1, 0.3
Dropout |  0.1, 0.3
Attention dropout | 0.1
Average Decay |  0, 0.0001

####  5.5.3 Human Evaluation of NMT

Morphological-rich languages, such as Irish, have a high degree of inflection
and free word order that gives rise to specific translation issues when
translating from English. Grammatical categories such as gender or case
inflections in nouns are often difficult to reliably generate in an Irish
translation.

One of the goals of this research is to explore how an NMT system handles
these issues compared with an RNN approach. Existing research suggests NMT
systems should improve these linguistic aspects. NMT, with its use of subword
models, implicitly addresses the problem in an unsupervised manner, without
understanding the actual formal rules of grammatical categories.

Previous human evaluation studies that evaluate EN$\leftrightarrow$GA MT
performance have focused on the differences between an SMT and an NMT approach
[37]. In the context of our research, human evaluation was conducted on purely
NMT methods, which included RNN and Transformer approaches. Furthermore, our
study is differentiated by using both SQM and MQM as our human evaluation
metrics.

It is clear from our earlier experimental findings, based solely on automatic
evaluation metrics, that a Transformer approach leads to significant
improvements compared to traditional RNN systems. However, as with most
automatic scoring methods, these simply provide an overall score for each
system but do not indicate the exact nature of the linguistic problems that
may be encountered in translation. Therefore, it can be said that automatic
evaluation does not address the question of the linguistic or grammatical
quality of the target output. Nuances, such as how gender or cases are
handled, are not covered by this approach.

To achieve a deeper understanding of the linguistic errors created by our RNN
and Transformer systems, a fine-grained human evaluation was conducted. The
outputs from these systems were systematically analysed and compared in a
manual error analysis. This approach captures the nature of the translation
errors for each of the evaluated systems. The output from this study forms the
basis of future work, which will help to improve the translation quality of
our models. The annotation framework, the overall annotation process and the
inter-annotator agreement are discussed below and broadly follow the approach
adopted by other fine-grained human evaluation studies [57].

##### Scalar Quality Metrics

SQM [40] adapts the WMT shared-task settings to collect segment-level scalar
ratings with a document context. SQM uses a scale from 0 to 6 for translation
quality assessment. This is a modification of the WMT approach [77], which
uses a range from 0 to 100.

With this evaluation approach, annotators must select a rating from 0 through
6 when presented with the source and target sentences. The SQM quality levels
for 0, 2, 4 and 6 are outlined in Table 5.2. Annotators may also choose
intermediate levels of 1, 3 and 5 in cases where the translations do not
exactly match the core SQM levels.

Table 5.2: SQM levels explained [40]. SQM Level |  Details of Quality
---|---
6 |  Perfect Meaning and Grammar: The meaning of the translation is completely consistent with the source and the surrounding context (if applicable). The grammar is also correct.
4 |  Most Meaning Preserved and Few Grammar Mistakes: The translation retains most of the meaning of the source. This may contain some grammar mistakes or minor contextual inconsistencies.
2 |  Some Meaning Preserved: The translation preserves some of the meaning of the source but misses significant parts. The narrative is hard to follow due to fundamental errors. Grammar may be poor.
0 |  Nonsense/No meaning preserved: Nearly all information is lost between the translation and source. Grammar is irrelevant.

##### Multidimensional Quality Metrics

As part of QTLaunchpad project111https://www.qt21.eu/ the MQM
framework222https://themqm.org/the-mqm-full-typology/ was developed to provide
a framework of how manual evaluation could be performed via a detailed error
analysis. A single metric for all uses is not imposed. Instead, a
comprehensive catalogue of quality issue types, with standardised names and
definitions, is provided. This catalogue may be customised for specific tasks.
In addition to forming a reliable methodology for quality assessment, it also
allows for us to specify which error tags were relevant to our task.

To adapt the generic MQM framework for our context, we followed the official
guidelines for scientific research [74]. The details of our customisation of
MQM are discussed below.

A large variety of tags, on several annotation layers, are proposed within the
original MQM guidelines. However, this full MQM tagset is too detailed for a
specific annotation task. Therefore, when evaluating our MT output, the
smaller default set of evaluation categories, specified in the core tagset,
was used. These standard top-level categories of accuracy and fluency, which
are proposed by the MQM guidelines, are illustrated in Figure 5.2. A special
non-translation error was used to tag an entire sentence, which was too badly
translated to allow for the identification of individual errors.

Figure 5.2: The core set of error categories proposed by the MQM guidelines.

Error severities are specified as either major or minor errors and are
assigned independently of category. These correspond to actual translation and
grammatical errors or smaller imperfections, respectively. The recommended
default weights [74] were used, which allocate a weight of 1 to minor errors
whereas major errors are assigned a weight of 10. Furthermore, the non-
translation category was allocated a weight of 25, an approach which is in
line with the best practice established in previous studies [40].

The annotators were instructed to identify all errors within each sentence of
the translated output for both systems. The error categories used by the
annotators are outlined in Table 5.3.

Table 5.3: Description of error categories within the core MQM framework [40]. Category |  Sub-Category |  Description
---|---|---
Non-translation |  |  Impossible to reliably characterise the 5 most severe errors.
Accuracy |  Addition |  Translation includes information not present in the source.
|  Omission |  Translation is missing content from the source.
|  Mistranslation |  Translation does not accurately represent the source.
|  Untranslated text |  Source text has been left untranslated.
Fluency |  Punctuation |  Incorrect punctuation
|  Spelling |  Incorrect spelling or capitalisation.
|  Grammar |  Problems with grammar, other than orthography.
|  Register |  Wrong grammatical register (e.g., inappropriately informal pronouns).
|  Inconsistency |  Internal inconsistency (not related to terminology).
|  Character encoding |  Characters are garbled due to incorrect encoding.

##### Annotation Setup

Annotations were carried out using the simpler SQM approach and a more
detailed, fine-grained MQM approach. The hierarchical taxonomy of our MQM
implementation is illustrated in Figure 5.2, whereas the SQM categories are
summarised in Table 5.2.

Two annotators with similar backgrounds were used for the annotation of
outputs from an RNN system and a Transformer system. Both annotators are
native speakers of Irish and neither had prior experience with MQM. Prior to
annotation, they were thoroughly familiarised with the process and the
official MQM annotation guidelines. These guidelines offer detailed
instructions for annotation within the MQM framework.

Both annotators have been very involved in the education sector for decades.
One of the annotators has edited numerous English-language and Irish-language
books during her career as a university lecturer. The second annotator has a
PhD in Irish-language place names. In addition, he has written numerous books
in both English and Irish. Given their experience and strong language
backgrounds, they were well-equipped to handle the task at hand.

Using a test set of 20 randomly selected sentences, the annotators were
presented with the English source text, an Irish reference translation and the
two unannotated system outputs: one generated using an RNN model and the other
created using a Transformer model. Potential bias was removed by using blind
annotation such that annotators did not know which model the translation
output came from. The annotators worked independently of each other but were
occasionally in contact to discuss the process and how to approach difficult
sentences.

Translations from the RNN and the Transformer system were annotated by both
annotators, meaning that each system translated the same 20 sentences and each
annotator annotated the resulting 40 translated sentences (20 source sentences
for 2 MT systems), producing a total of 80 annotated sentences. The annotated
dataset is publicly available on
GitHub.333https://github.com/seamusl/isfeidirlinn

Once the annotation data were extracted, each annotator analysed the output to
determine the performance of each system for each error category.

##### Inter-Annotator Agreement

Low inter-annotator agreement (IAA) scores are a common problem experienced
when using manual MT evaluation approaches such as MQM [76, 25].To determine
the validity of the findings of our research, it is important to check the
level of agreement between our annotators [8].

Cohen’s $kappa$ ($k$) [30] was used to determine inter-annotator agreement.
The agreement was calculated based on the annotations of each system, with the
agreement being observed at the sentence level. With this approach, the
differences in agreement across systems were explored and we also gained a
high-level view of overall agreement between the annotators. Furthermore,
Cohen’s $kappa$ was calculated separately for every error type and the
findings are outlined in Table 5.4.

Table 5.4: Inter-annotator agreement using Cohen values Error type | RNN | NMT |
---|---|---|---
Non-translation | 1.0 | 1.0 |
Accuracy | 1.0 | 1.0 |
Addition | 1.0 | 1.0 |
Omission | 1.0 | 1.0 |
Mistranslation | -0.071 | 1.0 |
Untranslated text | 0.0 | 1.0 |
Fluency |  |  |
Punctuation | 0.651 | 1.0 |
Spelling | 0.0 | 0.0 |
Grammar | 0.867 | 0.895 |
Register | 1.0 | 1.0 |
Inconsistency | 1.0 | 1.0 |
Character Encoding | 1.0 | 1.0 |

###  5.6 Empirical Evaluation

####  5.6.1 Experimental Setup

##### Datasets

The performance of the Transformer and RNN approaches is evaluated on a
publicly available EN$\leftrightarrow$GA parallel dataset from the Directorate
General for Translation
(DGT).444https://ec.europa.eu/info/departments/translation The Joint Research
Centre of the DGT has made all its translation memory (i.e. sentences and
their professionally produced translations) available, which covers the
official European Union languages [107]. Included in the training data are
parallel texts from the Digital Corpus of the European Parliament (DCEP) and
the DGT. Crawled data, from sites of a similar domain, are also incorporated.
This dataset is broadly categorised as generic and is publicly available.

##### Infrastructure

Model development was conducted using local workstations, each of which was
built with an AMD Ryzen 7 2700X processor, 16GB memory, a 256SSD and an NVIDIA
GeForce GTX 1080 Ti.

In addition, a Google Colab Pro subscription enabled rapid prototype
development and created zero-emission models. The available computing power of
the Google Cloud was much higher than our local infrastructure and provided
servers with 16GB graphic cards (NVIDIA Tesla P100 PCIe) and up to 27GB of
memory [19]. Larger Transformer models were built on local infrastructure
since long builds timed out on Colab due to Google restrictions. The Pytorch
implementation of OpenNMT 2.0, an open-source toolkit for NMT [56], was used
to train all MT models.

##### Metrics

The performance of all models was evaluated using the automated metrics of
BLEU [88], TER [105] and ChrF [90]. Case-insensitive BLEU scores are reported
at the corpus level.

####  5.6.2 Automatic Evaluation Results

##### Performance of Subword Models

The impact that choice of subword model has on translation is highlighted in
Tables 5.5 and 5.6. Incorporating any subword model type led to improvements
in model accuracy when training both RNN and Transformer architectures.

Table 5.5: RNN performance on DGT dataset of 52k Lines. There were zero carbon emissions in building these models since smaller RNN models were trained on Google Colab servers, which are carbon-neutral. Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps | Runtime (h) | kgCO2
---|---|---|---|---|---|---
dgt-rnn-base | 52.7 | 0.42 | 0.71 | 75k | 4.47 | 0
dgt-rnn-bpe8k | 54.6 | 0.40 | 0.73 | 85k | 5.07 | 0
dgt-rnn-bpe16k | 55.6 | 0.39 | 0.74 | 100k | 5.58 | 0
dgt-rnn-bpe32k | 55.3 | 0.39 | 0.74 | 95k | 4.67 | 0
dgt-rnn-unigram | 55.6 | 0.39 | 0.74 | 105k | 5.07 | 0
Table 5.6: Transformer performance on 52k DGT dataset. The highest-performing model uses 2 attention heads. All other models use 8 attention heads. Transformer models were long-running builds, which had to be carried out on local servers. Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps | Runtime (h) | kgCO2
---|---|---|---|---|---|---
dgt-trans-base | 53.4 | 0.41 | 0.72 | 55k | 14.43 | 0.81
dgt-trans-bpe8k | 59.5 | 0.34 | 0.77 | 200k | 24.48 | 1.38
dgt-trans-bpe16k | 60.5 | 0.33 | 0.78 | 180k | 26.90 | 1.52
dgt-trans-bpe32k | 59.3 | 0.35 | 0.77 | 100k | 18.03 | 1.02
dgt-trans-unigram | 59.3 | 0.35 | 0.77 | 125k | 21.95 | 1.24

A baseline RNN model, illustrated in Table 5.5, achieved a BLEU score of 52.7,
whereas the highest-performing BPE variant, using a 16k vocabulary, recorded
an improvement of nearly three points, with a score of 55.6.

In the context of Transformer architectures, highlighted in Table 5.6, the use
of subword models delivers significant performance improvements. The
performance gains for Transformer models are much higher than the improvements
recorded by the RNN models. A baseline Transformer model achieves a BLEU score
of 53.4, whereas a Transformer model, with a 16k BPE subword model, has a
score of 60.5, representing a BLEU score improvement of 13% at 7.1 BLEU
points.

For translating into a morphological-rich language, such as Irish, the ChrF
metric has proven successful in showing a strong correlation with human
translation [106]. In the context of our experiments, this worked well in
highlighting the performance differences between RNN and Transformer
architectures.

##### Transformer Performance Compared with RNN

The performance of RNN models is contrasted with the Transformer approach in
Figures 5.3 and 5.4. Transformer models, as anticipated, outperformed all
their RNN counterparts. It is interesting to note the impact of choosing the
optimal vocabulary size for BPE subword models. Choosing a BPE vocabulary of
16k words yields the highest performance.

Furthermore, the TER scores highlighted in Figure 5.4 reinforce the findings
that using 16k BPE subword models on Transformer architectures leads to a
better translation performance. The TER score for the 16k BPE Transformer
model is significantly better (0.33) when compared with the baseline
performance (0.41).

Figure 5.3: BLEU performance for all model architectures. The use of a BPE
subword model improved translation performance in all cases. The best-
performing model was built using a 16k BPE subword model on a Transformer
architecture.  Figure 5.4: TER performance for all model architectures. The
highest-performing model uses a 16k BPE subword model on a Transformer
architecture. In all instances, incorporating a subword model improves TER.

####  5.6.3 Human Evaluation Results

The aggregate total of errors found by annotators for each system is
highlighted in Table 5.7. Looking at the aggregate data alone, it is evident
that both annotators have judged that the RNN system contains more errors and
that the NMT system contains fewer errors.

Table 5.7: Total errors found by each annotator using the MQM metric. | Annotator 1 | Annotator 2
---|---|---
System | RNN | Transformer | RNN | Transformer
Total Errors | 41 | 23 | 43 | 23

While such a high-level view is instructive in determining which system is
better, it lacks the granularity required to pinpoint the linguistic aspects
of how these translations can be improved. To achieve a deeper insight, a
fine-grained analysis of the error types was conducted, the results of which
are displayed in Table 5.8. Categorised by error type, the sum of error tags
by each annotator for each system is outlined.

Table 5.8: Transformer and RNN approach compared using concatenated annotation data across both annotators. In all MQM error categories, the Transformer architecture performs better, apart from a tie in the omission category. | RNN | NMT
---|---|---
Error Type | Error | Error
Non-translation | 0 | 0
Accuracy |  |
Addition | 10 | 4
Omission | 12 | 12
Mistranslation | 26 | 14
Untranslated text | 4 | 1
Fluency |  |
Punctuation | 5 | 4
Spelling | 1 | 0
Grammar | 20 | 11
Register | 2 | 0
Inconsistency | 2 | 0
Character Encoding | 0 | 0
Total errors | 82 | 46

###  5.7 Environmental Impact

The environmental impact of all aspects of computing has received increased
research interest in recent times. Much of this effort has concentrated on
NMT’s carbon footprint [51, 15]. To assess the environmental impact of our NMT
models, we tracked energy consumption during their development.

Prototype model development was carried out using Google Colab which is a
carbon-neutral platform [62]. However, longer running Transformer experiments
were conducted on local servers using 324 gCO2 per
kWh555https://www.seai.ie/publications/Energy-in-Ireland-2020.pdf [100]. The
net result was just under 10 kgCO2, created for a full run of model
development. Models developed during this study will be reused for ensemble
experiments in future work.

The environmental costs of our model development were tracked to serve as a
benchmark for future work. Awareness of such costs will impose a discipline on
our work, such that we opt for carbon-neutral cloud providers. In cases where
models are developed on local infrastructure, this will encourage the use of
more efficient GPUs and the utilisation of techniques that result in faster
builds.

###  5.8 Discussion

Validation accuracy and model perplexity (PPL) in developing the baseline and
optimal Transformer models are illustrated in Figures 5.5 and 5.6. Training a
Transformer model with a 16k BPE subword model boosted the validation accuracy
by over 8% compared to its baseline.

Figure 5.5: Transformer baseline. Figure 5.6: Transformer 16k BPE subword
model

Rapid convergence was observed while training the baseline model such that
little accuracy improvement occurred after 20k steps. Including a subword
model led to slower converging models, with only marginal gains recorded after
60k steps. Examining Figures 5.5 and 5.6, we can see that PPL achieves a lower
global minimum when the Transformer approach is used with a 16k BPE subword
model. The PPL global minimum (2.7) is over 50% lower than the corresponding
PPL for the Transformer base model (5.5). This finding illustrates that
choosing an optimal subword model delivers significant performance gains.

Translation engine performance, at the corpus level, was benchmarked against
Google Translate’s666https://translate.google.com/ EN$\leftrightarrow$GA
translation service, which is freely available on the internet. Four random
samples were selected from the English source test file and are presented in
Table 5.9. Translation of these samples was carried out on the optimal
Transformer model and using Google Translate. Case-insensitive, sentence-level
BLEU scores were recorded and are presented in Table 5.10. It must be
acknowledged that this comparison is not entirely valid given that Google does
not have access to our training data, nor do we have unlimited access to the
Google Cloud infrastructure. Nonetheless, the results are encouraging and
indicate a good performance by our translation models on the DGT dataset.

Table 5.9: Random samples of human reference translations from the test dataset. Source Language (English) |  Reference Human Translation (Irish)
---|---
A clear harmonised procedure, including the necessary criteria for disease–free status, should be established for that purpose. |  Ba cheart nós imeachta comhchuibhithe soiléir, lena n-áirítear na critéir is gá do stádas saor ó ghalar, a bhunú chun na críche sin.
the mark is applied anew, as appropriate. |  déanfar an mharcáil arís, mar is iomchuí.
If the court decides that a review is justified on any of the grounds set out in paragraph 1, the judgment given in the European Small Claims Procedure shall be null and void. |  Má chinneann an chúirt go bhfuil bonn cirt le hathbhreithniú de bharr aon cheann de na forais a leagtar amach i mír 1, beidh an breithiúnas a tugadh sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní go hiomlán.
households where pet animals are kept; |  teaghlaigh ina gcoimeádtar peataí;
Table 5.10: Transformer model compared with Google Translate using random samples from the DGT corpus. Full evaluation of Google Translate’s engines on the DGT test set, with 1.3k lines, generated a BLEU score of 46.3 and a TER score of 0.44. Comparative scores on the test set using our Transformer model, with 2 attention heads and 16k BPE subword model realised 60.5 for BLEU and 0.33 for TER. Transformer (16k BPE) |  BLEU $\uparrow$ |  Google Translate |  BLEU $\uparrow$
---|---|---|---
Ba cheart nós imeachta soiléir comhchuibhithe, lena n-áirítear na critéir is gá maidir le stádas saor ó ghalair, a bhunú chun na críche sin. |  61.6 |  Ba cheart nós imeachta comhchuibhithe soiléir, lena n-áirítear na critéir riachtanacha maidir le stádas saor ó ghalair, a bhunú chun na críche sin. |  70.2
go gcuirtear an marc i bhfeidhme, de réir mar is iomchuí. |  21.4 |  cuirtear an marc i bhfeidhm as an nua, de réir mar is cuí. |  6.6
Má chinneann an chúirt go bhfuil bonn cirt le hathbhreithniú ar aon cheann de na forais a leagtar amach i mír 1, beidh an breithiúnas a thugtar sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní. |  77.3 |  Má chinneann an chúirt go bhfuil údar le hathbhreithniú ar aon cheann de na forais atá leagtha amach i mír 1, beidh an breithiúnas a thugtar sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní |  59.1
teaghlaigh ina gcoimeádtar peataí; |  100 |  teaghlaigh ina gcoinnítear peataí; |  30.2

The optimal hyperparameters selected in this discovery process are identified
in bold in Table 5.1. A higher initial learning rate of 2 coupled with an
average decay of 0.0001 led to longer training times but more accurate models.
Despite setting an early stopping parameter, many of the Transformer builds
continued for the full cycle of 200k steps over periods of 20+ hours.

Training Transformer models with a reduced number of attention heads led to a
marginal improvement in translation accuracy with a smaller corpus. Our best-
performing model achieved a BLEU score of 60.5 and a TER score of 0.33 with 2
heads and a 16k BPE subword model. By comparison, using 8 heads with the same
architecture and dataset yielded 60.3 for BLEU and 0.34 in terms of TER.

Transformer models developed, using state-of-the-art techniques, were
evaluated as part of the LoResMT2021 Shared Task [85]. Models developed using
our approach, as outlined above, were entered into the competition, and the
highest-performing EN$\rightarrow$GA direction system was submitted by our
team (ADAPT) [64].

####  5.8.1 Inter-Annotator Reliability

In Cohen’s original article [30], the interpretation of specific $k$ scores is
clearly outlined. There is no agreement with values $\leq$ 0, none to slight
agreement when scores are in the range of 0.01–0.20, fair agreement is
represented by 0.21–0.40, 0.41–0.60 is moderate agreement, 0.61–0.80 is
substantial agreement, and 0.81–1.00 is almost perfect agreement.

The literature [79] recommends a minimum of 80% agreement for good inter-
annotator agreement. As illustrated in Table 5.4, there is almost perfect
agreement between the annotators when evaluating output from the NMT models.
In the case of the RNN outputs, there is disagreement in the mistranslation
category but agreement in all other categories. Given these scores, we have a
high degree of confidence in our human evaluation of both the RNN and NMT
outputs.

####  5.8.2 Performance of Models Relative to Google

Using standard Transformer parameters, such as a batch size of 2048 and
setting the number of encoder and decoder layers to 6, were observed to
perform well. Increasing the regularisation dropout to 0.3 and reducing hidden
neurons to 256 improved translation performance. Consequently, these values
were selected when building all Transformer models.

####  5.8.3 Linguistic Observations

A linguistic analysis of the outputs from the Transformer-optimised model is
illustrated in Table 5.11. The English language source sentences and their
Irish language translations are presented. Sentences have been selected from
the fine-grained human evaluation since they highlight some of the key error
types that are encountered. The analysis focuses on the shortcomings of our
model outputs, which fall into the following categories: interpretative
meaning, core grammatical errors and commonly used irregular verbs. Finally,
using the human evaluation metrics of SQM and MQM, the performance of an RNN
approach is contrasted with that of the Transformer approach.

##### Interpreting Meaning

The generic Irish verb “déan” (to do or to make) is used to express more
precise concepts such as “to conduct”, “to put into effect” or “to carry out”.
Both the RNN and Transformer systems make use of “déan” in a generic way, but
they fail to capture the refinement of the concept expressed in each of these
meanings. An example of this problem is illustrated in GA-1 in Table 5.11. In
this context, a more natural and intuitive translation to capture the
expression “to conduct” would be to substitute “a dhéanamh” with “a sheoladh”.

A similar lack of refinement from both systems is also found in the usage of
other words. For example, “cuid” (part) is used to translate “operative part”
in GA-2. However, a more precise interpretation would be the usage of “gné”,
leading to the correct translation “gné oibríochtúil” i.e., “operative part”.

Another example where the translation models failed to correctly interpret the
true sense of an English source word into a corresponding Irish translation
can be seen in GA-3. The Irish verb “Mainnigh” meaning “to default” would not
be used in the context of the source text in EN-3. Using the Irish verb
“teip”, meaning “to fail”, is the correct translation of the idea “fails to
meet the performance requirements”: “má theipeann an t-oibreoir na ceanglais
feidhmíochta a chomhlíonadh.” This error was observed in both the RNN and
Transformer model outputs.

Table 5.11: Linguistic analysis of system outputs. Sources of errors are flagged in blue and in red. Type |  Sentence
---|---
EN-1 |  The lead supervisory authority may request at any time other supervisory authorities concerned to provide mutual assistance pursuant to Article 61 and may conduct joint operations pursuant to Article 62, in particular for carrying out investigations or for monitoring the implementation of a measure concerning a controller or processor established in another Member State.
GA-1 |  Féadfaidh an príomhúdarás maoirseachta iarraidh, tráth ar bith, ar bith eile lena mbaineann cúnamh frithpháirteach a chur ar fáil de bhun Airteagal 61 agus féadfaidh sé oibríochtaí comhpháirteacha a dhéanamh de bhun Airteagal 62, go háirithe maidir le himscrúduithe a dhéanamh nó maidir le faireachán a dhéanamh ar chur chun feidhme beart i ndáil le rialaitheoir nó próiseálaí atá bunaithe i mBallstát eile.
EN-2 |  The Office shall mention the judgment in the Register and shall take the necessary measures to comply with its operative part.
GA-2 |  Luafaidh an Oifig an breithiúnas sa Chlár agus glacfaidh sí na bearta is gá chun cloí lena chuid oibríochtúil.
EN-3 |  The competent authority may at any time wholly or partially suspend or terminate the contract awarded under this provision if the operator fails to meet the performance requirements.
GA-3 |  Féadfaidh an t-údarás inniúil an conradh a dámhadh faoin bhforáil seo a chur ar fionraí nó a fhoirceannadh go hiomlán nó go páirteach má mhainníonn an t-oibreoir na ceanglais feidhmíochta a chomhlíonadh.
EN-4 |  This Directive shall enter into force on the day following that of its publication in the Official Journal of the European Union.
GA-4 |  Tiocfaidh an Treoir seo i bhfeidhm an lá tar éis lá a fhoilsithe in Iris Oifigiúil an Aontais Eorpaigh.
EN-5 |  Such special measures are interim in nature, and shall not be subject to the conditions set out in Article 7(1) and (2).
GA-5 |  Tá bearta speisialta den sórt sin eatramhach, agus ní bheidh said faoi réir na gcoinníollacha a leagtar amach in Airteagal 7(1) agus (2) iad.

##### Core Grammatical Errors

Grammatical mistakes in the form of the misuse of lenitions (e.g., GA-4),
incorrect pronouns (e.g., GA-5) and register errors (e.g., GA-5) were observed
in both translation architectures. However, as is evident from both the
automatic and MQM evaluations, there were far fewer errors with the
Transformer model. Evidence of this can be seen in Table 5.11. In the case of
GA-4, the RNN model included the lenition in “a foilsithe”, whereas the
Transformer model correctly removed “h”. The correct use of the feminine noun
“treoir” requires the removal of “h” in “fhoilsithe”.

The misuse of pronouns was observed in the RNN translation model and, to a
lesser degree, in the Transformer model. In the case of GA-5, the RNN’s
incorrect use of the pronoun “ní bheidh siad” (they will not) is illustrated,
whereas the Transformer approach used the correct form “ní bheidh sé” (he will
not).

Within the same sentence, GA-5, there is also evidence of a register error. In
the English source text EN-5, the use of “shall not be subject to” expresses a
stipulation. This is not registered in the Irish translation of “ní bheidh
said”, which simply, and less forcefully, means “they will not”. This
incorrect use of register was observed with both the RNN and the Transformer
approaches. A more formal and closer interpretation of the English source
would be the use of the imperative mode: “ná bídís” (let it not be).

##### Commonly-Used Irregular Verbs

One of the main inadequacies observed in both the RNN and Transformer systems
is a lack of refinement of verbal usage, particularly when using the verbs
“déan” (to do or to make ) and “bí” (to be). As in many languages, the fact
that these are possibly the two most universally used verbs in Irish further
exacerbates the problem. An illustration of this problem can be seen in the
output GA-1, which highlights the incorrect usage of “déan”. Similarly, GA-5
demonstrates how the system misinterprets the usage of the verb “bí”, e.g.,
“ní bheidh said”.

##### Performance of RNN Approach Relative to Transformer Approach

There is a strong correlation between automatic and human evaluation of the
translation systems that we developed. The automatic BLEU scores are
contrasted with the human evaluation scores for both the RNN and Transformer
models in Table 5.12.

Table 5.12: Transformer approach compared to the RNN approach across all metrics for the DGT dataset. The results from our human evaluation, using SQM and MQM metrics, validate the BLEU automatic evaluation results. Approach |  |  BLEU $\uparrow$ |  SQM $\uparrow$ |  MQM $\uparrow$
---|---|---|---|---
Transformer |  |  60.5 |  4.53 |  77.92
RNN |  |  52.7 |  3.30 |  43.05

####  5.8.4 Limitations of the Study

Certain aspects of this study could be further developed, given more time and
resources. Although there is a high inter-annotator agreement, it would help
to have more annotators. In addition, the human evaluation of a greater number
of lines, coupled with a more detailed MQM taxonomy, may provide greater
insight into the MT outputs. This would help in uncovering other aspects, such
as how gender is handled by the MT models.

###  5.9 Conclusions and Future Work

With this research, we have presented the first human evaluation study that
compares the output of EN$\rightarrow$GA RNN systems with that of Transformer-
based EN$\rightarrow$GA systems. Automatic metrics were shown to differentiate
the systems and highlighted that Transformer models are superior to RNN
models. In our paper, we demonstrated that a random search approach to HPO
enabled the development of high-performing translation models. We have shown
there is a high level of correlation between a human evaluation and an
automatic approach. Both the automatic metrics and our human evaluation
demonstrated that the Transformer-based system is the most accurate.

The importance of selecting hyperparameters when training low-resource
Transformer models was also demonstrated. By increasing dropout and reducing
the number of hidden-layer neurons, our models performed significantly better
than Google Translate and our baseline models.

We have demonstrated that choosing the correct subword models is an important
performance driver for low-resource MT. Within the context of low-resource
EN$\rightarrow$GA translations, we achieved optimal performance on a 55k
generic corpus when a Transformer architecture with a 16k BPE subword model
was used. Improvements in the performance of our optimised Transformer models
were observed across all key indicators, namely, PPL was achieved at a lower
global minimum, with a lower post-editing effort and a higher translation
accuracy.

As part of future work, steps can be taken to deal with the inadequacies
highlighted in our linguistic analysis. The issue of misusing common irregular
verbs could be addressed by fine-tuning our models with a dataset specifically
tailored for that purpose. Similarly, fine-tuning after the careful selection
of training data would also reduce the register errors encountered in our
linguistic analysis. As it is difficult to train systems for all
eventualities, using post-editing tools would be the best approach to
correcting core grammatical errors involving pronouns, lenitions and
lemmatization.

Author Contributions: All authors have read and agreed to the published
version of the manuscript.

Funding: This work was supported by ADAPT, which is funded under the SFI
Research Centres Programme (Grant 13/RC/2016) and is co-funded by the European
Regional Development Fund. This research was also funded by the Munster
Technological University.

Informed Consent Statement: Informed consent was obtained from all subjects
involved in the study.

Data Availability Statement: The data presented in this study are openly
available at https://github.com/seamusl/isfeidirlinn

Acknowledgments: We would like to thank the annotators, Dr Éamon Lankford and
Ms. Máirín Lankford for their meticulous work in annotating the system
outputs.

Conflicts of Interest: The authors declare no conflict of interest. The
funders had no role in the design of the study; in the collection, analyses,
or interpretation of data; in the writing of the manuscript, or in the
decision to publish the results.

Glossary

Irish terms referenced and used in this manuscript:
Déan To do or to make Bí To be Ná bídís Let it not be Ní bheidh siad They will
not Ní bheidh sé He will not

##  Chapter 6 Design of an Open-Source Architecture for NMT

###  6.1 Context

With RQ4, the process of how NMT development, evaluation, and deployment could
be streamlined for both developers and translators was considered. The purpose
of this short paper for the inaugural CrowdMT workshop at Tampere was to
showcase the adaptNMT application [69]. As a workshop paper, this chapter is
effectively a compressed version of the longer and more comprehensive journal
paper outlined in Chapter 7. Readers who wish for an in-depth discussion of
the adaptNMT application are encouraged to go directly to Chapter 7, adaptNMT:
Open-Source Neural Machine Translation.

By simplifying NMT model development, building and deploying NMT models can be
streamlined. This approach recognises that newcomers to the field face
challenges in setting up the development environment and creating the
necessary data splits for training, validation, and testing. The community is
invited to contribute new ideas and improvements, and a spirit of
collaboration will be fostered among researchers. The research contribution is
to provide an open-source approach which can lead to the continuous
improvement and evolution of the adaptNMT project.

Design of an Open-Source Architecture for Neural Machine Translation

Séamus Lankford
Haithem Afli
Andy Way

crowdMT23: Workshop on Open Community-Driven Machine Translation

June 15, 2023

Tampere, Finland.

ADAPT Centre

Dublin City University

Ireland

https://aclanthology.org/2023.crowdmt-1.2.pdf

###  6.2 Abstract

adaptNMT is an open-source application that offers a streamlined approach to
the development and deployment of recurrent neural networks and Transformer
models. This application is built upon the widely-adopted OpenNMT ecosystem,
and is particularly useful for new entrants to the field, as it simplifies the
setup of the development environment and creation of training, validation, and
test splits. The application offers a graphing feature that illustrates the
progress of model training and employs SentencePiece for creating subword
segmentation models. Furthermore, the application provides an intuitive user
interface that facilitates hyperparameter customisation. Notably, a single-
click model development approach has been implemented, and models developed by
adaptNMT can be evaluated using a range of metrics. To encourage eco-friendly
research, adaptNMT incorporates a green report that flags the power
consumption and kgCO2 emissions generated during model development. The
application is freely available.111http://github.com/adaptNMT

###  6.3 Credits

This research is supported by Science Foundation Ireland through the ADAPT
Centre (Grant 13/RC/2106) (www.adaptcentre.ie) at Dublin City University. This
research was also funded by the Munster Technological University.

###  6.4 Introduction

Explainable Artificial Intelligence (XAI) [12] aims to ensure the outcomes of
AI solutions are easily comprehensible to humans. In light of this goal,
adaptNMT has been developed to provide users with a form of explainable neural
machine translation (XNMT). The typical neural machine translation (NMT)
process comprises several independent stages, including setting up the
environment, preparing the dataset, training subword models, setting the
parameters and training the main models, evaluating and deploying them. By
adopting a modular approach, this framework has established an effective NMT
model development process that caters to both technical and non-technical
practitioners in the field. To address the environmental impact of building
and running large AI models [47, 52], we have also produced a “green report”
that calculates carbon emissions. While primarily intended as an information
aid, this report will hopefully encourage the development of reusable and
sustainable models.

This research endeavours to create models and applications that address the
challenges of language technology, which will be particularly beneficial for
those new to the field of Machine Translation (MT) and those seeking to learn
more about NMT.

The application is built on OpenNMT222https://opennmt.net [56] and thus
inherits all of its features. Unlike many NMT toolkits, a command line
interface is not used, and the interface is designed and fully implemented in
Google Colab.333https://colab.research.google.com For both educational and
research purposes, a cloud-hosted solution like Colab is often more user-
friendly. Additionally, the training of models can be monitored and controlled
on a mobile phone using Google Colab’s responsive design, which is useful for
long-run builds. The adaptNMT framework also includes GUI controls that allow
for the customisation of all crucial parameters needed for NMT model training.

The application can be run in local mode to utilise existing infrastructure or
hosted mode for rapid infrastructure scaling. A deploy function is also
included to allow for the immediate deployment of trained models.

This paper begins by presenting background information on NMT and NMT tools in
Section 6.5, followed by a detailed description of the adaptNMT architecture
and its key features in Section 6.6. The system is discussed in Section 6.7
before concluding with a discussion of future work in Section 6.8. A more in-
depth system description, coupled with an empirical evaluation of models
developed using the application, is outlined in a separate paper [68].

###  6.5 Related Work

####  6.5.1 NMT

In addition to the ongoing research dedicated to developing state-of-the-art
(SOTA) NMT models, comprehensive descriptions of this technology are readily
available in the literature, making it accessible to individuals who are new
to the field or have limited technical expertise [116].

NMT has benefited from the availability of large parallel corpora, leading to
the development of high-performing MT models. The field of MT has experienced
significant advancements through the application of NMT, particularly after
the introduction of the Transformer [114] architecture, which has resulted in
SOTA performance across multiple language pairs [20, 21, 65, 64, 63, 66].

####  6.5.2 NMT Tools

In essence, adaptNMT is an IPython wrapper built on OpenNMT, enabling it to
benefit from OpenNMT’s extensive feature set and continuous code maintenance.
However, adaptNMT takes abstraction to a higher level than OpenNMT, with a
greater focus on usability, particularly for newcomers. As a result, adaptNMT
facilitates easy and fast deployment, offering features such as more pre-
processing, as well as GUI control over model creation. Moreover, it
incorporates green features in line with current research efforts towards
smaller models with reduced carbon footprints, making it suitable for
educational and research environments alike.

Other commonly used frameworks for developing NMT systems include
FAIRSEQ444https://github.com/facebookresearch/fairseq [87], an open-source
sequence modelling toolkit based on PyTorch that allows for training models
for translation, summarisation, and language modelling.
Marian555https://marian-nmt.github.io [53], on the other hand, is an NMT
framework based on dynamic computation graphs and developed using C++. OpenNMT
is an open-source NMT framework that has been widely adopted in the research
community and covers the entire MT workflow from data preparation to live
inference.

###  6.6 Architecture of adaptNMT

After providing a general overview of NMT and NMT development systems, we
introduce the adaptNMT tool, which enables users to configure the components
of the NMT development process. The platform’s system architecture is depicted
in Figure 6.1. The tool is built as an IPython notebook and leverages the
Pytorch implementation of OpenNMT for training models. Additionally,
SentencePiece is used to train subword models. Using a Jupyter notebook
facilitates sharing the application with other members of the MT community,
and the application’s setup is simplified since all necessary packages are
downloaded dynamically as the application runs.

Figure 6.1: Proposed architecture for adaptNMT: a language-agnostic NMT
development environment. The system is designed to run either in the cloud or
using local infrastructure. Models are trained using parallel corpora.
Visualisation and extensive logging enable real-time monitoring. Models are
developed using vanilla RNN-based NMT, Transformer-based approaches or
transfer learning using a fine-tuning approach. Translation and evaluation can
be carried out using either single models or ensembles.

The system has two deployment options: running it locally or as a Colab
instance via Google Cloud. To build translation models, the system requires
parallel text corpora for both the source and target languages. A Tensorboard
visualization allows for real-time monitoring of the model training process.
At run time, users can select to use the system for either model building or
translation services, or both. Additionally, as depicted in Figure 6.1, the
system enables the generation of an ensemble output during translation.
Finally, trained models can be easily deployed to a pre-configured location.

####  6.6.1 adaptNMT

The application may be run as an IPython Jupyter notebook or as a Google Colab
application. Given the ease of integrating large Google Drive storage into
Colab, the application has been used exclusively as a Google Colab application
for our experiments.

##### Initialisation and Logging

Initialisation enables connection to Google Drive to run experiments,
automatic installation of Python, OpenNMT,666https://opennmt.net
SentencePiece,777https://github.com/google/sentencepiece Pytorch and other
applications. The visualisation section enables real-time graphing of model
development. All log files are stored and can be viewed to inspect training
convergence, the model’s training and validation accuracy and changes in
learning rates.

##### Modes of Operation

There are two modes of operation: local and cloud. In local mode, the
application is run so that models are built using the user’s local GPU
resources. The option to use cloud mode enables users to develop models using
Google’s GPU clusters. For shorter training times, the unpaid Colab option is
adequate. However, for a small monthly subscription, the Google Colab Pro
option is worthwhile since users have access to improved GPU and compute
resources. Furthermore, using Google Cloud may be considered as the “green
option” since its platform uses 100% renewables [62].

##### Customisation of Models

The system has been developed to allow users to select variations to the
underlying model architecture. A vanilla RNN or Transformer approach may be
selected to develop the NMT model. The customisation mode enables users to
specify the exact parameters required for the chosen approach. One of the
features, AutoBuild, enables a user to build an NMT model in three simple
steps: (i) upload source and target files, (ii) select RNN or Transformer, and
(iii) click AutoBuild.

##### Use of Subword Segmentation

In the NMT development process, users can specify the type of optimiser for
learning and choose from different subword models. The subword model
functionality allows for the selection of a subword model type and the choice
of vocabulary size, currently offering either a SentencePiece unigram or a
SentencePiece BPE model.

A user may upload a dataset which includes the train, validation and test
splits for both source and target languages. In cases where a user has not
already created the required splits for model training, single source and
target files may be uploaded. Automated splitting of the uploaded dataset into
the train, validation, and test files is then performed based on the user’s
chosen split ratio.

Given that building NMT models typically demands long training times, an
automatic notification feature is incorporated that informs the user by email
when model training has been completed.

##### Translation and Evaluation

The application supports not only the training of models but also the
translation and evaluation of model performance. For translation using pre-
built models, users can specify the model name as a hyperparameter which is
subsequently used to translate and evaluate the test files. The option for
creating an ensemble output is also available, with users simply naming the
models to be used in generating the ensemble output.

Once the system has been built, the user can select the model to be used for
translating the test set. While human evaluation is often considered the most
insightful approach for evaluating translation quality, it can be limited by
factors such as availability, cost, and subjectivity. Thus, automatic
evaluation metrics are frequently employed, particularly by developers
monitoring the incremental progress of systems. A further discussion on the
advantages and disadvantages of human and automatic evaluation is available in
the literature [115].

Several automatic evaluation metrics provided by
SacreBleu888https://github.com/mjpost/sacrebleu [91] are used: BLEU [88], TER
[105] and ChrF [90]. Translation quality can also be evaluated using Meteor
[34] and F1 score [80]. Note that BLEU, ChrF, Meteor and F1 are precision-
based metrics, so higher scores are better, whereas TER is an error-based
metric and lower scores indicate better translation quality. Evaluation
options available include standard (truecase) and lowercase BLEU scores, a
sentence-level BLEU score option, ChrF1 and ChrF3.

There are three levels of logging: model development logs for graphing,
training console output and experimental results. A references section
outlines resources which are relevant to developing, using and understanding
adaptNMT. Validation during training is currently conducted using model
accuracy and perplexity (PPL).

####  6.6.2 Infrastructure

Rapid prototype development is possible through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16GB graphic cards and up to 27GB of
memory when available.

###  6.7 Discussion

Numerous tools have been developed to assess the carbon footprint of NLP [11].
The notion of sustainable NLP has also gained momentum as an independent
research track, with high-profile conferences such as the EACL 2021 Green and
Sustainable NLP track dedicating resources to this
area.999https://2021.eacl.org/news/green-and-sustainable-nlp

Given these developments, we have incorporated a “green report” into adaptNMT
that logs the kgCO2 generated during model development. This aligns with the
industry’s increasing focus on quantifying the environmental impact of NLP. In
fact, it has been demonstrated that high-performing MT systems can be
developed with much lower carbon footprints, leading to significant energy
cost savings for a real translation company [51].

The risks associated with relying on large language models (LLMs) have been
well-documented in the literature. The discussion surrounding these models
emphasises not only their environmental impact but also the inherent biases
and dangers they pose for low-resource languages [15]. It is important to note
that smaller, in-domain datasets can yield high-performing NMT models, and the
adaptNMT framework makes this approach easily accessible and understandable.

###  6.8 Conclusion and Future Work

We have introduced adaptNMT, an application that manages the entire NMT model
development, evaluation, and deployment workflow.

As for future work, our development efforts will be directed towards
incorporating new transfer learning methods and improving our ability to track
environmental costs. We will integrate modern zero-shot and few-shot
approaches, as seen in the GPT3 [23] and Facebook LASER [7] frameworks. While
the existing adaptNMT application is focused on customising NMT models, we
will also develop a separate application, adaptMLLM [67], for fine-tuning
multilingual language models (MLLMs) and LLMs. In particular, adaptMLLM will
cater for low-resource language pairs covered by Meta’s NLLB [32] models.

The green report integrated into the application represents our first
implementation of a sustainable NLP feature within adaptNMT. We plan to
enhance this feature by improving the user interface and providing
recommendations on how to develop greener models. As an open-source project,
we invite the community to contribute new ideas and improvements to the
development of this feature.

##  Chapter 7 adaptNMT: Open-Source Neural Machine Translation

###  7.1 Context

The motivation for the research on adaptNMT is to address several key
challenges in the field of NMT and NLP in general. Overall, the research
behind adaptNMT addresses critical issues in NMT development and deployment,
while also promoting sustainable and collaborative practices within the NLP
community.

The primary motivations for this research were initially raised in RQ4 and are
outlined in more detail below:

  * •

Streamlining Development and Deployment Processes: The primary goal of
adaptNMT is to simplify and streamline all aspects of developing and deploying
RNN and Transformer neural translation models. This includes processes like
setting up the development environment, creating data splits, and training the
models.

  * •

Accessibility for Both Developers and Translators: The application is designed
to be user-friendly for individuals with varying levels of technical
expertise. This inclusivity aims to make MT more accessible to a wider
audience, including those who may not have extensive technical backgrounds.

  * •

Support for New Entrants in the Field: adaptNMT is particularly valuable for
newcomers to the field of MT. By simplifying the setup and training process,
it lowers the barrier to entry, allowing more individuals to participate in
NLP research and development.

  * •

Built Upon the OpenNMT Ecosystem: By leveraging the widely adopted OpenNMT
ecosystem, adaptNMT benefits from a strong foundation of established tools and
resources in the NMT community.

  * •

Visualisation and Monitoring of Model Training: The application provides
visualisations to track the progress of model training. This feature aids
researchers and developers in understanding how their models are evolving
during the training process.

  * •

Utilisation of SentencePiece for Subword Segmentation Models: SentencePiece is
employed to create subword segmentation models. This helps in handling
morphologically rich languages and improving the overall performance of the
translation models.

  * •

Intuitive Hyperparameter Customisation: The application offers an intuitive
user interface (UI) for customising hyperparameters. This facilitates
experimentation with different settings to optimise model performance.

  * •

Efficient Model Development with Single-Click Approach: The one-click model
development approach simplifies and accelerates the process of creating and
fine-tuning translation models.

  * •

Comprehensive Model Evaluation and Deployment: Models developed with adaptNMT
can be rigorously evaluated using a range of metrics. Additionally, the
application allows for easy deployment of these models as a translation
service.

  * •

Environmental Considerations: A strong emphasis has been placed on eco-
friendly practices in NLP research. A “green report” feature that tracks the
power consumption and carbon emissions generated has been implemented which
promotes sustainable research practices.

  * •

Demonstrated Success in Shared Tasks: The performance of adaptNMT was
validated by generating an EN$\rightarrow$GA translation model, which achieved
1st place in the LoResMT2021 shared task. This success demonstrates the
practical effectiveness of the application.

  * •

Expanding to Fine-Tuning Large Language Models: While adaptNMT focuses on
customising NMT models, a separate application, adaptMLLM, has been developed
to fine-tune MLLMs and LLMs. This is particularly important for low-resource
language pairs.

adaptNMT: an open-source, language-agnostic development environment for Neural
MT

Séamus Lankford
Haithem Afli
Andy Way

Journal of Language Resources Evaluation, Springer

March 1st 2023

ADAPT Centre

Dublin City University

Ireland

https://link.springer.com/article/10.1007/s10579-023-09671-2

###  7.2 Abstract

adaptNMT streamlines all processes involved in the development and deployment
of RNN and Transformer neural translation models. As an open-source
application, it is designed for both technical and non-technical users who
work in the field of MT. Built upon the widely-adopted OpenNMT ecosystem, the
application is particularly useful for new entrants to the field since the
setup of the development environment and creation of training, validation and
test splits is greatly simplified. Graphing, embedded within the application,
illustrates the progress of model training, and SentencePiece is used for
creating subword segmentation models. Hyperparameter customisation is
facilitated through an intuitive user interface, and a single-click model
development approach has been implemented. Models developed by adaptNMT can be
evaluated using a range of metrics and deployed as a translation service
within the application. To support eco-friendly research in the NLP space, a
green report also flags the power consumption and kgCO2 emissions generated
during model development. The application is freely
available.111http://github.com/adaptNMT

###  7.3 Graphical abstract

Figure 7.1: Graphical abstract summarising the adaptNMT system

###  7.4 Introduction

Explainable artificial intelligence (XAI) [44, 12] seeks to ensure that the
results of AI solutions are easily understood by humans. It is against this
backdrop that adaptNMT has been developed to afford users a form of
explainable neural machine translation (XNMT). The stages involved in a
typical neural machine translation (NMT) process are broken down into a series
of independent steps including environment setup, dataset preparation,
training of subword models, parameterising and training of main models,
evaluation and deployment. This modular approach has created an effective NMT
model development process for both technical and less technical practitioners
in the field. Given the environmental impact of building and running large AI
models [108, 47, 52], we also compute carbon emissions in a “green report”,
primarily as an information aid, but hopefully as a way to encourage reusable
and sustainable model development.

An important part of this research involves developing applications and models
to address the challenges of language technology. It is hoped that such work
will be of particular benefit to newcomers to the field of machine translation
(MT) and in particular to those who wish to learn more about NMT.

To have a thorough understanding of how NMT models are trained, the individual
components and the mathematical concepts underpinning both RNN- and
Transformer-based models are explained and illustrated in this paper. The
application is built upon OpenNMT [56] and subsequently inherits all of its
features. Unlike many NMT toolkits, a command line interface approach is not
used. The interface is designed and fully implemented in Google
Colab.222https://colab.research.google.com For an educational setting, and
indeed for research practitioners, a Colab cloud-
hosted333https://cloud.google.com solution is often more intuitive to use.
Furthermore, the training of models can be viewed and controlled using the
Google Colab responsive website which is ideal for builds with long run times.
GUI controls, also implemented within adaptNMT, enable the customisation of
all key parameters required when training NMT models.

The application can be run in local mode enabling existing infrastructure to
be utilised, or in hosted mode which allows for rapid scaling of the
infrastructure. A deploy function allows for the immediate deployment of
trained models.

This paper is organised by initially presenting background information on NMT
and related work on system-building environments in Section 7.5. This is
followed by a detailed description of the adaptNMT architecture and its key
features in Section 7.6. An empirical evaluation of models is carried out in
Section 7.7. The system is discussed in Section 7.8 before drawing conclusions
and describing future work in Section 7.9. For newcomers to the field, we
suggest going straight to Section 7.6 to examine the platform’s capabilities,
and then discovering more about the various components and their statistical
underpinning in Section 7.5. This can be followed by the remaining sections in
their logical sequence.

###  7.5 Neural Networks for MT

####  7.5.1 Recurrent Neural Network Architecture

Recurrent neural networks (RNNs) [101, 104, 5] are often used for the tasks of
natural language processing (NLP), speech recognition and MT. RNNs, such as
long short-term memory (LSTM) [48], were designed to support sequences of
input data. LSTM models use an encoder-decoder architecture which enables
variable length input sequences to predict variable length output sequences.
This architecture is the cornerstone of many complex sequence prediction
problems such as speech recognition and MT.

RNN models enable previous outputs to be used as inputs through the use of
hidden states. In the context of MT, such neural networks were ideal due to
their ability to process inputs of any length. In the initial stages of NMT,
the RNN encoder-decoder framework was adopted and variable-length source
sentences were encoded as fixed-length vectors [28, 109]. An improvement upon
the basic RNN approach was proposed in [10] which enhanced translation
performance of the basic encoder-decoder architecture by replacing fixed-
length vectors with variable-length vectors. A bidirectional RNN was now
employed to read input sentences in the forward direction to produce forward
hidden states while also producing backward hidden states by reading input
sentences in the reverse direction. This development enabled neural networks
to more accurately process long sentences, which previously had served as
bottlenecks to performance, given their tendency to ‘forget’ words in long
input sequences which are ‘too far away’ from the current word being
processed.

More importantly, [10] introduced the concept of ‘attention’ to the basic RNN
architecture, similar in spirit and intention to ‘alignments’ in the
forerunner to NMT, statistical MT [84]. In attention-augmented NMT, the system
could now pay special heed to the most relevant other source-sentence words
and use them as contextual clues when considering how best to select the most
appropriate target words for translationally ambiguous words in the same
string.

####  7.5.2 Transformer Architecture

Following the introduction of the attention mechanism, a natural line of
investigation was to see whether attention could do most of the heavy lifting
of translation by itself. Accordingly, [114] proposed that “attention is all
you need” in their ‘Transformer architecture’, which has achieved state-of-
the-art (SOTA) performance on many NLP benchmarks by relying solely on an
attention mechanism, removing recurrence and convolution while allowing the
use of much simpler feed-forward neural networks.

Figure 7.2: The Transformer architecture using an encoder-decoder [114]. The
encoder maps an input sequence to the decoder. The decoder generates a new
output by combining the encoder output with the decoder output from the
previous step.

This approach follows an encoder-decoder structure and allows models to
develop a long memory which is particularly useful in the area of language
translation. The task of the encoder is to map an input sequence to a sequence
of continuous representations, which is then passed to a decoder to generate
an output sequence by using the output of the encoder together with the
decoder output from the previous time step. Both the encoder and decoder each
consist of a stack of 6 identical layers, whose structure is illustrated in
Figure 7.2. In the encoder, each layer is composed of two sub-layers: a multi-
head self-attention mechanism and a fully connected feed-forward network. In
the case of the decoder, there are three sub-layers: one which takes the
previous output of the decoder stack, another which implements a multi-head
self-attention mechanism, and the final layer which implements a fully
connected feed-forward network.

####  7.5.3 Attention

As illustrated in Figure 7.3, the attention function can be described as
mapping a query and a set of key-value pairs to an output, where the query,
keys, values, and output are all vectors. The output is computed as a weighted
sum of the values, where the weight assigned to each value is computed by a
compatibility function of the query with the corresponding key, as shown in
Equation 7.1.

Figure 7.3: Multi-head attention in the decoder [114]. In the decoder, a
multi-head layer receives queries from the previous decoder sublayer, and the
keys and values from the encoder output. The decoder can now attend to all
words in the input sequence.

The query, keys and values used as inputs to the attention mechanism are
different projections of the same input sentence (‘self-attention’) and
capture the relationships between the different words of the same sentence.

Both a scaled dot-product attention and a multi-head attention are used in the
Transformer architecture. With scaled dot-product attention, a dot product is
initially computed for each query $q$ with all of the keys $k$. The input
consists of queries and keys of dimension $d_{k}$. Subsequently, each result
is divided by $\sqrt{d_{k}}$ and a Softmax function is applied. The process
leads to the weights which are used to scale the values, $v$.

The Softmax function allows us to perform multiclass classification which
makes it a good choice in the final layer of neural network-based classifiers.
The function forces the outputs of the neural network to a total sum to 1,
which can be viewed as a probability distribution across multiple classes.
Therefore, Softmax is the ideal choice as the output activation function,
given that NMT is essentially a multiclass classification problem where the
output classes represent the words within the vocabulary.

Computations performed by scaled dot-product attention can be efficiently
applied to the entire set of queries simultaneously. To achieve this, the
matrices, $Q$, $K$ and $V$, are supplied as inputs to the attention function:

$attention(Q,K,V)=softmax(QK^{T}/\sqrt{d_{k}})V$ (7.1)

####  7.5.4 NMT

While much research effort concentrates on creating new SOTA NMT models,
excellent descriptions of the technology are also available within the
literature for those starting out in the field, or for those with a less
technical background [39, 116].

The availability of large parallel corpora has enabled NMT to develop high-
performing MT models. Breakthrough performance improvements in the area of MT
have been achieved through research efforts focusing on NMT [10] but the
advent of the Transformer architecture has greatly improved MT performance.
Consequently, SOTA performance has been attained on multiple language pairs
[20, 21, 65, 66, 63].

Similar to many deep-learning approaches, NMT development is underpinned by
the mathematics of probability. At a fundamental level, the goal is to predict
the probabilistic distribution $P(y|x)$ given a dataset $D$, where $x$
represents the source input sentence and $y$ represents the target output
sentence.

Supervised training of an NMT model develops the model weights by comparing
the predicted $P(y|x)$ with the correct $y$ sentences of the training dataset,
$D\textsubscript{Train}$. In evaluating the performance of an NMT model,
automatic evaluation results are determined when the predicted $P(y|x)$
sentences are compared with the correct $y$ sentences of the test dataset,
$D\textsubscript{Test}$.

In adopting a deep learning paradigm, MT inherits the mathematical first
principles which are inherent to this approach. To understand these
principles, the manner in which neural networks model a conditional
distribution is outlined. Furthermore, the encoder-decoder mechanism used for
training NMT models is presented in the modelling subsection, and model
optimisation using training objectives is outlined in the learning subsection.
Finally, the mathematics of how translated sentences are generated is explored
in the inference subsection.

##### Modelling

In NMT, sentence-level translation is modelled using input and output
sentences as sequences. Using this approach, an NMT model implements a
sequence-to-sequence model with a given source sentence, $x=(x_{1},...,x_{s})$
generating a target sentence $y=(y_{1},...,y_{t})$.

In effect, such a sequence-to-sequence NMT model acts as a conditional
language model. The decoder within the model predicts the next word of the
target sentence $y$, while such predictions are conditioned on the source
sentence $x$.

By applying the chain rule, a model’s prediction (i.e. translation $y$ of
length $T$) maximises the probability $P(y|x)$ identified in Equations 7.2 and
7.3:

$P(y|x)=P(y_{1}|x)P(y_{2}|y_{1},x)P(y_{3}|y_{1},y_{2},x)P(y_{T}|y_{1},...,y_{T-1},x)$
(7.2)

$P(y|x)=\prod_{t=1}^{T}P(y_{t}|y_{1},...,y_{T-1},x)$ (7.3)

Prior to Transformer, encoder-decoder models that incorporate RNNs were the
most common method of representing text sequences in NMT. RNNs are networks
which accumulate information composed of similar units repeated over time. In
NMT, a primary function of the RNN encoder is that it encodes text, i.e. it
turns text into a numeric representation. Neurons within an RNN are
illustrated in Figure 7.4.

Figure 7.4: Neurons within an RNN. At the input side, the neuron’s input at
time $t$ is a function of the encoded word (i.e. input vector $x_{t}$) and a
hidden state vector $h_{t-1}$ which contains the previous sequence. The output
generated by the neuron is represented by the vector $O_{t}$.

Decoders unfold the vector representing the sequence state and return text. An
important distinction between an encoder and a decoder is illustrated in
Figure 7.5, where it can be seen that both the encoder hidden state and the
output from the previous decoding state are required by the decoder.

Figure 7.5: Encoder-decoder architecture. The encoder encodes the entire input
sequence into a fixed-length context vector, $c$, by processing input time
steps. The function of the decoder is to read this context vector while
stepping through output time steps.

To kick-start the processing of the decoder, a special token $<start>$ is used
since there is no previous output. The calculations carried out by the encoder
are summarised in Equation 7.4:

$h_{t}=RNN_{ENC}(x_{t},h_{t-1})$ (7.4)

The $RNN_{ENC}$ function is iteratively applied over the input sequence to
generate the final encoder state, $h_{s}$ which is fed to the decoder. The
complete source sentence is effectively represented by $h_{s}$. The decoder
within the model predicts the next word of the target sentence y, while such
predictions are conditional on the source sentence $x$.

The RNN decoder, $RNN_{DEC}$, creates a state vector $s_{t}$ by compressing
the decoding history ${y_{0},…,y_{t-1}}$ which is described in Equation 7.5.
The distribution of target tokens is predicted by a classification layer which
typically uses the Softmax activation function.

$s_{t}=RNN_{DEC}(y_{t-1},s_{t-1})$ (7.5)

##### Learning

It is possible to optimise models using different types of training
objectives, although maximum log-likelihood (MLE) is the most commonly used
method. Given a set of training examples $D=\\{(x^{s},y^{s})\\}_{s=1}^{S}$,
the MLE is maximised according to Equations 7.6 and 7.7.

$\boldsymbol{\hat{\theta}}_{MLE}=\arg\max_{\theta}\\{\mathcal{L}(\theta)\\}$
(7.6)

$\mathcal{L}(\theta)\ =\sum_{s=1}^{S}logP(y^{s}|x^{s});\theta)$ (7.7)

The gradient of $\mathcal{L}$ with respect to $\theta$ is calculated using
back-propagation [97] as an automatic differentiation algorithm for
calculating gradients of the neural network weights, where $\theta$ is the set
of model parameters.

Many NMT approaches implement Stochastic Gradient Descent (SGD) as the
optimisation algorithm for minimising the loss of the predictive model with
regard to the training data. For reasons of computational efficiency, SGD
typically computes the loss function and gradients on a minibatch of the
training set. The standard SGD optimiser updates parameters of an NMT model
according to Equation 7.8, where the learning rate is specified by $\alpha$:

$\theta\leftarrow\theta-\alpha\bigtriangledown\mathcal{L}(\theta)$ (7.8)

There are several alternatives to using SGD for optimisation, among which the
ADAM optimiser has proven popular due to a reduction in training times [55].

Figure 7.6: Beam Search Algorithm

[120]

##### Inference

In the context of NMT, inference should ideally find the target translated
sentence $y$ from the source $x$ which maximises the model prediction
$P(y|x;\theta)$. However, in practice, it is often difficult to find the
translation with the highest probability due to the impractically large search
space. Accordingly, to find a good but not necessarily the very ‘best’ (i.e.
that with the highest probability given the model) translation, NMT usually
relies instead on local search algorithms such as greedy search or beam search
(cf. Figure 7.6). Translations are carried out by default using beam search,
although the option exists to switch to greedy search if needed. This approach
is consistent with many other NMT tools since beam search is a classic local
search algorithm. Using a pre-defined beam width parameter K, the beam search
algorithm keeps only the top-K possible translations as potential candidates.
With each iteration, a new potential translation is formed by combining each
candidate word with a new word. New candidate translations compete with each
other using log probability values to obtain the new top-K most probable
results. This process is continued until the end of the translation process,
and the 1-best translation is output.

####  7.5.5 Subword Models

Translation by its very nature requires an open vocabulary, but restricted
(e.g. 30k, 50k, or 70k) vocabularies are typically used for reasons of
computational efficiency. However, the use of subword models aims to address
this fixed vocabulary problem associated with NMT. The problem manifests
itself in how previously unseen ‘out-of-vocabulary’ (OOV) words are handled.
In such cases, a single ‘UNK’ (for ‘unknown’) token is used to ‘recognise’ the
OOV word. Encoding rare and unknown words into sequences of subword units
significantly reduces the problem and has thus given rise to a number of
subword algorithms.

Optimally, this will be performed via morphological processing [89], but good
quality wide-coverage morphological analysers are not always available.
Therefore it is common practice to use methods such as Byte Pair Encoding
(BPE) [41] to break down rare and previously unseen words into subword models
in order to significantly improve translation performance [103, 60].

Designed for NMT, SentencePiece [61], is a language-independent subword
tokenizer that provides an open-source C++ and a Python implementation for
subword units. An attractive feature of the tokenizer is that SentencePiece
trains subword models directly from raw sentences.

####  7.5.6 NMT Tools

[59] describe their Joey NMT platform444https://github.com/joeynmt/joeynmt as
a minimalist NMT toolkit, based on PyTorch, which is designed especially for
newcomers to the field. Joey NMT provides many popular NMT features in a
simple code base enabling novice users to easily adapt the system to their
particular requirements. The toolkit supports both RNN and Transformer
architectures.

Given that adaptNMT is essentially an IPython wrapper layered on top of
OpenNMT, it inherits all of OpenNMT’s features and continues to benefit from
the work which goes into developing and maintaining its code base. adaptNMT
offers a higher level of abstraction over OpenNMT where the focus is much more
on usability, especially for newcomers to the field. Accordingly, it provides
for easy and rapid deployment by enabling new features such as greater pre-
processing, as well as GUI control over model building. It also contains green
features in line with the current research drive towards smaller models with
lower carbon footprints (cf. Sections 7.7.4 and 7.8). Such features make
adaptNMT suitable for both educational and research environments. The key
features differentiating adaptNMT from Joey NMT are outlined in Table 7.1.

adaptNMT is built upon OpenNMT and subsequently inherits all of its features.
---
| The interface is designed and fully implemented in Google Colab.
---
| Colab is easier to follow for practitioners since each step can be executed
individually.
---
The approach is ideal in education since the progression of the pipeline is
demonstrated.
Training of models can be viewed and controlled using Colab Android or Apple
apps.
| adaptNMT can be run in local mode enabling existing infrastructure to be
utilised or in
---
hosted mode which allows rapid scaling of the infrastructure.
| Colab Pro+ provides individual researchers, or even small teams, the
capacity to build
---
large models on an excellent infrastructure with very little resources.
| GUI controls can split a corpus into training, validation and test datasets.
---
GUI controls are available for hyperparameter customisation in NMT training.
A green report outlines the country-specific kgCO2 generated when training a
model.
Autonotification notifies the user on the completion of training.
A deploy function enables the immediate deployment of trained models.
The functionality of serverNMT is not available within Joey NMT.
Table 7.1: Key features differentiating adaptNMT from Joey NMT

Other popular frameworks for NMT system-building include
FAIRSEQ555https://github.com/facebookresearch/fairseq [87], an open-source
sequence modelling toolkit based on PyTorch, that enables researchers to train
models for translation, summarisation and language modelling.
Marian666https://marian-nmt.github.io [53], developed using C++, is an NMT
framework based on dynamic computation graphs. OpenNMT777https://opennmt.net
[56] is an open-source NMT framework that has been widely adopted in the
research community. The toolkit covers the entire MT workflow from the
preparation of data to live inference.

####  7.5.7 Hyperparameter Optimisation

Hyperparameters are employed to customise machine learning models such as
translation models. It has been shown that machine learning performance may be
improved through hyperparameter optimisation (HPO) rather than just using
default settings [98].

The principal methods of HPO are grid search [81] and random search [16]. Grid
search is an exhaustive technique which evaluates all parameter permutations.
However, as the number of features grows, the amount of data permutations
grows exponentially making optimisation expensive in the context of developing
translation models which require long build times. Accordingly, an effective,
less computationally intensive alternative is to use random search which
samples random configurations.

Figure 7.7: Proposed architecture for adaptNMT: a language-agnostic NMT
development environment. The system is designed to run either in the cloud or
using local infrastructure. Models are trained using parallel corpora.
Visualisation and extensive logging enable real-time monitoring. Models are
developed using vanilla RNN-based NMT, Transformer-based approaches or
transfer learning using a fine-tuning approach. Translation and evaluation can
be carried out using either single models or ensembles.

###  7.6 Architecture of adaptNMT

Having described the individual components of RNN- and Transformer-based NMT
systems, we now present the adaptNMT tool itself, in which these components
can be configured by the user. A high-level view of the system architecture of
the platform is presented in Figure 7.7. Developed as an IPython notebook, the
application uses the Pytorch implementation of OpenNMT for training models
with SentencePiece used for training subword models. By using a Jupyter
notebook, the application may be easily shared with others in the MT
community. Furthermore, the difficulties involved in setting up the correct
development environment have largely been removed since all required packages
are downloaded on the fly as the application runs.

(a) Overview of adaptNMT. Key areas include initialisation, pre-processing,
environment setup, visualisation, auto and custom NMT, training of subword
model, training of main model, evaluation and deployment (cf. Section 7.6.1).

(b) Overview of serverNMT. Highlighted cells include initialisation,
environment setup, Anvil server, API functions, translation, model building,
adaptNMT and running the server (cf. Section 7.6.2).

Figure 7.8: adaptNMT and serverNMT

There are options to run the system on local infrastructure or to run it as a
Colab instance using Google Cloud. Translation models are developed using
parallel text corpora of the source and target languages. A Tensorboard
visualisation provides a real-time graphical view of model training. The
primary use cases for the system are model building and a translation service,
one or both of which can be selected at run-time. As illustrated in the system
diagram in Figure 7.7, generating an ensemble output while translating has
also been facilitated. Models may also be deployed to a pre-configured
location.

####  7.6.1 adaptNMT

The application may be run as an IPython Jupyter notebook or as a Google Colab
application. Given the ease of integrating large Google Drive storage into
Colab, the application has been used exclusively as a Google Colab application
for our experiments, some of which are described in Section 7.7. The key
features of the notebook are illustrated in Figure 7.8.

##### Initialisation and logging

Initialisation enables connection to Google Drive to run experiments,
automatic installation of Python, OpenNMT, SentencePiece, Pytorch and other
applications. The visualisation section enables real-time graphing of model
development. All log files are stored and can be viewed to inspect training
convergence, the model’s training and validation accuracy, changes in learning
rates and cross-entropy.

##### Modes of operation

There are two modes of operation: local and cloud. In local mode, the
application is run so that models are built using the user’s local GPU
resources. The option to use cloud mode enables users to develop models using
Google’s GPU clusters. For shorter training times, the unpaid Colab option is
adequate. However, for a small monthly subscription, the Google Colab Pro
option is worthwhile since users have access to improved GPU and compute
resources. Nevertheless, there are also environmental and running costs to
consider (cf. Sections 7.7.4 and 7.8), although the Google Cloud is run on a
platform which uses 100% renewables [62]. It is also a very cost-effective
option for those working in the domain of low-resource languages since
developing smaller models requires shorter training times. However, users
requiring long training times and very high compute resources will need to use
their own hardware and run the application in local mode unless they have
access to large budgets.

##### Customisation of models

The system has been developed to allow users to select variations to the
underlying model architecture. A vanilla RNN or Transformer approach may be
selected to develop the NMT model. The customisation mode enables users to
specify the exact parameters required for the chosen approach. One of the
features, AutoBuild, enables a user to build an NMT model in three simple
steps: (i) upload source and target files, (ii) select RNN or Transformer, and
(iii) click AutoBuild.

##### Use of subword segmentation

The type of optimiser to be used for learning can be specified, and users may
also choose to employ different types of subword models when building the
system. The subword model functionality allows the user to choose whether or
not to use a subword model. Currently, the user specifies the vocabulary size
and chooses either a SentencePiece unigram or a SentencePiece BPE subword
model (cf. Section 7.5.5).

A user may upload a dataset which includes the training, validation and test
splits for both source and target languages. In cases where a user has not
already created the required splits for model training, single source and
target files may be uploaded. The splits needed to create the training,
validation and test files are then automatically generated according to the
user-specified split ratio. Given that building NMT models typically demands
long training times, an automatic notification feature is incorporated that
informs the user by email when model training has been completed.

##### Translation and evaluation

In addition to supporting the training of models, the application also allows
for translation and evaluation of model performance. Translation using pre-
built models is also parameterised. Users specify the name of the model as a
hyperparameter which is then subsequently used to translate and evaluate the
test files. The option for creating an ensemble output is also catered for,
and users simply name the models which are to be used in generating the
ensemble output.

Once the system has been built, the model to be used for translating the test
set may be selected. To evaluate the quality of translation, humans usually
provide the best insight, but they may not always be available, do not always
agree, and are expensive to recruit for experiments. Accordingly, automatic
evaluation metrics are typically used, especially by developers monitoring the
incremental progress of systems (cf. [115] for more on the pros and cons of
human and automatic evaluation).

Several automatic evaluation metrics provided by
SacreBleu888https://github.com/mjpost/sacrebleu [91] are used: BLEU [88], TER
[105] and ChrF [90]. Translation quality can also be evaluated using Meteor
[34] and F1 score [80]. Note that BLEU, ChrF, Meteor and F1 are precision-
based metrics, so higher scores are better, whereas TER is an error-based
metric and lower scores indicate better translation quality. Evaluation
options available include standard (truecase) and lowercase BLEU scores, a
sentence-level BLEU score option, ChrF1 and ChrF3.

There are three levels of logging for model development, training and
experimental results. A references section outlines resources which are
relevant to developing, using and understanding adaptNMT. Validation during
training is currently conducted using model accuracy and perplexity (PPL).

####  7.6.2 serverNMT

A server application, serverNMT, was also developed and implemented as an
IPython notebook. It can be configured to run either as a translation server
or as a build server. A secure connection, implemented from serverNMT, can be
made to websites hosting embedded web apps. At the core of serverNMT, there
are two embedded Python web apps, one for translation services and another for
developing models, both of which use the anvil.works
platform.999https://anvil.works

As a build server, serverNMT enables a window to the underlying cloud
infrastructure in which NMT models can be trained. A web app hosted on another
system may connect to this infrastructure made available by serverNMT.

Using an Anvil server embedded within serverNMT, the application continuously
waits for communication to web apps and effectively enables a cloud
infrastructure for NMT. Written as a REST server, it acts as an API for
serving previously built models and facilitates the integration of translation
models with other systems.

###  7.7 Empirical Evaluation

Having described the theoretical background and the tool itself, we now
evaluate the effectiveness of the adaptNMT approach by training models for
English to Irish (EN$\rightarrow$GA) and Irish to English (GA$\rightarrow$EN)
translation in the health domain using the gaHealth [63]
corpus.101010https://github.com/seamusl/gaHealth All experiments involved
concatenating source and target corpora to create a shared vocabulary and a
shared SentencePiece subword model. To benchmark the performance of our
models, the EN$\leftrightarrow$GA test datasets from the LoResMT2021 Shared
Task111111https://github.com/loresmt/loresmt-2021 [85] were used. These test
datasets enabled the evaluation of the gaHealth models since the shared task
focused on an application of the health domain, namely the translation of
Covid-related data. Furthermore, using an official test dataset from a shared
task enables the direct comparison of our models’ performance with models
entered by other teams, as well as future implementations.

The hyperparameters used for developing the models are outlined in Table 7.2.
The details of the training, validation and test sets used by our NMT models
are outlined in Tables 7.3 and 7.4. In all cases, 502 lines were used from the
LoResMT2021 validation dataset whereas the test dataset used 502 lines for
EN$\rightarrow$GA translation and 250 lines for GA$\rightarrow$EN translation.
Both were independent health-specific Covid test sets which were provided by
LoResMT2021. There was one exception; due to a data overlap between the test
and train datasets, a reduced test set was used when testing the gaHealth
en2ga* system.

The results from the IIITT [92] and UCF [27] teams are included in Tables 7.5
and 7.6 so the performance of the gaHealth models can be easily compared with
the findings of the participating LoResMT2021 systems. IIITT fine-tuned an
Opus MT model121212https://github.com/Helsinki-NLP/Opus-MT [112] on the
training dataset. UCF used transfer learning [121], unigram and subword
segmentation methods for EN$\leftrightarrow$GA translation.

Hyperparameter | Values
---|---
Learning rate | 0.1, 0.01, 0.001, 2
Batch size | 1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers | 5, 6
Feed-forward dimension | 2048
Embedding dimension | 128, 256, 512
Label smoothing |  0.1, 0.3
Dropout | 0.1, 0.3
Attention dropout | 0.1
Average Decay | 0, 0.0001
Table 7.2: Hyperparameter optimisation for Transformer models. Optimal parameters are highlighted in bold. [65]. Team | System | Train | Validation | Test
---|---|---|---|---
adapt | covid_extended | 13k | 502 | 500
adapt | combined_domains | 65k | 502 | 500
IIITT | en2ga-b | 8k | 502 | 500
UCF | en2ga-a | 8k | 502 | 500
gaHealth | en2ga | 24k | 502 | 500
gaHealth | en2ga* | 24k | 502 | 338
Table 7.3: EN$\rightarrow$GA training, validation and test dataset distributions. The baseline gaHealth system was augmented with an 8k Covid dataset provided by LoResMT2021. Team | System | Train | Validation | Test
---|---|---|---|---
IIITT | ga2en-b | 8k | 502 | 250
UCF | ga2en-b | 8k | 502 | 250
gaHealth | ga2en | 24k | 502 | 250
Table 7.4: GA$\rightarrow$EN training, validation and test dataset
distributions. The baseline gaHealth system was augmented with an 8k Covid
dataset provided by LoResMT2021. All overlaps were removed from the gaHealth
corpus prior to training the gaHealth ga2en model.

####  7.7.1 Infrastructure

Rapid prototype development was enabled through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16GB graphic cards and up to 27GB of
memory when available [19]. All gaHealth MT models were trained using
adaptNMT.

####  7.7.2 Metrics

Automated metrics were used to determine the translation quality. To compare
against our previous work, the performance of models is measured using three
evaluation metrics, namely BLEU, TER and ChrF. These metrics indicate the
accuracy of the translations derived from our NMT systems.

Case-insensitive BLEU scores at the corpus level are reported. Model training
was stopped after 40k training steps or once an early stopping criterion of no
improvement in validation accuracy for four consecutive iterations was
recorded.

PPL is often used to evaluate language models within NLP. It measures the
effectiveness of a probability model in predicting a sample. As a metric for
translation performance, it is important to keep low scores so that the number
of alternative translations is reduced.

####  7.7.3 Results: Automatic Evaluation

The experimental results from LoResMT 2021 are summarised in Tables 7.5 and
7.6. In the LoResMT2021 Shared Task, the highest-performing EN$\rightarrow$GA
system was submitted by the ADAPT team [64]. The system uses an extended Covid
dataset, which is a combination of the 2021 MT Summit Covid baseline and a
custom ADAPT Covid dataset. The model, developed within adaptNMT, uses a
Transformer architecture with 2 heads. It performs well across all key
translation metrics (BLEU: 36.0, TER: 0.531 and ChrF3: 0.6). The training of
this EN$\rightarrow$GA model is illustrated in Figure 7.9. The model achieved
a maximum validation accuracy of 30.0% and perplexity of 354 after 30k steps.

Figure 7.9: adapt covid_extended system: training EN$\rightarrow$GA model with 13k lines consisting of the ADAPT 5k corpus and an 8k LoResMT2021 Covid corpus. The graph on the left illustrates accuracy and the graph on the right demonstrates perplexity. Team | System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---
UCF | en2ga-b | 13.5 | 0.756 | 0.37
IIITT | en2ga-b | 25.8 | 0.629 | 0.53
adapt | combined | 32.8 | 0.590 | 0.57
gaHealth | en2ga | 33.3 | 0.604 | 0.56
adapt | covid_extended | 36.0 | 0.531 | 0.60
gaHealth | en2ga* | 37.6 | 0.577 | 0.57
Table 7.5: EN$\rightarrow$GA system compared with LoResMT 2021
EN$\rightarrow$GA systems.

The results from the LoResMT2021 Shared Task were further improved by
developing models using a bespoke health dataset, gaHealth. Table 7.5 shows an
improvement of 1.6 BLEU points, a relative improvement of almost 4.5%,
although TER and ChrF3 scores are a little worse. Validation accuracy and PPL
in training the gaHealth models with adaptNMT are illustrated in Figures 7.10
and 7.11. Figure 7.9 illustrates model training using the covid_extended
dataset, also developed using adaptNMT. In training the gaHealth en2ga*
system, as highlighted in Figure 7.10, the EN$\rightarrow$GA model was trained
with the combined 16k gaHealth and 8k LoResMT2021 corpora. The model’s
validation accuracy of 38.5% and perplexity of 113 achieved a BLEU score of
37.6 when evaluated with the test data.

The training of the GA$\rightarrow$EN gaHealth ga2en system with the combined
16k gaHealth corpus and 8k LoResMT2021 Covid corpus is shown in Figure 7.11.
This model achieves a validation accuracy of 39.5% and perplexity of 116 which
results in a BLEU score of 57.6. This is significantly better (by 20 BLEU
points) than for the reverse direction, as it is well-known that translating
into a morphological-rich language like Irish is always more difficult
compared to when the same language acts as the source. This is confirmed by
comparing the results for the UCF (13.5 vs. 21.3 BLEU) and IIITT (25.8 vs.
34.6) systems in Tables 7.5 and 7.6.

Rapid convergence was observed while training the gaHealth models such that
little accuracy improvement occurs after 30k steps, 10K fewer than for the
reverse direction. Only marginal gains were achieved after this point and it
declined in the case of the system trained using the covid_extended dataset,
as the left-hand graph in Figure 7.9 shows.

Of the models developed by the ADAPT team, the worst-performing model uses a
larger 65k dataset. This is not surprising given that the dataset is from a
generic domain of which only 20% is health-related. The performance of this
higher-resourced 65k line model lags behind the augmented gaHealth model which
was developed using just 24k lines.

Team | System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---
UCF | ga2en-b | 21.3 | 0.711 | 0.45
IIITT | ga2en-b | 34.6 | 0.586 | 0.61
gaHealth | ga2en | 57.6 | 0.385 | 0.71
Table 7.6: GA$\rightarrow$EN gaHealth systems compared with LoResMT2021
GA$\rightarrow$EN systems

For GA$\rightarrow$EN translation, the best-performing model for the
LoResMT2021 Shared Task was developed by IIITT with a BLEU of 34.6, a TER of
0.586 and ChrF3 of 0.6. Accordingly, this serves as the baseline score by
which our GA$\rightarrow$EN model, developed using the gaHealth corpus, can be
benchmarked. The performance of the gaHealth model offers an improvement
across all metrics with a BLEU score of 57.6, a TER of 0.385 and a ChrF3
result of 0.71. In particular, the 66% relative improvement in BLEU score
against the IIITT system is very significant.

Figure 7.10: gaHealth en2ga* system: training EN$\rightarrow$GA model with
combined 16k gaHealth corpus and 8k LoResMT2021 Covid corpus. The graph on the
left illustrates OpenNMT accuracy and the graph on the right demonstrates
perplexity.

Figure 7.11: gaHealth ga2en system: training GA$\rightarrow$EN model with
combined 16k gaHealth corpus and 8k LoResMT2021 Covid corpus. The graph on the
left illustrates OpenNMT accuracy and the graph on the right demonstrates
perplexity.

####  7.7.4 Environmental Impact

We were motivated by the findings of [108] and [15] to track the energy
consumption required to train our models. Prototype model development used
Colab Pro, which as part of Google Cloud is carbon neutral [62]. However,
longer running Transformer experiments were conducted on local servers using
324 gCO2 per kWh131313https://www.seai.ie/publications/Energy-in-
Ireland-2020.pdf [100]. The net result was just under 10 kgCO2 created for a
full run of model development. Models developed during this study will be
reused for ensemble experiments in the future so that work will have a life
beyond this paper.

####  7.7.5 Stochastic Nuances

To evaluate the translation performance of an IPython-based application such
as adaptNMT, a comparison with a Python script version of the same
application, myNMT.py, was conducted. We built EN$\leftrightarrow$GA
translation models using this script. The models developed with adaptNMT were
trained on Google Colab using a 12GB Tesla K80 GPU, whereas the myNMT models
were trained on a local machine using a 12GB Gigabyte 3060 graphics card. The
results from evaluating these models are presented in Tables 7.7 and 7.8.

Despite setting the same random seed, it is clear from Tables 7.7 and 7.8 that
the translation performance of the adaptNMT models is better by 1.2 BLEU
points (3.3% relative improvement) in EN$\rightarrow$GA translation and 1.0
BLEU point (1.8% relative improvement) in GA$\rightarrow$EN translation.

Given the stochastic nature of machine learning, training models on different
systems can yield different results even with the same training, validation
and test data. The performance differences can be attributed to the stochastic
nature of the learning algorithm and evaluation procedure. Furthermore, the
platforms had different underlying system architectures which is another
source of stochastic error.

System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---
adaptNMT | 37.6 | 0.577 | 0.570
myNMT | 36.4 | 0.622 | 0.56
Table 7.7: Stochastic differences between EN$\rightarrow$GA systems System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---
adaptNMT | 57.6 | 0.385 | 0.71
myNMT | 56.6 | 0.399 | 0.703
Table 7.8: Stochastic differences between GA$\rightarrow$EN systems

###  7.8 Discussion

The mathematical first principles governing NMT development were presented to
demonstrate the mechanics of what happens during model training. Several
parameters in Equations 7.2 \- 7.8 are configurable within the adaptNMT
application.

The environmental impact of technology, and the measurement of its effects,
has gained a lot of prominence in recent years [47]. Indeed, this may be
viewed as a natural response to truly massive NLP models which have been
developed by large multinational corporations. In particular, HPO of NMT
models can be particularly demanding if hyperparameter fine-tuning is
conducted across a broad search space. As part of their work on NMT
architectures, the Google Brain team required more than 250,000 GPU hours for
NMT HPO [22]. Training of these models was conducted using Tesla K40m and
Tesla K80 GPUs with maximum power consumption between 235 W and 300 W, giving
rise to potentially in excess of 60 MWh of energy usage. Even though the
Google Cloud is carbon neutral, one must consider the opportunity cost of this
energy usage.

A plethora of tools to evaluate the carbon footprint of NLP [11] has
subsequently been developed and the concept of sustainable NLP has become an
important research track in its own right at many high-profile conferences
such as the EACL 2021 Green and Sustainable NLP
track.141414https://2021.eacl.org/news/green-and-sustainable-nlp In light of
such developments, a ‘green report’ was incorporated into adaptNMT whereby the
kgCO2 generated during model development is logged. This is very much in line
with the industry trend of quantifying the impact of NLP on the environment;
indeed, [51] have demonstrated that high-performing MT systems can be built
with much lower footprints, which not only reduce emissions but also in the
post-deployment phase deliver savings of almost 50% in energy costs for a real
translation company.

To evaluate system performance in translating health data in the
EN$\rightarrow$GA direction, we used the adaptNMT application to develop an MT
model for the LoResMT2021 Shared Task. The application was subsequently used
to develop an MT model for translating in the GA$\rightarrow$EN direction. In
both cases, high-performing models achieving SOTA scores were achieved by
using adaptNMT to develop Transformer models capable of generating high-
quality output.

The danger of relying on increasingly large language models (LLMs) has been
well-documented in the literature. Such discussion focuses not just on the
environmental impact but also highlights the impact of in-built bias and the
inherent risks that large models pose for low-resource languages [15]. Using
an easily-understood framework such as adaptNMT, the benefits of developing
high-performing NMT models with smaller in-domain datasets should not be
overlooked.

###  7.9 Conclusion and Future Work

We introduced adaptNMT, an application for NMT which manages the complete
workflow of model development, evaluation and deployment. The performance of
the application was demonstrated in the context of generating an
EN$\rightarrow$GA translation model which ranked 1st in the LoResMT2021 shared
task and validated against a standalone reimplementation of
EN$\leftrightarrow$GA systems outside the tool, where no drop-off in
performance was seen.

With regard to future work, development will focus more on tracking
environmental costs and integrating new transfer learning methods. Modern
zero-shot and few-shot approaches, adopted by GPT3 [23] and Facebook LASER [7]
frameworks, will be integrated. Whereas the existing adaptNMT application
focuses on customising NMT models, a separate application adaptMLLM will be
developed to fine-tune multilingual language models and LLMs, in particular
those that focus on low-resource language pairs such as NLLB [32].

The green report embedded within the application is our first implementation
of a sustainable NLP feature within adaptNMT. It is planned to develop this
feature further to include an improved user interface and user recommendations
about how to develop greener models. As an open-source project, we hope the
community will add to its development by contributing new ideas and
improvements.

##  Chapter 8 adaptMLLM: Fine-Tuning Multilingual Language Models

###  8.1 Context

Figure 8.1: Fine-tuned MLLM approach of adaptMLLM for EN$\leftrightarrow$GA
translation

As part of the research effort required to answer RQ4, two low-resource
language pairs (EN$\leftrightarrow$GA and EN$\leftrightarrow$MR) were chosen
for running fine-tuning experiments. The research contribution of this work
resulted in the development of the adaptMLLM application which was used to
carry out the fine-tuning experiments. The fine-tuned EN$\leftrightarrow$GA
and EN$\leftrightarrow$MR MLLM models demonstrated superior translation
quality when compared to baselines from the LoResMT2021 Shared Task.

The first suite of tests was conducted on the EN$\leftrightarrow$GA pair and
our approach in fine-tuning a pre-trained 3.3B parameter No Language Left
Behind (NLLB) model is illustrated in Figure 8.1. With EN$\rightarrow$GA
translation, the following scores were attained: 41.2 BLEU, 0.531 TER and 0.6
ChrF. In the GA$\rightarrow$EN direction, the system achieved 75.1 BLEU, 0.385
TER and 0.71 ChrF which surpassed by a wide margin the winning scores of the
shared task. An overview of the NLLB project is outlined in Section 8.5.2 and
a more in-depth discussion of the NLLB architecture is covered in Appendix A.

The approach taken in fine-tuning a pre-trained 3.3B parameter NLLB model on
the EN$\leftrightarrow$MR pair is illustrated in Figure 8.2. Again these
scores improved upon all previous scores which were achieved in LoResMT2021.
In the EN$\rightarrow$MR direction, the following scores were attained: 26.4
BLEU, 0.56 TER and 0.608 ChrF. In the MR$\rightarrow$EN direction 52.6 BLEU,
0.409 TER and 0.704 ChrF were achieved.

Figure 8.2:  Fine-tuned MLLM approach of adaptMLLM for EN$\leftrightarrow$MR
translation

adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages
with integrated LLM playgrounds

Séamus Lankford
Haithem Afli
Andy Way

Information, MDPI

November, 2023

ADAPT Centre

Dublin City University

Ireland

https://doi.org/10.3390/info14120638

###  8.2 Abstract

The advent of multilingual language models (MLLMs) and large language models
(LLMs) has spawned innovation in many areas of natural language processing.
Despite the exciting potential of this technology, its impact on developing
high-quality machine translation (MT) outputs for low-resource languages
remains relatively under-explored. Furthermore, an open-source application,
dedicated to both fine-tuning MLLMs and managing the complete MT workflow for
low-resources languages, remains unavailable. We aim to address these
imbalances through the development of adaptMLLM which streamlines all
processes involved in the fine-tuning of MLLMs for MT. This open-source
application is tailored for developers, translators, and users who are engaged
in MT. It is particularly useful for newcomers to the field, as it
significantly streamlines the configuration of the development environment. An
intuitive interface allows for easy customisation of hyperparameters, and the
application offers a range of metrics for model evaluation and the capability
to deploy models as a translation service directly within the application. As
a multilingual tool, we used adaptMLLM to fine-tune models for two low-
resource language pairs: English to Irish (EN$\leftrightarrow$GA) and English
to Marathi (EN$\leftrightarrow$MR). Compared with baselines from the
LoResMT2021 Shared Task, the adaptMLLM system demonstrated significant
improvements. In the EN$\rightarrow$GA direction, an improvement of 5.2 BLEU
points was observed and an increase of 40.5 BLEU points was recorded in the
GA$\rightarrow$EN direction representing relative improvements of 14% and 117%
respectively. Significant improvements in the translation performance of the
EN$\leftrightarrow$MR pair were also observed notably in the MR$\rightarrow$EN
direction with an increase of 21.3 BLEU points which corresponds to a relative
improvement of 68%. Finally, a fine-grained human evaluation of the MLLM
output on the EN$\rightarrow$GA pair was conducted using the Multidimensional
Quality Metrics and Scalar Quality Metrics error taxonomies. The application
and models are freely available.111http://github.com/adaptNMT/adaptMLLM

###  8.3 Graphical abstract

Figure 8.3: Graphical abstract summarising the adaptMLLM system

###  8.4 Introduction

Large language models (LLMs), are AI models that use deep learning techniques
to generate human-like text. These models are trained on vast amounts of text
data, often using unsupervised learning, to learn the patterns and
relationships within language. This results in models that can generate text
which is often indistinguishable from text written by a human.

The excitement surrounding LLMs stems from their potential to revolutionise
many fields, from language translation [32] and content generation [23] to
chatbots222https://openai.com/blog/chatgpt and virtual
assistants.333https://genie.stanford.edu/ With their ability to understand
natural language and generate complex responses, LLMs have the potential to
enhance human communication and productivity in ways that were previously
unimaginable. LLMs can also be used in creative applications, such as
generating music444https://soundraw.io/ or art.555https://labs.openai.com/

No Language Left Behind (NLLB) [32] represents a groundbreaking AI project in
the area of multilingual language models (MLLMs). The project has released
open-source models proficient in delivering high-quality translations across
200 languages and has enhanced translations for low-resource languages on
platforms like Facebook and Instagram. The NLLB-200 model, integrated into the
Wikimedia Foundation’s Content Translation tool, aids Wikipedia editors in
translating content into their preferred languages. These editors can now more
effectively translate articles from lesser-known languages, such as Luganda
and Icelandic, enriching Wikipedia’s language diversity. The open-sourced
nature of the NLLB-200 model also empowers the research community and
Wikipedia editor groups to expand upon their findings.

When building MLLMs and LLMs, the focus is on designing and training the model
architecture. This involves selecting the appropriate neural network
architecture and hyperparameters, as well as deciding on the training data and
optimisation techniques to use.

Tuning an MLLM or LLM, on the other hand, involves adjusting the parameters of
the model to improve its performance on a specific task. In neural networks
such as MLLMs and LLMs, the weights and biases are parameters that the network
adjusts through training to minimise a cost function. This is done by training
the model on a task-specific dataset and adjusting the model’s hyperparameters
to optimise its performance. Tuning an MLLM can be a challenging task, as the
model is often very complex and the training process can take a long time. Our
paper concentrates on fine-tuning pre-built MLLMs to enhance machine
translation (MT) with a particular focus on low-resource language pairs.

The process of fine-tuning an MLLM involves several distinct stages which are
broken down into individual steps. These steps include setting up the
environment, preparing the dataset, parameterising and fine-tuning the chosen
MLLM, and evaluating and deploying the model. This modular approach has proven
to be effective in fine-tuning MLLMs and we have structured our adaptMLLM
application to cater for both developers and translators. In light of the
environmental impact of developing and running large AI models [108, 47, 51],
we also calculate carbon emissions in a “green report”. It is envisaged that
such a report will incentivise more responsible and sustainable model
development.

A significant aspect of our research involves creating applications and models
to address language technology challenges. Similar to our previous work which
focused on developing NMT models [68], we hope this paper will be particularly
helpful for those new to MT wishing to learn more about fine-tuning MLLMs.

Unlike many translation toolkits, our application does not use a command line
interface. Instead, we have designed and fully implemented the interface in
Google Colab,666https://colab.research.google.com a cloud-hosted
solution777https://cloud.google.com that is more intuitive for both
educational and research settings. Furthermore, our application provides
graphical user interface (GUI) controls within adaptMLLM, enabling users to
customise all key hyperparameters required for MLLMs.

Our application is designed to operate as a platform as a service (PaaS) cloud
computing application, allowing for quick and efficient scaling of the
infrastructure. Additionally, the deploy function allows for immediate
deployment of trained models.

This paper is organised by initially presenting related work and background
information on MLLMs and LLMs in Section 8.5. This is followed by a
description of our datasets in Section 8.6. The key features of the adaptMLLM
architecture are discussed in Section 8.7 and an empirical evaluation of our
trained models, including a human evaluation is carried out in Section 8.8.
The system is discussed in Section 8.9 before drawing conclusions and
describing future work in Section 8.10.

###  8.5 Related Work

####  8.5.1 Transformer Architecture

Following the introduction of the attention mechanism, a natural line of
investigation was to see whether attention could do most of the heavy lifting
of translation by itself. Accordingly, [114] proposed that “attention is all
you need” in their Transformer architecture, which has achieved state-of-the-
art (SOTA) performance on many natural language processing (NLP) benchmarks by
relying solely on an attention mechanism, removing recurrence and convolution
while allowing the use of much simpler feed-forward neural networks. In the
context of our research, we have previously demonstrated that Transformer-
based models deliver high-functioning models for the low-resource
EN$\rightarrow$GA language pair [65].

The default Transformer architecture follows an encoder-decoder structure
generating its output without relying on recurrence and convolutions. The task
of the encoder is to map an input sequence to a sequence of continuous
representations, which is then passed to a decoder. The decoder generates an
output sequence by using the output of the encoder together with the decoder
output from the previous time step.

####  8.5.2 Multilingual Language Models - NLLB

MT has become a significant area of research in AI to eliminate language
barriers worldwide. However, the current focus is limited to a small number of
languages, neglecting the vast majority of low-resource languages. To address
this issue, the NLLB initiative was launched. This project aims to overcome
the challenges of using MT for low-resource language translation by developing
datasets and models that bridge the performance gap between low- and high-
resource languages. The NLLB team has also created architectural and training
enhancements tailored to support MT for low-resource languages. Their work is
open-source,888https://github.com/facebookresearch/fairseq/tree/nllb and many
of their models serve as baselines for fine-tuning with adaptMLLM.

####  8.5.3 Large Language Models

The increasing availability of large datasets provides the raw material for
LLM training [94, 31, 117] enabling performance improvement on NLP tasks which
can learn from a wide variety of sources.

Another key factor in driving the ubiquity of LLMs has been the growth in
computational power dedicated to the domain. As a consequence, more powerful
computers now have the capability to train LLMs on massive datasets, which in
turn has led to SOTA results on many common NLP tasks [35]. New training
algorithms developed through advancement in AI research have further boosted
LLM performance [71].

LLMs have the potential to improve the use of technology across a wide range
of domains, among which include medicine, education and computational
linguistics. In education, LLMs may be used for personalised student learning
experiences [54] while in the medical domain, analysing large amounts of
medical files can assist doctors in treating patients [49]. Of particular
interest to our research is how LLMs can be used within the realm of
computational linguistics, more specifically in the field of MT.

##### GPT-J

Transformers are increasingly the architecture of choice for NLP problems,
replacing recurrent neural networks (RNNs) such as long short-term memory
(LSTM) [48].

GPT-J is an open-source implementation of a particular class of LLMs known as