# A Simple Framework to Accelerate
Multilingual Language Model for Monolingual Text Generation

Jimin Hong &Gibbeum Lee
KRAFTON
{jimmy.h, pirensisco<EMAIL_ADDRESS>
&Jaewoong Cho

###### Abstract

Recent advancements in large language models have facilitated the execution of
complex language tasks, not only in English but also in non-English languages.
However, the tokenizers of most language models, such as Llama, trained on
English-centric corpora, tend to excessively fragment tokens in non-English
languages. This issue is especially pronounced in non-roman alphabetic
languages, which are often divided at a character or even Unicode level,
leading to slower text generation. To address this, our study introduces a
novel framework designed to expedite text generation in these languages. This
framework predicts larger linguistic units than those of conventional
multilingual tokenizers and is specifically tailored to the target language,
thereby reducing the number of decoding steps required. Our empirical results
demonstrate that the proposed framework increases the generation speed by a
factor of 1.9 compared to standard decoding while maintaining the performance
of a pre-trained multilingual model on monolingual tasks.

##  1 Introduction

The advent of large language models (LLMs) (OpenAI, 2023; Touvron et al.,
2023; Antropic, 2023) has revolutionized the field of generative artificial
intelligence, demonstrating remarkable capabilities to respond across diverse
tasks and languages (Eloundou et al., 2023; Solaiman et al., 2023). These
models, despite being predominantly trained on English data, exhibit
significant multilingual proficiency (Bandarkar et al., 2023).

However, when applied to non-alphabetic languages, these models often suffer
from slower text generation due to English-centric tokenization Rust et al.
(2021); Ahia et al. (2023); Petrov et al. (2023). Current tokenization
techniques used in Large Language Models (LLMs) are data-driven and optimize
segmentation based on the frequency of characters or bytes within a specific
corpus (Sennrich et al., 2016; Kudo, 2018). As a result, the tokenizers of
multilingual models, which are heavily influenced by English-dominant training
data, are predominantly composed of English subwords. This leads to excessive
fragmentation, where non-English words are overly segmented into a large
number of subword units Rust et al. (2021); Ahia et al. (2023); Petrov et al.
(2023). The autoregressive nature of LLMs further amplifies this inefficiency,
as it sequentially requires the generation of text.

Figure 1:  Overview of the proposed framework. Illustration of (Left) the
generation with a pre-trained multilingual model and (Right) the generation of
MuMo Framework. Given the Korean prefix “천왕성은” (Uranus is), the model
generates the consecutive phrase “태양으로부터”(from the Sun) that consisted of 3
morphemes (“태양”, “으로”, “부터”) in Korean. The generation with the pre-trained
multilingual model faces inefficiency due to excessive fragmentation,
requiring 12 steps to generate only 3 Korean morphemes. However, the MuMo
framework empowers the multilingual language model to generate multiple tokens
in a single iteration by extracting a word from the Korean Vocabulary,
requiring 3 steps.

To address these challenges, previous studies Wang et al. (2019b); Rust et al.
(2021); Cui et al. (2023) have proposed replacing or augmenting the existing
vocabulary of pre-trained multilingual models with language-specific
vocabularies to more effectively encode monolingual text corpora.
Specifically, Rust et al. (2021) improved mBERT Devlin et al. (2019) by
replacing its tokenizer with a monolingual one and incorporating an additional
100,000 pre-training steps. On the other hand, Cui et al. (2023) enhanced
Llama (Touvron et al., 2023) by expanding the Chinese vocabulary and further
pre-training it on a 120GB text corpus that includes Chinese texts. However,
this approach requires an extensive pre-training phase with a substantial
amount of data.

Another approach to address the challenges is the use of small draft models
(Leviathan et al., 2023; Chen et al., 2023a). These models generate draft
output tokens, which are then verified by the original language model.
However, a significant challenge arises when trying to identify or train a
suitable small model that can handle multiple languages with reliable
performance (Conneau et al., 2020; Bandarkar et al., 2023).

In response to these challenges, our research introduces MuMo, a novel
framework designed to accelerate Multilingual language models for targeted
Monolingual text generation, particularly in non-alphabetic languages. MuMo
incorporates a new vocabulary of a target language into the output layer, also
known as the Language Model (LM) head, and predicts the next token from this
expanded vocabulary. This approach requires training only the extended portion
of the output layer and specific layers of the feed-forward network.
Importantly, MuMo eliminates the need for extensive text corpora or a draft
model, requiring only a modest corpus of the target language, approximately
365M tokens in size.

Empirical results across summarization, and translation tasks in Korean and
Japanese demonstrate that MuMo significantly accelerates text generation,
achieving over a 1.9-fold increase in speed without compromising output
quality.

The contributions of our research are as follows:

  * •

We propose MuMo, a framework that accelerates the decoding speed of non-Roman
alphabetic languages without compromising the performance of the language
model.

  * •

Our study shows a novel approach to address the issue of slow inference speed
of non-English languages, a problem that arises due to the excessive
fragmentation in pre-trained models that are primarily English-centric.

  * •

It can be effectively employed even in the absence of extensive computing
resources or large monolingual text corpora.

##  2 Background

##### Language Modeling

The goal of auto-regressive language modeling is to create a probability
distribution, denoted as $p_{\theta}(\mathbf{x})$, where $\theta$ represents
the model parameter. This distribution is established over a sequence of
tokens, represented as $\mathbf{x}=[x_{1},\ldots,x_{T}]$. These tokens are
derived from a pre-established vocabulary, denoted as $\mathcal{V}$, where
each token $x_{t}$ is a member of $\mathcal{V}$. A conventional approach
involves maximizing the log-likelihood of a sequence:

$\max_{\theta}\mathcal{L}_{\text{MLE}}(p_{\theta},\mathbf{x})=\sum_{t=1}^{T}\log
p_{\theta}(x_{t}|\mathbf{x}_{<t}).$ (1)

##### Tokenization in LM

Tokenization is the process of segmenting text into subword chunks (Kudo,
2018; Sennrich et al., 2016). The process can be started with a base
vocabulary composed solely of characters, then, new vocabulary items are
subsequently added by recursively merging existing ones based on the frequency
statistics within a training corpus. However, in a multilingual setup, even a
base vocabulary composed solely of characters can be extensive, with over 130K
items (Wang et al., 2019a).

##### Byte-level Subword Tokenization

Radford et al. (2019) proposed using bytes as the basic units, with only 256
tokens required to form subwords. This Byte-level Byte Pair Encoding (BBPE)
does not suffer from out-of-vocabulary problem and has become the standard in
most pre-trained models (Brown et al., 2020; Muennighoff et al., 2022; Zhang
et al., 2022b; Scao et al., 2022). However, BBPE often produces longer
sequence length when handling multilingual scripts compared to character-based
BPE (Wang et al., 2019a; Petrov et al., 2023).

Lang Word Multilingual Tokens Ko 서울 (“서”, “\0xec”, “\0xb8”, “\0x9a”) Ja 発売
(“発”, “\0xe5”, “\0xa3”, “\0xb2

Table 1: Examples of the tokenization results. These examples are tokenized by
the Llama tokenizer. The target monolingual word are excessively segmented
into byte units, when a suitable match is not found in the multilingual
vocabulary.

##### Exccessive Fragmentation

We explore the issue of excessive fragmentation (Ahia et al., 2023; Petrov et
al., 2023) in multilingual setups, a common challenge in multilingual
tokenizers (Rust et al., 2021; Zhang et al., 2022a; Ahia et al., 2023; Petrov
et al., 2023). The problem occurs when a tokenizer produces too many tokens
due to significant differences between input texts and the pre-trained corpus
(Rust et al., 2021; Hong et al., 2021).

The difficulty of integrating numerous languages into one model while
balancing vocabulary composition contributes to this issue (Wang et al.,
2019b). As the number of vocabulary items increases, model parameters expand
exponentially, impacting training efficiency (Liang et al., 2023).

To investigate this, we compared the tokenization outputs of several
multilingual model tokenizers: Bloomz (Scao et al., 2022), Llama-2 (Touvron et
al., 2023), and GPT-3.5/4 (OpenAI, 2023). We used the FLoRes-200 (Goyal et
al., 2022) dataset, which includes Wikipedia articles in 200 languages. Table
2 indicates that multilingual tokenizers generate more tokens for Japanese and
Korean languages than their monolingual counterparts.

##  3 Proposed Framework

Figure 2:  Illustration of a single step prediction with MuMo. Initially, the
MuMo LM Head $f_{\text{mumo}}$ selects the top 6 candidates. Then, the pre-
trained multilingual model verifies the feasibility of the candidates. Among
the modules in MuMo, the Target Monolingual LM head (the Korean LM Head in the
figure) is only trained.

We propose a framework named MuMo to accelerate the inference speed of a pre-
trained multilingual LM for a non-alphabetic monolingual language via a given
small monolingual dataset. In the section, we introduce 1) the model
architecture, 2) the fine-tuning process on a small monolingual dataset, and
3) the inference process of the proposed framework.

Model # of Vocabulary en ja ko Bloomz 250K 26.0 46.5 72.4 Llama-2 32K 29.5
66.3 94.6 GPT-3.5 / 4 100K 25.9 58.6 61.3 Mono-Ja 30K - 28.1 - Mono-Ko 32K - -
32.3

Table 2:  Average number of tokens after tokenization. We evaluate
tokenization efficiency on the FLoRes-200 dataset for English, Japanese, and
Korean Languages. Mono-Ja, and Mono-Ko indicate the tokenizer of pre-trained
monolingual model for Japanese and Korean respectively.

###  3.1 Model Architecture

We illustrate the model architecture of MuMo in Fig. 2.

##### Pre-trained Multilingual Model

We consider a setting in which a pre-trained multilingual model
$f_{\text{multi}}$ is given. The model consists of 1) Transformer layers that
consist of attention and feed-forward network, and 2) an output embedding
layer called language model (LM) head. We denote $\mathcal{V}_{\text{multi}}$
as the multilingual vocabulary set of the model. objective, as
$\mathcal{L}_{\text{MLE}}(p_{\text{multi}},\mathbf{x})=\sum_{t=1}^{T}\log
p_{\text{multi}}(x_{t}|\mathbf{x}_{<t})$

##### Target Monolingual LM Head

The primary concept involves modifying pre-trained representations to predict
a singular token unit within a target monolingual vocabulary
$\mathcal{V}_{\text{mono}}$. We refer to the Target Monolingual LM head as
$f_{\text{mono}}$, which is composed of two main components: a feed-forward
network (FFN) and an output linear layer, represented as
$g_{\text{mono}}:\mathbb{R}^{d_{{}_{\text{mono}}}}\rightarrow\mathbb{R}^{|\mathcal{V}_{\text{mono}}|}$:

$\text{FFN}(h)=q(W_{1}^{\top}h)W_{2}\in\mathbb{R}^{d_{\text{mono}}}$ (2)

where $W_{1}\in\mathbb{R}^{d_{\text{multi}}\times d_{\text{ffn}}}$ and
$W_{2}\in\mathbb{R}^{d_{\text{ffn}}\times d_{\text{mono}}}$ are the weight
matrices, $q$ is non-linearity function, and $d_{\text{mono}}$ represents the
dimension of the target language representaiton. We set $d_{\text{ffn}}$ as
$d_{\text{multi}}/4$, and the non-linearity function $q$ as SwiGLU (Shazeer,
2020). The output linear layer $g_{\text{mono}}$ then generates a word.

$\begin{split}f_{\text{mono}}(h)=g_{\text{mono}}(\text{FFN}(h))\in\mathbb{R}^{|\mathcal{V}_{\text{mono}}|}\end{split}$
(3)

##### MuMo LM Head

Note that the output space of $f_{\text{mono}}$ is restricted to tokens in the
$\mathcal{V}_{\text{mono}}$. Inspired by Lan et al. (2023), we simply extend
the $f_{\text{mono}}$ by concatenating the output linear layer of pre-trained
multilingual model. This is particularly useful when there is no suitable
token in $\mathcal{V}_{\text{mono}}$ to predict, such as special symbols or
alphabet-based tokens for non-alphabet languages.

Formally, given context representation $h_{t-1}$, the output of the MuMo LM
head is computed as:

$\displaystyle f_{\text{mumo}}(h_{t-1})=$
$\displaystyle[f_{\text{multi}}(h_{t-1});f_{\text{mono}}(h_{t-1})]\in\mathbb{R}^{|\mathcal{V}_{\text{multi}}|+|\mathcal{V}_{\text{mono}}|}$
(4)

where the symbol ; indicates the concatenation of two vectors, and the
$f_{\text{mumo}}$ indicates the output of the MuMo LM head. Thus, the MuMo LM
head is composed of a combination of the pre-trained language model head and
Target Monolingual LM head.

###  3.2 Fine-tuning

In the proposed framework, we only fine-tune the target monolingual LM head
$f_{\text{mono}}$ leveraging a small given target monolingual dataset. Note
that the parameters of the pre-trained multilingual model remain frozen during
the process. The model is fine-tuned by maximizing the log-likelihood of a
sequence:

$\underset{{f_{\text{mono}}}}{\max}~{}\mathcal{L}_{\text{MLE}}(p_{\text{mumo}},\mathbf{x})=\sum_{t=1}^{T}\log
p_{\text{mumo}}(x_{t}|\mathbf{x}_{<t})$

(5)

where
$p_{\text{mumo}}(x_{t}|\mathbf{x}_{<t})=\operatorname{Softmax}(f_{\text{mumo}}(h_{t-1}))$.

###  3.3 Inference

Despite the availability of direct generation based on the $p_{\text{mumo}}$,
the newly initialized Target Monolingual LM head, which is trained on limited
data, may be constrained by generalization capabilities beyond the training
dataset. The key concept is to leverage the probabilistic knowledge acquired
by the pre-trained model $p_{\text{multi}}$, which has been extensively
trained on large text corpora.

####  3.3.1 Step 1: Top-$k$ Selection

Initially, we select top $k$ candidates based on the probability
$p_{\text{mumo}}(x_{t}|\mathbf{x}_{<t})$. Given the fact that we do not modify
the input embedding of the pre-trained model, we are unable to feed the
predicted word if a word does not belong in $\mathcal{V}_{\text{multi}}$
during the subsequent iteration. Instead, we input the predicted word as the
tokenization units of the pre-trained vocabulary. For example, let’s consider
the Korean word “수소”, which corresponds to a sequence of two tokens (“수”, “소”)
in $\mathcal{V}_{\text{multi}}$. If the Korean word “수소” is selected among the
Top-$k$ candidates, we employ these two multilingual tokens.

####  3.3.2 Step 2: Verification

Then, the feasibility of these potential completions is measured using the
log-joint probability distribution over $p_{\text{multi}}$. To account for
shorter sequences naturally having higher scores (Jean et al., 2015; Murray
and Chiang, 2018), we normalize each candidate’s score by its token length.

We measure the feasiblity for a candidate sequence as follows:

$\begin{split}\sigma(\mathbf{c}^{i})&=\frac{1}{l^{i}}\sum_{k=1}^{l^{i}}{\log
p_{\text{multi}}(c_{t+k}^{i}|c_{<t+k}^{i},\mathbf{x}_{<t})}\end{split}$ (6)

where $\mathbf{c}^{i}$ symbolizes a predicted token within the top-$k$
candidates, $p_{\text{multi}}$ represents the probability as determined by the
pre-trained multilingual model, and $l^{i}$ corresponds to the sequence length
of the candidate $\mathbf{c}^{i}$.

From the $k$ candidates, the ultimate prediction can be derived from both
deterministic and stochastic manners, depending on decoding strategies.

##  4 Experiments

###  4.1 Setup

##### Languages

We focus on two non-roman alphabetical languages: Korean and Japanese. These
languages constitute over 0.05% of the pre-training corpus used in Llama-2
(Touvron et al., 2023). They are of particular interest due to their
tokenization disparity when compared to English scripts (Ahia et al., 2023;
Petrov et al., 2023). Notably, previous work (Bandarkar et al., 2023) has
shown impressive results on natural language understanding tasks for these
languages using Llama-2.

Language Language Family Pre-trained Tokenizer Korean Koreanic
EleutherAI/polyglot-ko-12.8b Japanese Japonic rinna/japanese-gpt-neox

Table 3:  Selected languages and tokenizers.  We utilize the tokenizers to
construct $\mathcal{V}_{\text{mono}}$ in each language.

##### Model

We utilize the Llama-2 13B model (Touvron et al., 2023) for all experiments.
We observed some language alignment discrepancies between instructions and
responses when using the Llama-2 13B chat model.111meta-llama/Llama-2-13b-chat
To address the issue, we conduct multilingual instruction tuning (Muennighoff
et al., 2022) for English, Korean, and Japanese languages using the ShareGPT
and Alpaca (Chen et al., 2023c). This process improve the model’s fluency in
each language (Muennighoff et al., 2022; Chen et al., 2023b).

##### Implementation of MuMo

To construct targeted monolingual vocabularies in MuMo Framework, we levergage
the tokenizers from the off-the-shelf model, as shown in Table 3. We selected
monolingual tokens by filtering vocabulary items based on the Unicode range of
each monolingual script. Additionally, we excluded items from the selection if
they were already present in the pre-trained vocabulary.

Regarding the initialization of $g_{\text{mono}}$, we utilize the LM head of
the pre-trained multilingual model. For example, when the Korean word "태양" is
tokenized into subword units (“\0xed”, …, “\0x91”) using the pre-trained
vocabulary, we initialize the Korean LM head of "태양" by taking the mean of the
corresponding subword embeddings of the multilingual LM head. This process
ensures that the initialized embeddings of Target Monolingual head represent
the original word in the multilingual context.

##### Fine-tuning

We only train the Target Monolingual LM head $g_{\text{mono}}$ with the
translated ShareGPT and Alpaca datasets (Chen et al., 2023c) in Korean, and
Japanese. The training is done with 1500 steps, using 8 A100-40GB GPUs, with
one batch consisting of 128 examples. We use the AdamW (Loshchilov and Hutter,
2019) optimizer with a learning rate of 0.001, weight decay of 0.01, and 150
steps of warm-up.

Summarization (0-shot) Translation (3-shot) Lang Method ROUGE-2 ROUGE-L
Tokens/sec Speed Up BLEU Tokens/sec Speed Up Ko Vanilla Decoding 20.7 36.1
28.9 1.00x 21.2 29.8 1.00x Spec. (w/o Rejection) 18.7 33.5 35.2 1.21x 18.6
36.5 1.22x Spec. 20.3 35.2 27.5 0.95x 21.5 29.2 0.98x Shortlisting 20.5 36.3
30.6 1.06x 19.5 32.7 1.03x MuMo (Ours) 20.3 35.9 55.3 1.92x 21.7 50.9 1.70x Ja
Vanilla Decoding 11.3 26.6 29.3 1.00x 26.3 33.4 1.00x Spec. (w/o Rejection)
10.8 24.2 35.4 1.21x 22.7 39.9 1.21x Spec. 11.6 26.5 28.5 0.97x 26.0 29.7
1.03x Shortlisting 11.4 26.3 30.3 1.03x 25.2 34.9 1.04x MuMo (Ours) 11.6 26.3
59.2 2.02x 24.3 58.3 1.75x

Table 4:  Comparative study of Language Model (LM) inference speed. The column
labeled “Speed Up” represents the relative performance improvement in
inference speed compared to the vanilla decoding method. The highest
performance in each category is highlighted in Boldface, and the second
highest score is underlined. All models use sampling-based decoding. MuMo
outperforms the compared baselines in the inference speed. Detailed
information about the generation hyperparameters, including those used for
sampling-based decoding, can be found in Appendix A.

##### Evaluation

We choose two representative generation tasks: summarization and translation.
For summarization, we use 500 examples from XLSum (Hasan et al., 2021), and
for translation, we use 500 examples from the FLoRes-200 (Goyal et al., 2022)
dataset. We translate English sentences to each target language sentence.

For each task, we report 0-shot results for summarization, and 3-shot results
for translation. We set the maximum sequence length as 512. We utilize flash-
attention 2 (Dao, 2023) and bfloat16 types for text generation.

##### Metrics

In the summarization task, we gauge the reliability of the generated content
by calculating the ROUGE-2 and ROUGE-L (Lin, 2004) scores, averaging the
results across 5 different generated summaries. Likewise, for the translation
task, we measure the quality of the translations by computing the BLEU
(Papineni et al., 2002) score, again averaging over 5 translation
results.222We utilize SacreBLEU scores with the signature BLEU —nrefs:1
—case:mixed —eff:no —tok:ko,ja-mecab—smooth:exp —version:2.2.0. We report
Tokens/sec to measure the inference speed of the models.

###  4.2 Baselines

We consider the following several baselines for the comparison with the
proposed method.

##### Vanilla Decoding

The autoregressive generation is to sequentially sample the subsequent word
based on the probability distribution over the pre-trained vocabulary. This
approach serves as the standard against which improvements are measured.
Accounting for the nature of task, all the baselines and our framework
utilizes sampling-based decoding strategy with temperature as 0.1, $k$ as 10
for top-$k$ sampling (Fan et al., 2018) and $p$ as 0.7 for nucleus sampling
(Holtzman et al., 2020).

##### Speculative Decoding

Speculative decoding approach (Chen et al., 2023a; Leviathan et al., 2023)
employs a preliminary "draft" model to rapidly generate a set of token
candidates at each decoding step. Subsequently, these candidates undergo a
validation process by the original language model to ascertain their
likelihood as plausible continuations of the text. We implement two variants
of this method: one with the capability to reject unsuitable candidates
(Spec.) and another without its rejection module (Spec. w/o Rejection). For
the draft model, we utilize Llama-2 7B (Touvron et al., 2023). Following the
implementation of Chen et al. (2023a), we generate 5 draft tokens at each
iteration.

##### Lexical Shortlisting

Lexical Shortlisting (Shortlisting) (Abdaoui et al., 2020; Ushio et al.,
2023), or vocabulary selection, is the approach that optimizes the decoding
process by allowing it to generate a word within a set of tokens during the
inference stage (Ushio et al., 2023). We implement to filter out tokens that
are not present within the corresponding target language subset of the mC4
corpus (Xue et al., 2021), as Ushio et al. (2023).

Summarization (0-shot) Translation (3-shot) Method Update Param. Dataset size
(Tokens) ROUGE-2 ROUGE-L Morphemes/sec Speed Up BLEU Morphemes/sec Speed Up
Vanilla Fine-tuning 13.0B 365M 20.7 36.1 9.8 1.00x 21.2 10.1 1.00x Vocabulary
Expansion 13.1B 365M 13.7 23.1 17.1 1.92x 12.3 20.2 2.00x Vocabulary
Expansion† 13.1B 60B + 365M 20.3 37.3 20.5 2.12x 20.3 23.1 2.29x MuMo (Ours)
70M 365M 20.5 36.3 15.3 1.73x 21.7 17.2 1.71x

Table 5:  Comparsion with the fine-tuning strategies. The column labeled
“Speed Up” represents the relative performance improvement in inference speed
compared to Vanilla Fine-tuning. Vocabulary Expansion† was pre-trained on over
60B tokens, comprised of both Korean and English text corpora. Other methods
are only trained with the instruction dataset (365M) (Chen et al., 2023c),
ShareGPT and Alpaca translated in Korean. The Boldface signifies the superior
performances, and the second highest score is underlined.

###  4.3 Results

Table 4 shows the generation results in both summarization and translation
tasks. For the summarization task in Korean, MuMo outperforms all baselines in
terms of speed, achieving a 1.92x speed-up over the Vanilla Decoding while
maintaining competitive ROUGE scores. In translation, MuMo again demonstrates
superior efficiency with a 1.70x speed-up and even shows an improvement in
BLEU score compared to Vanilla Decoding.

In the case of Japanese, the results are similar, with MuMo achieving a 2.02x
speed-up in summarization and a 1.75x speed-up in translation. The ROUGE and
BLEU scores for MuMo are on par with or slightly below Vanilla Decoding,
indicating that the increase in speed does not significantly compromise the
quality of the output.

Shortlisting shows only marginal gains in speed across both languages and
every tasks, while preserving the generation capability. This is likely
because the relative computational cost of processing the embedding matrix is
reduced in larger models, making vocabulary reduction less impactful (Berard
et al., 2021; Ushio et al., 2023). On the other hand, the Spec. heavily relies
on the capacity of the draft model, as shown as the comparison with (Spec. w/o
Rejection). If the draft model lacks of sufficient multilingual capacity, it
may not generate high-quality candidates, leading to a lower acceptance rate
by the original model and thus reduced efficiency.

The superior performance of MuMo in terms of inference speed can be primarily
attributed to its capability to predict larger linguistic units compared to
those in the pre-trained vocabulary. We found that the target language tokens
in $\mathcal{V}_{\text{mono}}$ are typically tokenized into 3-4 separate
tokens in $\mathcal{V}_{\text{multi}}$, suggesting that the decoding step
could potentially be reduced by 3-4 times. It is hypothesized that the
inference speed is significantly influenced by the disparity between the pre-
trained multilingual vocabulary and the target language.

##  5 Further Analysis

Summarization (0-shot) Translation (3-shot) LM Head Initialization ROUGE-2
ROUGE-L BLEU Mono-init 20.7 36.2 21.5 Random-init 19.2 35.5 17.2 Multi-Init
20.3 36.3 21.7

Table 6: Comparative analysis for the initialization strategy. Mono-Init
signifies to leverage the pre-existing embedding representation. We use the
language model head of the monolingual model from EleutherAI/polyglot-
ko-12.8b. In the case of Random-Init, we randomly initialize with Gaussian
distribution. Multi-Init indicates to leverage multilingual model
representation by averaging its subword embedding as the main experiment. The
Boldface signifies the superior performances.

Summarization (0-shot) Translation (3-shot) Lang Method ROUGE-2 ROUGE-L
Tokens/sec BLEU Tokens/sec Ko MuMo 20.3 35.9 55.3 21.7 50.9 w/o Verification
11.0(-9/3) 26.4(-9.5) 60.8(+5.5) 16.3(-5.4) 62.3(+11.4) Ja MuMo 11.6 26.3 59.2
24.3 58.3 w/o Verification 6.7(-4.9) 20.4(-5.9) 69.1(+9.9) 10.8(-13.5)
73.6(+15.3)

Table 7: Ablation Study. While the exclusion of the verification accelerates
approximately 1.2 times in inference speed, it significantly compromises the
quality of the generation.

###  5.1 Comparative Analysis of Fine-Tuning Strategies

In the section, we provide a comparative analysis of three distinct fine-
tuning strategies for multilingual models. This analysis aims to highlight the
advantages and disadvantages of each strategy, particularly in terms of
dataset requirements and the number of parameters to train.

####  5.1.1 Setup

The two strategies compared in the analysis are:

1\. Vanilla Fine-tuning: This strategy, which serves as a baseline, involves
fine-tuning a standard multilingual model on the instruction dataset without
any modifications to the pre-trained vocabulary.

2\. Vocabulary Expansion: Inspired by prior work Chau et al. (2020); Cui et
al. (2023), this strategy involves expanding the vocabulary of the pre-trained
multilingual model and fine-tuning on the instruction dataset. This method,
unlike MuMo, expands not only the LM head but also the token embedding in the
input layer. Two implementations of this strategy are considered. The first
involves pre-training on large-scale text corpora (60B tokens)333We use the
off-the-shelf checkpoint from beomi/llama-2-koen-13b before fine-tuning on the
instruction dataset. This strategy is marked with a dagger in Table 5. The
second only undergoes the fine-tuning phase on the instruction dataset.

To account for the variability of token unit between the different strategies,
we report the inference speed with the morphemes per second (Morphemes/sec),
providing a standardized measurement.444python-mecab-ko.

####  5.1.2 Discussion

Table 5 reveals a consistent trend across both summarization and translation
tasks. The vocabulary expansion strategies, which expand the dimension of both
the token embeddings and LM head, exhibit significant increases in inference
speed, but this is accompanied by a substantial decrease in the quality of the
generated output when not trained on large-scale text corpora. This indicates
that merely fine-tuning with an expanded vocabulary on a limited downstream
dataset may not suffice to maintain high-quality text generation, as suggested
by Conneau et al. (2020). Furthermore, while vocabulary expansion with pre-
training achieves notable speed improvements, it does not exhibit significant
enhancements in generation quality.

In contrast, our proposed method exhibits a modest increase in speed while
also slightly improving BLEU scores relative to vanilla fine-tuning. The
principal advantage of our method lies in its capacity to attain these results
without necessitating vast monolingual text corpora. This approach not only
reduces the number of parameters that need to be fine-tuned, making it more
parameter-efficient but also lessens the dependency on large-scale data for
pre-training, making it a more data-efficient solution.

###  5.2 Initialization of Target Monolingual LM Head

We investigate the impact of 3 different initialization strategies on the
target monolingual LM head $g_{\text{mono}}$ in the Target Monolingual LM
head. The first strategy involves leveraging embeddings that correspond to the
pre-trained representation of a targeted monolingual LM head, termed as Mono-
init. The second strategy is initializing the parameters with random value
using Gaussian distribution (Random-init). Lastly, we utilize the embeddings
from the pre-trained multilingual LM head (Multi-init), as the main
experiment. This is achieved by averaging the output embeddings of the
multilingual model.

Table 6 shows that Multi-init achieves a ROUGE-L score of 36.3 and a BLEU
score of 21.7, which are close to the 36.2 ROUGE-L and 20.9 BLEU scores of
Mono-init. On the other hand, Random-init shows a decrease in performance,
with a ROUGE-L score of 35.5 and a BLEU score of 17.2.

The result demonstrates that the Multi-init approach is almost equally
effective with Mono-init. This suggests that our framework can be utilized
even without a pre-existing target monolingual model, making it applicable to
low-resource languages.

###  5.3 Effectiveness of Verification Step

We design an ablation study to investigate the role of the verification step
in the inference process ( Sec. 3.3.2). To assess the impact of the
verification step, we generated sequences without employing the verification
step.

From the results in Table 7, conducted in both Korean and Japanese, we notice
that the overall generation speed is approximately 1.2 times faster when the
verification is excluded. However, it is crucial to highlight that the
exclusion of the verification step in the inference phase leads to a
significant reduction in the generation quality. This is evident in the
decrease in ROUGE-2, ROUGE-L, and BLEU scores for both languages when the
verification module is not used, as shown in the table. This suggests that
while the verification step may slightly slow down the generation process, it
plays a vital role in preserving the model’s generation capability.

##  6 Related Work

##### Tokenization Disparity

Subword tokenization, a common approach in LMs, is typically data-driven. Most
of pre-trained tokenizers, which are often trained on predominantly English
corpora, frequently result in excessive fragmentation of non-English scripts
(Rust et al., 2021; Zhang et al., 2022a). Ahia et al. (2023); Petrov et al.
(2023) have found significant tokenization disparities across languages across
popular LLMs (Xue et al., 2021, 2022; Scao et al., 2022; OpenAI, 2023) with a
depth analysis. Our work endeavors to address the slowdown in inference that
arises due to tokenization disparity in non-alphabetic languages.

##### Modifying Pre-trained Vocabulary

Previous works have explored the adaptation of pre-trained vocabularies or the
addition of new tokens (Artetxe et al., 2020; Rust et al., 2021; Liu et al.,
2023), these methods often necessitate extensive pre-training to integrate the
new tokens effectively (Wang et al., 2019b; Chau et al., 2020; Cui et al.,
2023; Liu et al., 2023). In contrast, our MuMo framework sidesteps the need
for fine-tuning the parameters of pre-trained models to preserve the original
capabilities of the pre-trained language model. Efforts to select items of
pre-trained embedding matrix have been made (Abdaoui et al., 2020; Domhan et
al., 2022; Ushio et al., 2023), but these have not yielded significant speed
up in our experiment, where the relative size of the embedding layer is small
(Bogoychev et al., 2023).

##### Accelerating LLM Inference

The quest to accelerate inference in auto-regressive large language models
(LLMs) has led to a variety of approaches. There has been a proliferation of
systems specifically engineered for LLM inference, such as Orca (Yu et al.,
2022), FlexGen (Sheng et al., 2023), and StreamingLLM (Xiao et al., 2023). Our
proposed methodology can be harmonically integrated with the aforementioned
techniques. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023a)
have also been explored to increase inference velocity. However, the approach
often relies on the assumption that a small model can maintain high fidelity
when generating a series of multiple tokens. Moreever, acquiring a small yet
competitive model may be tricky, especially in a multilingual setup (Conneau
et al., 2020; Bandarkar et al., 2023). Our work distinguishes itself by
specifically solving the inference inefficiency that arises from excessive
fragmentation in the non-alphabetic context.

##### Parameter Efficient Cross-lingual Transfer Learning

The curse of multilinguality, which refers a trade-off between the language
coverage and model capacity (Conneau et al., 2020), is a significant issue
even in massively multilingual models, such as mBERT, XLM-R, and mT5 (Devlin
et al., 2019; Conneau et al., 2020; Xue et al., 2021; Ansell et al., 2021).
The problem has been mitigated through modular parameter-efficient adaptations
of the multilingual models through lightweight adapters (Houlsby et al.,
2019): additional trainable parameters inserted into the transformer layers of
model (Pfeiffer et al., 2020; Üstün et al., 2020; Vidoni et al., 2020; Parović
et al., 2022) for a target language. Previous studies focus on improving the
representational capability of multilingual models, rather than directly
addressing the issue of speeding up multilingual models. In contrast, our
research primarily aims to accelerate the inference for multilingual models in
specific target languages.

##  7 Conclusion

Our study has successfully tackled the challenges in generating text for non-
alphabet languages, particularly those associated with excessive fragmentation
issues. The approach not only speeds up text generation but also paves the way
for more efficient multilingual language applications. Our future work will
broaden our experimental scope to languages that were not substantially
represented in the pre-trained corpus.

## Acknowledgements

We thank Chaehun Park for their valuable discussions and feedback on this
paper.

## References

  * Abdaoui et al. (2020) Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020.  Load what you need: Smaller versions of multilingual bert.  In _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_.
  * Ahia et al. (2023) Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023.  Do all languages cost the same? tokenization in the era of commercial language models.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ansell et al. (2021) Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021.  MAD-G: Multilingual adapter generation for efficient cross-lingual transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Antropic (2023) Antropic. 2023.  Model card and evaluations for claude models.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Berard et al. (2021) Alexandre Berard, Dain Lee, Stephane Clinchant, Kweonwoo Jung, and Vassilina Nikoulina. 2021.  Efficient inference for multilingual neural machine translation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Bogoychev et al. (2023) Nikolay Bogoychev, Pinzhen Chen, Barry Haddow, and Alexandra Birch. 2023.  Large language model inference with lexical shortlisting.  _arXiv preprint arXiv:2311.09709_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chau et al. (2020) Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020.  Parsing with multilingual BERT, a small corpus, and a small treebank.  In _Findings of the Association for Computational Linguistics: EMNLP_.
  * Chen et al. (2023a) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023a.  Accelerating large language model decoding with speculative sampling.  _arXiv preprint arXiv:2302.01318_.
  * Chen et al. (2023b) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield. 2023b.  Monolingual or multilingual instruction tuning: Which makes a better alpaca.  _arXiv preprint arXiv:2309.08958_.
  * Chen et al. (2023c) Zhihong Chen, Shuo Yan, Juhao Liang, Feng Jiang, Xiangbo Wu, Fei Yu, Guiming Hardy Chen, Junying Chen, Hongbo Zhang, Li Jianquan, Wan Xiang, and Benyou Wang. 2023c.  MultilingualSIFT: Multilingual Supervised Instruction Fine-tuning.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Dao (2023) Tri Dao. 2023.  Flashattention-2: Faster attention with better parallelism and work partitioning.  _arXiv preprint arXiv:2307.08691_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Domhan et al. (2022) Tobias Domhan, Eva Hasler, Ke Tran, Sony Trenous, Bill Byrne, and Felix Hieber. 2022.  The devil is in the details: On the pitfalls of vocabulary selection in neural machine translation.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Eloundou et al. (2023) Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023.  Gpts are gpts: An early look at the labor market impact potential of large language models.  _arXiv preprint arXiv:2303.10130_.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021.  Xl-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics, ACL)_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Hong et al. (2021) Jimin Hong, Taehee Kim, Hyesu Lim, and Jaegul Choo. 2021.  Avocado: Strategy for adapting vocabulary to downstream domain.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Jean et al. (2015) Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015.  Montreal neural machine translation systems for WMT’15.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Lan et al. (2023) Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023.  Copy is all you need.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023.  Fast inference from transformers via speculative decoding.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Liang et al. (2023) Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023.  Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_.
  * Liu et al. (2023) Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023.  Chipnemo: Domain-adapted llms for chip design.  _arXiv preprint arXiv:2311.00176_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Murray and Chiang (2018) Kenton Murray and David Chiang. 2018.  Correcting length bias in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_.
  * Parović et al. (2022) Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022.  BAD-X: Bilingual adapters improve zero-shot cross-lingual transfer.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Petrov et al. (2023) Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023.  Language model tokenizers introduce unfairness between languages.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  Mad-x: An adapter-based framework for multi-task cross-lingual transfer.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021.  How good is your tokenizer? on the monolingual performance of multilingual language models.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, Franccois Yvon, and Matthias Gallé. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _ArXiv_ , arXiv preprint arXiv:2211.05100.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Shazeer (2020) Noam Shazeer. 2020.  Glu variants improve transformer.  _arXiv preprint arXiv:2002.05202_.
  * Sheng et al. (2023) Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E Gonzalez, et al. 2023.  High-throughput generative inference of large language models with a single gpu.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Solaiman et al. (2023) Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, et al. 2023.  Evaluating the social impact of generative ai systems in systems and society.  _arXiv preprint arXiv:2306.05949_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Ushio et al. (2023) Asahi Ushio, Yi Zhou, and Jose Camacho-Collados. 2023.  An efficient multilingual language model compression through vocabulary trimming.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): Findings_. Association for Computational Linguistics.
  * Üstün et al. (2020) Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020.  UDapter: Language adaptation for truly Universal Dependency parsing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Vidoni et al. (2020) Marko Vidoni, Ivan Vulić, and Goran Glavaš. 2020.  Orthogonal language and task adapters in zero-shot cross-lingual transfer.  _arXiv preprint arXiv:2012.06460_.
  * Wang et al. (2019a) Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2019a.  Neural machine translation with byte-level subwords.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Wang et al. (2019b) Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019b.  Improving pre-trained multilingual model with vocabulary expansion.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. Association for Computational Linguistics.
  * Xiao et al. (2023) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023.  Efficient streaming language models with attention sinks.  _arXiv preprint arXiv:2309.17453_.
  * Xue et al. (2022) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  In _Transactions of the Association for Computational Linguistics_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Yu et al. (2022) Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022.  Orca: A distributed serving system for Transformer-Based generative models.  In _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_.
  * Zhang et al. (2022a) Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022a.  How robust is neural machine translation to language imbalance in multilingual tokenizer training?  In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_.
  * Zhang et al. (2022b) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022b.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.

## Appendix

##  Appendix A Implementation Details

Hyperparameter Value temperature 0.1 sampling True $p$ for top-$p$ sampling
0.7 repetition penalty 1.05 exponential decay length penalty (256, 1.03) max
sequence length 512 $k$ for top-$k$ sampling 20

Table 8: Hyperparameters used for generation.