# Tokenizer Choice For LLM Training: Negligible or Crucial?

Mehdi Ali1,2 †, Michael Fromm1,2 †, Klaudia Thellmann3 †
Richard Rutmann1,2, Max Lübbering1,2, Johannes Leveling1, Katrin Klug1, Jan
Ebert4,
Niclas Doll1, Jasper Schulze Buschhoff1, Charvi Jain1,2, Alexander Arno
Weber1,2,
Lena Jurkschat3, Hammam Abdelwahab1 Chelsea John4, Pedro Ortiz Suarez5, Malte
Ostendorff5
Samuel Weinbach6, Rafet Sifa1, Stefan Kesselheim4, Nicolas Flores-Herr1

1Fraunhofer IAIS, 2Lamarr Institute, 3TU-Dresden, 4FZ Jülich, 5DFKI, 6Aleph
Alpha

###### Abstract

The recent success of Large Language Models (LLMs) has been predominantly
driven by curating the training dataset composition, scaling of model
architectures and dataset sizes and advancements in pretraining objectives,
leaving tokenizer influence as a blind spot. Shedding light on this
underexplored area, we conduct a comprehensive study on the influence of
tokenizer choice on LLM downstream performance by training 24 mono- and
multilingual LLMs at a 2.6 B parameter scale, ablating different tokenizer
algorithms and parameterizations. Our studies highlight that the tokenizer
choice can significantly impact the model’s downstream performance, training
and inference costs. In particular, we find that the common tokenizer
evaluation metrics fertility and parity are not always predictive of model
downstream performance, rendering these metrics a questionable proxy for the
model’s downstream performance. Furthermore, we show that multilingual
tokenizers trained on the five most frequent European languages require
vocabulary size increases of factor three in comparison to English. While
English-centric tokenizers have been applied to the training of multi-lingual
LLMs, we find that this approach results in a severe downstream performance
degradation and additional training costs of up to 68%, due to an inefficient
tokenization vocabulary.

†††Equal contribution.

##  1 Introduction

LLMs have shown impressive capabilities in many downstream tasks in a
zero/few-shot setting such as summarization, reading comprehension,
translation, and commonsense reasoning Brown et al. (2020b); Touvron et al.
(2023). To train a LLM, the currently established approach is to employ a
tokenizer that splits the training documents into tokens where a token
represents a word Bengio et al. (2000), a sub-word Schuster and Nakajima
(2012); Sennrich et al. (2015); Wang et al. (2020), or a single character Gao
et al. (2020b), and each token is represented in the model by an embedding
vector that can be further processed.

The quality of a tokenizer can be assessed intrinsically and extrinsically. An
intrinsic evaluation solely addresses the characteristics of tokenizers and
their generated output in isolation, whereas the extrinsic evaluation measures
the impact of the tokenizer on a downstream component, e.g., the Large
Language Model (LLM).

While many different tokenization approaches have been proposed, ranging from
character-based to word-based methods, the potential impact of different
tokenizers is neglected w.r.t. LLMs, especially in the context of multilingual
LLMs. Recent work proposed by Petrov et al. (2023) demonstrates that
carelessly designed tokenizers applied to the training of multilingual LLMs
result in severe inequalities and limitations across languages. In fact, text
passages translated into different languages resulted in tokenized sequences
that differ in length up to a factor of 15, affecting inference costs and
latency during inference. Furthermore, it is known that an essential property
for effectively learning transformer-based LLMs is the learning of long-range
dependencies Vaswani et al. (2017). Given a fixed sequence length, learning to
relate words far apart in the input text is impossible for languages whose
text is excessively fragmented by the tokenizer.

Despite the importance of tokenizers and the potentially severe impact of
poorly performing tokenizers, there exists so far no extensive study that
holistically investigates the intrinsic and extrinsic tokenizer performance in
a monolingual and multilingual setting with a focus on decoder-only models,
which represent the backbone of current LLMs.

Therefore, we address this gap and conduct an extensive study in which we
measure the impact of the tokenizer on the model performance. In particular,
we make the following contributions:

  * •

We conduct a study investigating the intrinsic tokenizer performance.

  * •

We conduct a study investigating the extrinsic tokenizer performance, i.e.,
the impact of the tokenizer on the model’s downstream performance.

  * •

Investigate whether a correlation between the intrinsic and the extrinsic
tokenizer performance exists.

##  2 Related Work

This section provides an overview of tokenization algorithms and their usage
in encoder- and decoder-only transformer models.

###  2.1 Tokenization Approaches

##### Word Tokenization.

The most basic tokenization approach is to split sequences based on white
spaces and consider each word as a token Bengio et al. (2000).

##### Subword tokenization.

_Subword tokenization_ algorithms are data-driven tokenization approaches
which can decompose words into subwords/multiple tokens and currently
represent the established tokenization approach upon which LLMs rely Kudo and
Richardson (2018); Petrov et al. (2023). Because subword tokenizers decompose
words into subwords, they can process out-of-vocabulary words by merging
subwords from the vocabulary Kudo and Richardson (2018). Examples of popular
subword tokenizers are WordPiece Schuster and Nakajima (2012), BPE Gage
(1994); Sennrich et al. (2015), Byte-Level BPE (BBPE) Wang et al. (2020), and
Unigram Kudo (2018).

##### Character Tokenization.

Tokenization can also be performed on a character level or based on UTF-8
bytes. However, this results in an increased sequence length, which becomes
computationally expensive in the transformer architecture, the current
predominated architecture for LLMs due to the quadratic complexity of the
self-attention layer in the sequence length Vaswani et al. (2017). Though,
several approaches have been proposed to address this limitation Gao et al.
(2020b); Tay et al. (2021); Xue et al. (2022); Clark et al. (2022); Yu et al.
(2023).

###  2.2 Tokenizers in Transformers Models

##### Tokenizers in Encoder Models

Most research on tokenization has been conducted on encoder models. Rust et
al. (2021) investigated whether the tokenizer choice impacts the downstream
performance of multi- and monolingual BERT Devlin et al. (2018) models. Zhang
et al. (2022) showed that better machine translation performance is often
obtained when languages are equally sampled during the tokenizer training.
Toraman et al. (2023) trained several medium-sized language models for Turkish
and suggested that different subword tokenizers perform roughly equivalent,
whereas word- and character-level tokenizers perform drastically worse on
downstream tasks. Finally, Chirkova and Troshin (2022) analyzed the effect of
employing different tokenizations on code-related tasks and demonstrated that
carefully configured tokenizers could reduce average sequence length up to 40%
or allow for small downstream performance improvements by up to 2% at a lower
compression rate.

##### Tokenizers in Decoder Models

An overview of current mono- and multilingual LLMs is provided in Lin et al.
(2022); Shliazhko et al. (2022); Scao et al. (2022). As part of their work,
Shliazhko et al. (2022) ablated different tokenizer pre-processing approaches
while keeping the tokenizer algorithm, the vocabulary size and the employed
implementation fixed. In none of the other major LLMs works the extrinsic
tokenizer performance has been studied.

##  3 Approach

To investigate the tokenizer impact on the model performance, we conducted an
extensive ablation study. In detail, we created dedicated datasets for the
training of the tokenizers and the models, trained BPE and Unigram tokenizers,
and for each tokenizer we trained decoder-only models with a size of 2.6B
parameters while keeping the remaining configuration (i.e., dataset and model
hyper-parameters) fixed. This allowed us to measure the tokenizer’s impact on
the model’s downstream performance in isolation.

###  3.1 Data

While creating our tokenizer and model training datasets, we ensure that the
mixture proportions of data domains (Wikipedia, books, web text) follow the
same distribution to avoid a domain shift between tokenizers training and
model training. We created _two datasets_ with 70B words where one of the
datasets is monolingual, containing English documents, and the second is a
multilingual dataset comprised of English, German, French, Italian, and
Spanish documents. Our datasets are filtered and deduplicated and consist of
web-crawled data (80%) and curated data (20%), comparable to related datasets
used to train LLMs. In the multilingual dataset, the amount of web-crawled
data is equally distributed across languages in terms of number of words.
Further details about our data pipeline and the data composition are described
in Table 5.

###  3.2 Tokenizer

Our studies rely on the two established tokenization algorithms, BPE and
Unigram, and their implementation in the Huggingface tokenizer library Moi and
Patry (2023) and the SentencePiece library Kudo and Richardson (2018). We
considered both libraries in order to investigate the effect of differences in
the pre-and post-processing steps and potential differences in the
implementations. Due to missing pre-processing options for Huggingface’s
Unigram implementation, which causes a large discrepancy in the resulting
vocabulary compared to SentencePiece’s implementation of Unigram, we omitted
the training of Unigram tokenizers based on Huggingface. Overall, we trained
24 different tokenizers, where one-half of the tokenizers were monolingual
English tokenizers, and the other half of the tokenizers were multilingual
tokenizers. Besides the tokenizer algorithm, language composition, and
employed tokenizer library, we also varied the vocabulary size. Concrete
tokenizer configurations are described in the Appendix B.

###  3.3 Models

To measure the impact of our trained tokenizers on the model downstream
performance, we trained one model for each tokenizer. In particular, for each
of our 24 trained tokenizers, we trained a 2.6B transformer-based decoder-only
model on up to 52B tokens following the scaling law proposed by Hoffmann et
al. (2022a). Additionally, serving as baselines, we trained a monolingual and
a multilingual model using the pre-trained GPT-2 tokenizer. All models have
been trained based on the causal language modeling training objective.

###  3.4 Evaluation

To assess the impact of the tokenizers on the model downstream performance, we
first performed an intrinsic tokenizer evaluation, followed by an extrinsic
evaluation, and finally, we investigated whether a correlation between both
evaluation approaches is given.

The intrinsic evaluation aims to assess the generated output of tokenizers
based on fertility and parity. Furthermore, the tokenizer’s vocabulary overlap
with other tokenizers is computed. The intrinsic evaluation does not assess
the impact of tokenizers on the model performance.

Fertility, the most common metric to evaluate a tokenizer’s performance Scao
et al. (2022); Stollenwerk (2023); Rust et al. (2021), is defined as the
average number of tokens that are required to represent a word or document.
For a tokenizer $T$ and dataset $A$, the fertility can be calculated as the
number of tokens in $A$ (when $T$ is applied) divided by the number of words
in $A$. We calculate the fertility on a held-out set (10,000 documents), which
was not used for the tokenizer training. For calculating the words of a
document, we used whitespace splitting. Higher fertility scores correspond to
weaker compression capabilities of the tokenizer.

Parity Petrov et al. (2023), which has been recently proposed, assesses how
fairly a tokenizer treats equivalent sentences in different languages. A
tokenizer $T$ achieves parity for language $A$ with respect to language $B$ if
$\frac{|T(s_{A})|}{|T(s_{B})|}\approx 1$, where $s_{A}$ and $s_{B}$ denote the
sets of all sentences in the corpora of language $A$ and $B$, respectively. We
use the FLORES-200 Goyal et al. (2022) parallel corpus, consisting of the same
sentences human-translated into 200 languages. We calculate the parity values
for each tokenizer and the four non-English languages with respect to English
(see Fig. 2 for an overview).

The extrinsic evaluation aims to explicitly assess the impact of a tokenizer
on the model’s downstream performance. We selected a comprehensive set of
downstream tasks covering commonsense reasoning, reading comprehension,
translation, and math tasks to measure the downstream performance. For an
exhaustive lists of all tasks, we refer the reader to Table 11 in Appendix E.

Additionally, we computed the impact of a tokenizer on the average
computational costs of a given model per word during training and inference.
The computational costs during training for one step including the forward and
the backward pass can be estimated by

$C=96Bslh^{2}\left(1+\dfrac{s}{6h}+\dfrac{V}{16lh}\right),$ (1)

given a model with batch size $B$, sequence length $s$, $l$ layers, hidden
size $h$ and vocabulary size $V$ Narayanan et al. (2021). The costs per token
can be derived by $C_{\text{token}}=C/Bs$ and the average costs per word by
$C_{\text{word}}=C_{\text{token}}\cdot\text{fertility}$.

##  4 Intrinsic Tokenizer Evaluation

In our intrinsic evaluation, we first compare the fertility and parity of the
trained tokenizers (Section 4.1) and the overlap of their vocabularies
(Section 4.2).

###  4.1 Fertility & Parity

(a) Non-English, multilingual documents

(b) English documents

Figure 1: Comparison of fertility scores between monolingual (English)
tokenizer and multilingual tokenizers applied to (a) English and (b) non-
English, multi-lingual documents.

Applying the described fertility and parity evaluation to the
mono-/multilingual tokenizers, our analysis highlights the following two major
aspects, as visible in Fig. 1 and Fig. 2.

Firstly, it can be observed that applying a monolingual tokenizer to
multilingual data results in significantly higher fertility and parity scores
(see Fig. 1(a) and Fig. 2). While multilingual tokenizers have lower fertility
than monolingual English tokenizers on all non-English documents by a large
margin, they are only slightly worse on tokenizing English documents, as shown
in Fig. 1(b).

Secondly, with increasing vocabulary size, fertility and parity reduce in all
cases, which can be explained by the tokenizer requiring fewer sub-word tokens
when tokenizing text given a larger vocabulary. However, it can be observed
that for monolingual English tokenizers, the fertility is less dependent on
the vocabulary when tokenizing English documents, implying that 33k might be a
sufficiently large vocabulary. This is especially interesting considering the
fact that the LLaMA/LLaMA2 models have been trained with a vocabulary size of
33k Touvron et al. ; Touvron et al. (2023).

Figure 2: Comparison of parity scores between monolingual (English) tokenizer
and multilingual tokenizers applied multi-lingual documents.

###  4.2 Vocabulary Overlap

To analyze the tokenizer similarity, we calculated the vocabulary overlap.
Particularly, we assess Huggingface’s and SentencePiece’s BPE implementations.
Their vocabulary overlap is depicted in Table 1.

The overlap is roughly constant across different vocabulary sizes, and the
total overlap tends to be rather low, despite being the identical algorithm
only implemented by two different libraries. Consequently, the tokenizers
produce different tokenized sequences, possibly affecting model training and
downstream performance. Investigating the underlying reasons, the low overlap
might be attributed to different configuration and pre-processing options in
these libraries. Due to the larger thesaurus in multilingual documents, the
overlap for the multilingual tokenizer is lower than for the English
tokenizers.

| 33k | 50k | 82k | 100k
---|---|---|---|---
English | 0.77 | 0.76 | 0.74 | 0.74
Multilingual | 0.62 | 0.62 | 0.62 | 0.61
Table 1: Vocabulary overlap between the HuggingFace and SentencePiece BPE
tokenizer for different vocab sizes.

(a) Non-English documents

(b) English documents

(c) German documents

Figure 3: Average compute (GFLOPs) required to process a single word within
(a) multilingual, (b) English, and (c) German documents within a full training
pass (including the backward pass).

##  5 Extrinsic Tokenizer Evaluation

In the following, we describe the results of our extrinsic evaluation of
tokenizers. Section 5.1 describes the experimental setup, Section 5.2 presents
the downstream performance of the trained models based on the investigated
tokenizers, and Section 5.3 analyses the computational costs associated with
each tokenizer when employed in a specific model.

###  5.1 Experimental Setup

To assess the impact of the tokenizers on the model downstream performance, we
trained a decoder-only transformer model of size 2.6 B for each tokenizer. We
trained our models for 52.6 B tokens following the scaling law proposed by
Hoffman et al. Hoffmann et al. (2022b) based on the causal language modeling
training objective. The hyper-parameters are described in Table 8 in the
Appendix C.

We evaluated our models in zero-shot settings on a wide range of tasks,
covering commonsense reasoning (HellaSwag Zellers et al. (2019), WinoGrande
Sakaguchi et al. (2020), ARC easy and challenge Clark et al. (2018)), world
knowledge (TriviaQA Joshi et al. (2017)), reading comprehension (LAMBADA
Paperno et al. (2016) and their machine translated versions for German,
Spanish, French, and Italian, RACE Lai et al. (2017), BoolQ Clark et al.
(2019)), math (MATH Hendrycks et al. (2021)), multilingual natural language
inference (XNLI Conneau et al. (2018)).

###  5.2 Downstream Performance

We split our analysis of the downstream performance into several parts. First,
we discuss the overall results obtained for the investigated tokenizers,
followed by presenting the impact of the tokenizer library (Section 5.2.1),
the impact of the tokenizer algorithm (Section 5.2.2), and the impact of the
vocabulary size (Section 5.2.3).

(a) Monolingual tokenizers.

(b) Multilingual tokenizers.

Figure 4: Top-1/-2/-3 count for mono-/multilingual tokenizers across mono-/multilingual tasks the tasks were evaluated on the accuracy metric. | Model | DE | FR | IT | ES | EN
---|---|---|---|---|---|---
EN | BPE-HF | - | - | - | - | 45.44
BPE-SP | - | - | - | - | 46.31
UNI-SP | - | - | - | - | 46.01
MULTI | BPE-HF | 36.72 | 36.55 | 35.34 | 42.28 | 44.45
BPE-SP | 36.96 | 36.95 | 36.12 | 41.96 | 44.92
UNI-SP | 37.38 | 37.05 | 35.70 | 42.56 | 44.21
Table 2: Impact of the tokenizer algorithm and the tokenizer training library, scores are averaged over the vocabulary sizes. | Vocab | DE | FR | IT | ES | EN
---|---|---|---|---|---|---
EN | 32 | - | - | - | - | 46.34
50 | - | - | - | - | 46.30
82 | - | - | - | - | 45.29
100 | - | - | - | - | 45.88
MULTI | 32 | 36.72 | 36.22 | 35.20 | 41.76 | 44.71
50 | 36.57 | 36.34 | 34.60 | 42.22 | 43.91
82 | 37.40 | 37.22 | 36.21 | 42.30 | 44.75
100 | 37.17 | 37.45 | 36.19 | 42.64 | 44.55
Table 3: Impact of the vocabulary size on the downstream accuracy. The
accuracy scores are averaged over the libraries and tokenizer algorithms

To provide a holistic performance overview, we assess the impact of the
tokenizers on the model performance based on various tasks and analyze the
results from different perspectives instead of providing solely a single
aggregated metric.

This holistic evaluation is presented in Fig. 4, which expresses how often a
tokenizer is among the top-1/-2/-3 performing tokenizers, Fig. 7, which shows
the distribution of the accuracy scores across tasks, and in Table 9 which
summarizes the average performances across all tasks. We dissect the results
from different perspectives in the following subsections.

##### Monolingual Tokenizer

Among the monolingual tokenizers, there can be significant performance
differences. For instance, Fig. 4(a) depicts that the GPT-2 tokenizer is
overall most often across the top-$k$ (with $k\in\\{1,2,3\\}$) performing
tokenizers, whereas the EN-BPE-HF-32 tokenizer is never the best-performing
tokenizer. Though the count-based metric already provides a first overview of
the model performance, it does not illustrate the difference in performance.

Therefore, we also looked at the accuracy distribution across all tasks that
employ accuracy as the evaluation metric. Fig. 7(a) demonstrates that the EN-
BPE-SP-32 tokenizer, on average, is across all tasks among the best-performing
tokenizers and has the highest median value. At the same time, it also
expresses a large variance across tasks compared to other tokenizers such as
EN-BPE-HF-32, implying that for a set of tasks, the tokenizer performs very
well, while on other tasks, it performs comparably poorly.

Aggregated metrics provide a good impression of the overall performance but do
not demonstrate potentially large performance differences for single tasks.
Therefore, we listed in Table 11 the performance difference across all tasks
between the best-performing tokenizer and worst-performing tokenizer on this
task. The performance difference is huge for various tasks, which is
especially interesting since only the tokenizer has been changed while the
model configuration is the same.

Overall, the GPT-2 tokenizer and the EN-BPE-SP-32 tokenizer are strong
tokenizers for monolingual English models. Interestingly, SentencePiece’s
implementation of BPE with a vocabulary size of 32k has been used for LLaMA2.
Our results provide empirical and transparent evidence that both
tokenizers/tokenizer settings are reasonable choices for training monolingual
English models.

##### Multilingual Tokenizer

Our multilingual tokenizer experiments further support that different
tokenizers can express large performance differences. Based on our count-based
metric (Figure 4(b)), the differences across tokenizers are even larger than
for monolingual tokenizers. While Multi-UNI-SP-100 and Multi-UNI-SP-82 are
most often among the strongest tokenizers, Multi-BPE-HF-32 is only a single
time across the top-performing tokenizers.

The accuracy distributions depicted in Figure 7(b) further show that both the
tokenizers mentioned above are among the best-performing tokenizers and that
specific tokenizers such as the already mentioned tokenizer perform worse.
Figure 7(b) additionally demonstrates that the GPT-2 tokenizer performs
poorly. Compared to monolingual models, we can observe that all tokenizers
express a similar variance across tasks.

In reference to Table 11, the analysis of single tasks reveals that for
multilingual tokenizers, the performance difference between tasks can be huge.

Multi-UNI-SP-100 and Multi-UNI-SP-82 are excellent choices, while employing a
pre-trained GPT-2 tokenizer should be omitted to train multilingual models.

####  5.2.1 Impact of the Tokenizer Library

Table 2 demonstrates that BPE-SP, on average, outperforms BPE-HF across all
tasks. In the multilingual setting, BPE-SP outperforms BPE-HF for all
languages except for Spanish. The performance differences might be attributed
to the differences in implementation details of the tokenisers’ pre-and
postprocessing, which could affect the vocabulary creation (see Section 4.2)
and, consequently, the downstream performance.

####  5.2.2 Impact of the Tokenizer Algorithm

Table 2 shows that for English tokenizers and models, BPE works better
compared to Unigram. For multilingual models and tokenizers, we can see that
for Spanish, German and French unigram can result in an improvement in
performance. German and the romance languages are generally inflected
languages, while English, although primarily analytic, does have some
agglutinative features. Recent work showed Unigram tokenizers are better able
to find morphologically accurate segmentations Bostrom and Durrett (2020);
Park et al. (2021) which might help these languages.

####  5.2.3 Impact of the Tokenizer Vocabulary

Analyzing the impact of the vocabulary size revealed that in the monolingual
English setting, the smaller/medium-sized vocabulary performs better (Table 3)
whereas in the multilingual setting, larger vocabulary sizes result in better
downstream performance(Table 3). This is in line with the vocabulary size of
current English LLMs such as LLama.

###  5.3 Computational Costs

Even with an identical vocabulary size, two trained tokenizers most likely
split a text passage in sequences of different lengths, resulting in different
computation costs when processed by a model.

Our analysis revealed the following two insights. Firstly, increasing the
vocabulary size decreases computational costs (Fig. 3). This implies that the
reduction in the overall number of tokens represented by lower fertility (Fig.
1) compensates for the additional computational costs that a larger vocabulary
size introduces in the model. In contrast, during inference, the computational
costs increase with increasing vocabulary size and at least a vocabulary size
of 50K, i.e., solely the forward pass without the backward pass does not
compensate for the additional computational costs introduced by a larger
vocabulary size in the model.

Secondly, the computational training costs for multilingual documents are
significantly lower for multilingual tokenizers than monolingual English
tokenizers for training (Fig. 3(a)) and for inference (in Appendix Fig. 8). In
extreme cases, up to 68% additional training costs (see EN-UNI-SP-100352 vs.
Multi-UNI-SP-50176 in Appendix Table 10 and Fig. 3(c)) as it is the case for
training. At the same time, the difference in computational costs between
multilingual and monolingual English tokenizers for processing English
documents is much lower (Fig. 3(b)). A complete overview of the computational
costs can be found in Table 10.

##  6 Correlation Between Intrinsic And Extrinsic Tokenizer Performance

Figure 5: Correlation of fertility/parity scores and downstream task
performance for all five languages. We evaluated English models on English
tasks, whereas multilingual models are evaluated across all four non-English
tasks.

This section investigates a possible predictive relationship of intrinsic
tokenizer metrics (fertilty and parity) to the extrinsic model downstream
performance.

As highlighted in the correlation heatmaps in Fig. 5, we find that there is no
distinct correlation across all tasks and languages, demanding a more granular
analysis. While for non-English tasks, we mainly observe a correlation between
low fertility and higher downstream performance, the non-English tasks yield
seemingly random positive and negative correlations.

Taking the fertility trends with varying vocabulary sizes (see Fig. 1) into
consideration, we hypothesize that fertility only correlates with downstream
performance in certain language-specific vocabulary size limits. For the
English language, the tokenizers already provide low, close-to-convergence
fertility scores for vocabulary sizes of 32k tokens. While additional tokens
yield only minute fertility improvements, we presume that they do not capture
morphological segmentations and, thus, can harm downstream performance and
significantly increase the computation costs (see Section 5.3) in the end.

In contrast, for multilingual tokenizers, we observe significant fertility
improvements with increasing vocabulary sizes. Due to the larger thesaurus
induced by the additional languages, the tokenizer requires a larger
vocabulary to allow a model to perform convincingly on all languages.
Therefore, only within the non-convergence vocabulary range, we achieve a
strong, negative correlation between fertility and downstream performance with
varying vocabulary sizes.

In conclusion, intrinsic tokenizer metrics such as fertility and parity need
to be taken with a grain of salt and supposedly are only predictive of
downstream model performance in certain bounds. Furthermore, the analysis
showed that low fertility scores can be regarded as a necessary criterion but
not as a sufficient one.

##  7 Conclusion

This work represents a fundamental step to a better understanding of the
impact of the tokenizer on the models’ downstream performance. We have shown
that training tokenizers with a balanced share across languages achieve
comparable low fertility and parity scores across all languages, which has
important implications. Lower fertility results in less computational costs
and enables the model to learn long-range dependencies in limited context
windows.

Furthermore, we highlight that the tokenizer choice can significantly impact
the model’s downstream performance. We could show that it is beneficial to
employ Unigram tokenizers with a large vocabulary size to train multilingual
models. This is contrary to the current monolingual setting where BPE-based
tokenizers are used together with a smaller/medium-sized vocabulary size. This
also implies that insights obtained for monolingual settings are not
transferable to multilingual settings and, therefore, require dedicated
studies.

Finally, we could show that there is no clear correlation between intrinsic
and extrinsic tokenizer performance, but the correlation is rather task-
specific.

## Acknowledgements

This work was funded by the German Federal Ministry for Economic Affairs and
Climate Action (BMWK) through the project OpenGPT-X (project no. 68GX21007D)
as well as by the Federal Ministry of Education and Research of Germany and
the state of North-Rhine Westphalia as part of the Lamarr-Institute for
Machine, LAMARR22B. The authors gratefully acknowledge the Gauss Centre for
Supercomputing e.V. (www.gauss-centre.eu) for funding this project by
providing computing time on the GCS Supercomputer JUWELS at Jülich
Supercomputing Centre (JSC) as well as the Center for Information Services and
High Performance Computing [Zentrum für Informationsdienste und
Hochleistungsrechnen (ZIH)] at TU Dresden for providing its facilities for
high throughput calculations.

## References

  * Abadji et al. (2021) Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021.  Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus.  In Harald Lüngen, Marc Kupietz, Piotr Bański, Adrien Barbaresi, Simon Clematide, and Ines Pisetta, editors, _Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)_ , pages 1 – 9. Leibniz-Institut für Deutsche Sprache, Mannheim.
  * Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000.  A neural probabilistic language model.  _Advances in neural information processing systems_ , 13.
  * Bostrom and Durrett (2020) Kaj Bostrom and Greg Durrett. 2020.  Byte pair encoding is suboptimal for language model pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4617–4624, Online. Association for Computational Linguistics.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Brown et al. (2020b) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chirkova and Troshin (2022) Nadezhda Chirkova and Sergey Troshin. 2022.  CodeBPE: Investigating subtokenization options for large language model pretraining on source code.  In _Deep Learning for Code Workshop_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL-HLT (1)_ , pages 2924–2936. Association for Computational Linguistics.
  * Clark et al. (2022) Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022.  Canine: Pre-training an efficient tokenization-free encoder for language representation.  _Transactions of the Association for Computational Linguistics_ , 10:73–91.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457.
  * Computer (2023) Together Computer. 2023.  Redpajama: An open source recipe to reproduce llama training dataset.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: evaluating cross-lingual sentence representations.  In _EMNLP_ , pages 2475–2485. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gage (1994) Philip Gage. 1994.  A new algorithm for data compression.  _The C Users Journal archive_ , 12:23–38.
  * Gao et al. (2020a) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020a.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gao et al. (2020b) Yingqiang Gao, Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. 2020b.  Character-level translation with self-attention.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1591–1604, Online. Association for Computational Linguistics.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Graën et al. (2019) Johannes Graën, Tannon Kew, Anastassia Shaitarova, and Martin Volk. 2019.  Modelling large parallel corpora: The zurich parallel corpus collection.  In _Proceedings of the 7th Workshop on Challenges in the Management of Large Corpora (CMLC)_ , pages 1–8. Leibniz-Institut für Deutsche Sprache.
  * Graën et al. (2014) J. Graën, D. Batinic, and M. Volk. 2014.  Cleaning the Europarl corpus for linguistic applications.  In _Konvens 2014_. Stiftung Universität Hildesheim.
  * Hajlaoui et al. (2014) Najeh Hajlaoui, David Kolovratnik, Jaakko Vaeyrynen, Ralf Steinberger, and Dániel Varga. 2014.  DCEP - Digital corpus of the European parliament.  In _Proc. LREC 2014 (Language Resources and Evaluation Conference). Reykjavik, Iceland_ , pages 3164–3171.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the MATH dataset.  In _NeurIPS Datasets and Benchmarks_.
  * Hoffmann et al. (2022a) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. 2022a.  An empirical analysis of compute-optimal large language model training.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 30016–30030. Curran Associates, Inc.
  * Hoffmann et al. (2022b) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022b.  Training compute-optimal large language models.  _CoRR_ , abs/2203.15556.
  * Höfler and Piotrowski (2011) Stefan Höfler and Michael Piotrowski. 2011.  Building corpora for the philological study of Swiss legal texts.  _Journal for Language Technology and Computational Linguistics_ , 26(2):77–89.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _ACL (1)_ , pages 1601–1611. Association for Computational Linguistics.
  * Koehn (2005) P. Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Machine Translation Summit, volume 5_ , pages 79––86. Asia-Pacific Association for Machine Translation (AAMT).
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _EMNLP 2018_ , page 66.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017.  RACE: large-scale reading comprehension dataset from examinations.  In _EMNLP_ , pages 785–794. Association for Computational Linguistics.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and tv subtitles.  In _Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC-2016)_.
  * Moi and Patry (2023) Anthony Moi and Nicolas Patry. 2023.  HuggingFace’s Tokenizers.
  * Narayanan et al. (2021) Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021.  Efficient large-scale language model training on gpu clusters using megatron-lm.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’21, New York, NY, USA. Association for Computing Machinery.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _ACL (1)_. The Association for Computer Linguistics.
  * Park et al. (2021) Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021.  Morphology Matters: A Multilingual Language Modeling Analysis.  _Transactions of the Association for Computational Linguistics_ , 9:261–276.
  * Petrov et al. (2023) Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023.  Language model tokenizers introduce unfairness between languages.  _arXiv preprint arXiv:2305.15425_.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021.  How good is your tokenizer? on the monolingual performance of multilingual language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3118–3135, Online. Association for Computational Linguistics.
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  Winogrande: An adversarial winograd schema challenge at scale.  In _AAAI_ , pages 8732–8740. AAAI Press.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5149–5152. IEEE.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mgpt: Few-shot learners go multilingual.  _arXiv preprint arXiv:2204.07580_.
  * Stollenwerk (2023) Felix Stollenwerk. 2023.  Training and evaluation of a multilingual tokenizer for gpt-sw3.  _arXiv preprint arXiv:2304.14780_.
  * Tay et al. (2021) Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021.  Charformer: Fast character transformers via gradient-based subword tokenization.  In _International Conference on Learning Representations_.
  * Toraman et al. (2023) Cagri Toraman, Eyup Halit Yilmaz, Furkan Sahinuc, and Oguzhan Ozcelik. 2023.  Impact of tokenization on language models: An analysis for turkish.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ , 22(4).
  * (45) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: open and efficient foundation language models, 2023.  _URL https://arxiv. org/abs/2302.13971_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wang et al. (2020) Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020.  Neural machine translation with byte-level subwords.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):9154–9160.
  * Xue et al. (2022) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  _Transactions of the Association for Computational Linguistics_ , 10:291–306.
  * Yu et al. (2023) Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023.  Megabyte: Predicting million-byte sequences with multiscale transformers.  _arXiv preprint arXiv:2305.07185_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _ACL (1)_ , pages 4791–4800. Association for Computational Linguistics.
  * Zhang et al. (2022) Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022.  How robust is neural machine translation to language imbalance in multilingual tokenizer training? In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 97–116, Orlando, USA. Association for Machine Translation in the Americas.

##  Appendix A Corpora

Name | Language | #Words
---|---|---
Oscar | DE | 11.200.000.000
Oscar | ES | 11.200.000.000
Oscar | EN | 11.200.000.000
Oscar | IT | 11.200.000.000
Oscar | FR | 11.200.000.000
Pile | DE | 13.838.432
Pile | ES | 21.990.512
Pile | EN | 4.334.313.669
Pile | IT | 7.946.402
Pile | FR | 15.857.811
RedPajama | DE | 143.907.461
RedPajama | ES | 112.950.000
RedPajama | EN | 4.663.646.781
RedPajama | IT | 137.802.711
RedPajama | FR | 139.749.147
RedPajama | Code | 2.052.228.788
Misc | DE | 600.844.912
Misc | ES | 186.934.269
Misc | EN | 1.337.030.904
Misc | IT | 19.810.753
Misc | FR | 211.147.445
Total |  | 70.000.000.000
Table 4: Overview of the multilingual 70B words dataset with language, number of sampled words Name | Language | # words
---|---|---
Oscar | EN | 56.000.000.000
Pile | EN | 4.893.724.288
RedPajama | EN | 5.308.974.750
RedPajama | Code | 2.299.301.635
Misc | EN | 1.497.999.327
Total |  | 70.000.000.000
Table 5: Overview of the English 70B words dataset with language, number of
sampled words

Our web documents in the corpora consist of Oscars111https://oscar-
project.org/ Abadji et al. (2021), that were generated by the ungoliant
pipeline222https://github.com/oscar-project/ungoliant based on three Common
Crawl WET Archives (2022-27, 2022-49 and 2023-14).

The curated datasets consist of The Pile Gao et al. (2020a), RedPajama
Computer (2023), and single datasets that do not belong to a collection. From
the Pile subcorpora, we selected: Phil Archive, PMC Abstracts, PMC Extracts,
OpenWebText, NIH Exporterm, and Free Law Opinions V2. From RedPajama we use:
ArXiv, Books, Github, StackExchange, and Wikipedia.

The remaining datasets are:

Hyper-Parameter | Value(s)
---|---
model_type | Unigram $|$ BPE
vocab_size | 33k $|$50k $|$ 82k $|$ 100k
character_coverage | 0.9999
split_by_number | True
allow_whitespace_only_pieces | True
add_dummy_prefix | True
user_defined_symbols_1 | <s>, </s>,<pad>,<eod>
user_defined_symbols_2 | …, <placeholder_255>
byte_fallback | True
max_sentence_length | 4192
normalization_rule_name | NFKC
train_extremely_large_corpus | True
remove_extra_whitespaces | False
split_by_whitespace | True
Table 6: Overview of the SentencePiece options that we used for the training of our tokenizers. Hyper-Parameter | Value(s)
---|---
model_type | BPE
vocab_size | 33k $|$50k $|$ 82k $|$ 100k
limit_alphabet | 512
use_nfkc_normalizer | True
use_lowercase_normalizer | False
use_strip_accents_normalizer | True
pre_tokenizer | ByteLevel, Digits
Table 7: Overview of the Sentencepiece Options that we used for the training
of our tokenizers.

  1. 1.

All the News V2.0333https://metatext.io/datasets/all-the-news-2.0 is a corpus
of newspaper articles crawled from over 26 different publications from January
2016 to April 1, 2020.

  2. 2.

Bundestag -
Plenarprotokolle444https://www.bundestag.de/dokumente/protokolle/plenarprotokolle
comprises transcripts of sessions of the German Bundestag.

  3. 3.

Bundesgerichtshof -
Entscheidungen555https://www.bundesgerichtshof.de/DE/Entscheidungen/entscheidungen_node.html
is a collection of decisions of the German Federal Court.

  4. 4.

CoStEP666https://pub.cl.uzh.ch/wiki/public/costep/start is a cleaned-up and
corrected version of the EuroParl corpusGraën et al. (2014). Koehn (2005)

  5. 5.

DCEP777https://joint-research-centre.ec.europa.eu/language-technology-
resources/dcep-digital-corpus-european-parliament_en is a companion corpus to
CoStEP, containing documents published by the European Parliament. Hajlaoui et
al. (2014)

  6. 6.

DNB
Dissertations888https://www.dnb.de/DE/Professionell/Services/Dissonline/dissonline_node.html
is a collection of dissertations from the Deutsche Nationalbibliothek.

  7. 7.

MAREC/IREC999https://researchdata.tuwien.ac.at/records/2zx6e-5pr64: The
MAtrixware REsearch Collection / The Information retrieval facility Research
Collection is a patent corpus of over 19 million documents from the EP, WO,
US, and JP patent offices.

  8. 8.

Medi-Notice101010https://pub.cl.uzh.ch/wiki/public/pacoco/medi-notice is part
of the Zurich Parallel Corpus Collection. It is a multilingual corpus compiled
from information leaflets for medications and pharmaceutical products
published by the Swiss Agency for Therapeutic Products.Graën et al. (2019)

  9. 9.

Swiss
Policy111111https://pub.cl.uzh.ch/wiki/public/pacoco/swiss_legislation_corpus
contains documents of the Swiss Legislation Corpus Höfler and Piotrowski
(2011)

  10. 10.

OpenSubtitles
2018121212https://opus.nlpl.eu/OpenSubtitles-v2018.php131313https://www.opensubtitles.org/de/index.cgi
is a collection of translated movie subtitles. Lison and Tiedemann (2016)

##  Appendix B Tokenizer

In our experiments, we focused on the Huggingface tokenizer library Moi and
Patry (2023) and the SentencePiece library Kudo and Richardson (2018). We use
the standard settings of the SentencePiece library if not stated otherwise in
Table 6. For the HuggingFace tokenizer library Table 7 shows where we deviated
from the standard values.

##  Appendix C LLM Architecture and Hyperparameters

Regarding the training architecture of our 2.7B parameter models, we followed
closely the architecture of GPT-3 Brown et al. (2020a). An overview of the
used architecture details and hyperparameters is given in Table 8.

parameter | value
---|---
# Hidden Dimension | 2560
# Layers | 32
# Attention-Heads | 32
Sequence-Length | 2048
Optimizer | Adam
Adam$-\beta_{1}$ | 0.9
Adam$-\beta_{2}$ | 0.9
Learning rate | 1.6e-4
Learning rate decay | Cosine
Precision | BF16
FlashAttention | 2.0
Position-Embeddings | Rotary
Table 8: Overview of the LLM hyperparameters that we used for the training.

##  Appendix D Intrinsic Tokenizer Evaluation

Besides studying the overlap of the same algorithm on the same thesaurus, we
were also interested in vocabulary overlaps across algorithms and thesauruses
see Fig. 6. What we can observe is that multilingual vocabulary and English
vocabulary have a rather small overlap between 24% and 34% that remains
similar across increasing vocabulary sizes. Across algorithms, we can see that
Unigram and BPE of SentencePiece have a slightly higher overlap than Unigram
of SentencePiece and BPE of Huggingface. We think this might be due to
library-specific preprocessing steps and more similar hyperparameters.

Figure 6: Vocabulary overlap between the examined tokenizers

##  Appendix E Extrinsic Tokenizer Evaluation

| Accuracy
---|---
Model | EN | MULTI
BPE-HF-100 | 45.60 | 38.50
BPE-HF-32 | 45.34 | 38.12
BPE-HF-50 | 45.85 | 38.15
BPE-HF-82 | 44.97 | 38.67
BPE-SP-100 | 46.06 | 38.92
BPE-SP-32 | 47.06 | 38.01
BPE-SP-50 | 46.85 | 38.41
BPE-SP-82 | 45.25 | 38.64
UNI-SP-100 | 45.97 | 39.34
UNI-SP-32 | 46.62 | 38.18
UNI-SP-50 | 45.81 | 38.66
UNI-SP-82 | 45.65 | 39.21
Table 9: Average accuracy of monolingual and multilingual tokenizers across
all downstream tasks

(a) Monolingual tokenizers

(b) Multilingual tokenizers

Figure 7: Accuracy distribution for mono-/multilingual tokenizers across
mono-/multilingual tasks. The "+" represents the average performance across
all tasks.

(a) Non-English documents.

(b) English documents.

(c) German documents.

Figure 8: Average compute (GFLOPs) required to process a single word within
(a) multilingual, (b) English, and (c) German documents within an inference
pass.

###  E.1 Performance Gains on Task-Level

In table Table 11, we show how much influence the choice of a tokenizer has on
each downstream task. Therefore, we compare the results of the best- and
worst-performing models for each task in our experiments. Performance
increasements reach from +5,3% up to 380,9%, again showing the importance of
choosing the right tokenizer on the downstream performance.

Model | Non-English | English | German
---|---|---|---
EN-BPE-HF-33 | 3,8 | 2,32 | 4,52
EN-BPE-HF-50 | 3,79 | 2,38 | 4,45
EN-BPE-HF-82 | 3,88 | 2,55 | 4,51
EN-BPE-HF-100 | 3,96 | 2,67 | 4,58
EN-BPE-SP-33 | 3,86 | 2,37 | 4,66
EN-BPE-SP-50 | 3,89 | 2,42 | 4,68
EN-BPE-SP-82 | 4,02 | 2,59 | 4,78
EN-BPE-SP-100 | 4,11 | 2,71 | 4,84
EN-UNI-SP-32 | 4,01 | 2,36 | 4,73
EN-UNI-SP-50 | 4,02 | 2,42 | 4,75
EN-UNI-SP-82 | 4,12 | 2,59 | 4,83
EN-UNI-SP-100 | 4,21 | 2,71 | 4,88
MULTI-BPE-HF-33 | 2,71 | 2,46 | 3,04
MULTI-BPE-HF-50 | 2,7 | 2,5 | 3,01
MULTI-BPE-HF-82 | 2,8 | 2,65 | 3,09
MULTI-BPE-HF-100 | 2,88 | 2,76 | 3,17
MULTI-BPE-SP-33 | 2,68 | 2,55 | 2,99
MULTI-BPE-SP-50 | 2,67 | 2,57 | 2,95
MULTI-BPE-SP-82 | 2,76 | 2,72 | 3,03
MULTI-BPE-SP-100 | 2,85 | 2,82 | 3,1
MULTI-UNI-SP-33 | 2,68 | 2,55 | 2,94
MULTI-UNI-SP-50 | 2,66 | 2,58 | 2,91
MULTI-UNI-SP-82 | 2,76 | 2,73 | 2,99
MULTI-UNI-SP-100 | 2,84 | 2,83 | 3,07
Table 10: Computational costs per word (GFLOPs) for different tokenizers. | Task | Metric | Worst | Best | Difference
---|---|---|---|---|---
EN | arc challenge | acc | 0,21 | 0,27 | +28,4 %
arc easy | acc | 0,50 | 0,59 | +18,6 %
boolq | acc | 0,56 | 0,61 | +8,9 %
cb | acc | 0,30 | 0,54 | +76,5 %
cb | f1 | 0,18 | 0,36 | +94,9 %
copa | acc | 0,67 | 0,73 | +9,0 %
hellaswag | acc | 0,34 | 0,41 | +19,7 %
lambada openai | acc | 0,47 | 0,56 | +20,0 %
mathqa | acc | 0,22 | 0,24 | +8,8 %
mnli | acc | 0,35 | 0,37 | +7,7 %
mnli mismatched | acc | 0,35 | 0,38 | +9,4 %
mrpc | acc | 0,54 | 0,69 | +26,1 %
mrpc | f1 | 0,66 | 0,81 | +22,7 %
gnad10 | acc | 0,15 | 0,43 | +187,7 %
gnad10 | f1 | 0,07 | 0,35 | +380,9 %
gnad10 | recall | 0,14 | 0,42 | +207,6 %
pawsx es | acc | 0,48 | 0,56 | +17,2 %
piqa | acc | 0,67 | 0,72 | +7,2 %
pubmedqa | acc | 0,45 | 0,59 | +32,1 %
race | acc | 0,31 | 0,34 | +11,8 %
rte | acc | 0,53 | 0,57 | +8,9 %
sst | acc | 0,51 | 0,75 | +48,2 %
triviaqa | acc | 0,02 | 0,05 | +171,3 %
wic | acc | 0,50 | 0,55 | +11,4 %
winogrande | acc | 0,49 | 0,56 | +12,4 %
wnli | acc | 0,42 | 0,54 | +26,7 %
| wsc | acc | 0,36 | 0,61 | +70,3 %
MULTI | xstory cloze es | acc | 0,49 | 0,60 | +24,3 %
mgsm de | acc | 0,00 | 0,03 | -
lambada de | acc | 0,11 | 0,33 | +186,4 %
lambada es | acc | 0,17 | 0,32 | +91,8 %
lambada fr | acc | 0,20 | 0,38 | +90,5 %
lambada it | acc | 0,16 | 0,37 | +132,3 %
wino de | acc | 0,50 | 0,61 | +20,9 %
xcodah de | acc | 0,27 | 0,42 | +52,1 %
xcodah es | acc | 0,28 | 0,43 | +52,1 %
xcodah fr | acc | 0,27 | 0,39 | +43,7 %
xcsqa de | acc | 0,19 | 0,30 | +56,4 %
xcsqa es | acc | 0,19 | 0,29 | +57,3 %
xcsqa fr | acc | 0,17 | 0,29 | +69,2 %
xnli de | acc | 0,32 | 0,48 | +51,0 %
xnli en | acc | 0,49 | 0,52 | +5,3 %
xnli es | acc | 0,34 | 0,48 | +41,0 %
xnli fr | acc | 0,37 | 0,49 | +30,3 %
xstance de | acc | 0,49 | 0,52 | +6,0 %
xstance de | f1 | 0,34 | 0,47 | +41,2 %
xstance de | recall | 0,49 | 0,52 | +7,7 %
Table 11: Differences between our best and worst performing tokenizers per
task.