# Tagengo: A Multilingual Chat Dataset

Peter Devine
Lightblue KK. (Tokyo, Japan)
<EMAIL_ADDRESS>
###### Abstract

Open source large language models (LLMs) have shown great improvements in
recent times. However, many of these models are focused solely on popular
spoken languages. We present a high quality dataset of more than 70k prompt-
response pairs in 74 languages which consist of human generated prompts and
synthetic responses. We use this dataset to train a state-of-the-art open
source English LLM to chat multilingually. We evaluate our model on MT-Bench
chat benchmarks in 6 languages, finding that our multilingual model
outperforms previous state-of-the-art open source LLMs across each language.
We further find that training on more multilingual data is beneficial to the
performance in a chosen target language (Japanese) compared to simply training
on only data in that language. These results indicate the necessity of
training on large amounts of high quality multilingual data to make a more
accessible LLM.

##  1 Introduction

Recently, open source large language models (LLMs) have grown drastically in
both popularity and performance. Models such as Llama 3 AI@Meta (2024b) have
exceeded the performance of previous state-of-the-art proprietary models like
GPT3.5 Ouyang et al. (2022) on popular robust benchmarks including the Chatbot
Arena leaderboard Chiang et al. (2024). These open source LLMs are also
increasingly being used in commerical AI chat products such as the Meta AI
assistant AI@Meta (2024a).

However, many current LLMs exhibit lower performance on languages outside of
English Achiam et al. (2023). Indeed, Llama 3 itself is currently an English-
only LLM, meaning that even when it is prompted in a language besides English,
it often replies in English. This limits the potential user base of these LLMs
due to the fact that less than 1.5 billion of the world’s more than 8 billion
population can speak English Central Intelligence Agency (2021); Eberhard et
al. (2024). Therefore, we set out to train a state-of-the-art open source LLM
(Llama 3) to be able to chat not only in English, but in many languages.

In order to make English-focused LLMs accessible in other languages, previous
work has fine-tuned these models on non-English data Sasaki et al. (2023);
Sengupta et al. (2023); Nguyen et al. (2023).

Many multilingual chat datasets such as MultiAlpaca Wei et al. (2023) and Aya
Singh et al. (2024) cover many languages and tasks but can also lack natural
prompts and high quality responses.

For this reason, we created a large, diverse, high quality multilingual
dataset using more than 70k human generated prompts in 74 languages and
generated responses from these using state-of-the-art proprietary chat models.
We used this dataset to train two models, a multilingual LLM and a Japanese-
only LLM, both supervised fine-tuned models based on the Llama 3 8B Instruct
model.

We found that our model achieved better evaluation scores on multilingual chat
benchmarks compared to the similarly sized state-of-the-art open source
models, indicating the high quality and diversity of our training dataset. We
also find that our multilingual-trained LLM performs better on Japanese chat
benchmarks compared to our Japanese-only-trained LLM, indicating that transfer
learning from training on other languages is beneficial for training even
monolingual models outside of English.

Our findings combine to inform the community of exactly how to fine-tune
monolingual LLMs to create a strong multilingual model.

We make our training data111https://huggingface.co/datasets/lightblue/tagengo-
gpt4, training code222https://github.com/lightblue-tech/suzume, and trained
models333https://huggingface.co/lightblue/suzume-
llama-3-8B-multilingual444https://huggingface.co/lightblue/suzume-
llama-3-8B-japanese publicly available for free use online.

Figure 1: Distribution of the languages found in the Tagengo dataset

##  2 Related Work

In the literature, strong foundation models such as Llama 2 Touvron et al.
(2023) and Gemma Team et al. (2024) have been subsequently fine-tuned on data
from a specific language or languages, including Japanese Sasaki et al.
(2023), Arabic Sengupta et al. (2023), and South-East Asian languages Nguyen
et al. (2023). Fine-tuning has often shown to improve the accuracy of the
resultant LLM on tasks in that language. However, the training dataset of
these models are often not shared, making it difficult to create a truly
multilingual LLM across many languages.

Some multilingual chat datasets do exist that can be used for training LLMs.
MultiAlpaca Wei et al. (2023) is a multilingual dataset of 133K prompt-
response pairs covering 11 languages that were generated in a similar manner
to Alpaca Taori et al. (2023). This dataset was created by generating
synthetic prompts from a small number of English seed prompts and then
answering these prompts using an large-scale LLM, GPT3.5 Ouyang et al. (2022).
Because these prompts are generated synthetically, this data may not reflect
the sorts of prompts that real users may use with an LLM, potentially limiting
the ability of models trained on this data to be used practically. Moreover,
the prompts and responses for this dataset were generated using GPT3.5,
meaning that the quality of the data may not be as high as if a state-of-the-
art LLM was used, like GPT4 Achiam et al. (2023).

xP3 (Crosslingual Public Pool of Prompts) Muennighoff et al. (2022) is a
dataset of more than 78 million examples covering 46 languages. This dataset
was generated by templating other datasets (e.g. translation datasets,
classification datasets) into a prompt-response forma. While this dataset is
large, the templating process limits the usefulness of the dataset as it
results in prompts that are not necessarily similar to what an actual user of
an LLM would ask. The templating process can also result in unnatural answers,
with single word answers being given where a fuller answer may be more
appropriate from an LLM.

Aya Singh et al. (2024) is a dataset of 204k human-annotated prompt-completion
pairs covering 65 languages. The majority of this dataset was generated by
first translating and templating datasets from various languages, which were
then corrected and annotated by human labellers. While the human labelling
process will prevent as many unnatural utterances enter the dataset, the
templating of datasets means that the prompts will still not necessarily be
the kind of prompts that an end-user of LLMs would use. Hence, the usefulness
of this dataset in training multilingual LLMs is limited by its data-
generation process.

The ShareGPT dataset used by models such as Vicuna Chiang et al. (2023) and
OpenChat Wang et al. (2023) contain approximately 70k open source
conversations between users and GPT3.5 Ouyang et al. (2022) and 6k
conversations between users and GPT4 Achiam et al. (2023), meaning that the
prompts used in these datasets are often much more naturalistic to a real LLM
use-case. However, the majority of these prompts are in English, meaning that
this dataset is limited in its use in training multilingual models. Moreover,
due to the fact that that majority of this dataset contains data generated
from GPT3.5, its usefulness in training is limited as many other models have
now surpassed the performance of GPT3.5 in English Zhu et al. (2023); AI@Meta
(2024b). The amount of multilingual data in the higher quality GPT4 subset of
the ShareGPT dataset is small, meaning that its usefulness in training is
constrained by its size.

To address the shortcomings in existing public datasets, we created a large,
diverse, high quality multilingual dataset using more than 70k human generated
prompts in 74 languages and generated responses from these using state-of-the-
art proprietary chat models.

##  3 Method

In this section, we detail how we generated our training dataset, our training
method, and finally our evaluation techniques.

###  3.1 Tagengo Dataset Creation

First, to generate our dataset, we sampled prompts from the million row LMSYS-
Chat-1M dataset Zheng et al. (2023). These prompts were collected from users
speaking to one of 25 LLMs on the Vicuna demo and Chatbot Arena
website555https://chat.lmsys.org/.

We cleaned this dataset by first removing all prompts which contains an OpenAI
Moderation Endpoint666https://platform.openai.com/docs/guides/moderation flag
in order to remove explicit, sexual, or illegal material.

We then removed all prompts which were listed as a non-recognised or fictional
language (unknown, Klingon, xx, zp, and zzp).

We removed any prompts which contained the string “name” when lower-cased, as
NAME0, NAME1 etc. was used as the placeholders for anonymised material.
Effectively, this removed any anonymised prompts from our dataset.

We then removed any prompts which contained the following keywords: “gpt”,
“vicuna”, “alpaca”, “llama”, “koala”, “claude”, “guanaco”. This was done to
remove prompts which referred explicitly to the LLMs that were being tested in
the Chatbot Arena as many prompts asked about the LLM specifically, which we
theorize is less useful in a more general context.

We then used the FastText Joulin et al. (2016) LangDetect
library777https://github.com/zafercavdar/fasttext-langdetect to determine the
confidence level of classifying a particular language. We filtered out all
prompts in which the confidence level of the language indicated in the
original LMSYS-Chat-1M paper was less than 80%. This was done to filter out
ambiguous language examples, as we later sample per-language.

Finally, we analysed the number of tokens of both the first prompt and LLM
response, and removed any prompts in which the combined token total of the
first prompt and LLM response amounted to more than 512 tokens. This was done
to prevent very long prompts or prompts which elicited very long responses
being used in our dataset in order to minimise costs when generating data with
these prompts using GPT4.

We then sampled a maximum of 25,000 prompts from each language, which
effectively meant we sampled the English prompts in this dataset as only
English (380,138) had more than 25,000 examples, while the next most popular
language Chinese (21,057) had less than 25,000. This was done to counteract
the outweighed prevalence of English within this dataset.

For each language, we then embedded each prompt using the BGE M3 embedding
model Chen et al. (2024), which is a state-of-the-art embedding model that
supports more than 100 languages. We then compared the prompt embeddings
pairwise using the dot product to obtain a similarity score for each prompt
pair. We perform fuzzy deduplication by removing one of any prompt pairs which
have a similarity score of greater than 0.8 in order to bolster the diversity
of our dataset. The amount of data removed from each language varied widely
with languages such as Chinese having a very high rate of deduplication
($\sim$75%) and other such as Portuguese having a lower rate of deduplication
($\sim$40%). This may be due to the biases of the embedding model or due to
the kind of prompts submitted to the original dataset in different languages.

A table of the number of prompts filtered at each stage of our cleaning
process can be found in Table 1.

Stage | Number of prompts
---|---
Start | 1,000,000
| OpenAI Moderation
---
check
964,464
| Remove unknown
---
languages
936,468
Remove anonymised data | 753,731
| Remove references
---
to models
735,390
| Language detection
---
confidence score >80%
556,368
| Remove prompt plus
---
responses with more
than 512 tokens
513,011
| Random sampling
---
of 25,000 prompts
per language
157,873
Fuzzy deduplication | 78,057
| Remove uncompleted/
---
unanswered prompts
76,338
Table 1: Table describing the number of prompts after each cleaning stage.

We used these prompts to generate responses using an Azure OpenAI deployment
of a state-of-the-art proprietary LLM, GPT4 (0125-Preview), with the
generation temperature set to 0 and setting a maximum number of response
tokens to be 2,048.

Due to the fact that generating high quality responses for all of these
prompts manually for each language would be prohibitively expensive, we
decided to generate these responses using a state-of-the-art model. We
hypothesize that using an LLM much larger - rumoured to be 1.8 trillion
parameters Schreiner (2023) \- than nearly all other open source models to
generate responses will lead to high quality responses that can then be used
to improve existing open source models. When viewed in this way, this training
can be viewed as a form of model distillation Buciluǎ et al. (2006); Hinton et
al. (2015).

We finally removed any responses which GPT4 did not answer or was not able to
complete within the 2,048 token limit. The number of prompts in our resultant
Tagengo dataset can be found in Table 1 and a breakdown of the prompts by
language can be found in Fig 1.

We share our dataset creation code and this training dataset on Huggingface at
lightblue/tagengo-gpt4888https://huggingface.co/datasets/lightblue/tagengo-
gpt4.

|  | Llama 3 8B
---
Instruct
| Suzume 8B
---
multilingual
| Suzume 8B
---
Japanese
| Starling
---
7B beta
| GPT3.5
---
Turbo
Chinese | - | 7.11 | - | 6.97 | 7.55
French | - | 7.66 | - | 7.29 | 7.74
German | - | 7.26 | - | 6.99 | 7.68
Japanese | - | 6.56 | 6.24 | 6.22 | 7.84
Russian | - | 8.19 | - | 8.28 | 7.94
English | 7.98 | 7.73 | - | 7.92 | 8.26
Table 2: Average MT-Bench scores across 6 languages for each LLM evaluated.

###  3.2 Training

For training data, we add two more datasets to the Tagengo dataset which we
regard as high quality chat datasets. The first is the Megagon Instruction
dataset Hayashibe (2023), a manually annotated dataset of 669 Japanese prompt-
response pairs. The second is the 6k GPT4 subset of the ShareGPT
dataset999https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset/blob/main/sharegpt_gpt4.json,
which has a majority of prompts in English but also includes responses in
other languages. We combined and randomly shuffled these three datasets to use
as a 83,213 prompt-response pair training dataset for the multilingual model.

We used our training data to train a Llama 3 8B Instruct
model101010https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct with the
Axolotl LLM training package111111https://github.com/OpenAccess-AI-
Collective/axolotl. We trained for one epoch using full fine tuning, using
sample packing Brown et al. (2020) and a context length of 8,096. We name this
model Suzume 8B multilingual and the full training configuration for this
model can be found on our model
card121212https://huggingface.co/lightblue/suzume-llama-3-8B-multilingual.

We also prepared a subset of the above three datasets that only included
Japanese data from each dataset, which amounted to 3,318 prompt-response
pairs. This was prepared to isolate the effect of monolingual training
compared to multilingual training on our data. We trained our model in the
same manner as the multilingual model with the name Suzume 8B Japanese. Full
details for how the training was conducted can be found on our model
card131313https://huggingface.co/lightblue/suzume-llama-3-8B-japanese.

###  3.3 Evaluation

We tested our models by using a forked version of the original MT-Bench
evaluation suite Zheng et al. (2024). The MT-Bench evaluation benchmark is a
set of prompts and responses in English that cover 8 broad categories of
prompts: writing, roleplay, extraction, reasoning, math, coding, STEM
knowledge, and humanities knowledge. Responses to these prompts are generated
using an LLM, and those responses are then evaluated using an evaluation model
such as GPT4.

We added publicly available translated versions of the original MT-Bench
dataset in Chinese, French, German, Japanese, and Russian that had been human-
verified by a native speaker of that language.

Note that the Russian translation did not contain reference answers for the
math, coding, and reasoning questions, so our evaluation does not include
math, coding, and reasoning problems in Russian.

Finally, we added the phrase “Your evaluation should also consider whether the
prompt responded in the correct language and the fluency and naturalness of
this response.” to the original MT-Bench evaluation criteria to ensure that
the LLM judge would not simply evaluate factually correct responses in English
to non-English prompts as correct. We conducted these evaluations using the
“gpt-4-turbo model” from OpenAI as the judge LLM.

We make our evaluation code freely available
online141414https://github.com/Peter-
Devine/multilingual_mt_bench/tree/main/fastchat/llm_judge.

As baselines, we also evaluate the original Llama 3 8B Instruct model AI@Meta
(2024b), GPT3.5-Turbo Ouyang et al. (2022), and the Starling 7B Beta Zhu et
al. (2023) which is the highest rated similarly sized multilingual model on
the Chatbot Arena leaderboard Chiang et al. (2024) and has been trained on the
ShareGPT dataset amongst other data.

##  4 Results

The MT-Bench scores for each model evaluated can be found in Table 2.

We first found that we were able to train Llama 3 8B Instruct to output
responses in the same language as the prompt. This means that we achieved our
base objective of enabling a monolingual model (Llama 3) to be able to output
multilingual chat.

Secondly, English performance of the multilingual trained model only dropped
slightly compared to the base Llama 3 8B Instruct model. This indicates that
English chat performance does not considerably drop even when training on a
majority of non-English data.

Thirdly, we found that the multilingual trained model performs better compared
to the Starling 7B Beta across 5 out of 6 non-English languages tested.
However, also we found that our multilingual model achieved lower evaluation
scores compared to the proprietary GPT3.5 on 5 of 6 non-English languages.
This indicates that our model has achieved state-of-the-art performance in
multilingual chat for open-source models of its size, but has not achieved
state-of-the-art performance more generally.

Finally, the Suzume 8B multilingual model achieves higher MT-Bench scores on
the Japanese benchmark compared to the Suzume 8B Japanese model, indicating
that transfer learning from training on other languages is beneficial for
training even monolingual models outside of English.

##  5 Discussion

Our results indicate the need for large, high quality, multilingual datasets
when training multilingual models. We find that with such a dataset, we can
train a state-of-the-art monolingual model such as Llama 3 to achieve state-
of-the-art multilingual performance.

We also found that training on additional non-Japanese data improves the
performance of our LLM on Japanese benchmarks when compared to training solely
on Japanese data, indicating that there is a collective improvement effect
between languages when training using multilingual data. This adds to the body
of work that indicates that training on multiple languages enables the LLM to
better generalise to other languages Nguyen and Chiang (2017); Schuster et al.
(2018). This suggests that generating an even larger, more diverse dataset in
the future could further aid the performance of LLMs on low-resource
languages.

##  6 Future Work

Our work could be built upon and improved in the following ways.

Our training dataset mainly consisted of single prompt-response pairs, but
many chats between users and LLMs extend beyond a single conversation turn.
Therefore, future work could include creating a dataset that contains multiple
turns of conversation, with the prompts either generated by humans or by high
quality LLMs.

Future work could also include adding more languages to our dataset. Our
dataset only included 74 languages, and crucially omits any languages in the
Niger–Congo language family, one of the most diverse language families in the
world Good (2017). Therefore, future work could involve sampling initial
prompts from a wider range of sources (possibly by advertising free chatbot
access to people in areas with many speakers of underrepresented languages)
and generating responses based on these prompts. This would help to both
improve an LLMs linguistic understanding of these low-resource languages as
well as improve their understanding of the topics and questions that people
from that language and culture may ask.

Finally, future work could include generating preference data, such as was
done in English in the Nectar dataset Zhu et al. (2023), for use with
contrastive learning techniques such as Direct Preference Optimisation
Rafailov et al. (2024) and Odds Ratio Preference Optimisation Hong et al.
(2024). These techniques have been shown to further improve the accuracy of
LLMs, suggesting that training using these techniques may also improve the
performance of LLMs in multilingual chat.

##  7 Conclusion

In this study, we successfully trained a state-of-the-art monolingual Llama 3
LLM to chat multilingually using a new, diverse dataset comprising over 70k
human-generated prompts in 74 languages paired with high-quality synthetic
responses. Our multilingual model showcased superior performance across
multiple languages compared to similar-sized open-source models on various
chat benchmarks. Interestingly, training using a multilingual dataset also
enhanced the performance on specific monolingual tasks, implying beneficial
cross-linguistic transfer effects. The outcomes underlines the importance of
using rich, diverse multilingual data for improving the capabilities of LLMs
in global, multilingual applications.

## Limitations

The three main limitations of this paper concern our prompt diversity, our
data generation methodology, and our model evaluation methodology.

Firstly, as stated in Section 6, our training data has a paucity of low-
resource languages represented within it. While we try to focus on non-English
data in our work by sampling a maximum of 25,000 prompts per language, this
still does not counteract the fact that the prompts in the LMSYS-Chat-1M
dataset Zheng et al. (2023) are disproportionately from a small set of
languages. These prompts are collected from users on the Chatbot Arena LLM
demo site, meaning that the speakers of low-resource languages may be too few,
unable, unaware, or unwilling to talk to an LLM chatbot in their native
language. This means that current open source LLMs will continue to have lower
performance on low-resource languages if this problem is not resolved.

Secondly, we generate our responses to prompts using GPT4, which means that
all training data will be in the worldview and within the domain of knowledge
that GPT4 exhibits. This biases the model as many LLMs have been shown to have
both political Feng et al. (2023) and cultural biases Cao et al. (2023) in the
text they generate, meaning that what may be deemed acceptable by one user may
not be deemed acceptable by another. Moreover, while GPT4 is state-of-the-art
and has been shown to generate more accurate information compared to previous
models Achiam et al. (2023), it is still capable of generating incorrect data
in response to a prompt, meaning that our training data may contain incorrect
statements or otherwise inaccurate data.

Finally, our evaluation methodology is biased by the fact that our 6
evaluation languages are all languages that are within the top 10 most popular
languages in our training data. This means that our evaluation does not
consider the performance of our models on low resource languages, limiting the
usefulness of our results to speakers of low resource languages.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI@Meta (2024a) AI@Meta. 2024a.  [link].
  * AI@Meta (2024b) AI@Meta. 2024b.  Llama 3 model card.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 535–541.
  * Cao et al. (2023) Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023.  Assessing cross-cultural alignment between chatgpt and human societies: An empirical study.  _arXiv preprint arXiv:2303.17466_.
  * Central Intelligence Agency (2021) Central Intelligence Agency. 2021.  _The World Factbook 2021_.  Central Intelligence Agency, Washington, DC.
  * Chen et al. (2024) Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024.  Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.  _arXiv preprint arXiv:2402.03216_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2(3):6.
  * Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024.  Chatbot arena: An open platform for evaluating llms by human preference.
  * Eberhard et al. (2024) David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2024.  _Ethnologue: Languages of the World_ , twenty-seventh edition.  SIL International, Dallas, Texas.  Online version: http://www.ethnologue.com.
  * Feng et al. (2023) Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023.  From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models.  _arXiv preprint arXiv:2305.08283_.
  * Good (2017) Jeff Good. 2017.  Niger-congo languages.  _The Cambridge handbook of areal linguistics_ , pages 471–499.
  * Hayashibe (2023) Yuta Hayashibe. 2023.  megagonlabs/instruction_ja: Japanese instructions data for llm.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. 2024.  Reference-free monolithic preference optimization with odds ratio.  _arXiv preprint arXiv:2403.07691_.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Nguyen and Chiang (2017) Toan Q Nguyen and David Chiang. 2017.  Transfer learning across low-resource, related languages for neural machine translation.  _arXiv preprint arXiv:1708.09803_.
  * Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2023.  Seallms - large language models for southeast asia.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Sasaki et al. (2023) Akira Sasaki, Masato Hirakawa, Shintaro Horie, and Tomoaki Nakamura. 2023.  Elyza-japanese-llama-2-7b.
  * Schreiner (2023) Maximilian Schreiner. 2023.  Gpt-4 architecture, datasets, costs and more leaked.
  * Schuster et al. (2018) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2018.  Cross-lingual transfer learning for multilingual task oriented dialog.  _arXiv preprint arXiv:1810.13327_.
  * Sengupta et al. (2023) Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023.  Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models.
  * Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. 2024.  Aya dataset: An open-access collection for multilingual instruction tuning.  _arXiv preprint arXiv:2402.06619_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 3(6):7.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023.  Openchat: Advancing open-source language models with mixed-quality data.  _arXiv preprint arXiv:2309.11235_.
  * Wei et al. (2023) Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023.  Polylm: An open source polyglot large language model.  _arXiv preprint arXiv:2307.06018_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2023.  Lmsys-chat-1m: A large-scale real-world llm conversation dataset.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhu et al. (2023) Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. 2023.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif.

##  Appendix A Appendix: Full Language Counts

Language | Number of examples |  | Language | Number of examples
---|---|---|---|---
English | 15,404 |  | Croatian | 11
Portuguese | 12,412 |  | Lithuanian | 11
Spanish | 8,098 |  | Slovenian | 10
Russian | 8,005 |  | Mongolian | 6
Italian | 6,874 |  | Basque | 6
German | 5,430 |  | Serbian | 6
Chinese | 5,309 |  | Icelandic | 5
French | 5,163 |  | Macedonian | 5
Japanese | 2,482 |  | Albanian | 5
Korean | 1,604 |  | Malay | 5
Polish | 1,029 |  | Sinhala | 4
Arabic | 783 |  | Latin | 4
Vietnamese | 428 |  | Tamil | 4
Turkish | 401 |  | Azerbaijani | 4
Dutch | 375 |  | Amharic | 3
Ukrainian | 322 |  | Armenian | 3
Greek | 293 |  | Urdu | 3
Swedish | 241 |  | Afrikaans | 2
Indonesian | 234 |  | Kazakh | 2
Hungarian | 210 |  | Belarusian | 2
Persian | 183 |  | Waray | 2
Czech | 176 |  | Yiddish | 2
Thai | 131 |  | Uyghur | 2
Hebrew | 117 |  | Burmese | 2
Finnish | 88 |  | Tibetan | 1
Catalan | 73 |  | Turkmen | 1
Romanian | 71 |  | Breton | 1
Danish | 64 |  | Lao | 1
Bulgarian | 56 |  | Georgian | 1
Bangla | 28 |  | Sanskrit | 1
Norwegian | 25 |  | Khmer | 1
Hindi | 20 |  | Kannada | 1
Latvian | 20 |  | Luxembourgish | 1
Estonian | 18 |  | Odia | 1
Esperanto | 17 |  | Marathi | 1
Slovak | 17 |  | Malayalam | 1
Tagalog | 15 |  | Uzbek | 1
Table 3: The number of prompt-response examples in our Tagengo dataset for
each language


# Data Driven Grapheme-to-Phoneme Representations for a Lexicon-Free
Text-to-Speech

###### Abstract

Grapheme-to-Phoneme (G2P) is an essential first step in any modern, high-
quality Text-to-Speech (TTS) system. Most of the current G2P systems rely on
carefully hand-crafted lexicons developed by experts. This poses a two-fold
problem. Firstly, the lexicons are generated using a fixed phoneme set,
usually, ARPABET or IPA, which might not be the most optimal way to represent
phonemes for all languages. Secondly, the man-hours required to produce such
an expert lexicon are very high. In this paper, we eliminate both of these
issues by using recent advances in self-supervised learning to obtain data-
driven phoneme representations instead of fixed representations. We compare
our lexicon-free approach against strong baselines that utilize a well-crafted
lexicon. Furthermore, we show that our data-driven lexicon-free method
performs as good or even marginally better than the conventional rule-based or
lexicon-based neural G2Ps in terms of Mean Opinion Score (MOS) while using no
prior language lexicon or phoneme set, i.e. no linguistic expertise.

Index Terms—  Grapheme-to-Phoneme, data-driven G2P, Text-to-Speech, lexicon-
free TTS, self-supervised learning

##  1 Introduction

Text-to-Speech synthesis has been the subject of extensive research for
several decades [1, 2]. Initially, concatenative speech synthesis models were
developed to address this task by assembling waveforms from a pre-existing
database of speech. Subsequently, statistical approaches were introduced to
generate speech features from the text, which were then fed to a vocoder to
produce the final output. The results of these methods are unsatisfactory due
to the unnaturalness and mispronunciations in the generated speech.

Grapheme-to-Phoneme (G2P) models are an integral part of current Text-to-
Speech (TTS) engines [3, 4]. The initial work in this field was done using
rule-based and joint sequence models. However, with the rise of deep learning
methods, RNN, and even more recently, Transformer based architectures have
been used to perform a variety of tasks such as Grapheme-to-Phoneme, Automatic
Speech Recognition[5, 6, 7], Machine Translation, and Text-to-Speech Synthesis
[3, 4, 8, 9]. As G2P conversion is essentially a sequence-to-sequence modeling
task, using Encoder-Decoder architectures[10, 11] helped obtain improvements
on Grapheme-to-Phoneme conversion [3].

Traditional G2Ps [12] typically use a large lexicon to perform dictionary
searches of the most frequent words and use hand-crafted rules to generate
pronunciations for out-of-vocabulary words. Neural G2Ps [3], in contrast, use
lexicons as their data for training their neural network and use the obtained
network for predicting pronunciations. While both of these methods have been
instrumental in building modern-day TTS systems, they have severe limitations
in terms of requiring an external lexicon. Building a lexicon is an expensive
and highly cumbersome task, as it requires multiple language experts to
propose and then verify its validity.

Recently there have been efforts to build a massively multi-lingual ByT5 G2P
[13]. It uses a T5 transformer based encoder-decoder architecture [14] and
uses an UTF8 based input tokenization to handle scripts in multiple languages.
It uses publicly available lexicons from the internet covering upto 100
languages with varying size and quality of the lexicons. The difficulty in
procuring high quality lexicon for all languages and the decreasing accuracy
of g2p models for languages with limited or noisy data can be clearly seen
from this paper.

In this paper, we propose a new mechanism to train the Grapheme-to-Phoneme
model without the need of any lexicon. We use Text-to-Speech as a use-case to
show the effectiveness of our G2P model. We first use unlabeled speech data to
pre-train a HuBERT [15] for three iterations. Once we have a pre-trained
HuBERT model, we input the labeled speech data and use a specific transformer
layer to extract the speech features and apply k-means clustering on them to
obtain frame-level phoneme targets. We use paired phoneme targets and labeled
speech data to train our G2P transformer model. We then use the trained G2P to
train a Tacotron 2 [1] model.

##  2 Neural Grapheme-to-Phoneme

###  2.1 G2P Architecture

Transformer-based Grapheme-to-Phoneme models have been found to outperform
LSTM / Recurrent Encoder-Decoder models [3, 16, 17] while also having the
advantage of faster parallel training resulting in significantly less training
time.

The neural Grapheme-to-Phoneme model takes the grapheme sequence
$X=(x_{1},x_{2},\dots,x_{n})$ as input. It then generates representations
$L=(l_{1},l_{2},\dots,l_{n})$ in a latent space which are then passed to a
decoder. Finally, the decoder tries to predict the corresponding phoneme
sequence $Y=(y_{1},y_{2},\dots,y_{m})$ in an autoregressive manner. Here $n,m$
is the token length of input and output respectively.

###  2.2 Text-to-Speech System

To evaluate our G2P for the TTS task, we use Tacotron 2 [1] as our text-to-
speech model. Although the original Tacotron 2 uses text, many studies [18,
19] have found phones to produce superior results. Hence, we use phones as
inputs to the Tacotron model for our experiments.

Fig. 1: Flow chart demonstrating proposed algorithm

##  3 Self Supervision in Speech Recognition

Models like wav2vec2.0 [20] quantize and mask input audio signals, after which
they use contrastive loss to allow the model to learn from the unlabeled data.
In contrast, models like HuBERT [15] separate the acoustic unit discovery step
from the masked prediction representation learning stage, allowing for a more
direct predictive loss computation. We use these acoustic units in Section 4
for training our G2P models. We outline the pre-training approach for
obtaining these acoustic units in HuBERT below.

Given speech representation feature sequence $X=[x_{1},x_{2},...,x_{T}]$ of
length $T$, $M\subset[T]$, a corrupted sequence $\tilde{X}$ is obtained by
replacing input representation at time $t$, $x_{t}$ with $\tilde{x}$ if $t\in
M$, where $\tilde{x}$ is a learned vector. HuBERT encoder $e$ takes
$\tilde{X}$ as input and predicts an output distribution over target
vocabulary at each time step $p(z_{t}|\tilde{X},t)$, where $z_{t}$ is frame
level target obtained via acoustic unit discovery.

For obtaining frame-level targets, an acoustic unit discovery model is used.
Given $X$, acoustic unit discovery model $h$ takes $X$ as input to produce
$h(X)=Z=[z_{1},z_{2},...,z_{T}]$ where each $z_{t}$ belongs to a $C$-class
categorical distribution and $h$ is a clustering model. We use the same
clustering model as [15] i.e., k-means.

While training the model, $M$ is calculated by randomly selecting $p\%$ of
timesteps as the starting indices and masking $l$ steps starting from them.
Cross entropy loss is then calculated over the masked timesteps.

$L_{m}(e;X,M,Z)=\Sigma_{t\in M}log\ p(z_{t}|\tilde{X},t),$ (1)

Encoder $e$ is trained to minimize $L_{m}$ in the pre-training stage. The pre-
training is done in multiple iterations. In the initial iteration, the
clustering model $h$ uses MFCC-generated features for producing $Z$. However,
as the training proceeds, to obtain better targets $Z$, $h$ uses latent
features extracted from the HuBERT model pre-trained in the previous iteration
at some intermediate layer.

##  4 Data Driven Grapheme-to-Phoneme for Text-to-Speech

Recent TTS systems [1, 21] have shown remarkable results in producing almost
natural-sounding audios. However, these TTS systems require equally high-
performing G2P systems to produce these high-quality audios. Building a well-
performing G2P poses a couple of challenges for both traditional lexicon based
G2Ps and neural G2Ps. For traditional G2P systems, obtaining a lexicon is a
tedious task and requires experts. Moreover, even if the lexicon can be
obtained, making rules for out-of-vocabulary words poses another issue.
Although neural G2P does not use a lexicon for inference, it uses a lexicon as
training data.

While obtaining a lexicon for a language is laborious, speech data is
generally available in abundance. Obtaining a text transcription corresponding
to a few hours of speech data is much more feasible. Starting with unlabeled
speech data and a small amount of labeled data, we propose a novel training
strategy to (1) Obtain a data-driven phoneme set in contrast to the
traditional sets like CMU (ARPABET) or IPA, (2) Train a neural G2P on this new
phoneme set, (3) We subsequently use the trained G2P for training a TTS
system.

For achieving this, we use the acoustic units produced by a HuBERT model
during the discovery stage as our phonetic representation. Using the number of
clusters (i.e., $k$-value), we can control the size of the phoneme set. We
present our training algorithm with the availability of unlabeled speech
dataset $S_{u}$ and a small labeled dataset
$S_{l}=[(s_{1},t_{1}),...(s_{n},t_{n})]$, where $(s_{i},t_{i})$ is a single
data point with $s_{i}$ audio utterance paired with $t_{i}$ text. A flow chart
of the algorithm is shown in Fig. 1 and explained below.

  1. 1.

We use the unlabeled speech data $S_{u}$ to pre-train a HuBERT for three
iterations. Using MFCC for the first iteration, then using some intermediate
transformer layer. See Section 5.

  2. 2.

Once we have a pre-trained HuBERT model, we input the labeled speech data,
$(s_{1},...,s_{n})$, and use the 9th transformer layer to extract the speech
features and apply k-means clustering on them to obtain frame-level phoneme
targets $(P_{1},....,P_{n})$, where each
$P_{i}=h(q_{i})=[z^{i}_{1},...z^{i}_{T}]$ and $q_{i}$ is the 9th layer output
of $e(s_{i})$.

  3. 3.

Once we have $P_{i}$ for all $s_{i}$ in $S_{l}$, we use the paired
$(P_{i},t_{i})$ to train our G2P transformer model as described in Section 2.

  4. 4.

We then use the trained G2P model to train a Tacotron 2 model described in
Section 2.

Using the above training algorithm, we can leverage a large amount of
available unlabeled speech data and train high-quality TTS models with a
minimal amount of labeled training data.

##  5 Experiments

###  5.1 HuBERT model

As a first step, We trained the HuBERT Base model [15] with 95M parameters. 39
dimension MFCC features were used for obtaining cluster targets in the initial
iteration, after which the 9th transformer layer outputs were used. A total of
3 iterations were performed, and the 9th transformer layer outputs of the
final iteration were also used in obtaining final phoneme targets to train the
G2P model.

We used the LibriSpeech dataset to pre-train the model. We only use the 960h
training set for pre-training HuBERT model and no fine-tuning stage was
performed.

For masking, we randomly mask $8\%$ of the timesteps with a mask length of 10
steps. An initial value of k=100 for k-means clustering was used in the first
iteration, after which k=500 was used. However, while obtaining phone targets,
k=100 was used. Adam optimizer with linear decay and peak learning rate of
$5e^{-4}$ was used. A learning rate warmup was also used for the first $8\%$
of the training steps.

###  5.2 G2P Data

To obtain training data for our G2P model, we used the publically available
LJSpeech Dataset [22] with $\sim$ 24hrs of single speaker data at a sampling
frequency of 22050. We split the dataset into standard training, validation,
and testing splits of 8:1:1, respectively. Audio data from LJSpeech was
downsampled to 16000 before passing it through HuBERT. And as mentioned in
Section 5.1, we used k=100 for k-means clustering to obtain phone targets.

To train our rule-based and neural baselines mentioned in Section 2, we use
the CMU dict dataset. The training, validation, and testing splits were the
same as [3] to make the results directly comparable.

###  5.3 G2P model

For the G2P transformer model, we used a modified version of the open-source
toolkit Deep Phonemizer111https://github.com/as-ideas/DeepPhonemizer [23]. We
use 512 as the size for transformer input features. Both the encoder and
decoder have 4 layers of transformers. We use 1024 as the size for the feed-
forward network. A dropout rate of 0.1 was used along with a multi-head
attention consisting of 4 heads. We used a learning rate of 0.0001 with the
Adam optimizer. The input text was chunked into words before passing them
through the network.

###  5.4 Tacotron 2

We adopted the open-source TTS library CoquiTTS222 https://github.com/coqui-
ai/TTS for building and training our TTS models. Original sampling rate data
of 22050 Hz was used without any downsampling. Adam optimizer was used with
$\beta_{1}=0.9,\beta_{2}=0.998,\epsilon=10^{-6}$ and a learning rate of
$1e^{-4}$. Wav portions smaller than 40dB were considered silent and were
eliminated from the dataset.

###  5.5 Evaluation

We calculate the usability of our models and compare our model with baselines
using the golden mean opinion score (GMOS) and mean opinion score (MOS)
respectively. For calculating GMOS and MOS, we synthesized 40 sentences from
the test set. Each sentence was rated by 10 Human evaluators on a 10 points
Likert scale from 0.5 (“Bad”) to 5 (“Excellent”) with a step size of 0.5. We
report the results with 95% confidence interval.

Table 1: Comparison of our transformer G2P model with other previous works. Model | PER% | WER%
---|---|---
Bi-directional LSTM (3 layers) [17] | 5.45 | 23.55
Encoder CNN, decoder Bi-LSTM [24] | 4.81 | 25.13
Ours | 5.25 | 26.34

##  6 Results

To verify the quality of our G2P transformer model, we first train our model
and obtain the results on the standard CMU dict dataset. As can be seen from
Table 1, our models are comparable to the previous state-of-the-art. We used
the ARPABET phoneme set as the output phoneme set for training. Phoneme Error
Rate (PER) is obtained by calculating the Levenshtein distance between the
predicted phoneme sequence and the reference phoneme sequence in CMU dict. In
case of multiple pronunciations, the one with the minimum distance is used.
Word Error Rate (WER) measures how many word phoneme sequences were predicted
exactly by the transformer model.

To evaluate the usability of our approach, we calculate the GMOS using the
HuBERT model. In this setup, we don’t use any G2P model instead use the HuBERT
model directly to produce phoneme sequences using the test audio. All the
HuBERT configurations were the same as those mentioned earlier. As we use
phonemes obtained from HuBERT model to train our G2P model, these phoneme
sequences are like golden targets for us, and their good quality would
indicate the usefulness of exploring the usage of HuBERT outputs to train G2P
systems. Using test audios to produce phonemes, we obtained a MOS of $4.2\pm
0.06$, which is significantly better than the baseline, demonstrating the
usage of HuBERT outputs as phonemes for TTS systems333Audio samples are
available at : https://abhinavg4.github.io/g2p/.

Furthermore, we compare our G2P trained model using a custom phoneme set
produced by the algorithm outlined in Section 4 with traditional rule-based
G2P [25] and neural G2P trained using a lexicon. The results shown in Table 2
clearly indicate that our algorithm produces superior results as compared to
the traditional approach or even the neural approach. Apart from producing
better results, our approach does not need a lexicon and hence can be used
more widely.

###  6.1 Ablation Studies

Finally, we perform some ablation studies by varying the number of values of
clusters ($k$) when producing phoneme targets for the G2P model. Intuitively a
large value of $k$ would allow a wider range of phonemes to be produced, which
in turn would allow for better audio production in TTS. However, a very large
value of $k$ would add little to the variety and would make the model bulkier.
From Table 3 we observe that the value of 100 works best in the case of the
English language. While the value of 50 is too small to represent the variety
of sounds present, the value of 150 is too high and leads to worse
performance.

Table 2: MOS results for comparison between our proposed neural G2P, traditional baseline and neural baseline. Model | Mean Opinion Score (MOS)
---|---
Rules based G2P | 3.8 $\pm$ 0.04
Neural G2P using Lexicon | 3.91 $\pm$ 0.06
Proposed Neural G2P | 3.96 $\pm$ 0.05
Natural Speech | 4.32 $\pm$ 0.07
Golden Hubert Labels | 4.2 $\pm$ 0.06
Table 3: Ablation results for impact of cluster size on MOS. k | Mean Opinion Score (MOS)
---|---
50 | 3.71 $\pm$ 0.08
100 | 3.96 $\pm$ 0.06
150 | 3.82 $\pm$ 0.06

##  7 Conclusion

In this work, we explore a data-driven way to obtain phoneme representation
for a language. We then used these phoneme representations to train a neural
Grapheme-to-Phoneme model and used that model to train a high-quality TTS
system. Apart from performing comparable to previous G2Ps, our G2P does not
require any domain experts to build the lexicon. We perform our experiments on
the English language to compare our results with previous works. Finally, we
present ablation studies on the size of phoneme set optimal for the English
Language. As our methods does not require linguist expertise our method can be
easily extended for low-resource languages as a future work.

## References

  * [1] Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannakis, and Yonghui Wu,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  in 2018 IEEE ICASSP.
  * [2] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,”  in The 9th ISCA Speech Synthesis Workshop September 2016. 2016, p. 125, ISCA.
  * [3] Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth,  “Transformer based grapheme-to-phoneme conversion,”  in Interspeech 2019. sep 2019, ISCA.
  * [4] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu,  “Neural speech synthesis with transformer network,”  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6706–6713, 07 2019.
  * [5] Abhinav Garg, Dhananjaya Gowda, Ankur Kumar, Kwangyoun Kim, Mehul Kumar, and Chanwoo Kim,  “Improved multi-stage training of online attention-based encoder-decoder models,”  in 2019 IEEE ASRU.
  * [6] Abhinav Garg, Gowtham P. Vadisetti, Dhananjaya N. Gowda, Sichen Jin, Aditya Jayasimha, Youngho Han, Jiyeon Kim, Junmo Park, Kwangyoun Kim, Sooyeon Kim, Kyung-Joong Min, and Chanwoo Kim,  “Streaming on-device end-to-end asr system for privacy-sensitive voice-typing,”  in Interspeech, 2020.
  * [7] Abhinav Garg, Ashutosh Gupta, Dhananjaya N. Gowda, Shatrughan Singh, and Chanwoo Kim,  “Hierarchical multi-stage word-to-grapheme named entity corrector for automatic speech recognition,”  in Interspeech, 2020.
  * [8] Linhao Dong, Shuang Xu, and Bo Xu,  “Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition,”  in ICASSP, 2018.
  * [9] Jiyeon Kim, Mehul Kumar, Dhananjaya Gowda, Abhinav Garg, and Chanwoo Kim,  “A comparison of streaming models and data augmentation methods for robust speech recognition,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 989–995.
  * [10] Chanwoo Kim, Abhinav Garg, Dhananjaya Gowda, Seongkyu Mun, and Changwoo Han,  “Streaming end-to-end speech recognition with jointly trained neural feature enhancement,”  in ICASSP 2021 - 2021.
  * [11] Dhananjaya N. Gowda, Abhinav Garg, Kwangyoun Kim, Mehul Kumar, and Chanwoo Kim,  “Multi-task multi-resolution char-to-bpe cross-attention decoder for end-to-end speech recognition,”  in Interspeech, 2019.
  * [12] “Cmudict (the carnegie mellon pronouncing dictionary),” https://github.com/cmusphinx/cmudict.
  * [13] Jian Zhu, Cong Zhang, and David Jurgens,  “Byt5 model for massively multilingual grapheme-to-phoneme conversion,”  in Interspeech, 2022.
  * [14] Markéta Rezácková, Jan Svec, and Daniel Tihelka,  “T5g2p: Using text-to-text transfer transformer for grapheme-to-phoneme conversion,”  in Interspeech, 2021.
  * [15] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [16] Kanishka Rao, Fuchun Peng, Haşim Sak, and Françoise Beaufays,  “Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks,”  in 2015 IEEE ICASSP, 2015, pp. 4225–4229.
  * [17] Kaisheng Yao and Geoffrey Zweig,  “Sequence-to-sequence neural net models for grapheme-to-phoneme conversion,”  arXiv preprint arXiv:1506.00196, 2015.
  * [18] Rafael Valle, Kevin J. Shih, Ryan Prenger, and Bryan Catanzaro,  “Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis,”  in International Conference on Learning Representations, 2021.
  * [19] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Frank Soong, Tao Qin, Sheng Zhao, and Tie-Yan Liu,  “Naturalspeech: End-to-end text to speech synthesis with human-level quality,” 2022.
  * [20] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 12449–12460, Curran Associates, Inc.
  * [21] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Robert A. J. Clark, and Rif A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,”  in INTERSPEECH, 2017.
  * [22] Keith Ito and Linda Johnson,  “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [23] Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth,  “Transformer based grapheme-to-phoneme conversion,”  in Interspeech 2019. Sep 2019, ISCA.
  * [24] Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth,  “Grapheme-to-phoneme conversion with convolutional neural networks,”  Applied Sciences, 2019.
  * [25] Josef Robert Novak, Nobuaki Minematsu, and Keikichi Hirose,  “Phonetisaurus: Exploring grapheme-to-phoneme conversion with joint n-gram models in the wfst framework,”  Natural Language Engineering, vol. 22, no. 6, pp. 907–938, 2016.


# Pretraining Data and Tokenizer for Indic LLM

Author 1  Author n
Address line
Address line  Rahul Kumar, Shubham Kakde, Divyansh Rajput,
Daud Ibrahim, Rishabh Nahata, Pidathala Sowjanya, Deepak Kumar

###### Abstract

We present a novel approach to data preparation for developing multilingual
Indic large language model. Our meticulous data acquisition spans open-source
and proprietary sources, including Common Crawl, Indic books, news articles,
and Wikipedia, ensuring a diverse and rich linguistic representation. For each
Indic language, we design a custom preprocessing pipeline to effectively
eliminate redundant and low-quality text content. Additionally, we perform
deduplication on Common Crawl data to address the redundancy present in 70% of
the crawled web pages. This study focuses on developing high-quality data,
optimizing tokenization for our multilingual dataset for Indic large language
models with 3B and 7B parameters, engineered for superior performance in Indic
languages. We introduce a novel multilingual tokenizer training strategy,
demonstrating our custom-trained Indic tokenizer outperforms the state-of-the-
art OpenAI Tiktoken tokenizer, achieving a superior token-to-word ratio for
Indic languages.

Pretraining Data and Tokenizer for Indic LLM

Rahul Kumar, Shubham Kakde, Divyansh Rajput, Daud Ibrahim, Rishabh Nahata,
Pidathala Sowjanya, Deepak Kumar

##  1 Introduction

In the ever-evolving realm of Natural Language Processing (NLP), the
development of Large Language Models (LLMs) have seen a meteoric rise since
the inception of transformers. The introduction of LLMs has ushered in a new
era in NLP, where the boundaries of what machines could achieve with human
language are constantly pushed to astonishing limits. OpenAI’s ChatGPT and
Google’s BARD has been a pioneering force for redefining the landscape of
language modeling performance and has also illuminated the vast societal
implications inherent in these technological advancements. Alongside ChatGPT
and BARD, various open source and proprietary LLMs have showcased remarkable
natural language understanding and generation. There have been public releases
by some key players in the realm of LLM including Llama 2 Touvron et al.
(2023a), Mistral Jiang et al. (2023), Falcon, MPT, etc creating more
affordable, efficient, and high-performing language models. The chat-models of
these pretrained LLMs showcase comparable results on few benchmarks with
proprietary LLM chat-models including ChatGPT, BARD and Claude. LLMs such as
Llama and Falcon have outlined their data refinement and pre-training steps in
their respective technical reports. We draw inspiration from the meticulous
analysis of RefinedWeb Dataset by Falcon in our data filtering and
deduplication approaches.

Indic culture boasts linguistic diversity, encompassing Indo-Aryan, Dravidian
and Munda languages. Indo-Aryan and Dravidian languages constitute 96% of
India’s spoken languages. Despite this richness, most open language models
lack Indic language support, hindering innovation due to limited high-quality
data and complex tokenization challenges AI (2023). Notable corpora like
EMILLE/CIIL, Wikipedia for Indian languages, Samantar Corpus, and AI4Bharat-
IndicNLP provide valuable resources Kunchukuttan et al. (2020). EMILLE/CIIL
spans 14 languages with 92 million words, Wikipedia for Indian languages is
limited, Samantar Corpus offers 49.7 million parallel sentences across 11
languages, and AI4Bharat-IndicNLP Corpus contains 2.7 billion words for 10
languages Kunchukuttan et al. (2020). IndicCorp with 8.8 billion tokens across
11 languages, supplements linguistic resources Kakwani et al. (2020). Despite
these efforts, it is noteworthy that Hindi being the third most spoken
language, does not rank among the top 20 languages in processed CommonCrawl
documents, highlighting the scarcity of India-specific data for open large
language model training Penedo et al. (2023).

In this study, we provide a technical report on clean indic dataset
preparation and tokenizer training for Indic LLM : India’s own foundational
model with indic-rich context. The study highlights a comprehensive analysis
of available open source and proprietary datasets and data refinement steps.
We also devise a state-of-art indic tokenizer through rigorous
experimentations and validated the performance through a pretraining model.

##  2 Related Works

Large language models (LLMs) owe their remarkable learning capabilities to
massive model sizes and extensive training datasets. At present, there are
numerous foundational models spanning from open source and proprietary LLMs.
Some noteworthy open source foundational LLMs include Llama 2 Touvron et al.
(2023a), Mistral 7B Jiang et al. (2023), Falcon Penedo et al. (2023), MPT Team
(2023), Bloom Workshop et al. (2022), etc. whereas proprietary foundational
LLMs comprise of GPT4 Achiam et al. (2023), LaMDA Thoppilan et al. (2022),
etc. LLaMA 2 developed by Meta AI and Microsoft focuses on multilingual
capabilities and is optimized for swift training and inference. MPT-7B by
MosaicML and Mistral-7B by Mistral AI are 7-billion-parameter models that have
demonstrated efficient open-source training code, promoting transparency and
ease of use. These models have showcased superiority over other open-source
models in the 7B-20B range. Falcon-40B developed by Technology Innovation
Institute (TII) has 40-billion-parameters and is a causal decoder-only model
trained on a causal language modeling task. It is trained on a large dataset
and has demonstrated superior performance to GPT-3. BLOOM is the world’s
largest open multilingual language model, with 176-billion-parameters.
Generally, proprietary systems are more expensive and offer product solutions
that can be tailored to fit very specific business needs. Open source models
usually offer more affordable and customizable options but may lack the
performance level and specialization of proprietary LLMs.

Despite the widespread availability of LLMs for public exploration, the lack
of transparency regarding training datasets, especially for state-of-the-art
models, hinders research on addressing relevant biases. Furthermore, LLMs are
known to generate text lacking sufficient grounding to knowledge sources, thus
posing risks of misinformation and hallucination Li et al. (2023). This
challenge is exacerbated largely in multilingual learning scenarios where
datasets are often inadequately collected. Researchers have been advancing in
the development of LLMs tailored to specific regional languages Cui et al.
(2023) Balachandran (2023) Kunchukuttan et al. (2020). For development of a
robust multilingual LLM, two pivotal components are: presence of abundant
multilingual data and a diverse vocabulary Yuan et al. (2023). The current
state-of-art LLMs provide a rigid multilingual support, this is due to less
multilingual data in the pre-training corpus. For example, Llama models
Touvron et al. (2023a) Touvron et al. (2023b) have leveraged a vast pre-
training corpus with over 1.6 trillion tokens, but less than 4.5% is
multilingual data, over 20 different languages. This number is further
enhanced in Llama 2 models where the proportion of multilingual data is
increased to approximately 11% and the number of languages to around 26.
CulturaX Nguyen et al. (2023) is another multilingual dataset with 6.3
trillion tokens across 167 languages. The dataset is created after
meticulously cleaning and deduplication steps to facilitate advancements in
multilingual LLMs. However, out of 167 languages only 14 languages amount to
90.38%, thus creating a very low pre-training corpus for developing an
efficient and versatile LLM having contextual understanding of indic
languages.

To cater this issue of procuring massive datasets for LLM development,
researchers have been relying on open source datasets such as web-crawled
Common Crawl, Wikipedia, Public domain books spanning from cultural and
historical facets Gao et al. (2020), Stack Exchange and Github archives,
Journal articles and educational resources Lewkowycz et al. (2022), News
archives, Government and institutional legal repositories, Multimedia
transcripts. In this study, we aim to exploit the aforementioned data sources
for developing extensive indic data corpus.

Moreover, these massive corpora need meticulous rigorous data refinement and
deduplication to ensure data quality while maintaining integrity. Recent LLMs
have demonstrated a robust work on data preparation and filtering, The
RefinedWeb Penedo et al. (2023) has been pivotal in providing an insightful
technical background in this context, significantly enhancing our
understanding of the nuances involved. RefinedWeb has executed filtering and
deduplication techniques on Common Crawl corpus, these filters include
incorporation of threshold-based language filtering, url-filtering, line-wise
correction filters, document deduplication, etc. MassiveText Rae et al. (2021)
has defined rules for reducing nuances from the text documents by implementing
extensive quality filtering techniques. Upon rigorous filtering and
deduplication steps, a large amount of noisy data is removed from the original
corpora. It has been observed that the open source language datasets do hold a
high number of boilerplate texts and similar-context text documents Penedo et
al. (2023) Lee et al. (2021).

Consequently, deduplication process is a crucial step for producing high
quality pre-training corpus. Various deduplication algorithms have been
established in literature spanning from exact matching with suffix arrays
Manber and Myers (1993), largest substring matching , minhash Broder (1997),
simhash Charikar (2002), and other fuzzy techniques in order to minimize
memory usage while maximizing efficiency. In this work, we have methodically
incorporated the data filtering and deduplication techniques, drawing
inspiration from the research findings presented in RefinedWeb and MassiveText
and also proposed our own innovative filters for data preprocessing.

Conventional tokenization approaches often involve a complex preprocessing
pipeline and are language-specific. A simplistic and multilingual tokenizer is
required for diverse natural language processing (NLP) tasks. These
tokenization techniques generally include BPE Sennrich et al. (2015),
Wordpiece Wu et al. (2016) , Sentencepiece Kudo and Richardson (2018),
IndicBERT Kakwani et al. (2020), Spacy AI (Accessed: 2024). Kunchukuttan et
al. (2020) proposed an Indic NLP tokenizer Kunchukuttan (2020) which emerges
as an effective tokenization tool for indic-languages (Assamese, Bengali,
Gujarati, Hindi, Marathi, Odia, Punjabi, Kannada, Malayalam, Tamil, Telugu)
including English. In addition, Stanford NLP Al-Rfou and Skiena (2013) has
proposed Indic tokenizer Manning et al. (2014) which supports English, Indo-
aryan and Dravidian languages along with preprocessing functionalities.
SentencePiece Kudo and Richardson (2018) tokenizer enables a fully end-to-end
language-independent system by directly training its subword models from raw
input sentences. This approach eliminates the need for pre-tokenized word
sequences. In this study, we experiment with the SentencePiece Kudo and
Richardson (2018) tokenizer and fine-tune it based on our indic-rich corpora.

In this study, we meticulously compile data corpus from open source and
proprietary datasets for a comprehensive selection of 12 indic languages
including English. These languages span from Assamese, Bengali, English,
Kannada, Gujarati, Hindi, Marathi, Malayali, Punjabi, Odia, Tamil, Telugu. Our
contributions in paper are highlighted as :
1\. Development of High-Quality Multilingual Indic Data
2\. Novel Multilingual Tokenizer Training Strategy

##  3 Pre-training Data

###  3.1 Common Crawl

Common Crawl is an open repository that houses extensive web crawl data. Since
its inception in 2008, the archive has amassed petabytes of data and continues
to perform crawls almost monthly. The data, accessible via Common Crawl’s
public S3 bucket, is provided in three distinct archive formats: WAT, WET, and
WARC files. The WAT (Web Archive Transformation) files encompass the metadata
of the crawl, including HTTP headers, elements from the HTML <head> (such as
title, meta tags, and scripts), and links from the websites. WET (Web
Extracted Text) files contain the text extracted directly from the HTML of the
crawl. Meanwhile, WARC (Web ARChive) files comprise the complete crawl data,
encompassing both the metadata and the full HTML response. While WET files
could have served as a straightforward source for text data, our
experimentation with various HTML scraping tools revealed that the text
extracted in WET files often lacks cleanliness. Therefore, we opted to process
the WARC files, which allowed us to scrape cleaner text from their
comprehensive HTML archives.

To date, we have processed a total of 93 Common Crawl snapshots, with CC-
MAIN-2023-50 being the latest snapshot at the time of writing this paper. Our
processing of the Common Crawl data is divided into three major steps:

  1. 1.

Preprocessing : It involves raw text extraction.

  2. 2.

Postprocessing : Entails Language Detection and the application of Heuristic
Filters.

  3. 3.

Deduplication : Aimed at removing duplicate components.

Figure 1: Common Crawl Dataset Processing Pipeline

Common Crawl Preprocessing Among the various data sources utilized during the
pretraining phase, the Common Crawl dataset presented the most significant
challenge, primarily due to its vast scale, which encompasses approximately 7
petabytes of data. We employed the warcio library in Python, which is adept at
efficiently streaming WARC files instead of loading them entirely into memory.
This approach is crucial given the substantial size of WARC files, often
reaching gigabytes, making complete loading into memory impractical. This
streaming process necessitated a separate post-processing pipeline for
language identification and heuristic feature computation, which proved more
efficient when conducted in batches rather than on individual records.

In our exploration of various open-source text extraction libraries, including
unstructured.io and trafilatura, we found that trafilatura delivered the most
effective results. While streaming through the warcio iterator, we extracted
clean text, markdown text, and URLs from each web page using trafilatura.

Our analysis encompassed a total of 5,455,398 WARC files across 93 snapshots.
We initially conducted a Proof of Concept (POC) for WARC preprocessing using a
PySpark pipeline. In this pipeline, we read multiple WARC files as RDDs
(Resilient Distributed Datasets) and processed them concurrently. Each RDD in
this pipeline was a tuple containing the file path and the content of the WARC
file as a byte array. We employed the same warcio library in the map partition
User-Defined Function (UDF) for processing these files.

Additionally, we established a vanilla Python processing pipeline utilizing
multiprocessing, which enabled parallel processing of multiple WARC files,
fully utilizing the cores of a machine. The extraction of text using
trafilatura from each WARC file took approximately 40 to 60 minutes, varying
with the number of web pages archived in each file. Both the PySpark and
multiprocessing vanilla Python pipelines demonstrated similar processing times
for the WARC files. However, we opted to process the files using our
multiprocessing vanilla Python pipeline. This decision was driven by the
significant reduction in compute costs it offered. Furthermore, this approach
allowed us to design our pipeline more fault-tolerantly, leveraging a custom
orchestration pipeline that ran on multiple AWS EC2 instances.

Common Crawl Postprocessing Following the preprocessing step for the WARC
files, we obtain a CSV file corresponding to each WARC file. These files
comprise columns detailing the scraping date, clean text, markdown text, and
URL. As our focus is on developing Large Language Models (LLMs) for Indic
languages, an initial step involves applying a language detection filter to
segregate non-Indic languages, with the exception of English. In our
exploration of various open-source language detection models, including
AI4Bharat and FastText, we found that FastText provided the most accurate
results. Currently, the text extracted from the Common Crawl dataset is being
classified into English and 11 specific Indic languages.

Post this language filtration, approximately 50% of the initial data remains.
Subsequently, we compute heuristic features such as token count, mean sentence
length, mean word length, symbol-to-word ratio, perplexity (ppl) score,
fraction of duplicate lines, fraction of characters in duplicate lines,
fraction of characters in the most common 2-11-grams, fraction of lines ending
with an ellipsis, and fraction of lines starting with a bullet point. Through
exploratory data analysis conducted for each language, we determined that
applying these filters within the range of 0-90th percentiles effectively
eliminates unclean and gibberish documents. Additionally, we examined the
impact of applying filters solely on mean word length, mean sentence length,
language threshold, and symbol-to-word ratio. The outcomes were found to be
similar to those obtained when applying the full range of heuristic
features.Table 1 presents the number of documents and token count subsequent
to the application of these filters.

Language |  Lang(0.1) + Basic filter |  Lang(0.6) + Basic filter |  Dedup + Lang(0.6) + Basic filter |  Dedup + Lang(0.6) + All filter
---|---|---|---|---
Hi |  108 |  85 |  54 |  51.2
Bn |  49.5 |  34 |  18.6 |  16.3
Ta |  33.6 |  22.3 |  6.2 |  5.1
Ml |  14.3 |  9.5 |  3.2 |  2.7
Mr |  11.2 |  8.1 |  2.7 |  2.4
Te |  10.5 |  4 |  1.8 |  1.6
Kn |  6.2 |  4.2 |  1.4 |  1.1
Gu |  6.1 |  4.5 |  1.5 |  1.3
Pa |  5.1 |  2.7 |  0.88 |  0.69
Or |  1.2 |  0.9 |  0.4 |  0.3
As |  0.8 |  0.6 |  0.01 |  0.01
Table 1: Comparative Analysis of Token Counts(in millions) Before and After
Application of Basic, Full Range Filters and Deduplication

Common Crawl Deduplication As noted on the Common Crawl website, each snapshot
of their web crawl typically encompasses approximately 3 billion web pages.
Remarkably, 2 billion of these pages have been previously crawled in earlier
snapshots. This results in an average of about 66% duplicate content per
snapshot. When considering all snapshots cumulatively, the proportion of
duplicate content is substantially higher. Consequently, the implementation of
a deduplication process is crucial. It not only ensures the high quality of
data but also significantly reduces the computational resources required for
training.
In our deduplication pipeline, we have employed the Minhash Locality Sensitive
Hashing (LSH) technique. The Minhash LSH process involves three primary steps:
first is shingling, which converts documents into set representations; second
is min-hashing, which transforms these large sets into shorter signatures
while preserving their similarity; and finally, LSH, which identifies likely
candidates for similarity based on these signatures. We have opted for 5-gram
shingles for the initial conversion of each document into sets. In the min-
hashing step, we set the signature dimension to 250. By establishing a
similarity threshold of 0.7, we determined that the optimal configuration
consists of 25 bands and 10 rows per band for this process.
While duplicates were present across snapshots, we chose not to run our
deduplication pipeline across all snapshots to avoid excessive data loss.
Additionally, we noted that if a URL is re-crawled within approximately a
year, its content is likely to remain largely unchanged, although the converse
may not hold true. Therefore, we segmented our deduplication process,
considering snapshots spanning 1-2 years as a single batch, which roughly
equates to 10 million documents. Thus, we divided the 93 snapshots for each
language into batches of 10 million documents and executed the deduplication
pipeline on each batch. This approach effectively removes recent duplicated
data while preserving older, potentially unique data.The number of documents
before and after deduplication is presented in Tabel 1

Language |  Token count |  Mean word length |  symbol to word ratio |  mean sentence length
---|---|---|---|---
Hi |  [50,10000] |  [3 10] |  0.22 |  >4.2
Bn |  [40,10000] |  [4,9] |  0.24 |  >4.4
Ta |  [45,10000] |  [6,9] |  0.32 |  >5.1
Ml |  [55,10000] |  [6,10] |  0.25 |  >3.5
Mr |  [43,10000] |  [4,6] |  0.31 |  >4.3
Te |  [52,10000] |  [5,8] |  0.28 |  >5.4
Kn |  [48,10000] |  [5,8] |  0.32 |  >3.6
Gu |  [51,10000] |  [4,6] |  0.23 |  >3.4
Pa |  [55,10000] |  [3,6] |  0.24 |  >3.7
Or |  [47,10000] |  [4,7] |  0.32 |  >4.2
As |  [49,10000] |  [4,7] |  0.25 |  >3.8
Table 2: Comparative Analysis of Filter Thresholds Language-wise

###  3.2 Newspaper

There are plethora of newspapers across various indic languages which are
regarded as a knowledge-rich source of data for pre-training our indic-rich
context LLM. We accumulated language-specific newspapers and downloaded the
digital version of all the historical editions. These newspapers generally
comprise blocks of images, tables and advertisements which are noisy data in
pre-training corpora. Henceforth, a robust and scalable algorithm is necessary
to identify individual useful text-blocks from the news article and
subsequently extract data out of it. This elementary problem of article
extraction is challenging due to composition of a wide range of layouts random
arranged in the target page. Research has been prevailing for development of
algorithms addressing the text extraction issues . However, most of these
methods are based on a set of heuristic rules. In this study, we construct an
article-extraction pipeline by experimenting with existing open source
algorithms and frameworks. Our goal in this extraction process is to identify
bounding boxes of layouts in an article and then detect relevant text-blocks
for respective languages. Initially, we experiment with an open source python
package, layoutparser, prolific in document-image analysis tasks. LayoutParser
provides a rich repository of deep learning models for layout detection as
well as a set of unified APIs for using them.LayoutParser comes with a set of
layout data structures with carefully designed APIs that are optimized for
document image analysis tasks. In addtion, we train models on indic-data
comprising of various handwritten documents and newspapers. In this step, we
experiment with deep learning-based models by detectron2 from layoutparser
library are leveraged to detect text-snippets from the layouts of the article.
Furthermore, this algorithm also detects tables which are redundant for the
text corpus. After rigorous experimentation, we resolved to fine-tune a
MaskRCNN based model for our extraction process.

Annotation All the segmented blocks of the newspaper images are labelled into
4 classes namely headings, text, images and non-text by leveraging label
studio. The text class focuses mainly on the article block and this is what we
intend to extract. The headings class contains all the main and sub headings
in the newspaper image. The non-text class taks care of the tables, summaries
and quotes.

Model fine-tuning MaskRCNN based model is a three stage model that predicts a
class label, bounding box offset and object mask for each Region of Interest
(ROI). It can produce more accurate and fine-grained masks for each object,
which can capture the shape and contour details better than bounding boxes. It
can also handle overlapping and occluded objects better.

Model evaluation The performance of the object detection and localization
algorithm is evaluated by a metric called Average Precision (AP) and mean
average precision (MAP). AP is calculated with the help of several other
metrics such as Intersection over Union (IoU), confusion matrix (TP, FP, FN),
precision and recall as shown in the figure below. IoU quantifies the
closeness of the two bounding boxes (ground truth and prediction). It’s a
value between 0 and 1. The IoU is calculated by taking the ratio between the
area of intersection and the area of the union of two bounding boxes. Average
Precision is the area under the PR curve. AP summarizes the PR Curve to one
scalar value. Average precision is high when both precision and recall are
high, and low when either of them is low across a range of confidence
threshold values. The range for AP is between 0 to 1. AP value can be
calculated for each class. The mean average precision is calculated by taking
the average of AP across all the classes under consideration.

Model Inference The model inference pipeline takes up one newspaper image and
then bounding box prediction is done on the entire newspaper image along with
the class. Masking of the heading, non-text and images class bounding boxes is
done on the image. For each text class bounding box, a tesseract based OCR is
used for extraction of text.

###  3.3 Books

The web crawled data from Common Crawl has limited distribution of Indic
language specific content. The distribution of deduplicated content is indeed
further lesser. This is a huge challenge especially when we want to perform
LLM pre-training with Indic languages. In order to increase the number of
tokens as well as the breadth of Indic specific content, we leverage the open
source pdfs that mostly contain books, periodicals, magazines, and financial
reports. The pdfs are downloaded using a pipeline across the indic languages.
The pdfs have broadly two types of format present in it, image or text
embedded on pdf. A pipeline is built that detects whether an image is embedded
on the pdf and then based on that, appropriate extraction pipeline runs.
Tesseract OCR is leveraged to extract text out of these pdf documents with
images embedded on it. There is a challenge of detecting a script so that
tesseract OCR engine with appropriate language is used for extraction. We
overcome this with our pipeline that detects the script and extracts text out
of it. It is then scaled up with multiprocessing and across multiple nodes.

##  4 Tokenizer

Tokenization is a preprocessing step in natural language processing (NLP),
facilitating machine comprehension and interpretation of human language by
breaking down text into fundamental units such as words, characters and
subwords. Among the various types of tokenization tools, one widely employed
approach is SentencePiece, which sets itself apart by directly training
subword models from raw sentence data. This method offers a fully end-to-end
and language-agnostic system, eliminating the need for pre-tokenized word
sequences.

In our study, we adopted Byte Pair Encoding (BPE) as the tokenization
strategy. We assessed the quality of tokenization using two key metrics: the
token-to-word ratio and the exact score, which measures the proportion of
correctly tokenized units out of the total tokens.

Language | 70k | 85k | 100k
---|---|---|---
en | 1.52 | 1.49 | 1.46
hi | 1.38 | 1.36 | 1.35
mr | 1.83 | 1.83 | 1.79
te | 2.58 | 2.45 | 2.39
ta | 2.51 | 2.41 | 2.30
as | 2.06 | 2.03 | 2.00
gu | 2.05 | 1.92 | 1.90
kn | 2.51 | 2.32 | 2.29
ml | 3.01 | 2.98 | 2.86
od | 1.94 | 1.89 | 1.85
pn | 1.58 | 1.66 | 1.63
Table 3: Token-to-Word Ratio for Different Vocabulary Sizes

###  4.1 Tokenizer Experiments

Corpus Size For training the tokenizer, we utilized sample data from various
sources such as Common Crawl, Wikipedia, and Books. This sampled data
constitutes the corpus size employed during tokenizer training. The data in
the corpus is sampled in proportion to the availability of data from these
different sources. The vocabulary size in the context of a tokenizer refers to
the number of unique tokens or subword units that the tokenizer can produce or
handle. We trained the tokenizer with different corpus sizes and vocabulary
sizes. The results in corpus and vocab table indicate that increasing the
corpus size does not significantly improve the performance of the tokenizer.
However, it is evident that increasing the vocabulary size significantly
improves the token-to-word ratio. This finding is logical because increasing
the corpus size does not necessarily enhance tokenizer performance once a
critical mass of major words has been covered. If the vocabulary size remains
constant, the same or similar tokens will be generated, and the token-to-word
ratio will remain unchanged, even with an increased corpus size. Thus, beyond
a certain point, expanding the corpus size yields diminishing returns in terms
of tokenizer improvement.

Vocabulary Size Additionally, we experimented with various corpus sizes. The
results, as presented in the corpus table, demonstrate that a tokenizer
trained with a 225 million sampled corpus size achieves a token-to-word ratio
similar to that of a tokenizer trained with a 12 billion sampled corpus size.
Increasing the vocabulary size allows for the formation of more eligible words
or sub-words as tokens, thereby improving the tokenizer’s quality, as
indicated by the improved token-to-word ratio. However, this improvement comes
with a trade-off: a tokenizer with a larger vocabulary size will have reduced
inference speed compared to one with a smaller vocabulary size.

Character Coverage We also experimented with different character coverage
levels during tokenizer training. Setting character coverage to 1 resulted in
the inclusion of numerous gibberish characters, such as emojis and special
characters, in the tokenizer vocabulary. These characters, despite having low
frequencies, were included due to the high character coverage. To address
this, we tested various character coverage settings. As presented in the
character coverage table, we found that a character coverage of 0.997 is
optimal. This setting ensures that the characters included in the Hindi
tokenizer vocabulary closely match the typical set of characters in the Hindi
language, which consists of approximately 50 characters (36 consonants and 12
vowels).

Language | GPT-4o | Indic
---|---|---
en | 1.33 | 1.48
hi | 1.62 | 1.39
mr | 2.53 | 1.80
te | 3.15 | 2.29
ta | 3.16 | 2.30
as | 2.70 | 1.99
gu | 2.28 | 1.91
kn | 3.03 | 2.31
ml | 3.37 | 2.71
od | 6.39 | 1.80
pn | 2.68 | 1.56
Table 4: Token-to-Word Ratios for GPT-4o and Indic 100k tokenizer on ai4bharat
sangraha Corpus

###  4.2 Tokenizer Training

Initially, we trained a tokenizer with the optimal vocabulary size, corpus
size, and character coverage determined from our previous experiments. Despite
sampling a dataset focused on Indic content, we observed a significant number
of false positives, which introduced characters from other languages, such as
Chinese and French, into our Indic LLMs. To address this issue and create a
clean Indic tokenizer, we implemented a series of steps specifically targeting
Indic words.

First, we trained a dummy tokenizer using the sampled data from various
sources. With the assistance of linguistic experts, we manually removed
gibberish characters and characters from other languages from the dummy
tokenizer vocabulary. Next, we encoded and decoded the sampled dataset using
this dummy tokenizer, with the byte fallback flag set to false. This process
converted all gibberish words and non-Indic language words identified by
linguists to UNK tokens.

With this cleaned dataset, we then trained our final tokenizer. This refined
tokenizer demonstrated optimal vocabulary size, corpus size, and character
coverage, with minimal inclusion of gibberish and foreign language words. We
compared the performance of our tokenizer with the OpenAI Tiktoken tokenizer
across 11 Indic languages. The results, as shown in the tokenizer table,
clearly indicate that our tokenizer’s metrics significantly outperform those
of the Tiktoken tokenizer.

##  5 Conclusion

In conclusion, our meticulous data acquisition and custom preprocessing
pipeline have effectively curated a high-quality multilingual dataset,
significantly enhancing the performance of our Indic large language models.
Through rigorous filtering and deduplication processes, we ensured the
elimination of redundant and low-quality content, optimizing the data for
training. Our novel multilingual tokenizer training strategy demonstrated
superior token-to-word ratios for Indic languages compared to the state-of-
the-art OpenAI Tiktoken tokenizer. The experiments underscore the importance
of tailored preprocessing and tokenizer design, paving the way for more
accurate and efficient language models in Indic-rich multilingual contexts.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI (Accessed: 2024) Explosion AI. Accessed: 2024.  spaCy tokenizer api.  https://spacy.io/api/tokenizer.
  * AI (2023) SARVAM AI. 2023.  Openhathi series: An approach to build bilingual llms frugally.
  * Al-Rfou and Skiena (2013) Rami Al-Rfou and Steven Skiena. 2013.  Speedread: A fast named entity recognition pipeline.  _arXiv preprint arXiv:1301.2857_.
  * Balachandran (2023) Abhinand Balachandran. 2023.  Tamil-llama: A new tamil language model based on llama 2.  _arXiv preprint arXiv:2311.05845_.
  * Broder (1997) Andrei Z Broder. 1997.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)_ , pages 21–29. IEEE.
  * Charikar (2002) Moses Charikar. 2002.  Similarity estimation techniques from rounding algorithms.  pages 380–388.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020.  Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4948–4961.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_.
  * Kunchukuttan (2020) Anoop Kunchukuttan. 2020.  The IndicNLP Library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.
  * Kunchukuttan et al. (2020) Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Avik Bhattacharyya, Mitesh M Khapra, Pratyush Kumar, et al. 2020.  Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages.  _arXiv preprint arXiv:2005.00085_.
  * Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.  Deduplicating training data makes language models better.  _arXiv preprint arXiv:2107.06499_.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _Advances in Neural Information Processing Systems_ , 35:3843–3857.
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6449–6464.
  * Manber and Myers (1993) Udi Manber and Gene Myers. 1993.  Suffix arrays: a new method for on-line string searches.  _siam Journal on Computing_ , 22(5):935–948.
  * Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014.  The stanford corenlp natural language processing toolkit.  In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , pages 55–60.
  * Nguyen et al. (2023) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023.  Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.  _arXiv preprint arXiv:2309.09400_.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only.  _Computing Research Repository_ , arXiv:2306.01116.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Team (2023) MosaicML NLP Team. 2023.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  Accessed: 2023-05-05.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yuan et al. (2023) Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. 2023.  How multilingual is multilingual llm?  _arXiv preprint arXiv:2311.09071_.