# IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning
Datasets for Indian Languages

Mohammed Safi Ur Rahman Khan∗1 Priyam Mehta$~{}~{}^{1}$ Ananth Sankar1
Umashankar Kumaravelan1 Sumanth Doddapaneni1,2 Suriyaprasaad
G1,422footnotemark: 2
Varun Balan G1,522footnotemark: 2 Sparsh Jain1,622footnotemark: 2 Anoop
Kunchukuttan1,2,3
Pratyush Kumar1,2,7 Raj Dabre2,8 Mitesh M. Khapra1,2

1Nilekani Centre at AI4Bharat 2Indian Institute of Technology, Madras
3Microsoft
4Sant Longowal Institute of Engineering and Technology 5IIIT D&M Kancheepuram
6Maharaja Agrasen Institute of Technology 7Sarvam AI
8National Institute of Information and Communications Technology, Kyoto, Japan
Equal Contribution. All author contributions listed in Author
Contributions.Work done during an internship at Nilekani Center at AI4Bharat
Corresponding Author: Mitesh Khapra<EMAIL_ADDRESS>
###### Abstract

Despite the considerable advancements in English LLMs, the progress in
building comparable models for other languages has been hindered due to the
scarcity of tailored resources. Our work aims to bridge this divide by
introducing an expansive suite of resources specifically designed for the
development of Indic LLMs, covering 22 languages, containing a total of 251B
tokens and 74.8M instruction-response pairs. Recognizing the importance of
both data quality and quantity, our approach combines highly curated manually
verified data, unverified yet valuable data, and synthetic data. We build a
clean, open-source pipeline for curating pre-training data from diverse
sources, including websites, PDFs, and videos, incorporating best practices
for crawling, cleaning, flagging, and deduplication. For instruction-fine
tuning, we amalgamate existing Indic datasets, translate/transliterate English
datasets into Indian languages, and utilize LLaMa2 and Mixtral models to
create conversations grounded in articles from Indian Wikipedia and Wikihow.
Additionally, we address toxicity alignment by generating toxic prompts for
multiple scenarios and then generate non-toxic responses by feeding these
toxic prompts to an aligned LLaMa2 model. We hope that the datasets, tools,
and resources released as a part of this work will not only propel the
research and development of Indic LLMs but also establish an open-source
blueprint for extending such efforts to other languages. The data and other
artifacts created as part of this work are released with permissive licenses
at https://github.com/AI4Bharat/IndicLLMSuite

##  1 Introduction

Figure 1: Overview of the different components present in IndicLLMSuite.

Building Large Language Models (LLMs) is an inherently data-intensive process
requiring a comprehensive set of resources for pre-training (Raffel et al.,
2020; Xue et al., 2021; Gao et al., 2021; Penedo et al., 2023; Nguyen et al.,
2023a; Abadji et al., 2022) and fine-tuning (Longpre et al., 2023; Conover et
al., 2023; Köpf et al., 2023; Ding et al., 2023a). The last year has seen
remarkable progress in building English LLMs, thanks to open-source models
(Touvron et al., 2023a; b; Jiang et al., 2023; 2024a; Almazrouei et al., 2023)
developed using comprehensive datasets containing such resources. Nonetheless,
this progress has largely bypassed low and mid-resource languages due to the
lack of data resulting from the lack of open-source pipelines for curating
data for such languages from diverse sources such as websites (which require
crawling and extraction), books (which require OCR) and videos (which require
transcription). Further, for instruction fine-tuning, English LLMs now rely on
model-generated data such as ShareGPT111https://sharegpt.com/, Self-Instruct
(Wang et al., 2023b), Evol-Instruct (Xu et al., 2023a), UltraChat (Ding et
al., 2023a), etc. However, for low and mid resource languages, this option is
not available due to lack of high-quality LLMs, leading to a chicken and egg
problem, further widening the gap between the haves and the have-nots.

A case in point is that of languages from the Indian sub-continent, which
collectively are spoken by over 1.4 billion people. We focus on the 22
languages recognised in the 8th schedule of the Indian constitution. These
languages, despite their significant number of speakers, receive minimal
representation in the training datasets and tokenizers of current open-source
LLMs (Touvron et al., 2023b; Jiang et al., 2024a; Almazrouei et al., 2023)
leading to a notable exclusion of their rich cultural contexts and nuances. In
this work, we address this disparity by making the following contributions, as
summarised in Figure 1:

1\. Sangraha: Pre-training data containing 251B tokens222We built a custom
tokenizer which supports English and Indian languages and has an average
fertility of 1.3 to 2.79 across the 22 languages. We use this tokenizer for
all the reported statistics unless mentioned otherwise. summed up over 22
languages, summarized in Table 1, extracted from curated URLs, existing
multilingual corpora, and large-scale translations.

2\. Setu: Spark-based (Zaharia et al., 2016) distributed pipeline customized
for Indian languages for extracting content from websites, PDFs and videos,
with in-built stages for cleaning, filtering, toxicity removal and
deduplication.

3\. IndicAlign - Instruct: A diverse collection of 74.7 million prompt-
response pairs across 20 languages, summarized in Table 7, collected through
four methods: aggregating existing Instruction Fine-Tuning (IFT) datasets,
translating English datasets into 14 Indian languages using an open-source
translation model, creating context-grounded conversations from India-centric
Wikipedia articles using open source LLMs, and establishing a crowdsourcing
platform called Anudesh for prompt collection. We also create a novel IFT
dataset to teach the model language and grammar, by leveraging IndoWordNet
(Bhattacharyya, 2010), a lexically rich but rather neglected resource in the
era of LLMs.

4\. IndicAlign - Toxic: 123K pairs of toxic prompt and non-toxic responses
generated using open source English LLMs and translated to 14 Indian languages
for safety alignment of Indic LLMs.

We collectively refer to the above as IndicLLMSuite. We try to balance quality
and quantity while acknowledging recent trends of using synthetic data for
building powerful LLMs for English (Gunasekar et al., 2023; Li et al., 2023d)
as well as low resource languages (Nguyen et al., 2023b; Li et al., 2023c). To
ensure quality, we take help from humans to verify websites to flag noisy or
machine translated content and to create toxicity lists for Indian languages.
On the other hand, to ensure explicit representation of prompt-response pairs
grounded in the Indian context we take the help of powerful open-source LLMs
to generate grounded conversations from India-centric Wikipedia articles. We
recognize the need to represent diverse knowledge and alignment information in
Indic languages for better performance of LLMs in Indic languages. Hence, we
undertake large-scale machine translation of rich English resources like
Wikimedia as well as English fine-tuning datasets into Indian languages using
SOTA open-source MT models.

We thus balance source original data with translated and LLM-generated data to
create the above collection.

We believe that these choices can be replicated across other languages to
create LLMSuites. All the code, tools and datasets developed as a part of this
work will be publicly released and hopefully advance the development of LLMs
for Indian languages. Given that LLM training is an expensive exercise, we
plan to undertake community-effort to train LLMs, where multiple groups can
pool together computing resources to build a high-quality Indic language LLM.

##  2 Related Works

We organise the Related Work into 3 sections in line with our main
contributions.

###  2.1 Multilingual Datasets

Wikipedia has consistently served as the go to repository of multilingual data
and continues to be an important contributor for training data. Prior works
such as OSCAR (Abadji et al., 2022), CC100 (Conneau et al., 2020)
(encompassing 100 languages), and mC4 (Xue et al., 2021) (encompassing 101
languages) have been instrumental in generating data for a large set of
languages through the meticulous processing of common crawl
dumps333https://commoncrawl.org/. The overarching methodology across all the
works includes the systematic extraction of data, language identification, and
subsequent stages of filtering and deduplication. Notably, CCNet (Wenzek et
al., 2020) employs filters based on n-gram language models, while OSCAR and
mC4 (Xue et al., 2021) leverage heuristic-based filtering mechanisms. Efforts
like Samanantar (Ramesh et al., 2022) and ROOTS (Laurençon et al., 2023) have
underscored the importance of aggregating existing datasets as an initial step
towards multilingual corpus construction. ROOTS Laurençon et al. (2023), with
a specific focus on 59 languages, has additionally drawn attention to
potential data duplications across disparate pipelines. Building on these
works, MADLAD-400 (Kudugunta et al., 2023) extends to 419 languages, while
introducing human audit of data and iterative refinement processes, coupled
with language family-specific filters. CulturaX (Nguyen et al., 2023a), merges
mC4 and all versions of OSCAR, followed by a rigorous cleaning pipeline to
produce corpora in 167 languages.

###  2.2 Data Curation

Kreutzer et al. (2022a) has unilaterally showed the importance of auditing
datasets. They discovered problems like wrong language, bad quality, offensive
content, etc. Other works Rae et al. (2021); Penedo et al. (2023) reinforced
the idea that using clean data is key to making better models.

##### Sources & Scraping.

Majority of data curation pipelines start with Common Crawl as the internet
source, followed by a series of cleaning steps to produce the final dataset
(Raffel et al., 2020; Xue et al., 2021; Conneau et al., 2020; Penedo et al.,
2023). Wikipedia is another common source of _high quality_ data that is used
across many works (Gao et al., 2021; Computer, 2023; Soldaini et al., 2024).
Research (Brown et al., 2020; Touvron et al., 2023a) further demonstrates the
benefits of augmenting datasets with high-quality content, such as Wikipedia
and Books corpus, for improved language model training. The first major hurdle
for scraping data is extracting the actual text content from the HTML files
and filtering out any boilerplate and unwanted HTML content. Tools like
Trafilatura (Barbaresi, 2021) and jusText (Endrédy & Novák, 2013) are widely
used in recent works like RefinedWeb (Penedo et al., 2023) and Pile (Gao et
al., 2021).

##### Language Identification.

The subsequent challenge in the data cleaning process involves language
detection, a task for which several LID tools have been employed in previous
studies, namely, cld3 (Botha et al., 2017)444https://github.com/google/cld3/,
langdetect555https://github.com/shuyo/language-detection, fasttext (Wenzek et
al., 2020; Costa-jussà et al., 2022), SSLID (Kudugunta et al., 2023), among
others. These tools demonstrate proficiency in identifying 55 to 500
languages; however, they fall short in encompassing all Indian languages
considered in this study. Complications arise, particularly in the case of
low-resource languages sharing the same script, leading to potential instances
of mislabeling (e.g., Hindi and Marathi languages), necessitating the need for
focused language family-specific detectors (Madhani et al., 2023a).

##### Filtering.

One of the pivotal components in the data processing pipeline involves
implementing various heuristics to ensure the quality of the processed data.
These heuristics encompass rule-based approaches, targeting elements such as
punctuation, repetitions, special characters, templated content, code
snippets, etc. (Wenzek et al., 2020; Raffel et al., 2020; Xue et al., 2021;
Laurençon et al., 2023; Rae et al., 2021). Additionally, model-based tools,
including n-gram language models perplexities (Wenzek et al., 2020; Laurençon
et al., 2023; Nguyen et al., 2023a) and ML classifier-based approaches (Brown
et al., 2020), play a crucial role in filtering out data that falls below
certain quality thresholds. A prominent emphasis within the data cleaning
process is the removal of toxic and harmful content, a priority shared across
various datasets. Techniques employed for this purpose often include the
utilization of word lists (Raffel et al., 2020), blocklists of URLs (Penedo et
al., 2023), and leveraging Google Safe Search (Rae et al., 2021). Notably,
MADLAD-400 (Kudugunta et al., 2023) adopts language-specific heuristics,
incorporating a manual audit of a small portion of data to enhance the
cleaning process. This multifaceted approach underscores the importance of
employing diverse strategies to ensure the reliability and safety of processed
data in various contexts.

##### Deduplication.

Recent research findings shed light on the significant impact of deduplication
on language models (LMs). Carlini et al. (2023) demonstrated that
deduplication effectively reduces memorization within LMs, while Lee et al.
(2022) highlighted its role in enhancing LM performance. Furthermore, recent
work Hernandez et al. (2022) underscored the detrimental effects of data
repetition on model performance, particularly as model size increases. These
insights underscore the critical importance of rigorous deduplication
procedures in dataset preprocessing. Common approaches to deduplication
encompass various techniques such as deduplication based on URLs, fuzzy
techniques like MinHash (Broder, 1997) and SimHash (Charikar, 2002), alongside
embedding techniques like those proposed by SemDedup (Abbas et al., 2023).

With the rapid expansion of LLM development, a significant challenge arises
from the proliferation of machine-generated text on the internet. It is
important to devise effective strategies for filtering out such content to
ensure the creation of high-quality data intended for human consumption. This
necessitates a careful curation process that distinguishes between machine-
generated and human-written text. Additionally, another issue pertains to the
crawling and inclusion of benchmark/test data in the pre-training mixture
i.e., Data Contamination (Sainz et al., 2023; Golchin & Surdeanu, 2024).
BigBench (Srivastava et al., 2022) advocates for filtering based on matching
key strings, while recent research suggests encrypting benchmark data with
passwords before distribution (Jacovi et al., 2023).

###  2.3 Supervised Fine-Tuning Datasets

Pre-training demands huge amounts of data to effectively train on diverse
linguistic patterns, while fine-tuning, specifically instruction tuning,
necessitates comparatively smaller yet high-quality datasets (Zhou et al.,
2023). Broadly there exist two approaches for creating these datasets: (i)
_human-generated_ , involving the manual input of humans for prompt creation
and/or answer generation; and (ii) _model-generated_ , where both prompts and
answers are generated by models.

##### Human Generated

Dolly (Conover et al., 2023) and Open-Assistant (Köpf et al., 2023) are
created through a comprehensive human annotation process from start to finish.
Dolly, for instance, was curated by around 5000 Databricks employees to create
nearly 15k instructions, whereas Open-Assistant emerged from a collaborative
crowd-sourced initiative, accumulating 10k conversations spanning across 35
languages. These datasets encompass a wide array of tasks including question-
answering, creative writing, classification, etc. On the flip side, datasets
like ShareGPT666https://sharegpt.com/ and WildChat Zhao et al. (2024) are
created by gathering real human interactions with
ChatGPT777https://chat.openai.com/. Similarly, HC3 Guo et al. (2023) gathered
both human and ChatGPT responses for questions generated by humans on public
datasets. These variations span diverse topics due to the extensive
participation in creating these datasets.

##### Model Generated

Distilling data from powerful models for the creation of datasets tailored for
SFT tasks has become a widespread practice. Self-Instruct (Wang et al., 2023a)
introduced an almost annotation-free data creation pipeline. Starting with
just 175 seed prompts, they expand the dataset to 52k instructions
iteratively. Alpaca (Taori et al., 2023) streamlined the Self-Instruct
process, leading to increased efficiency and cost reduction. Taking a further
stride, Evol-Instruct (Xu et al., 2023a) introduces a novel method of evolving
the seed prompts iteratively along various axes. This resulted in the
generation of an extensive dataset comprising 250k instructions. Further works
generate data by simulating conversations between two or more model agents,
with at least one acting as a User and the other acting as the Assistant.
CAMEL (Li et al., 2023a), for instance, crafted 115k instructions via multi-
agent role play, while Ultrachat (Ding et al., 2023a) produced 1.5 million
multi-turn dialogues. Similarly, Baize (Xu et al., 2023c) generated 115k
dialogues through self-chat with ChatGPT.

##  3 Sangraha

Code | SV | SS | SU | Total Tokens
---|---|---|---|---
asm | 292.1 | 11,696.4 | 17.5 | 12,006.0
ben | 10,604.4 | 13,814.1 | 5,608.8 | 30,027.5
brx | 1.5 | - | - | 1.5
doi | 0.06 | - | - | 0.06
eng | 12,759.9 | - | - | 12,759.9
gom | 10.1 | - | - | 10.1
guj | 3,647.9 | 12,934.5 | 597.0 | 17,179.4
hin | 12,617.3 | 9,578.7 | 12,348.3 | 34,544.3
kan | 1,778.3 | 12,087.4 | 388.8 | 14,254.5
kas | 0.5 | - | - | 0.5
mai | 14.6 | - | - | 14.6
mal | 2,730.8 | 13,130.0 | 547.8 | 16,408.6
mar | 2,827.0 | 10,816.7 | 652.1 | 14,295.8
mni | 7.4 | - | - | 7.4
npi | 1,822.5 | 10,588.7 | 485.5 | 12,896.7
ori | 1,177.1 | 11,338.0 | 23.7 | 12,538.8
pan | 1,075.3 | 9,969.6 | 136.9 | 11,181.8
san | 1,329.0 | 13,553.5 | 9.8 | 14,892.3
sat | 0.3 | - | - | 0.3
snd | 258.2 | - | - | 258.2
tam | 3,985.1 | 11,859.3 | 1,515.9 | 17,360.3
urd | 3,658.1 | 9,415.8 | 1,328.2 | 14,402.1
tel | 3,706.8 | 11,924.5 | 647.4 | 16,278.7
Total | 64,306.1 | 162,707.9 | 24,307.7 | 251,321.0
Table 1: Number of tokens (in Millions) in each split of Sangraha. (SV:
Sangraha Verified, SS: Sangraha Synthetic, SU: Sangraha Unverified). We
represent the languages in this document using the ISO 639-3 standard codes

In this section, we describe the composition and curation process of Sangraha
spanning verified (64B), unverified (24B), and synthetic (162B) content for a
total of 251B tokens. Table 1 shows the language level tokens distribution in
each of the splits.

###  3.1 Sangraha Verified

We introduce Sangraha Verified, a high-quality dataset, which adds human
verification at various stages of its curation. A major chunk of this includes
data crawled from high-quality, manually verified Indic language websites.
Additionally, recognizing the fact that a significant amount of Indic language
text is locked in PDFs and audio, we also collect data from various
books/documents and videos resulting in a total of 64B tokens. Table 2 shows
the language level tokens distribution across the Web, PDF, and Speech data in
Sangraha Verified.

| Sangraha Verified
---|---
Code |  Web |  PDFs |  Speech
asm | 128.6 | 162.9 | 0.6
ben | 6,398.8 | 4,132.5 | 73.0
brx | 1.3 | 0.1 | -
doi | 0.03 | 0.03 | -
eng | 12,190.9 | 542.9 | 26.0
gom | 9.3 | 0.7 | -
guj | 2,370.2 | 1,256.6 | 21.0
hin | 9,312.5 | 2,435.7 | 869.0
kan | 1,415.1 | 350.2 | 12.9
kas | 0.3 | 0.06 | 0.04
mai | 14.5 | 0.06 | -
mal | 2,539.5 | 78.8 | 112.4
mar | 2,415.5 | 397.2 | 14.2
mni | 0.9 | 5.8 | 0.6
npi | 1,809.0 | 13.3 | -
ori | 653.3 | 523.3 | 0.4
pan | 863.1 | 211.6 | 0.5
san | 54.3 | 1,274.6 | -
sat | 0.06 | - | 0.2
snd | 257.9 | 0.23 | -
tam | 3,345.3 | 611.1 | 28.6
tel | 2,934.8 | 753.3 | 18.5
urd | 1,836.5 | 1,798.4 | 23.1
Total | 48,552.8 | 14,550.0 | 1,203.2
Table 2: Number of tokens (in Millions) in each component of Sangraha
Verified.

####  3.1.1 Web Data

Our web data, constituting most of Sangraha, diverges from traditional Common
Crawl-based approaches by prioritizing data quality. This involves manual
verification of each website before scraping. We adopt a three-fold strategy
to collect a comprehensive collection of verified websites for scraping.
Firstly, we extend the efforts of Kakwani et al. (2020) and Doddapaneni et al.
(2023) of discovering web sources using existing news repositories and
automated web searches using popular keywords to discover a large list of
Indic language websites. But, unlike the previous efforts, we do not restrict
ourselves to just news websites. Secondly, we identify various domains such as
Indian Culture, Food, Health, Travel, among others and enlist volunteers to
gather websites within these domains, prioritizing those in Indic languages or
in English but pertaining to Indian context. Thirdly, we collect base URLs
from mC4 (Xue et al., 2021), focusing on websites with high amount of content,
and get them verified by volunteers. Additionally, we also include all the
Indian Government websites888https://igod.gov.in/, which serves as a valuable
resource, given their multilingual content.

Volunteers review all the websites collected via automated methods and decide
on acceptance or rejection based on the defined criteria. A website can be
rejected if either of the below conditions were met:

  * •

Website is non-Indic or non-English.

  * •

Website is an adult, gambling, or a general toxic website.

  * •

Website has content that clearly appears to be machine-translated.

(a) Accepted Website statistics - Domain information  (b) Rejected Website
statistics - Rejection Reason

Figure 2:

Figure 2 presents the verification outcomes, highlighting a significant
rejection rate due to website inactivity, particularly those sourced from mC4.
This means that the information in existing collections is becoming outdated
because of defunct websites. We make available both the verification portal
and the list of validated URLs for further research utilization. We then used
the open-source webcorpus999https://github.com/AI4Bharat/webcorpus toolkit to
crawl the verified websites.

####  3.1.2 PDF Data

Given that a lot of Indic language content is locked in various digitized PDF
documents, we focus on text extraction from them using high-quality OCR
systems. We employ GCP’s Vision Tool for performing OCR as it is known to give
good performance across different categories (Dilmegani, 2023). We source the
PDFs from 7 broad sources as shown in Table 3.

PDF Sources | #PDFs | #Pages
---|---|---
Internet Archive | 437,225 | 74M
eGyanKosh | 5,133 | 88K
Indian Parliament | 30,964 | 2.7M
AIR News | 74,353 | 148K
Govt. Magazines | 895 | 46K
School Books | 4,315 | 359K
Miscellaneous | 27,988 | 4.6M
Total | 507,419 | 82M
Table 3: Sangraha PDF sources - The final statistic of the PDFs on which OCR
has been performed.

Internet Archive

Utilizing the official API of the Internet
Archive101010https://archive.org/developers/internetarchive/, we collected
approximately 921K PDF documents across all Indic languages. This collection
spans diverse categories such as religious texts, news articles, fiction,
educational materials, and scientific literature. We subsequently filtered out
PDFs incompatible with GCP
Vision111111https://cloud.google.com/vision/docs/languages, specifically
excluding PDFs in languages like Bodo, Dogri, Kashmiri, Konkani, Maithili,
Manipuri, and Sindhi, with plans for future inclusion.

To optimize for quality and manage costs, OCR was performed only on high-
quality PDFs. We first remove all the corrupted and encrypted PDFs.
Additionally, resource limitations from GCP Vision necessitated the filtering
of PDFs exceeding 2000 pages. We also filter out all PDFs having less than 25
pages as these are often incoherent documents such as glossaries, comics,
bills, and receipts. This was followed by filtering out scanned PDFs with a
Pixel Per Inch (PPI) value below 300 to filter out blurry PDFs. Additionally,
we analyzed images from 10 consecutive pages of each PDF, measuring the
average area covered by images and their brightness. Pages with images were
considered for further analysis if they covered less than 50% of the page area
and had a brightness level above 200. Table 13 shows the statistics of PDFs
filtered after each filtering stage.

eGyanKosh

eGyanKosh, India’s National Digital Repository, is a repository for digital
learning resources from Open and Distance Learning Institutions. We collect
PDF documents covering various subjects, such as History, Economics, Political
Science, Public Administration, and Sociology, across various Indian
languages.

Indian Parliament

This source comprises manually compiled summaries of debates and discussions
from the Indian Parliament and various State Legislative Assemblies. These
form a rich source of local and culturally relevant data. We collect all the
publicly available Parliamentary and State Assembly materials. Table 14 shows
the state-wise statistics of the collected documents.

AIR News

All India Radio (AIR) is the national radio broadcaster of India, a Prasar
Bharati division, that streams radio programs in all major Indian languages.
Following the approach of (Bhogale et al., 2022), we download all news
bulletins for 12 Indian languages. Table 15 shows the language level
statistics of the collected data.

Government Magazines

We aggregated content from magazines published by various governmental
agencies, which include annual reports, details on governmental schemes,
initiatives, cabinet decisions, and current affairs, that are published in
multiple Indian languages.

School Textbooks

This set includes publicly available textbooks from various Indian states and
those published by the National Council of Educational Research and Training
(NCERT), providing a rich source of educational content in multiple Indian
languages. Table 16 shows the statistics of the books collected from different
states.

In addition to the above categorized sources, we also incorporated a variety
of documents from government and public domains, focusing on content either in
Indic languages or in English with relevance to India. Our future work will
continue to explore digitization and OCR of new public sources.

####  3.1.3 Speech Data

Similar to PDFs, a bulk of language data is present in audio forms either in
videos, podcasts, radio broadcasts, etc. This data captures the most natural
way of human interactions in the form of conversations. We collect a variety
of both manually as well as automatically transcribed sources, covering a
broad variety of content, as shown in Table 4.

Source | Number of Instances
---|---
YouTube - Hindi | 276K videos
Open Subtitles | 14K movies
NPTEL - Transcripts | 1.4K courses
Mann Ki Baat | 1.4K podcasts
Others | 15K
Total | 309K
Table 4: Statistics of the various sources of Speech Data collected

Youtube - Hindi

Following the approach of Anonymous (2024), we collect around 80K hours of
audio data from YouTube videos in Hindi language. We then chunk it into
smaller segments by detecting silences using WebRTC
VAD121212https://github.com/wiseman/py-webrtcvad and get each chunk
transcribed using the Hindi Conformer model. We then piece together the
transcripts of each individual chunk to get the transcript of the whole video.

OpenSubtitles

Following Gao et al. (2021), we collect all the Indic Language subtitles from
OpenSubtitles131313https://www.opensubtitles.org/. We first process the SRT
files using simple regex-based patterns to remove the timestamps and extract
the text. We then define regex patterns to filter out other noisy content like
character cues, continuation ellipses, etc. We then combine the different
parts to form a single document per SRT file. Table 17 shows the language-wise
statistics of Subtitles.

NPTEL - Transcripts

The National Programme on Technology Enhanced Learning
(NPTEL)141414https://nptel.ac.in/ is an Indian e-learning platform for
university-level science, technology, engineering, and mathematics subjects
that is jointly developed by various Indian Institutes. Although the course
content developed by NPTEL is primarily in English, much of it has been
manually transcribed and translated into 11 different Indian Languages and
reviewed before being made publicly available. The translated content has been
compiled and released as course textbooks. Table 18 shows the statistics of
the course transcripts available in different languages.

Mann Ki Baat

Mann Ki Baat is an Indian Radio programme hosted by the Indian Prime Minister,
usually with a frequency of 1 programme per month. This is transcribed and
then manually translated into 13 Indian languages. Table 19 shows the
language-wise statistics.

###  3.2 Sangraha Synthetic

There is a huge disparity between the information-rich digital content and
knowledge available in English compared to Indian languages. To address this
disparity, we introduce Sangraha Synthetic, an initiative to democratize
access to knowledge by translating a knowledge-rich English corpus into Indian
languages. Utilizing IndicTrans2 (Gala et al., 2023), we translated the
entirety of English Wikimedia into 14 Indian languages resulting in nearly 90B
tokens. Since IndicTrans2 operates at the sentence level and does not retain
the document level formatting such as newlines, markdowns and other
structures, we developed the Setu-Translate pipeline, described in Section
4.5. This pipeline facilitates the translation of documents and conversations
while preserving the original document structure.

Recognizing the prevalent trend of “Romanized” Indic language usage,
particularly in informal settings and in digital communication, we extend
Husain et al. (2024) and transliterate the above-translated content in 14
languages to Roman script using IndicXlit (Madhani et al., 2023b) resulting in
about 72B tokens. Going forward, we will extend Sangraha Synthetic to cover
all the 22 scheduled languages of India as well as translate other knowledge-
rich sources.

###  3.3 Sangraha Unverified

We introduce the Sangraha Unverified split to expand the Sangraha corpus while
ensuring high quality. We employ a perplexity filtering pipeline, inspired by
CCNet (Wenzek et al., 2020), to collect all the high-quality tagged documents
from CulturaX (Nguyen et al., 2023a) and MADLAD-400 (Kudugunta et al., 2023).
We consider CulturaX and MADLAD-400 as these represent the latest and most
comprehensive multilingual collections of Web data.

We first randomly sample 200,000 documents from the Sangraha Verified split
for each language. We then normalize each document by converting text to
lowercase, removing accents from characters, normalizing numbers to a uniform
representation (specifically converting all digits to ”0”), replacing a
predefined set of Unicode punctuations with their ASCII counterparts, and
removing non-printing characters. We then train a sentencepiece tokenizer and
tokenize all of the sampled data. Then, we train a 5-gram Kneser-Ney model
using the KenLM (Heafield, 2011) library. We binarize these models for quicker
inference.

For deciding the language-specific thresholds, we create a validation set by
sampling another 100,000 documents from Sangraha Verified and calculate the
perplexity of each document using the trained n-gram models. We then sort the
perplexities and choose the 80th percentile value as the threshold for each
language. Table 5 shows the thresholds chosen for each language. Higher
percentile thresholds can be chosen to prefer more quality over volume, but
that may result in reduced diversity and representativeness of the resultant
data.

Lang |  | Min
---
Perplexity
| Max
---
Perplexity
| Mean
---
Perplexity
| Perplexity
---
Threshold
| Total
---
Docs
| Chosen
---
Docs
| Filtering
---
Rate
asm | 27.4 | 65155.6 | 1013.9 | 1216 | 25,617 | 18,713 | 26.9%
ben | 6.7 | 22941.5 | 286.6 | 606.7 | 6,838,196 | 6,274,727 | 8.24%
guj | 7.8 | 23184.4 | 421.7 | 792.5 | 640,843 | 586,977 | 8.4%
hin | 5.7 | 160264.7 | 230.44 | 378.8 | 19,362,407 | 17,271,194 | 10.8%
kan | 8.6 | 25413.1 | 74.5 | 103.4 | 748,914 | 623,662 | 9.1%
mal | 5.6 | 43419.9 | 65.8 | 61.4 | 1,723,524 | 1,012,425 | 41.25%
mar | 8.3 | 16032.2 | 214.2 | 277.8 | 1,322,324 | 1,051,722 | 20.4%
nep | 7.1 | 20334.8 | 140.0 | 120.32 | 1,625,754 | 961,637 | 40.84%
ori | 5.8 | 166,311 | 160.0 | 170.8 | 61,692 | 44,298 | 28.1%
pan | 8.0 | 23375.0 | 232.6 | 229.7 | 302,421 | 195,115 | 35.48%
san | 32.8 | 5919.0 | 823.8 | 1397.7 | 3,332 | 2,993 | 10.17%
tam | 6.2 | 22583.3 | 157.6 | 262.3 | 2,416,008 | 2,089,674 | 13.5%
tel | 12.6 | 65297.8 | 139.3 | 377 | 930,407 | 898,991 | 3.37%
urd | 2.4 | 25206.5 | 158.4 | 316.8 | 1,502,769 | 1,372,703 | 8.65%
Table 5: Perplexity Statistics of CulturaX and MADLAD-400 datasets. Perplexity
is calculated using n-gram language models trained on data sampled from
Sangraha Verified.

We clean the entire CulturaX and MADLAD-400 corpora using Setu and deduplicate
it with the entire Sangraha Verified split. Finally, we calculate the
perplexities of each document and filter out those that are above the chosen
threshold. Table 5 shows the final number of documents chosen after perplexity
based filtering.

##  4 Setu: A Comprehensive Pipeline for Data Synthesis, Cleaning, Filtering,
and Deduplication

Figure 3: Overview of Setu, the data cleaning pipeline used for curating the
Sangraha Verified corpus

To clean, filter, and deduplicate Web, PDF, and Speech data, we create Setu, a
pipeline built on Apache Spark (Zaharia et al., 2016) which broadly has 4
stages, as shown in Figure 3 \- Document Preparation, Cleaning and Analysis,
Flagging and Filtering, and Deduplication. The Document Preparation stage
focuses on extracting the text from our diverse sources and creating text
documents for further processing. The Cleaning and Analysis stage performs
cleaning on each document to reduce the noise, performs Language
Identification by using an ensemble of different models, and then computes
various statistical signals for each document. In the Flagging and Filtering
stage, we apply various filters based on previously computed signals to filter
out the noisy documents. Finally, the Deduplication stage performs fuzzy
deduplication using MinHashLSH. We also introduce the Setu-Translate and the
Setu-Transliterate pipelines for performing large-scale structure-preserving
translations and transliterations of documents and conversations. We discuss
the details of these pipelines in this section.

###  4.1 Document Preparation

This stage focuses on the extraction of text from varied data sources,
ensuring the retention of main content while eliminating extraneous
information and then preparing the notion of a document that is preserved
throughout the pipeline. Due to the different modalities of content, this
stage is different for each of Web, PDF, and Speech data.

####  4.1.1 Web Data

Preparation of the document for Web data is quite straightforward. We use
trafilatura (Barbaresi, 2021) to extract the text from the HTML pages that are
scraped by webcorpus scraper. Although trafilatura is reportedly the best non-
commercial library (Scrapinghub, 2021), we still notice a considerable amount
of noise in the outputs, specifically in dynamic webpages. Figure 4 shows an
example of noisy content extracted using trafilatura. In Web data, each
webpage after text extraction is considered as a document.

Figure 4: Example showing noisy content being extracted from the HTML using
trafilatura

####  4.1.2 PDF Data

Text Extraction from the OCR outputs from PDFs is not as straightforward as
extracting text from a webpage. When utilizing Google Vision OCR for
extracting text from PDF documents, the output is a structured JSON file that
contains detailed information about the detected text. This information is
organized hierarchically from larger text blocks down to individual
characters. This hierarchical structure allows for a nuanced understanding of
the document’s layout and content. Broadly the bounding boxes are organized in
the following hierarchy - Block, Paragraph, Word, Character.

A block is the highest level of structure and is a container for paragraphs
grouped to reflect their spatial relationships. Paragraphs are subdivisions of
blocks and represent cohesive units of text, typically separated from other
units by new lines or indentation. Words are the basic units of text and
meaning within a paragraph. Each word is identified and extracted as a
separate entity in the OCR output. Characters are the most granular level of
text extraction, representing individual letters, numbers, punctuation marks,
and other textual symbols.

Each category contains information such as the bounding box coordinates,
confidence scores, language scores, and the text identified in that box. We
observe that directly consuming the text from the OCR is not good as it
contains a lot of noise coming in due to incorrect layout parsing. We also
observed that due to the skewness and quality of images, we had multiple
instances where we had bounding box overlaps, bounding box
mismatch/misalignment, text overlaps, and language script mismatches. To
resolve these and extract the highest quality text, we develop bounding-box
based filters. We list the filters below:

  * •

Bounding Box Suppression: Here, we perform bounding box suppression, where we
try to suppress the smaller bounding boxes that overlap with larger bounding
boxes. For each pair of overlapping bounding boxes, we calculate the ratio of
the area of intersection over the area of the smaller bounding box. We
suppress the smaller bounding box if this ratio exceeds a chosen threshold.
Figure 5a shows an example of a page where bounding box suppression is
applied.

  * •

Removing Horizontally sparse pages: Here, we identify and remove pages that
exhibit a significant lack of content across the horizontal span of the page.
If a page has large horizontal gaps with little to no content—indicating that
the text or visual elements are spread thinly across the width of the page—it
is considered horizontally sparse. Such pages are often less informative or
relevant, like index pages and table of contents among others. Figure 5b shows
an example of a page flagged as horizontally sparse.

  * •

Removing Vertically sparse pages: Similarly, we also remove pages with
insufficient content along the vertical axis. Pages containing large vertical
gaps, such as excessive spacing between paragraphs or sections without
meaningful content, are deemed vertically sparse. These pages are also less
informative, like pages having publisher information, colophons, comic strips,
etc. Figure 5c shows an example of a page flagged as vertically sparse.

  * •

Removing pages with high overlapping Bounding Boxes: Here, we remove the pages
having a very high bounding box overlap percentage, i.e., greater than a
chosen threshold as shown in Figure 5d.

  * •

Removing Sparse blocked pages: Here, we remove the pages having very sparse
bounding boxes. A block bounding box is considered sparse if the difference
between the total area of the block bounding box and the total area of
paragraph bounding boxes enclosed in it is greater than a chosen threshold. By
this, we remove pages with tables, large images, and forms among others.

  * •

Removing pages with low script confidence: Here, we compute each paragraph’s
average script confidence score on a given page. Paragraphs with scores below
our confidence thresholds are flagged for potential exclusion. Subsequently,
the entire page is discarded if the number of flagged paragraphs exceeds an
allowable limit. This ensures a balance between rejecting poor-quality OCR
output and retaining usable content.

(a) Bounding Box Suppression: Page in which smaller bounding boxes are
suppressed as these can lead to false flagging of pages or misaligned text.

(b) Horizontally Sparse: Page filtered out due to less horizontal text
coverage, this can be indicative of very small lines, lists, index etc.

(c) Vertically Sparse: Page filtered out due to less vertical text coverage.
This can be indicative of title pages, comics, etc.

(d) High Bounding Box Overlap: Page filtered out due to high bounding box
overlap. This high overlapping can lead to disordered parsing of text, break
in continuity, etc.

Figure 5: Illustrative examples of pages flagged in various bounding box
filters.

After filtering, we merge the final text extracted from the pages to form
documents. To maintain textual continuity as well as to get as many long-form
documents as possible, we concatenate the text of only consecutive batches of
pages of a given PDF together. Table 20 shows the average number of pages per
language that are merged to form a document.

####  4.1.3 Speech Data

For speech data, we currently only handle data from SRT files and the
transcripts obtained using Automatic Speech Recognition (ASR) models. For SRT
files, we first extract the text by removing the timestamps. We further define
simple regex patterns to filter out other noisy content like character cues,
continuation ellipses, etc. A single processed SRT file is considered as a
document.

The automated transcriptions obtained using ASR models are first pieced
together to form larger chunks of text. We then use IndicPunct (Gupta et al.,
2022) punctuation models to add appropriate punctuation marks in the
transcripts. Finally, all chunks belonging to a single video are merged to
form a document.

###  4.2 Cleaning and Analysis

This stage primarily focuses on performing in-document cleaning and language
identification. Additionally, we compute various statistics for performing
analysis and further filtering. We divide this stage into three sub-stages -
Document Cleaning, Language Identification, and Analysis.

####  4.2.1 Document Cleaning

Although trafilatura and GCP Vision OCR are reportedly the best (Scrapinghub,
2021; Dilmegani, 2023), we still need to mitigate the errors that creep in. We
define the below filters that clean a document.

  * •

Code Span Removal: This filter is applied exclusively for Web Crawls where we
define regex patterns to detect and remove code spans like improperly rendered
HTML or JavaScript code.

  * •

Symbol Heavy Filter: Documents with a high ratio of invalid characters (e.g.,
punctuation, emojis, and other symbols) to total characters exceeding a
predefined threshold are discarded. Figure 6 shows an example of a symbol-
heavy document.

  * •

Terminal Punctuation Filter: This filter is again exclusively for web crawls,
it removes text segments lacking valid terminal punctuation, effectively
filtering out clickbait text, menus, and incomplete sentences. Figure 7 shows
an example of content removed using this filter.

  * •

Symbol Only Chunk Filter: This filter removes all the text chunks with only
numbers or symbols.

  * •

Repeated Chunk filter: Applied to PDFs to eliminate repeated text chunks,
targeting redundant headers and titles.

  * •

Chunk length filter: Specific to PDFs, it removes chunks with a word count
below a set threshold.

Figure 6: Document flagged by symbol heavy filter in the Cleaning and Analysis
stage. Figure 7: Cleaning performed by ‘terminal punctuation filter’ in
Cleaning and Analysis stage.

####  4.2.2 Language Identification

To address the issues of accuracy that may occur while relying on a singular
model highlighted in Appendix A, we use an ensemble approach using three LID
models - IndicLID (Madhani et al., 2023a),
cld3151515https://github.com/google/cld3, NLLB (Costa-jussà et al., 2022).
Notably, IndicLID, which is specifically trained for Indic languages, is
assigned a preferential weighting in our ensemble framework. However, if both
cld3 and NLLB agree on a different language and are very confident about it
(beyond a chosen threshold), we consider their prediction instead. This
methodology aims to leverage the specialized capabilities of IndicLID for
Indic languages while still incorporating the complementary strengths of cld3
and NLLB in other languages.

####  4.2.3 Document Analysis

We compute various document-specific statistics for performing subsequent
filtering. The metrics and their descriptions are outlined in Table 6.

Metrics |  Description
---|---
bytes |  size of the document interms of bytes,
word_count |  no.of words present in a document
char_count |  no.of characters present in a document
lines_count |  total no.of sentences present in a document
mean_line_length |  mean sentence length interms of words of a document
min_line_length |  minimum sentence length interms of words of a document.
max_line_length |  max sentence length interms of words of a document
nsfw_words_count |  no.of NSFW words present in a document
non_li_character_count |  no.of non-latin/non-indic characters in a document
10_gram_characters_repetition_score |  score used for filtering documents using 10-gram character repetition filter
5_gram_words_repetition_score |  score used for filtering documents using 5-gram word repetition filter
Table 6: Showing all the metrics that are calculated in analysis stage

###  4.3 Flagging and Filtering

Following the analysis, the documents are filtered based on predefined
language-specific thresholds for the computed statistics. This step is
essential to eliminate residual noise that might have survived the initial
cleaning process. We include filters inspired from various previous works like
Roots (Laurençon et al., 2023), Gopher (Rae et al., 2022) and C4 (Raffel et
al., 2020) among a few.

  * •

NSFW word ratio filter: In an effort to reduce corpus toxicity, documents with
a high ratio of NSFW (Not Safe For Work) words to total words are excluded.
This approach aligns with that of IndicCorp v2, involving the development of
an NSFW word list specifically tailored for Indic languages. This list is made
available to the research community to encourage further studies.

  * •

Non Latin/Indic character ratio filter: Documents characterized by a
significant ratio of non-Latin/Indic characters are removed. This filter
eliminates content erroneously classified as Indic by the Language
Identification (LID) stage. Figure 9 shows an example of the type of content
removed by this filter.

  * •

Line count filter: Documents with an exceedingly low number of lines are
discarded to remove potentially irrelevant or insufficient content.

  * •

Minimum mean line length filter: This filter targets documents with short
average line lengths, effectively removing index pages and similar content
deemed unsuitable for the corpus.

  * •

5-gram word repetition: Inspired from Roots, we create a filter for the
repetitions by looking at the occurrences of the 5-gram word sequences. We
define the word repetition ratio as the ratio of the sum of the occurrences
greater than or equal to the sum of all occurrences, and we discard documents
with too high a ratio.

  * •

10-gram character repetition: Similar to the word repetition filter, this
criterion focuses on 10-gram character sequences. Documents exhibiting a high
ratio of such repetitions are excluded, based on methodology inspired by
Roots.

Figure 8: Figure showing the type of content flagged by the Non Latin/Indic
Filter. Ideally, these types of documents should not be rejected since they
contain valid math characters. These are some of the limitations of our
current pipeline.

Figure 9: Figure showing the type of content flagged by the Line count filter.

###  4.4 Deduplication

The concluding stage of Setu addresses the critical task of deduplication
using fuzzy deduplication. Following CulturaX, we use the Python
implementation of MinHashLSH from the text-
dedup161616https://github.com/ChenghaoMou/text-dedup/tree/main repository. We
efficiently identify and remove duplicate documents within the corpus by
utilizing 5-grams and a similarity threshold of 0.7, based on Jaccard
similarity. This procedure is executed separately for each language, utilizing
a computing node with 256 CPUs.

###  4.5 Setu-Translate

Majority of the machine translation systems are trained as sentence-level
translators, which often struggle to preserve various entities like inter-
sentence separators, new-line characters, tab spaces, markdowns, bullet
points, etc. Simple sentence-tokenizers present in the packages like NLTK
(Loper & Bird, 2002) and IndicNLP Library (Kunchukuttan, 2020) are not capable
of retaining these inter-sentence separators and markdowns. We introduce Setu-
Translate, a robust translation pipeline for mass-translation of both pre-
training and Instruction fine-tuning data while preserving the structure of
the document and the conversation. Overall, Setu-Translate focuses on
accurately identifying the parts of the document that must be sent to the
translation model and then replacing the translated sentences in the overall
document, thereby preserving the overall structure of the translated document.
The three main stages of Setu-Translate are described in this section.

Templating

Using regex patterns, we identify the parts of the documents we intend to
translate. The goal of this stage is to preserve the structure of the
document. The regex patterns defined ignore markdown structures, code snippets
(enclosed in backticks), bullet points, paragraph indicators, Roman numerals,
etc., and extract only the sentences. After performing unicode-normalization
and deduplication on the extracted sentences, a global sentence-level dataset
is created.

Inference

We binarize the data first and then utilize IndicTrans2 (Gala et al., 2023)
for translating English into Indic languages. We leverage both GPUs and TPUs
for large-scale translation. To benefit the community, we open-source the flax
port for IndicTrans2 model for TPU inferencing.

Replace

Once we have the translated sentences, we perform a regex-based replacement of
the original sentences with the translated ones. This ensures that only
sentences are replaced and the other structure of the document is retained as
is.

###  4.6 Setu-Transliterate

Similar to translation, we also release the Setu-Transliterate pipeline. Since
transliteration is done at a word level and doesn’t consider the context of
the remaining words, we follow the regular word replacement strategy. We
maintain a continuously updating mapping of Indic words to their Roman
counterparts in a prefix-based hierarchical format, which we feel is the key
to speedup and rapid access to the required word pairs.

Word Mapping Dictionary

For the creation of the initial mapping, we use Aksharantar(Madhani et al.,
2023b) dataset, which is the largest publicly available transliteration
dataset for Indic languages, as the starting point. We convert Aksharantar
into the said prefix-based hierarchical format. This mapping is continuously
updated with new mappings as we discover new un-romanized words further in our
pipeline.

Word Replacement

Word-level replacement has 2 main challenges: (i) identifying words to replace
while preserving the entire document structure; and (ii) unordered replacement
leading to sub-word replacement instead of the entire word. We address (i)
using the same regex-based approach used in Setu-Translate. To address (ii),
we sort the mapping based on source-language word length in descending order
before feeding the mapping to the regex-based ‘replace’ module.

Inference

During the first ‘replace’ pass, we log the un-romanized words, the words
whose mapping is not available in the current word mapping dictionary. In the
‘inference’ stage, we transliterate these words using IndicXlit(Madhani et
al., 2023b) to get an updated word-mapping dictionary. We then repeat the word
replacement until all the words are properly romanized.

##  5 IndicAlign

In this section, we describe the composition and the curation process of
IndicAlign, comprising of around 74.7 million diverse, human, and synthetic
prompt response pairs. Majority of the high-quality synthetic supervised fine-
tuning data released has been created with proprietary models like ChatGPT and
GPT-4, which renders them unusable in commercial settings. We therefore
consider only license-friendly datasets and models for curating IndicAlign for
different Indian languages. Further, we use the Setu-Translate and Setu-
Transliterate pipelines discussed in Section 4 for translating and
transliterating the conversations, thereby maintaining the structure and the
markdown of the responses. IndicAlign comprises two distinct splits:
IndicAlign - Instruct and IndicAlign - Toxic as shown in Table 7.

Component |  | Prompt
---
source
| Response
---
source
| Original/
---
Translated
#Examples |  | Avg.
---
Turns
| Avg.
---
Inst. Len
| Avg.
---
Out. Len
#Lang. |  | Lexical
---
Diversity
Indic ShareLlama | H | M | T | 21.1k | 1 | 60.45 | 267.98 | 15 | 57.69
Dolly-T | H | H | T | 15.0k | 1 | 12.34 | 59.38 | 15 | 47.23
OpenAssistant-T | H | H | T | 19.9k | 2.98 | 25.72 | 136.37 | 15 | 59.75
WikiHow | H | H | T | 20.3k | 1 | 43.85 | 327.95 | 15 | 23.87
IndoWordNet | H | H | O | 74,272.2k | 1 | 19.74 | 14.84 | 18 | 37.24
Anudesh | H | M | T | 43.3k | 1.58 | 12.4 | 149.28 | 20 | 51.69
Wiki-Conv | M | M | T | 144k | 9.14 | 7.09 | 11.22 | 15 | 23.17
Wiki-Chat | M | M | T | 202k | 2.8 | 23 | 227.75 | 15 | 56.67
HH-RLHF-T | H | M | T | 32.6k | 1 | 14.11 | 64.88 | 15 | 79
Toxic Matrix | M | M | T | 90.3k | 1 | 33.68 | 89.64 | 15 | 86.57
Table 7: Overall statistics of IndicAlign. Dolly-T represents Dolly
Translated, OpenAssistant-T represents OpenAssistant Translated. Lexical
Diversity is computed averaging the MTLD score over each utterance.
Interaction lengths are reported in number of words.

###  5.1 IndicAlign - Instruct

The IndicAlign - Instruct split encompasses datasets that can be used to
imbibe instruction-following ability in Large Language Models. Firstly we
amalgamate different existing Instruction Fine-tuning (IFT) datasets with
prompts authored by humans and responses generated by humans or open, license-
friendly models. To complement this human-centric approach, which is often too
expensive and time-consuming, we turn to synthetic data generation using
existing chat-aligned models following the works of Ding et al. (2023b), Li et
al. (2023b), and Xu et al. (2023b). We ensure that our outputs are always from
open, license-friendly models and are always grounded in context.

####  5.1.1 Indic-ShareLlama

We collect conversations from
ShareGPT171717https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered,
a platform where users share their interesting conversations with
ChatGPT181818https://chat.openai.com/. We then collect the user prompts from
the first turn of the conversations and prompt Llama2-70B Chat (Touvron et
al., 2023b) model for the responses in Indian contexts. We explicitly exclude
all the non-English, coding, and math-related prompts to get around 21K
conversations and then translate and transliterate them into 14 Indian
languages to form Indic-ShareLlama.

####  5.1.2 Dolly-Translated

Dolly-15K (Conover et al., 2023), introduced by Databricks, is an open-source
conversation dataset aimed at democratizing the capabilities of LLMs. It
consists of 15K high-quality, human generated prompt-response pairs, authored
by around 5000 Databricks employees. Following Gala et al. (2024) and Husain
et al. (2024), we translate and transliterate these conversations into 14
Indian languages to form Dolly-Translated.

####  5.1.3 OpenAssistant-Translated

OpenAssistant Conversations (OASST1) (Köpf et al., 2023), is a collection of
human-generated, human-annotated assistant style conversation corpus
consisting of around 3K conversation trees and around 20K conversations.
Extending Gala et al. (2024); Husain et al. (2024), we release the translated
and transliterated versions in 14 Indian languages as OpenAssistant-
Translated.

####  5.1.4 WikiHow

Wikihow191919https://www.wikihow.com/ is a collaborative online wiki-style
platform that serves as a valuable resource for a diverse array of how-to
guides. It covers various aspects of life, including technology, arts,
entertainment, home and garden, health, and more. Each piece is typically
structured with step-by-step instructions, supplemented by illustrations and
videos, to help readers achieve their goals. The questions users pose in these
articles closely align with potential use cases for any model, making it a
rich training resource. Gala et al. (2024) curate around 20,400 and 6000
instruction-answer pairs in English and Hindi. The data is formulated as a
completion task given either a question or a question along with a few initial
steps. We extend their efforts and translate and transliterate the English
conversations into 14 Indian languages.

####  5.1.5 IndoWordNet

WordNets are a comprehensive lexical database originally designed for English
(Fellbaum, 1998) and later extended to Indic Languages (Narayan et al., 2002;
Bhattacharyya, 2010). It organizes words into sets of synonyms called synsets,
providing short definitions and usage examples. Beyond mere dictionaries,
WordNet also captures the various semantic relationships between words. We
leverage this rich semantic information to create instruction fine-tuning data
to teach the model grammar and language creativity.

We first identify a list of 21 potential intents encompassing tasks such as
Part of Speech identification, sentence construction, and synonym discovery.
We craft 5 prompt-response templates for each intent, resulting in a
repository of 105 distinct templates. Then we iterate through the lexicon in
IndoWordNet using pyiwn (Panjwani et al., 2018), randomly sampling 100
templates for each word yielding around 74M pairs for 18 Indic languages.
Figure 10 shows some examples of templates. Table 21 shows each language’s
final statistics of the prompt-answer pairs.

Figure 10: Example prompt templates for three sample intents in the
IndoWordNet Instruction fine-tuning data.

####  5.1.6 Anudesh

Here, we introduce a novel dataset of real user interactions with
conversational models, leveraging open, license-compatible models such as
Llama2-70B Chat Touvron et al. (2023b). Recognizing the limitations imposed by
OpenAI’s terms of use202020https://openai.com/policies/terms-of-use on
existing crowd-sourced model interaction datasets, such as ShareGPT and
WildChat (Zhao et al., 2024), our dataset aims to provide a resource, free
from such constraints, thereby facilitating broader applicability in training
diverse conversational models.

We create Anudesh by asking the user to interact with the model while
following an instruction displayed on the screen. Occasionally, we allow
unrestricted interactions to collect more diverse and creative prompts. Each
displayed instruction is based on three axes that guide the user -

  * •

Intent \- Defines the purpose and goal behind the interaction, such as
summarization, recommendation seeking, etc.

  * •

Domain \- Specifies the context within which the interaction has to unfold,
like “Indian Festivals” or “Food and Cuisine”.

  * •

Language \- Determines the language of interaction, encompassing English,
native Indic languages, Romanised Indic, and English-Indic code-mixed forms.

Intent |  Domain
---|---
Information seeking |  Education and Academia, Science, Technology, History, Humanities, etc.
Detailed Topic Exploration |  Environmental Studies, Economics, Finance, Arts and Culture, Travel, Geography, etc.
Seeking Clarification |  Information Technology, Mathematics, Language and Linguistics, Physics, Chemistry, History etc.
Personal well-being |  Fitness and Nutrition, Mental Health, Lifestyle, Self-improvement, Relationships and Family, Spirituality, etc.
Seeking recommendations |  Home and Garden, Personal Finance, Healthcare, Work and Career, Education, etc.
Summarizing something |  Movies and Entertainment, Books, Politics, Current affairs, Science and Technology, Travel and Adventure, etc.
Table 8: Sample Intent and Corresponding Domains

Table 8 shows some examples of the Intents and Domains. Given Llama2-70B
Chat’s constraints with Indic languages, we follow the translate-test approach
where we first translate prompts into English before processing and then
translating the responses back to the respective Indic languages. Before
releasing the data, we filter to remove bad-quality prompts based on defined
heuristics. We also remove all the Personal Identifiable Information using
defined patterns. We discuss further user demographic and procedure details in
the Appendix B.

####  5.1.7 Wiki-Conv

We create Wiki-Conv, a synthetic dataset created by prompting a model to
generate an entire conversation spanning multiple turns between a user and an
assistant. We first collect all the “India-centric” English Wikipedia articles
using Wiki Export212121https://en.wikipedia.org/wiki/Special:Export and
Wikimedia API MediaWiki (2023). We then chunk the articles to create context
passages of around 1000 words. We also collect all the “India-centric”
WikiInfoboxes using wptools222222https://github.com/siznax/wptools. An Infobox
is a fixed-format table added to Wikipedia articles that summarizes important
facts, statistics, and important points in an easy-to-read format. We prompt
Llama2-70B Chat (Touvron et al., 2023b) to generate an entire conversation in
a user-assistant format using either the Wiki passage or a WikiInfobox as a
context. As shown in Table 7, these conversations span multiple turns but are
more focused on shorter and to-the-point answers. We perform filtering on the
generated conversations to remove noisy conversations and translate and
transliterate them to 14 Indian languages to form Wiki-Conv. Figure 28 shows
the prompt template used to generate this data.

####  5.1.8 Wiki-Chat

Figure 11: Overview of the Wiki-Chat pipeline. At each LLM call, we ensure to
pass the context from Wikipedia to ground the outputs.

To enhance the collection of open-generation conversations, we follow the
approaches tried out by UltraChat (Ding et al., 2023a), Camel (Li et al.,
2023b), and others of simulating interactions between two models.
Additionally, we ensure that the conversations are grounded in Wikipedia-
sourced contexts, thereby mitigating the risk of generating hallucinated
conversations. We show the overview of the entire pipeline in Figure 11.

Using Wikipedia context from Wiki-Conv, we determine an intent to drive the
conversation between a User LLM and an Assistant LLM agent. We use Llama2-70B
Chat (Touvron et al., 2023b) and Mixtral-8x7B-v0.1 (Jiang et al., 2024b) to
simulate the conversations, which are then translated and transliterated to 14
Indian languages forming Wiki-Chat. This simulation broadly involves four
different LLM agents:

  * •

Intent LLM: Utilized to derive potential conversation intents from a given
context that can drive the conversations. Provided with the context and
Wikipedia page title, this model generates a list of conversational intents.

  * •

Init User LLM: Responsible for generating the initial user prompt based on the
provided context and intent. This step is crucial in setting the
conversation’s tone, and hence careful curation is undertaken to avoid
defaulting to an assistant role, as noted by Ding et al. (2023a).

  * •

Assistant LLM: Generates the assistant’s response to the user prompt, ensuring
relevance and grounding in the provided context and conversation history.

  * •

Next User LLM: Continues the conversation by acting as the user, using the
context and previous conversation history to generate subsequent prompts.

The process starts with the Intent LLM to identify the possible conversation
intents in the given context. Following this, the Init User LLM crafts the
initial user prompt, which is then addressed by the Assistant LLM, completing
one conversation turn. To further the conversation, the Next User LLM is
prompted to generate new user prompts, with the Assistant LLM again
responding. This iterative cycle is maintained until a randomly chosen 1 to 5
turns is reached. We show the prompt templates for each LLM agent in Figure
11. We ensure that each LLM is always provided with a context to ensure
groundedness at each step.

Data Cleaning

Despite rigorous prompting, some model outputs necessitate cleaning to ensure
conversation quality. Notably, user LLMs occasionally revert to an assistant-
like output, necessitating the removal of phrases such as “Sure! Here is
something a user may ask …”. Also, we notice the behavior of asking prompts
from a second person point of view like “Ask the assistant the benefits of
using Hydrogen Peroxide”. We make sure to explicitly detect and filter out
these noisy prompts. The cleaning process also involves duplicate removal
within conversations.

Comparison of Llama2-70B Chat and Mixtral-8x7B-v0.1 models

Table 9 shows the statistics of the conversations generated by Llama2-70B Chat
and Mixtral-8x7B-v0.1 models. We observe that conversations generated using
Mixtral-8x7B-v0.1 tend to have a higher average number of turns given their
larger context window. Since we pass the context to the model as part of each
prompt, Llama2-70B Chat fails in conversations involving a higher number of
turns due to the smaller context window. Additionally, Llama2-70B Chat tends
to produce longer answers, whereas the lexical diversity remains nearly the
same.

Model | #Examples | Avg Turns |  | Avg Instruction
---
Length
| Avg Output
---
Length
Lexical Diversity
Llama2-70B Chat | 93K | 2.59 | 24.74 | 280 | 56.89
Mixtral-8x7B-v0.1 | 108K | 2.99 | 21.71 | 189 | 56.51
Table 9: Analysis of conversations generated using Llama2-70B Chat and
Mixtral-8x7B-v0.1

###  5.2 IndicAlign - Toxic

Aligning chat models to responsibly handle toxic prompts is crucial to
developing ethically responsible models. This work presents our initial steps
towards creating datasets to refine model responses to toxic inputs. We use
both human and synthetic data collection strategies and introduce two distinct
datasets: HH-RLHF-Translated, comprising human-curated data, and Toxic Matrix,
a novel toxic alignment dataset created synthetically. Figure 12 summarizes
the entire pipeline used for creating IndicAlign - Toxic.

Figure 12: Flowchart illustrating the creation process of IndicAlign - Toxic.

####  5.2.1 HH-RLHF - Translated

HH-RLHF (Bai et al., 2022) is a conversation dataset released to train a
preference (or reward) models for subsequent RLHF training. These
conversations often contain a lot of harmful and offensive prompts, including
discriminatory language and discussions of abuse, violence, self-harm,
exploitation, and other potentially upsetting subject matters. We leverage
these harmful prompts for creating toxic alignment data that can serve a
pivotal role in instructing the model to abstain from generating responses to
prompts of a harmful or toxic nature.

We first extract the initial user prompts from the dataset. Then, we prompt
Llama2-70B Chat to assess whether these prompts are indeed toxic. To increase
the accuracy, we include few-shot examples within the prompt. In addition to
identifying toxic prompts, we prompt Llama2-70B Chat for explanations
regarding the rationale behind the toxicity flagging. Figure 27 shows the
detailed prompt template. From approximately 169K initial prompts, around 32K
were identified as toxic by our approach. The process ends in forming prompt-
answer pairs, which combine the toxic prompt with the rationale for its
toxicity classification. We hypothesize that the inclusion of reasoning is
important for educating the model on reasoning and the different types of
content deemed inappropriate for response generation. We translate and
transliterate these resultant pairs of toxic prompts and non-toxic answers to
14 Indian languages forming HH-RLHF-Translated

####  5.2.2 Toxic Matrix

We introduce a novel approach to generate toxic alignment data synthetically
using a well-defined taxonomy. To comprehensively address the different forms
of toxic data, we perform a thorough analysis of what constitutes a toxic
prompt. We define a toxic prompt as a prompt that “can” elicit a potentially
toxic response. We note that not all toxic prompts can have a toxic answer.
Figure 13 shows one example where the same prompt has a toxic and a non-toxic
answer. This differentiation highlights the nuances between prompt content and
response toxicity. Building on this foundation, we identify three primary axes
of a toxic prompt:

  * •

Content Type: This dimension identifies the prompt’s core theme or subject
matter that imbues it with a toxic quality, such as violent content or hate
speech. It essentially captures the underlying intent of the toxic prompt.
Various examples of content types are cataloged in Table 10.

  * •

Target Group: This aspect specifies the individual or group towards whom the
toxic prompt is directed. Table 10 lists down a few examples of target groups.

  * •

Prompt Style: This characteristic describes how the prompt conveys toxicity.
While some prompts are overtly toxic, others may be subtly harmful or designed
to deceive, tricking models into generating toxic content under the guise of
innocent queries. Different prompt styles and their descriptions are listed in
Table 11.

Figure 13: Example of a prompt with a toxic and a non-toxic answer. Content Type |  Fraudulent activities, Harassment on Accent, Vaccine Misinformation, Kidnapping, Harassment on Appearance, Ethnic Insults, Suicidal Ideation
---|---
Target Group |  Children with Disabilities, Bengalis, Gujaratis, South Indians, Adolescents, Heterosexuals, Adults (30-49yrs), Sardarjis
Prompt Style |  Direct, Indirect, Misleading, Long Con, Fooling, Provocative, Role-Play, Exploitative, Manipulative
Table 10: Examples for each axis in the Toxic Matrix taxonomy Prompt Style |  Description
---|---
Direct |  Prompts explicitly asking for harmful or unethical content without ambiguity
Indirect |  Prompts that indirectly seek harmful information through references, comparisons or allusions.
Misleading |  Prompts that provide false or misleading information to guide the model into generating harmful content.
Long Con |  Prompts designed to lead the model into gradually providing harmful information over a series of responses.
Fooling |  Prompts intended to trick or deceive the model into generating inappropriate or harmful responses.
Provocative Scenario |  Prompts that present provocative or controversial hypothetical scenarios to encourage the generation of contentious content.
Role-Play |  Prompts that instruct the model to assume a specific role and generate content from that perspective, potentially encouraging the generation of biased or harmful content.
Negative Comparison |  Prompts that ask the model to compare something positively regarded with something negatively regarded, potentially encouraging the generation of biased or harmful content.
Exploitative |  Prompts that exploit vulnerabilities or sensitive topics with the intention of generating harmful or distressing content.
Manipulative Emotional Appeal |  Prompts that use emotional language or appeals to manipulate the model into generating content that may evoke strong negative emotions.
Table 11: Different Prompt Styles in IndicAlign - Toxic

Table 10 shows examples of categories under each of the axes. We then leverage
a combination of a relatively less toxic aligned model - Mistral-7B Chat
(Jiang et al., 2023) \- and a highly toxic aligned model - Llama2-70B Chat
(Touvron et al., 2023b) \- to generate toxic prompt and refusal answer pairs.
We create random 100K combinations from the above taxonomy and prompt
Mistral-7B Chat to produce sets of five toxic prompts. Figure 26 shows the
prompt template used for generating toxic prompts. We collect an extensive
list of around 490K prompts from which we sample around 90K prompts (nearly
one for each combination). We then prompt Llama2-70B Chat to get a response
for that prompt. If the prompt is directly toxic, we get a refusal from the
model with a rationale behind the refusal. Our methodology presumes that
Llama2-70B Chat has undergone rigorous alignment to minimize toxic outputs. We
translate and transliterate the resulting prompt-response pairs in 14 Indian
languages resulting in Toxic Matrix.

Although previous works have shown different ways to distill instruction
following alignment from strong models, we propose this method as one of the
ways to distill toxic alignment using a combination of a weakly and a strongly
toxic-aligned model. This approach, while still under development, offers a
promising direction for improving the ethical alignment of conversational
models. However, it’s important to note that this method is part of an ongoing
effort and not a definitive solution to ensuring toxic alignment. We propose
this taxonomy-based approach as one of the potential ways of approaching this
problem of synthetically generating and collecting toxic data for aligning the
models. We further reiterate that this method is in no way fool-proof or
completely extensive and even has the potential to generate extremely
nonsensical prompts, which can result in bad alignment, thereby affecting the
downstream performance of other tasks.

##  6 Data Analysis

###  6.1 Sangraha

The final statistics of Sangraha are shown in Table 1.

Comparison with other Multilingual Corpora

Figure 14: Comparison of the number of tokens (in Millions) in - IndicCorp v1,
IndicCorp v2 and Sangraha Verified \+ Sangraha Unverified

We compare Sangraha Verified split with other Indic-only corpora - IndicCorp
v1 (Kakwani et al., 2020), IndicCorp v2 (Doddapaneni et al., 2023) and
Wikipedia. Figure 14 shows the distribution of the number of tokens for
different Indic languages. We observe a significant increase in the size of
all languages, especially in the lower resource languages. Overall, Sangraha
Verified contains 64.3B tokens and is 2.6$\times$ bigger than IndicCorp v2. We
show a detailed language-wise comparison in Table 22

Average document length comparison across languages

Figure 15 compares the average document length across various languages in
terms of the number of words. For Web Data, a single webpage is a document,
whereas for PDF data, a batch of consecutive pages is considered a document.
We observe that Dravidian languages, i.e., Tamil, Malayalam, Kannada, and
Telugu, show considerably smaller document lengths, primarily because of the
agglutinative nature of these languages. Agglutination allows for the
construction of complex expressions in single words, potentially affecting the
overall document length by reducing the number of words needed to convey the
information.

Figure 15: Average Document Size for Web and PDF documents in number of words.

How much data gets filtered by Setu?

Figure 16: Percentage drop across the different stages of Setu when cleaned on
Sangraha Verified

We present a comprehensive analysis of the attrition in token count observed
across the various stages of the Setu pipeline in Figure 16. Notably, the
Deduplication stage exhibits the most significant reduction in tokens, which
can be attributed to the fact that a lot of web content for Indic Languages
comprises news articles with similar content distributed across various
platforms.

Uncleanliness of Existing Corpora

We clean the entirety of CulturaX and MADLAD-400 datasets using our Setu
cleaning pipeline and show the drop in the number of words and documents
across the stages. This helps us identify the type of noise present in these
datasets. Figure 18 shows the drop in the number of tokens in these datasets
respectively. We see a significant drop in both from Stage-1 to Stage-2
showing that a lot of noise in the form of Menu Items, Index lists, etc. must
have crept in despite they being cleaned using their existing cleaning
pipelines. We show a few examples of the kind of noisy text being filtered out
in Figure 19. Table 12 shows the overall statistics of the CulturaX data
filtered out at each stage in Setu.

Perplexity Analysis - Sangraha Unverified

Figure 17: Log Perplexity distributions of Cleaned CulturaX and MADLAD-400
using 5-gram language models trained on Sangraha Verified

Figure 17 shows the perplexity distributions of the cleaned CulturaX and
MADLAD-400 data using the n-gram language models trained on Sangraha Verified.
we observe that certain languages, specifically Hindi, Malayalam, and Marathi,
exhibit relatively tight distributions of perplexity values. This indicates a
higher degree of similarity in the statistical properties of these language
datasets to the Sangraha Verified training data. Conversely, we note that some
languages, particularly those classified as low-medium resource, show more
dispersed perplexity distributions.

Figure 18: Number of tokens (in Billions) dropped at each stage in CulturaX and MADLAD-400 when cleaned using Setu. | Stage-1 | Stage-2 | Stage-3
---|---|---|---
Language | Words | Docs | Words | Docs | Words | Docs
asm | 22M | 43K | 16M | 42K | 14M | 33K
ben | 4199M | 11721K | 3812M | 11305K | 3653M | 10099K
brx | - | - | 476 | 29 | 77 | 1
doi | - | - | 11K | 11K | 10922 | 53
eng | - | - | 17M | 70K | 12M | 33K
guj | 524M | 1084K | 462M | 1049K | 460M | 1027K
hin | 10664M | 18740K | 8985M | 17950K | 8897M | 17055K
kan | 436M | 1225K | 403M | 1198K | 366M | 1108K
kas | - | - | 50K | 3440 | 17811 | 61
kok | 0.16M | 444 | 0.17M | 912 | 164978 | 369
mai | 1195 | 47 | 1284 | 46 | 319 | 5
mal | 698M | 2480K | 635M | 2408K | 601M | 2200K
mni | - | - | 0.1M | 1092 | 0.06M | 89
mar | 934M | 2180K | 857M | 2138K | 845M | 2065K
nep | 1154M | 3047K | 1082M | 2983K | 1022M | 2660K
ory | 39M | 124K | 32M | 117K | 30M | 107K
pan | 369M | 597K | 284M | 499K | 281M | 466K
san | 3M | 11K | 1M | 11K | 1M | 3300
sat | - | - | 536 | 105 | - | -
snd | 83M | 91K | 76M | 85K | 75M | 76K
tam | 1607M | 4295K | 1485M | 4166K | 1384M | 3633K
tel | 583M | 1657K | 546M | 1599K | 523M | 1495K
urd | 1872M | 2538K | 1729M | 2435K | 1699M | 2209K
Total | 23195M | 49843K | 20429M | 48081K | 19872M | 44277K
Table 12: Statistics of the number of words and documents getting filtered out
at each stage while cleaning CulturaX through the Setu pipeline.

(a) CulturaX (b) MADLAD-400

Figure 19: Examples of noisy content being filtered out using Setu from the
already “cleaned” CulturaX and “cleaned” MADLAD-400 data corpus. The Left
shows the original document and the right shows the cleaned version. The text
in Red shows the noise that is removed.

###  6.2 IndicAlign

Table 7 shows the detailed statistics of IndicAlign data.

Number of Turns

Our curated dataset exhibits a wide range across various dimensions.
Specifically, the range of dialogue turns spans from an average of 9.27 to a
minimum of 1, which will result in the trained model’s capability to support
dialogues of both short and extended lengths. Furthermore, the variation in
average instruction and output lengths will underscore the model’s proficiency
in processing and generating content of diverse lengths.

Lexical Diversity of IndicAlign data

To show the lexical diversity of the prompts, following the work of UltraChat
(Ding et al., 2023b) we use the Measure of Textual Lexical Diversity (MTLD)
score (McCarthy & Jarvis., 2010). As seen in Table 7, the OpenAssistant
dataset has the highest lexical diversity, attributable to its sourcing from
approximately 13,500 volunteers. Additionally, the lexical diversity of the
Wiki-Chat dataset is on par with other human-generated datasets such as Indic
ShareLlama and Dolly, indicating that our methodology of using intents to
drive conversations is effective in producing prompts with diversity
comparable to those collected from human participants.

Intent Diversity Analysis

Figure 20 shows the distribution of intents within the Wiki-Chat dataset.
Notably, since we have used Wikipedia as the context, we understandably see a
majority of the interactions revolving around Information seeking. We also
observe the diversity of intents centered around various real-world scenarios
showing the real-world applicability of our data. To compare with other
datasets, we follow the approach of Self-Instruct (Wang et al., 2023a) and
show the most common root Verb-Noun analysis. Figure 21 shows the analysis on
a random sample of 20K prompts from each set.

Figure 20: Wiki-Chat Intent Analysis - The different kinds of intents based on
which Wiki-Chat conversations are simulated

(a) Wiki-Conv (b) Wiki-chat

(c) Dolly (d) IndicSharellama

(e) OpenAssistant

Figure 21: Comparative analysis of root Verb-Noun usage patterns across five
datasets. The inner circle represents the most commonly occurring Verbs, and
the outer circle denotes their 4 direct Noun objects

##  7 Conclusion

In summary, our work addresses the under-representation of low and mid-
resource languages, specifically focusing on the 22 constitutionally
recognised languages. We introduce IndicLLMSuite, a comprehensive framework
encompassing Sangraha pre-training data, Setu a Spark-based pipeline for data
curation, IndicAlign - Instruct a diverse prompt-response collection, and
IndicAlign - Toxic containing aligned toxic responses for Indic LLMs. By
striking a balance between human-verified content and model-generated data, we
aim to provide equitable access to information for diverse linguistic
communities. We encourage community collaboration in the costly endeavor of
LLM training, advocating for the pooling of resources to build high-quality
fully open-source Indic language LLMs. Through the public release of our tools
and datasets, we hope to inspire advancements in LLM development for Indian
languages and beyond.

## Limitations

Despite our efforts to curate and manually verify data, the intrinsic
variability in quality across different sources, including websites, PDFs, and
videos, remains a challenge. This variability may affect the consistency and
reliability of the models trained on this dataset. Also, despite wide
coverage, the representation of some languages, especially low-resource
languages, is limited. This shortfall is due to the challenge of gathering
resources for languages with scant digital presence. Furthermore, the
representativeness of each language in terms of dialects, regional variations,
and sociolects may not be fully comprehensive. This issue may impact the
model’s performance in accurately handling the nuances of each language. The
crowdsourced data exhibits low representation from higher age groups, uneven
coverage across Indian states and a lack of comprehensive inclusion for low-
resource languages.

Additionally, a significant portion of our dataset comprises translated data
to augment the original, curated content. While this method increases
diversity, it might not fully capture real-world language use, potentially
affecting the model’s ability to generate natural responses in some contexts.
We leave the analysis of the effect of synthetic data on model performance for
future work.

While our dataset and tools are extensive, the evaluation of models trained on
this suite across a wide range of downstream tasks for each of the 22
languages is beyond the scope of this work. Future research is needed to
evaluate the dataset’s effectiveness across various applications and domains,
which is essential for understanding its practical utility and identifying
performance variations.

## Ethics Statement

In developing Sangraha, we have sourced data from various formats, including
websites, PDFs, and videos. While enhancing dataset diversity, this approach
necessitates careful consideration of privacy, consent, and the ethical use of
data. To mitigate risks, we have implemented rigorous data-cleaning steps to
remove explicit, toxic, and personally identifiable (PII) content. However, we
rely on NSFW word detection for toxic data detection, which does not fully
capture or mitigate toxicity and sometimes results in false positives. We call
upon the community to create better toxic data detection techniques for all
Indian languages.

The legal landscape regarding the use of web-sourced content for training
models remains ambiguous across different jurisdictions. This ambiguity is
challenging for both data creators and consumers, especially where the
principle of fair use is not universally applicable. Additionally, the public
nature of our data sources introduces the risk of inherent biases, which could
be transferred to models trained on this dataset. We leave the analysis on
potential biases and debiasing techniques for future work.

All individuals involved in this effort, including annotators and developers,
were adequately compensated for their work, adhering to all relevant norms and
regulations of our country. The volunteers engaged in the curation of crowd-
sourced data were duly informed about the public release of the data.

#### Author Contributions

##### Sangraha:

Mohammed Safi Ur Rahman Khan, Priyam Mehta, Umashankar Kumaravelan, Sumanth
Doddapaneni, Sparsh Jain, Suriyaprasaad G, and Varun Balan G

##### Setu:

Priyam Mehta, Umashankar Kumaravelan, Ananth Sankar, Mohammed Safi Ur Rahman
Khan, Sumanth Doddapaneni

##### IndicAlign - Instruct:

Mohammed Safi Ur Rahman Khan, Ananth Sankar, Suriyaprasaad G and Varun Balan G

##### IndicAlign - Toxic:

Priyam Mehta and Mohammed Safi Ur Rahman Khan

##### Research Leads:

Mitesh M. Khapra, Raj Dabre, Pratyush Kumar and Anoop Kunchukuttan

#### Acknowledgments

We would like to thank EkStep Foundation and Nilekani Philanthropies for their
generous grant towards building datasets, models, tools and other resources
for Indian languages. We also thank Google for their valuable grant that
facilitated OCR through Google Cloud Vision and for access to free TPUs as
part of the TPU Research Cloud (TRC) initiative. We thank the Sarvam AI team
for their insightful discussions and constructive comments on this project. We
are also immensely grateful to the volunteers from the AI4Bharat team for
their motivation and meticulous efforts in conducting manual audits.

## References

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 4344–4355, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.463.
  * Abbas et al. (2023) Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication.  _CoRR_ , abs/2303.09540, 2023.  doi: 10.48550/ARXIV.2303.09540.  URL https://doi.org/10.48550/arXiv.2303.09540.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of open language models, 2023.
  * Anonymous (2024) Author Anonymousand Anonymous.  Anonymous title.  2024\.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * Barbaresi (2021) Adrien Barbaresi.  Trafilatura: A web scraping library and command-line tool for text discovery and extraction.  In Heng Ji, Jong C. Park, and Rui Xia (eds.), _Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL 2021 - System Demonstrations, Online, August 1-6, 2021_ , pp. 122–131. Association for Computational Linguistics, 2021.  doi: 10.18653/V1/2021.ACL-DEMO.15.  URL https://doi.org/10.18653/v1/2021.acl-demo.15.
  * Bhattacharyya (2010) Pushpak Bhattacharyya.  Indowordnet.  In _Lexical Resources Engineering Conference 2010 (LREC 2010)_ , Malta, May 2010.
  * Bhogale et al. (2022) Kaushal Santosh Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra.  Effectiveness of mining audio and text pairs from public data for improving asr systems for low-resource languages, 2022.
  * Botha et al. (2017) Jan A. Botha, Emily Pitler, Ji Ma, Anton Bakalov, Alex Salcianu, David Weiss, Ryan T. McDonald, and Slav Petrov.  Natural language processing with small feed-forward networks.  In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pp. 2879–2885. Association for Computational Linguistics, 2017.  doi: 10.18653/V1/D17-1309.  URL https://doi.org/10.18653/v1/d17-1309.
  * Broder (1997) Andrei Z. Broder.  On the resemblance and containment of documents.  In Bruno Carpentieri, Alfredo De Santis, Ugo Vaccaro, and James A. Storer (eds.), _Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings_ , pp. 21–29. IEEE, 1997.  doi: 10.1109/SEQUEN.1997.666900.  URL https://doi.org/10.1109/SEQUEN.1997.666900.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang.  Quantifying memorization across neural language models.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023.  URL https://openreview.net/pdf?id=TatRHT_1cK.
  * Charikar (2002) Moses Charikar.  Similarity estimation techniques from rounding algorithms.  In John H. Reif (ed.), _Proceedings on 34th Annual ACM Symposium on Theory of Computing, May 19-21, 2002, Montréal, Québec, Canada_ , pp. 380–388. ACM, 2002.  doi: 10.1145/509907.509965.  URL https://doi.org/10.1145/509907.509965.
  * Computer (2023) Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.  Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.  URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  * Costa-jussà et al. (2022) Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.  No language left behind: Scaling human-centered machine translation.  _CoRR_ , abs/2207.04672, 2022.  doi: 10.48550/ARXIV.2207.04672.  URL https://doi.org/10.48550/arXiv.2207.04672.
  * Dilmegani (2023) Cem Dilmegani.  Ocr in 2024: Benchmarking text extraction/capture accuracy.  2023\.  URL https://research.aimultiple.com/ocr-accuracy.
  * Ding et al. (2023a) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pp. 3029–3051. Association for Computational Linguistics, 2023a.  URL https://aclanthology.org/2023.emnlp-main.183.
  * Ding et al. (2023b) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations, 2023b.
  * Doddapaneni et al. (2023) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar.  Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 12402–12426, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.693.  URL https://aclanthology.org/2023.acl-long.693.
  * Endrédy & Novák (2013) István Endrédy and Attila Novák.  More effective boilerplate removal - the goldminer algorithm.  _Polibits_ , 48:79–83, 2013.  doi: 10.17562/PB-48-10.  URL https://doi.org/10.17562/pb-48-10.
  * Fellbaum (1998) Christiane Fellbaum (ed.).  _WordNet: An Electronic Lexical Database_.  MIT Press, Cambridge, MA, 1998.
  * Gala et al. (2023) Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan.  Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages, 2023.
  * Gala et al. (2024) Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy, and Anoop Kunchukuttan.  Airavata: Introducing hindi instruction-tuned llm, 2024.
  * Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The pile: An 800gb dataset of diverse text for language modeling.  _CoRR_ , abs/2101.00027, 2021.  URL https://arxiv.org/abs/2101.00027.
  * Golchin & Surdeanu (2024) Shahriar Golchin and Mihai Surdeanu.  Time travel in llms: Tracing data contamination in large language models, 2024.
  * Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need, 2023.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _CoRR_ , abs/2301.07597, 2023.  doi: 10.48550/ARXIV.2301.07597.  URL https://doi.org/10.48550/arXiv.2301.07597.
  * Gupta et al. (2022) Anirudh Gupta, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, Priyanshi Shah, Harveen Singh Chadha, and Vivek Raghavan.  indic-punct: An automatic punctuation restoration and inverse text normalization framework for indic languages, 2022.
  * Heafield (2011) Kenneth Heafield.  KenLM: Faster and smaller language model queries.  In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pp. 187–197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics.  URL https://aclanthology.org/W11-2123.
  * Hernandez et al. (2022) Danny Hernandez, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish.  Scaling laws and interpretability of learning from repeated data.  _CoRR_ , abs/2205.10487, 2022.  doi: 10.48550/ARXIV.2205.10487.  URL https://doi.org/10.48550/arXiv.2205.10487.
  * Husain et al. (2024) Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Ratish Puduppully, and Anoop Kunchukuttan.  Romansetu: Efficiently unlocking multilingual capabilities of large language models models via romanization, 2024.
  * Jacovi et al. (2023) Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg.  Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pp. 5075–5084. Association for Computational Linguistics, 2023.  URL https://aclanthology.org/2023.emnlp-main.308.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Jiang et al. (2024a) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts.  _CoRR_ , abs/2401.04088, 2024a.  doi: 10.48550/ARXIV.2401.04088.  URL https://doi.org/10.48550/arXiv.2401.04088.
  * Jiang et al. (2024b) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts, 2024b.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar.  IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.  In Trevor Cohn, Yulan He, and Yang Liu (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4948–4961, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.445.  URL https://aclanthology.org/2020.findings-emnlp.445.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.  Openassistant conversations - democratizing large language model alignment.  _CoRR_ , abs/2304.07327, 2023.  doi: 10.48550/ARXIV.2304.07327.  URL https://doi.org/10.48550/arXiv.2304.07327.
  * Kreutzer et al. (2022a) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Javier Ortiz Suárez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balli, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Trans. Assoc. Comput. Linguistics_ , 10:50–72, 2022a.  doi: 10.1162/TACL“˙A“˙00447.  URL https://doi.org/10.1162/tacl_a_00447.
  * Kreutzer et al. (2022b) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, 2022b.  doi: 10.1162/tacl˙a˙00447.  URL https://aclanthology.org/2022.tacl-1.4.
  * Kudugunta et al. (2023) Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat.  Madlad-400: A multilingual and document-level large audited dataset, 2023.
  * Kunchukuttan (2020) Anoop Kunchukuttan.  The indicnlp library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf, 2020.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.  Openassistant conversations – democratizing large language model alignment, 2023.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite.  The bigscience roots corpus: A 1.6tb composite multilingual dataset, 2023.
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pp. 8424–8445. Association for Computational Linguistics, 2022.  doi: 10.18653/V1/2022.ACL-LONG.577.  URL https://doi.org/10.18653/v1/2022.acl-long.577.
  * Li et al. (2023a) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.  CAMEL: communicative agents for ”mind” exploration of large scale language model society.  _CoRR_ , abs/2303.17760, 2023a.  doi: 10.48550/ARXIV.2303.17760.  URL https://doi.org/10.48550/arXiv.2303.17760.
  * Li et al. (2023b) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.  Camel: Communicative agents for ”mind” exploration of large language model society, 2023b.
  * Li et al. (2023c) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin.  Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation, 2023c.
  * Li et al. (2023d) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report, 2023d.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts.  The flan collection: Designing data and methods for effective instruction tuning, 2023.
  * Loper & Bird (2002) Edward Loper and Steven Bird.  Nltk: The natural language toolkit, 2002.
  * Madhani et al. (2023a) Yash Madhani, Mitesh M. Khapra, and Anoop Kunchukuttan.  Bhasa-abhijnaanam: Native-script and romanized language identification for 22 Indic languages.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 816–826, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.71.  URL https://aclanthology.org/2023.acl-short.71.
  * Madhani et al. (2023b) Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul Nc, Ruchi Khapra, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Khapra.  Aksharantar: Open Indic-language transliteration datasets and models for the next billion users.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 40–57, Singapore, December 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.4.  URL https://aclanthology.org/2023.findings-emnlp.4.
  * McCarthy & Jarvis. (2010) Philip M. McCarthy and Scott Jarvis.  Mtld, vocdd, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment, 2010.
  * MediaWiki (2023) MediaWiki.  Api:rest api/reference — mediawiki,, 2023.  URL https://www.mediawiki.org/w/index.php?title=API:REST_API/Reference&oldid=6225064.  [Online; accessed 12-February-2024].
  * Narayan et al. (2002) Dipak Narayan, Debasri Chakrabarti, Prabhakar Pande, and Pushpak Bhattacharyya.  An experience in building the indo wordnet-a wordnet for hindi.  In _First International Conference on Global WordNet_ , Mysore, India, 2002.
  * Nguyen et al. (2023a) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen.  Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023a.
  * Nguyen et al. (2023b) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing.  Seallms – large language models for southeast asia, 2023b.
  * Panjwani et al. (2018) Ritesh Panjwani, Diptesh Kanojia, and Pushpak Bhattacharyya.  pyiwn: A python based API to access Indian language WordNets.  In Francis Bond, Piek Vossen, and Christiane Fellbaum (eds.), _Proceedings of the 9th Global Wordnet Conference_ , pp. 378–383, Nanyang Technological University (NTU), Singapore, January 2018. Global Wordnet Association.  URL https://aclanthology.org/2018.gwc-1.47.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only.  _CoRR_ , abs/2306.01116, 2023.  doi: 10.48550/ARXIV.2306.01116.  URL https://doi.org/10.48550/arXiv.2306.01116.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher.  _CoRR_ , abs/2112.11446, 2021.  URL https://arxiv.org/abs/2112.11446.
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2022.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Ramesh et al. (2022) Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra.  Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages.  _Transactions of the Association for Computational Linguistics_ , 10:145–162, 2022.  doi: 10.1162/tacl˙a˙00452.  URL https://aclanthology.org/2022.tacl-1.9.
  * Sainz et al. (2023) Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 10776–10787, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.722.  URL https://aclanthology.org/2023.findings-emnlp.722.
  * Scrapinghub (2021) Scrapinghub.  Article extraction benchmark, 2021.  URL https://github.com/scrapinghub/article-extraction-benchmark.  GitHub repository.
  * Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _CoRR_ , abs/2206.04615, 2022.  doi: 10.48550/ARXIV.2206.04615.  URL https://doi.org/10.48550/arXiv.2206.04615.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023b.
  * Wang et al. (2023a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pp. 13484–13508. Association for Computational Linguistics, 2023a.  doi: 10.18653/V1/2023.ACL-LONG.754.  URL https://doi.org/10.18653/v1/2023.acl-long.754.
  * Wang et al. (2023b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions, 2023b.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pp. 4003–4012, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.494.
  * Xu et al. (2023a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _CoRR_ , abs/2304.12244, 2023a.  doi: 10.48550/ARXIV.2304.12244.  URL https://doi.org/10.48550/arXiv.2304.12244.
  * Xu et al. (2023b) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions, 2023b.
  * Xu et al. (2023c) Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pp. 6268–6278. Association for Computational Linguistics, 2023c.  URL https://aclanthology.org/2023.emnlp-main.385.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer, 2021.
  * Zaharia et al. (2016) Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica.  Apache spark: a unified engine for big data processing.  _Commun. ACM_ , 59(11):56–65, oct 2016.  ISSN 0001-0782.  doi: 10.1145/2934664.  URL https://doi.org/10.1145/2934664.
  * Zhao et al. (2024) Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.  (inthe)wildchat: 570k chatGPT interaction logs in the wild.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=Bl8u7ZRlbM.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy.  LIMA: less is more for alignment.  _CoRR_ , abs/2305.11206, 2023.  doi: 10.48550/ARXIV.2305.11206.  URL https://doi.org/10.48550/arXiv.2305.11206.

##  Appendix A Issues with Language Identification of Existing Corpora

The evolution of Language Identification (LID) models has predominantly
focused on European languages, leading to significant challenges in accurately
identifying languages from diverse linguistic families, notably Indic
languages. Kreutzer et al. (2022b) highlights a significant concern regarding
the mislabeling of languages in existing multilingual corpora, an issue that
undermines the reliability of language identification (LID) models. In this
small study, we analyze 200,000 documents per Indic language from the mC4 (Xue
et al., 2021) and OSCAR (Abadji et al., 2022) datasets, employing the IndicLID
model for its superior performance on Indic languages and support for
Romanized text (Madhani et al., 2023a). mC4 uses only cld3 model whereas OSCAR
defines an even stricter pipeline for identifying the language. It combines
sentence-level LID and aggregates them based on certain thresholds to classify
a document as multilingual or monolingual.

Our analysis uncovers a significant discrepancy in the accuracy of LID across
various Indic languages within the mC4 dataset. The languages sharing a common
script, such as Hindi, Marathi, and Nepali, experience higher rates of
mislabeling. This contrasts with languages with unique scripts showing
significantly lower mismatch percentages.

Conversely, the application of a more sophisticated LID methodology in the
OSCAR dataset markedly diminishes these inaccuracies, showing the
effectiveness of a refined approach to language identification. This
observation demonstrates the necessity for the development of language family-
specific identification models (Madhani et al., 2023a), as well as the
incorporation of better LID modules within data-cleaning pipelines.

Figure 22: % mismatch of the tagged language and the language predicted by
IndicLID

##  Appendix B Anudesh - User Base Analysis

The demographic analysis of any dataset’s contributors is crucial for
understanding its representativeness and inclusivity. Each user is prompted
first with a declaration - ”I consent to release my conversations under the
Creative Commons Attribution 4.0 International (CC BY 4.0) license.” as shown
in Figure 24 that the user has to accept before starting any interaction.

Figure 23: The User Agreement form

Figure 24: Anudesh chat page

Geographically, the user base is predominantly from Karnataka, Maharashtra,
and Tamil Nadu, as shown in Figure 25, with a notable underrepresentation of
users from other states, especially the North Eastern states. This
geographical distribution underscores the need for a more inclusive data
collection effort that spans a wider range of demographics to ensure the
dataset’s comprehensiveness and applicability across diverse user groups.

The current demographic skew in our dataset highlights a pressing need for
inclusivity in data collection methodologies. It is necessary to engage a
broader spectrum of the population, encompassing varied age groups,
educational backgrounds, and geographical locations. Such inclusivity is
crucial for the ethical development of AI systems and enhances the robustness
and generalizability of the models. Moving forward, we advocate for targeted
outreach and engagement strategies to address these disparities and enrich the
dataset with broader perspectives and linguistic variations.

(a) Participants distribution by Age-group (b) Participants distribution by
Qualification

(c) State-wise percentage distribution of participants across India

Figure 25: User Demographic Analysis of Anudesh Language | Original Count |  | After Validity
---
Check
| After Page
---
Count Check
| After Image
---
Filters
Hindi | 349,365 | 344,454 | 106,112 | 102,164
Urdu | 177,867 | 157,121 | 127,495 | 73,966
Sanskrit | 88,238 | 84,804 | 76,401 | 70,663
Bengali | 59,636 | 55,023 | 50,825 | 45,272
Tamil | 52,199 | 49,924 | 37,243 | 29,755
Telugu | 50,320 | 48,919 | 40,860 | 38,243
Gujarati | 43,677 | 42,021 | 34,514 | 34,038
Malayalam | 34,858 | 31,594 | 11,627 | 4,725
Kannada | 24,446 | 23,589 | 18,661 | 17,493
Punjabi | 13,898 | 12,932 | 7,397 | 5,617
Marathi | 9,710 | 9,174 | 7,875 | 7,478
Assamese | 2,424 | 2,408 | 2,205 | 2,408
Nepali | 1,545 | 1,497 | 836 | 671
Oriya | 4,972 | 4,733 | 2,439 | 4,732
Table 13: Statistics of PDFs filtering from Internet Archive State | Number of PDFs
---|---
Andhra Pradesh | 3383
Bihar | 306
Gujarat | 3241
Haryana | 433
Himachal Pradesh | 2035
Jharkhand | 124
Karnataka | 8405
Kerela | 2039
Madhya Pradesh | 656
Maharashtra | 544
Punjab | 287
Rajasthan | 7
Tamil Nadu | 680
Indian Parliament | 14896
Total | 37036
Table 14: Statistics of the PDFs collected from Indian Parliament and other State Assemblies Language | Number of PDFs
---|---
Bengali | 5721
Gujarati | 5586
Hindi | 18560
Kannada | 4888
Konkani | 471
Malayalam | 5665
Marathi | 8958
Nepali | 1686
Oriya | 5769
Punjabi | 885
Sanskrit | 730
Tamil | 7002
Telugu | 5555
Urdu | 2877
Total | 74353
Table 15: Language-wise statistics of PDFs collected from AIR - NewsOnAir State | Number of PDFs
---|---
Andhra Pradesh | 126
Assam | 61
Bihar | 426
Goa | 31
Haryana | 31
Himachal Pradesh | 1909
Karnataka | 502
Kerala | 121
Maharashtra | 76
Manipur | 70
Meghalaya | 293
Mizoram | 40