# SpeechAlign: a Framework for Speech Translation
Alignment Evaluation

Belen Alastruey1 , Aleix Sant2∗, Gerard I. Gállego2, David Dale1, Marta R.
Costa-jussà1
Meta AI1
TALP Research Center, Universitat Politècnica de Catalunya, Barcelona2
alastruey<EMAIL_ADDRESS>
gerard.i.gallego@upc.edu,daviddale<EMAIL_ADDRESS>Equal
contribution.

###### Abstract

Speech-to-Speech and Speech-to-Text translation are currently dynamic areas of
research. To contribute to these fields, we present SpeechAlign, a framework
to evaluate the underexplored field of source-target alignment in speech
models. Our framework has two core components. First, to tackle the absence of
suitable evaluation datasets, we introduce the Speech Gold Alignment dataset,
built upon a English-German text translation gold alignment dataset. Secondly,
we introduce two novel metrics, Speech Alignment Error Rate (SAER) and Time-
weighted Speech Alignment Error Rate (TW-SAER), to evaluate alignment quality
in speech models. By publishing SpeechAlign we provide an accessible
evaluation framework for model assessment, and we employ it to benchmark open-
source Speech Translation models.

##  1 Introduction

Speech-to-text Translation (S2TT) and Speech-to-speech Translation (S2ST)
refer to the task of converting spoken language into respectively written text
or speech in a different language. This tasks are increasing their popularity,
and can be used for applications such as subtitling videos in a different
language, translating between languages that do not have a written form, and
in general, ensuring seamless communication across people worldwide.

Figure 1: Example of a S2ST alignment in the Speech Gold Alignment dataset.

The initial approach to S2TT and S2ST involved the integration of distinct
models, forming what is nowadays known as a cascade system Ney (1999). This
systems consist of an Automatic Speech Recognition (ASR) model that
transcribes the spoken sentence, and a Machine Translation (MT) model that
translates the sentence into another language. In the case of S2ST an
additional speech synthesizer is needed, that is utilized to generate the
corresponding speech from the translated text. However, recent advancements
have led to the development of end-to-end models, that perform translation
from speech to text or to speech, without requiring an intermediate
transcription step, or a translated transcript. Known as direct Speech
Translation systems, these models have quickly progressed, and currently, they
can achieve state-of-the-art results comparable to those of cascade models
Ansari et al. (2020); Bentivogli et al. (2021). Nevertheless, the performance
of both cascade and end-to-end architectures remains far from optimal compared
to text translation systems, indicating that research in these areas is still
ongoing.

The recent growth of end-to-end models and the shift in the field towards
using them has raised the need to understand their inner workings. One related
task is source-target alignment, which involves analysing how models use the
provided source to make predictions, and whether they follow common human
intuition in this process.

This alignment task has been widely explored in the context of text
translation Ghader and Monz (2017); Ferrando et al. (2022). The task is
commonly evaluated using Alignment Error Rate (AER) Och and Ney (2003), a
metric that measures the differences between a gold-standard alignment and a
hypothesized one. For this aim, human-labeled alignment datasets have been
published in the context of text translation, such as Vilar et al. (2006) for
translation between English and German.

In speech-related fields, little interpretability work regarding alignments
has been done. Some previous studies have focused on analysing the self-
attention in the encoder of speech recognition, Zhang et al. (2021); Shim et
al. (2022) and speech translation Alastruey et al. (2022) systems. However,
these models' decoder, and consequently, its alignment capabilities, have yet
to be explored, potentially due to the absence of suitable datasets and
metrics for evaluating the task in this setting.

In light of this, we introduce SpeechAlign framework111Available on request.
Link will be published upon conference acceptance., which serves as a solution
to the stated lack of resources. SpeechAlign is formed of two core components:
a novel dataset and an evaluation framework founded on our proposed metrics.

The dataset, named Speech Gold Alignment, is specifically created to evaluate
alignment in S2TT and S2ST. This dataset is an extension of the text
translation gold alignment dataset introduced by Vilar et al. (2006). To
create it, we employ a Text-to-Speech (TTS) model to generate synthetic speech
for the sentences in the dataset. The utilization of a TTS model offers a main
advantage: apart from generating audio, it also provides timestamps denoting
the beginning and end of each word. Annotating such timestamps would be very
resource-intensive if using non-synthetic audios. Gathering the audios and the
timestamps, we are able to build the Speech Gold Alignment dataset, formed of
samples such as the one shown in Figure 1.

In terms of metrics, we adapt the AER for the speech domain, introducing two
novel metrics: Speech Alignment Error Rate (SAER) and Time-weighted SAER (TW-
SAER). These metrics quantify the alignment error models have, with the key
distinction that the former treats each word equally, and the latter factors
in word durations.

To sum up, the main contribution of this paper is the release of SpeechAlign,
a framework designed to simplify metrics computation using our dataset.
Additionally, we employ this framework to benchmark various open-source
models. Through these efforts, we aim to contribute to the exploration of
alignments in the domain of speech translation.

(a) T2T Translation.

(b) S2T Translation

(c) S2S Translation

Figure 2: Original alignment by Vilar et al. (2006) and our extensions.

##  2 Related Work

Over the past decades, considerable interest has been directed toward
comprehending the alignment capabilities of text translation models. In this
trajectory, both datasets and metrics have been developed to evaluate this
task.

Numerous authors have published alignment datasets Lambert et al. (2005);
Vilar et al. (2006); Kruijff-Korbayová et al. (2006); Graça et al. (2008);
Macken (2010); Holmqvist and Ahrenberg (2011) for the evaluation of alignments
in translations in languages such as English, Spanish, German, Dutch, and
Czech. In this work, we hone in on the dataset introduced by Vilar et al.
(2006)222https://www-i6.informatik.rwth-aachen.de/goldAlignment/ for text
translation between English and German. This dataset comprises 508 paired
sentences in the specified languages, along with precise information regarding
the alignment of words between these two languages. These sentences are
sourced directly from the EuroParl dataset Koehn (2005), which contains
transcripts and translations of speeches delivered in the European Parliament.
We opt for this dataset due to its coverage of the English-German translation
pair, which is extensively studied in the field of speech translation Agarwal
et al. (2023). Moreover, our work requires the generation of speech utterances
for the sentences in the dataset. Focusing on well-resourced languages like
English and German provides greater confidence in the quality of the speech
generated by the TTS model.

As for metrics, a singular measure has predominantly been used to evaluate
alignments. Alignment Error Rate (AER), introduced by Och and Ney (2003), is a
measure of alignment quality between a source sentence and its translation. It
is calculated as the ratio of alignment errors, where an alignment error
occurs when a unit in the translated sentence is not aligned with the correct
unit in the source. The score is computed based on a manually annotated gold-
standard alignment of a parallel corpus. Given a reference alignment,
consisting of a set $S$ of “Sure”, unambiguous alignment points, and a set $P$
of “Possible”, ambiguous alignment points, with $S\subseteq P$, the AER of an
alignment $A$ is defined to be:

$\text{AER}(S,P;A)=1-\frac{|A\cap S|+|A\cap P|}{|A|+|S|}$ (1)

##  3 Speech Gold Alignment Dataset

The dataset we introduce, Speech Gold Alignment, extends the bilingual text
alignment dataset presented by Vilar et al. (2006) by adding speech utterances
to each pair of English and German sentences. Additionally, for each audio
file, the dataset contains a dictionary defining all the words in each
sentence and their corresponding start and end time stamps, what gives us a
text-to-audio mapping. Once we have this, we incorporate the gold alignment
correspondences from the original dataset to obtain the alignments between
speech segments.

This augmented dataset, can either be considered as two distinct datasets
supporting S2TT from English to German (and vice versa), or as a unified S2ST
dataset by combining both S2TT alignments. In Figure 2 we show the three
different modalities of the dataset. Figure 2(a) shows a sample from the
original dataset presented by Vilar et al. (2006), and Figures 2(b) and 2(c)
show our extension for S2TT and S2ST settings respectively.

As part of the SpeechAlign framework, we publish a pipeline to prepare the
dataset, following the steps that are described in section 3.1.

###  3.1 Methodology

The construction of this dataset can be divided into two primary steps. First,
we employed the VITS model Kim et al. (2021) to generate synthetic speech for
all the sentences, as detailed in section 3.1.1. Subsequently, we aligned each
word to its corresponding time interval in the produced speech signal, as
explained in section 3.1.2. While integrating the datasets, we found specific
cases where alignment was not immediate or direct. We address these
complexities in section 3.1.3.

####  3.1.1 Speech Generation

To produce synthetic speech for the sentences in the Gold Alignment dataset,
we employed the VITS model. This TTS system uses a phonemizer to obtain the
phonemes corresponding to the input sequence. Then, to generate the speech
output, the model uses a stochastic duration predictor that assigns a duration
to each phoneme. The chosen duration is randomly sampled from each phoneme's
durations distribution. By doing this, the model is able to synthesize natural
speech and can generate different speech utterances for the same input text.

To build our dataset, we generated separate synthetic versions for the 508
sentences in both English and German. In English, we utilized LJ Speech Ito
and Johnson (2017), while for the German language, the Thorsten voice Müller
and Kreutz (2021) was employed. This task was done using the VITS model
available through the Coqui toolkit Eren and The Coqui TTS Team (2021).

####  3.1.2 Word-Audio Matching

The Gold Alignment dataset constitutes a word-to-word alignment reference, to
which we add our newly generated audios, product of VITS. Nevertheless, to
achieve an alignment between speech intervals, we first need to establish a
linkage between audio segments and words in the original dataset.

The approach followed to accomplish this starts by acquiring intermediate
representations from VITS. Specifically, we gather the output generated by the
phonemizer, which is the phonemized sentence, as well as the output of the
duration predictor. This predictor crates a dictionary containing duration in
integer units of each phoneme. With this information in hand, we perform a
two-step matching procedure, that ultimately yields the mapping from audio to
words, via the intermediate representation of phonemes:

  1. 1.

Phoneme-Word Matching. In this stage, we focus on aligning the phonemes with
the words present in the original dataset.

  2. 2.

Phoneme-Audio Matching. In this phase, we establish a time mapping between the
audio and its corresponding sequence of phonemes.

Figure 3 provides a visual representation of the sequential steps followed for
deriving both the waveform and the alignment between words and audio, which
constitute the dataset we present.

Figure 3: Pipeline used to generate the dataset.

With the basic steps outlined, now we will dive deeper into the details of
each of the phases to obtain the audio-word matching.

###### Phoneme-Word Matching.

The goal of this phase is to achieve a mapping between the sequence of
phonemes extracted from the phonemizer and the sequence of words in the
original dataset Vilar et al. (2006). To do so, we use blank spaces as
delimiters for words in the phonemes sequence, and we monotonically map them
with the sequence of words. It is important to note that the original dataset
underwent tokenization through Moses, introducing some challenges in this
process that are outlined in detail in Section 3.1.3.

###### Phoneme-Audio Matching.

After obtaining the correspondence between words and phonemes, we now need to
map phonemes to the audio. Ideally, the entire audio must be partitioned into
separate time intervals, each containing the pronunciation of a single word.
To accomplish this, it is necessary to compute the overall duration of each
individual word.

To compute the total duration of each word, we take the output of the duration
predictor and sum the duration in units of all the phonemes belonging to a
same word. As previously stated, blank spaces are used as delimiters between
words in the phoneme transcription. Consequently, the duration assigned to a
blank space is equally distributed and added to the neighboring words, both
preceding and succeeding the blank space. The same approach applies to units
attributed to punctuation marks, that we decided not to include in our
alignment dataset given that they cannot be found explicitly in speech
utterances.

Next, our objective is to establish the corresponding word duration in seconds
based on their duration in units. To achieve this, we divide the total length
of the audio by the aggregate duration in units of all the phonemes in the
sentence. This computation establishes a correlation between VITS duration
units and the equivalent time in seconds. Using this derived relationship, we
convert the word durations from units to seconds and find the start and end
times for each word.

####  3.1.3 Special Cases

During the two phases of the dataset construction we encounter some special
challenges that need special handling.

###### Phonemic Fusion.

In the majority of instances, phonemized words align with the original text
words, primarily through sentence segmentation using blank spaces.
Nevertheless, in certain cases the phonemizer merges adjacent words during
phonetic transcription, creating what we name as phonemic fusion. This
occurrence is primarily observed in short English words such as prepositions,
articles, and pronouns, which are pronounced seamlessly without pauses. Table
1 provides examples of this phenomenon. In such particular instances, we first
determine the combined duration of these merged words and subsequently
distribute the total time proportional to the length among the constituent
words. While this approach may not be entirely precise, we believe the
approximation is enough, given its applicability to very short words and few
cases.

Phonemic Fusion
---
Words: I am
Phonemes: /aIam/
Words: of the
Phonemes: /@vði/
Words: as it is
Phonemes: /æzItIz/
Words: that the
Phonemes: /ðætði/ ssssssssssssssssssssssss

Phonemic Fragmentation
---
Words: 124
Phonemes: /wVn hVndr@d twEntI fO/ sssssss
Words: 34%
Phonemes: /ðEtI fO p@sent/
Words: 1996
Phonemes: /naIntIn naIntI sIks/

Table 1: Examples of the special cases encountered when aligning words and
their phonemization.

###### Phonemic Fragmentation.

Furthermore, we have encountered a contrasting phenomenon in comparison to
phonemic fusion. The phonemizer carries out a normalization process on the
text before phonemization. Occasionally, this normalization procedure results
in the conversion of single words into multiple words – a phenomenon we refer
to as Phonemic Fragmentation. This behavior is particularly noticeable in
cases involving numbers, percentages, years, and similar elements. To address
this matter, we aggregate the durations of all the split words and attribute
the total duration to the original solitary word.

##### Possesives ('s)

The original Gold Alignment dataset does not provide alignments between
natural sentences, but for sentences tokenized with Moses. However, VITS works
on natural text, and this missmatch created some difficulties along the
matching process. This is the case of words such as "Parliament's", that is
considered a single word when dealing with VITS ("Parliament's" $\rightarrow$
/pal@m@nts/) , but it is actually two different words with independent
alignments in the original dataset ("Parliament 's"), due to Moses
tokenization.

This is a case of Phonemic Fusion (and it's addressed as such). However,
unlike previously shown cases caused by the phonemizer, this fusion stems from
the tokenization in the original dataset.

###### Percent Sign (%)

A similar behaviour arises when dealing with percent signs. These signs appear
alongside numbers in natural text ("34%"), but in the Gold Alignment dataset,
they're separate tokens due to Moses tokenization ("34 %"). However, as
illustrated in Table 1, percents are a case of Phonemic Fragmentation, with
the phonemizer breaking this construction into multiple phonemized words
("34%" $\rightarrow$ /ðEtI fO p@sent/).

In this particular cases of Phonemic Fragmentation, we aim to separate the
expanded phonetic text into two segments: the first containing phonemized
words associated with the number (/ðEtI fO/), and the last containing the
phonemized word corresponding to the percent (/p@sent/). In this instances,
the merging of time intervals encompasses all words except the final one in
the expansion.

###### German Phonemizer

In our utilization of the German phonemizer, we have noticed that it
occasionally produces inaccurate phonetic transcriptions for certain single
input words. These inaccuracies tend to occur with special symbols (e.g., "%",
"/"), years (e.g., "1996"), acronyms (e.g., "EU", "Nr"), compound nouns (e.g.,
"EU-Staate"), among others. To rectify these inaccuracies in phonetic
transcription, we have replaced specific words in the input sentences with
their expanded and "spoken" format ("EU" $\rightarrow$ "E U", "1996"
$\rightarrow$ "nineteen ninety six"). This adjustment assists the phonemizer
in producing more accurate transcriptions.

###  3.2 Dataset Quality Assessment

Within this section, we aim to examine the quality of the synthetic audio
produced by VITS. We conduct an assessment comparing EuroParl ST Iranzo-
Sánchez et al. (2020) test set and our own synthesized data, which is also
derived from a subset of the EuroParl dataset Koehn (2005). With this aim, we
evaluate the performance of the Whisper Tiny model Radford et al. (2022) on
the task of speech recognition on these two datasets. This strategy allows us
to understand the implications of using synthetic audio without the influence
of content domain. We choose to perform this evaluation in the setting of
speech recognition, and not in translation, because of the simplicity of the
former due to its monotonic alignment process. This ensures that the overall
model performance and the complexity of the task are less likely to influence
the results. We have opted to conduct this evaluation using the smallest
Whisper model. Our hypothesis behind this choice is that if no issues arise in
the smallest model, they are unlikely to manifest in larger models.

In Table 2, we present the Word Error Rate (WER) results obtained on both
datasets, and we observe that our synthetic audios results in a lower WER than
standard EuroParl ST dataset. Consequently, we can conclude that the
synthesized data does not pose a problem and appears to be easily handled by
the models, possibly due to the clarity of the generated audios compared to
European Parliament recordings.

Dataset | Language | WER
---|---|---
EuroParl ST | En | 29.7
Speech Gold Alignment | En | 3.9
EuroParl ST | De | 31.0
Speech Gold Alignment | De | 23.1
Table 2: Quality assessment results.

##  4 Proposed Evaluation

The objective of this section is to define an evaluation procedure and metrics
that are able assess models' ability to establish source-target alignments. To
analyse this capability, our focus is on the contribution maps generated by
the models. These maps indicate the relationship between source and target
tokens, such that the contribution of a source token to a target one is always
a non-negative value, and that the sum of contributions from all source tokens
to a target token must equal 1 (i.e. attention weights or more advanced
interpretability methods Kobayashi et al. (2021); Ferrando et al. (2022)).
Then, to measure the alignments, we build new metrics around the intuition of
the Alignment Error Rate (AER) score, initially introduced by Och and Ney
(2003) and defined in section 2. However, extracting the alignments from the
contribution map and adapting AER for speech sequences is not straightforward
process.

Figure 4: Example of the token-to-token attention weights of a S2TT decoder
layer on Whisper Small.

Input:

$C\\_t2t$: token-to-token contribution matrix,

$src$: source words & durations,

$tgt$: tgt words & durations

Output:

$C\\_w2w$: word-to-word contribution matrix

$max\\_duration\\_src\leftarrow src[-1][end]$

$max\\_tokens\\_src\leftarrow C\\_t2t.shape[1]$

for  _$word,word\\_idx\leftarrow src$_ do

$s\\_time\leftarrow src[word][start]$

$e\\_time\leftarrow src[word][end]$

$s\\_token\leftarrow ceil(s\\_time*max\\_tokens\\_src/max\\_duration\\_src)$

$e\\_token\leftarrow floor(e\\_time*max\\_tokens\\_src/max\\_duration\\_src)$

$C\\_w2t[:,word\\_idx]\leftarrow\text{sum}(C\\_t2t[:,s\\_token:e\\_token],dim=1)$

$max\\_duration\\_tgt\leftarrow tgt[-1][end]$

$max\\_tokens\\_tgt\leftarrow C\\_t2t.shape[0]$

for  _$word,word\\_idx\leftarrow tgt$_ do

$s\\_time\leftarrow tgt[word][start]$

$e\\_time\leftarrow tgt[word][end]$

$s\\_token\leftarrow ceil(s\\_time*max\\_tokens\\_tgt/max\\_duration\\_tgt)$

$e\\_token\leftarrow floor(e\\_time*max\\_tokens\\_tgt/max\\_duration\\_tgt)$

$C\\_w2w[word\\_idx,:]\leftarrow\text{avg}(C\\_w2t[s\\_token:e\\_token,:],dim=0)$

Algorithm 1 Contributions Preprocessing

###  4.1 Preprocessing

The metric of AER assesses the error rate between a hypothesis and a target
alignment. Hence, to compute this score, we need a gold alignment dataset. In
most text alignment datasets, such as the one we extend Vilar et al. (2006),
these alignments are provided as word-to-word relations. Consequently, the
hypothesis alignment needs to be structured in a word-to-word format too.
However, in speech settings, the system input tokens correspond to frames of a
spectrogram or ranges of a waveform. As a consequence, the contribution maps
usually extract token-to-token interactions, being each token a speech frame.
Thus, a conversion process is necessary to derive word-to-word alignments from
a tokens-to-tokens contribution map, and consequently being able to evaluate
the alignment to obtain an AER score.

Nonetheless, a similar challenge is faced in the setting of text translation,
where tokens are often sub-words rather than complete words. In this case, the
conversion from tokens to words involves a two-step process.When dealing with
sub-words in the source, their contributions are aggregated by summing them
together. This approach is rooted in the principle that the combined
contribution of two tokens to a target is the sum of their individual
contributions. Handling sub-words in the target sequence proves to be more
complex. Each token has a distinct distribution of contributions across the
source. To address this, the average of each sub-word distribution is
computed. By following this approach, we are able to effectively establish the
alignment between words despite the presence of sub-word units.

In the case of speech, we propose to employ a similar approach when
aggregating tokens from each word, in order to obtain a word-to-word
contributions plot. Leveraging our dataset, which provides details into the
correspondence between segments of input/output audio and individual words, we
define which tokens correspond to each word under with the assumption of a
linear relation between tokens and audio, and dismissing any overlap. Doing
this allows us to employ a similar approach to the one used for merging sub-
words, but this time, we apply it to the set of all tokens linked to a single
word. Given a contributions map $C$ where $c_{i,j}$ is the contribution of the
$j$-th source token to the $i$-th prediction, the resulting word-to-word
contributions map is computed using our dataset as shown in Algorithm 1. In
Figures 4 and 5 we show an example of a contributions map before and after the
preprocessing.

Following this conversion and before computing the alignment scores, we derive
the hard alignments. This is accomplished by aligning each target word with
the source word that has the highest contribution.

Size | Parameters |  SAER(%, $\downarrow$) |  TW-SAER(%,$\downarrow$) |  BLEU($\uparrow$)
---|---|---|---|---
Tiny | 39M | 75.3 | 70.1 | 3.6
Base | 74M | 72.9 | 67.8 | 8.4
Small | 244M | 70.7 | 65.7 | 15.4
Medium | 769M | 69.5 | 64.1 | 20.2
Large | 1.55B | 68.9 | 63.5 | 22.1
Large-v2 | 1.55B | 77.0 | 70.6 | 22.3
Table 3: Benchmarking of different sizes of Whisper models on De-En S2TT.

###  4.2 Speech Alignment Error Rate

Once we have the hard alignments, we define the Speech Alignment Error Rate
(SAER) in the same manner the AER is defined. This is, given a set set $S$ of
unambiguous alignments, a set $P$ of ambiguous alignment and a set $A$ of
hypothesis alignment:

$SAER=1-\frac{|A\cap S|+|A\cap P|}{|A|+|S|}$ (2)

However, it's important to note that SAER doesn't fully address a key aspect
in the speech setting – the noticeable disparity in the number of different
tokens that form each word, which corresponds to audio durations. Instead,
when computing SAER each word contributes equally to the final score,
regardless of its duration.

Figure 5: Example in Figure 4 after the preprocessing, obtaining a word-to-
word contributions map.

###  4.3 Time-Weighted SAER

To address the limitations of the SAER, we define the Time-weighted SAER, a
metric that accounts for the variability in word durations. To do so, we
introduce a new element – the incorporation of a weight for each alignment.
These weights are defined using the area of each alignment, as shown in Figure
6, and defined as follows:

$w_{i,j}=\begin{cases}s_{j}\cdot s_{i}&\text{if S2ST}\\\ s_{j}\cdot 1&\text{if
S2TT}\end{cases}$ (3)

where $w_{i,j}$ is the weight of an alignment between the $j$-th source word
and the $i$-th target word, and $s_{i}$, $s_{j}$ is the duration in seconds of
these words respectively. Therefore, given a set set $S$ of unambiguous
alignments and a set $P$ of ambiguous alignment, the TW-SAER is defined as the
sum of areas of the alignments in $A\cap S$ plus the sum of areas in $A\cap
P$, divided by the total alignment area of $A$ and $S$:

$TW-SAER=1-\frac{\sum_{i,j\in A\cap S}w_{i,j}+\sum_{i,j\in A\cap
P}w_{i,j}}{\sum_{i,j\in A}w_{i,j}+\sum_{i,j\in S}w_{i,j}}$ (4)

By including the weights we account for the temporal duration of each word
within the audio, refining our evaluation process. Note that SAER and TW-SAER
are equivalent if $w_{i,j}=1\hskip 2.0pt\forall i,j$.

Figure 6: TW-SAER weights.

##  5 SpeechAlign

The main contribution of this paper is the release of SpeechAlign, an
accessible open-source framework that encompasses the Speech Gold Alignment
dataset presented in section 3 and the SAER and TW-SAER metrics defined in
section 4. This tool seamlessly handles raw token-to-token alignment maps and
computes both proposed alignment error rates. This framework is versatile, and
can be used in attention weights or more sophisticated contribution maps. The
pipeline starts by taking the given contribution maps and converts them into
word-to-word equivalents. To achieve this, the alignment dataset is utilized
to account for varying word durations. Following this conversion, we derive
the hard alignments. The outcome is a definitive set of hypothesis alignments,
that are used to compute both SAER and TW-SAER scores.

To enhance the comprehension of the process, we include a notebook for
visualization of the alignments and contributions maps. This tool can be used
to visualize token-to-token and the extracted word-to-word representations,
and also the obtained hard alignments. By publishing this framework, we aim to
facilitate the use of our dataset by other researchers.

Finally, using SpeechAlign, we benchmark some S2TT models. For simplicity, we
decide to analyze alignments based on models' cross-attention weights. We
decide not benchmark the S2ST task due to the current lack of open-source
models, being the recently published SeamlessM4T SeamlessCommunication et al.
(2023) the only one available as of now. This model comprises two consecutive
Transformers, each containing its own decoder. Consequently, it presents
significant challenges in terms of obtaining a contributions map based on
attention weights, and developing further interpretability methods lies beyond
scope of this paper.

##### Models Benchmarking

Table 3 presents an evaluation of various sizes of the Whisper model Radford
et al. (2022) on De-En S2TT. Each model's performance is assessed through the
BLEU score on our test set, and the SAER and TW-SAER. The latter are computed
on the attention weights of each decoder layer, and in Table 3 we report the
best obtained score. This analysis uncovers a correlation between the
performance metrics and the alignment score. This correlation is also observed
to align with the model's size. Intriguingly, an outlier appears in the case
of the Large V2 model, which fails to deliver proper alignment despite
achieving the highest performance metrics.

##  6 Conclusion

In conclusion, this paper introduces SpeechAlign, a framework to evaluate
alignment in speech models. SpeechAlign has two main components. Firstly,
we've created the Speech Gold Alignment dataset, being the first of its kind
and created to address the lack of suitable evaluation data for the task.
Secondly, we have presented the two first evaluation metrics for speech
alignment, Speech Alignment Error Rate (SAER) and Time-weighted Speech
Alignment Error Rate (TW-SAER), to assess how well speech models perform on
the alignment task. SpeechAlign provides an accessible way to evaluate speech
models, and we have used it to benchmark various open-source models.

## Limitations

Our study only focuses on high resource languages, German and English,
stemming from the limitation of the original Gold Alignment dataset Vilar et
al. (2006). Additionally, it's worth noting the potential challenges that
might arise due to the limited domain covered by the dataset, centered on
speeches from the European Parliament. This might cause some issues when
evaluating models trained with a limited amount of data and from other
domains. However, this case would be a limitation of the model itself,
denoting its restricted capabilities. Using data from European Parliament's
speeches is a common practice in the Machine Translation field Koehn (2005);
Iranzo-Sánchez et al. (2020).

## References

  * Agarwal et al. (2023) Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello Federico, Souhir Gahbiche, Barry Haddow, Benjamin Hsu, Phu Mon Htut, Hirofumi Inaguma, Dávid Javorský, John Judge, Yasumasa Kano, Tom Ko, Rishu Kumar, Pengwei Li, Xutai Ma, Prashant Mathur, Evgeny Matusov, Paul McNamee, John P. McCrae, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Ha Nguyen, Jan Niehues, Xing Niu, Atul Kr. Ojha, John E. Ortega, Proyag Pal, Juan Pino, Lonneke van der Plas, Peter Polák, Elijah Rippeth, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Yun Tang, Brian Thompson, Kevin Tran, Marco Turchi, Alex Waibel, Mingxuan Wang, Shinji Watanabe, and Rodolfo Zevallos. 2023\.  FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 1–61, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Alastruey et al. (2022) Belen Alastruey, Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. 2022.  On the locality of attention in direct speech translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 402–412, Dublin, Ireland. Association for Computational Linguistics.
  * Ansari et al. (2020) Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondrej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stuker, Marco Turchi, Alex Waibel, and Changhan Wang. 2020.  Findings of the iwslt 2020 evaluation campaign.  In _IWSLT 2020_.
  * Bentivogli et al. (2021) Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021.  Cascade versus direct speech translation: Do the differences still make a difference? _CoRR_ , abs/2106.01045.
  * Eren and The Coqui TTS Team (2021) Gölge Eren and The Coqui TTS Team. 2021.  Coqui TTS.
  * Ferrando et al. (2022) Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costa-jussà. 2022.  Towards opening the black box of neural machine translation: Source and target interpretations of the transformer.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8756–8769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ghader and Monz (2017) Hamidreza Ghader and Christof Monz. 2017.  What does attention in neural machine translation pay attention to? In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 30–39, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Graça et al. (2008) João Graça, Joana Paulo Pardal, Luísa Coheur, and Diamantino Caseiro. 2008.  Building a golden collection of parallel multi-language word alignment.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Holmqvist and Ahrenberg (2011) Maria Holmqvist and Lars Ahrenberg. 2011.  A gold standard for English-Swedish word alignment.  In _Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011)_ , pages 106–113, Riga, Latvia. Northern European Association for Language Technology (NEALT).
  * Iranzo-Sánchez et al. (2020) Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020\.  Europarl-st: A multilingual corpus for speech translation of parliamentary debates.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8229–8233.
  * Ito and Johnson (2017) Keith Ito and Linda Johnson. 2017.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.
  * Kobayashi et al. (2021) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021.  Incorporating Residual and Normalization Layers into Analysis of Masked Language Models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4547–4568, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86, Phuket, Thailand.
  * Kruijff-Korbayová et al. (2006) Ivana Kruijff-Korbayová, Klára Chvátalová, and Oana Postolache. 2006\.  Annotation guidelines for Czech-English word alignment.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Lambert et al. (2005) Patrik Lambert, Adrià De Gispert, Rafael Banchs, and José B. Mariño. 2005.  Guidelines for word alignment evaluation and manual alignment.  _Language Resources and Evaluation_ , 39(4):267–285.
  * Macken (2010) Lieve Macken. 2010.  An annotation scheme and gold standard for Dutch-English word alignment.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Müller and Kreutz (2021) Thorsten Müller and Dominik Kreutz. 2021.  Thorsten - open german voice (neutral) dataset.  Please use it to make the world a better place for whole humankind.
  * Ney (1999) H. Ney. 1999.  Speech translation: coupling of recognition and translation.  In _1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258)_ , volume 1, pages 517–520 vol.1.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational Linguistics_ , 29(1):19–51.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.
  * SeamlessCommunication et al. (2023) SeamlessCommunication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023.  Seamlessm4t—massively multilingual & multimodal machine translation.  _arXiv_.
  * Shim et al. (2022) Kyuhong Shim, Jungwook Choi, and Wonyong Sung. 2022.  Understanding the role of self attention for efficient speech recognition.  In _International Conference on Learning Representations_.
  * Vilar et al. (2006) David Vilar, Maja Popović, and Hermann Ney. 2006.  AER: do we need to “improve” our alignments?  In _Proc. International Workshop on Spoken Language Translation (IWSLT 2006)_ , pages 205–212.
  * Zhang et al. (2021) Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. 2021.  On the usefulness of self-attention for automatic speech recognition with transformers.  In _2021 IEEE Spoken Language Technology Workshop (SLT)_ , pages 89–96.


# FuLG: 150B Romanian Corpus for Language Model Pretraining

Vlad-Andrei Bădoiu  University Politehnica of BucharestBucharestRomania ,
Mihai-Valentin Dumitru  University Politehnica of BucharestBucharestRomania ,
Alexandru M. Gherghescu  University Politehnica of BucharestBucharestRomania ,
Alexandru Agache  University Politehnica of BucharestBucharestRomania and
Costin Raiciu  University Politehnica of BucharestBroadcom
Inc.BucharestRomania

###### Abstract.

Research in the field of language models is rapidly evolving, with many open
models being released to the public. Openly available pretraining corpora
usually focus on only a handful of languages, with many others either missing
completely or extremely underrepresented. In this report, we introduce FuLG
111https://hf.co/datasets/faur-ai/fulg, a hundred-fifty-billion-token Romanian
corpus extracted from CommonCrawl. We present our methodology for filtering
FuLG and compare it via ablation studies against existing Romanian corpora.

##  1\. Introduction

In a world where Large Language Models (LLMs) have shown great potential for
natural language tasks, the corpora used for pretraining play a central role
in their overall capabilities. However, most state-of-the-art models are not
accompanied by their training datasets and authors only sparsely discuss their
composition.

In response to this, researchers and practitioners have produced numerous data
corpora for widely spoken languages on the Internet. These are often derived
from CommonCrawl 222https://commoncrawl.org, a public repository of crawled
web pages, which has indexed more than 250 billion pages. Other datasets go
further by including curated data such as books, social media discussions, and
research papers. Trillion-token datasets such as Dolma (Soldaini et al.,
2024), FineWeb (Penedo et al., 2024), or RedPajama (Computer, 2023) have
enabled the development of billion-parameter models with loss-optimal
training, but they cover only a small fraction of languages.

This limitation is evident in the language understanding capabilities of open
models, as they are often trained on publicly available data. For instance,
OLMo models (Groeneveld et al., 2024) support only six languages, while other
initiatives such as LLM360 K2 (Liu et al., 2023) are English-centric. Even
models developed by tech giants, such as Llama (Touvron et al., 2023) or Gemma
(Team et al., 2024), have limited output capabilities in less commonly spoken
languages. Only recently have we seen some developments in this area, with
Llama 3 supporting 30 languages, albeit with lower performance levels compared
to English and limited information about the training dataset.

To improve the standing of the Romanian language in future models, we delve
into filtering Romanian content from CommonCrawl, documenting the process and
releasing an open dataset. We built a pipeline that employs deduplication
techniques, common signal filters, and FastText for language detection. We
document the steps taken to obtain the final text corpus. As a result, we
release FuLG, an openly available corpus that is three times larger than
existing Romanian corpora.

In this work we make two main contributions. We release FuLG, a 156B-token
(589GB tokenized), or 220B tokens with the Llama 3 tokenizer, corpus for LLM
pretraining and fine-tuning in the Romanian language. Second, we document the
process of filtering data from CommonCrawl, which may be employed for other
underrepresented languages.

##  2\. Related Work

A crucial factor in model performance is the size, quality and diversity of
the pretraining corpus. These datasets typically comprise content from various
sources, primarily web pages, due to their sheer number and availability,
followed by social media discussions, academic papers, books, code
repositories, and encyclopedias, covering a wide range of topics. The main
sources of data are either private crawls of the Internet or the publicly
available CommonCrawl repository.

While information about closed models’ training data is scarce, even open
models like Llama or Gemma lack transparency regarding their training
datasets, as it is a central part of the competitive advantage. We are thus in
a peculiar scenario, where the best open-source models aren’t truly open-
source when it comes to pretraining data, which deeply affects any smaller
initiative.

On the other hand, open datasets are essential components for truly open
models and play a vital role in democratizing future LLMs. Currently, numerous
open datasets exist, many derived from CommonCrawl. Notable examples include:
Dolma (Soldaini et al., 2024) (3T tokens), C4 (Dodge et al., 2021) (175B
tokens), The Pile (Gao et al., 2020) (387B tokens), ROOTS (Laurençon et al.,
2022) (400B tokens), RefinedWeb (Penedo et al., 2023) (600B tokens), RedPajama
v2 (Computer, 2023) (30T tokens), FineWeb (Penedo et al., 2024) (15T tokens),
Zyda (Tokpanov et al., 2024) (1.3T tokens), LLM360 Amber (Liu et al., 2023)
(1.2T tokens), MAP-Neo (Zhang et al., 2024) (4.5T tokens), OSCAR (Ortiz Suárez
et al., 2019). While these datasets provide sufficient quality and size for
English-based models, they often lack adequate representation of less commonly
spoken languages. For example, after training a Romanian GPT-NeoX tokenizer on
the Romanian part of OSCAR, we obtain a number of only 10B tokens on OSCAR,
and 41B tokens on mC4 (Raffel et al., 2019) (slightly higher number of tokens
if using an English tokenizer, similar to Llama or OLMo, due to worse
compression ratio). However, considering the scaling laws of dataset size
relative to model size, these quantities are insufficient for optimal LLM
training. Beyond these, several smaller datasets exist for various Natural
Language Processing (NLP) tasks in Romanian (Váradi et al., 2020; Mititelu et
al., 2018; Ion et al., 2012; Midrigan-Ciochina et al., 2020; Schöch et al.,
2021; Manolescu and Çöltekin, 2021) but they generally contain less than 500M
tokens, which is far from adequate for LLM development.

##  3\. Data Acquisition and Filtering

FuLG is derived from CommonCrawl, a vast collection of web page crawls dating
back to 2007. CommonCrawl releases snapshots of portions of the Internet
several times annually. These snapshots exhibit low similarity rates between
releases, enabling us to consistently extract new data from each snapshot. Our
work encompasses snapshots from 2013 to May 2024.

We developed a pipeline based on existing software, which we elaborate on
shortly. Given the petabytes of data requiring filtration, we utilized a
distributed environment with multiple nodes for data acquisition. For
deduplication and quality filtering, we employed a single large-memory node.

### Data Acquisition

To process CommonCrawl snapshots, we leveraged the CCNet pipeline (Wenzek et
al., 2019). CCNet facilitates distributed processing of snapshots, including
downloading in WET format, language identification via the FastText algorithm
(Joulin et al., 2016), and deduplication of common paragraphs. We encountered
two primary challenges with CCNet:

  1. (1)

Due to the fact that CCNet development has stalled, we quickly hit roadblocks
related to package versions and environment-specific problems with our SLURM
setup, which needed modifications to the source code.

  2. (2)

Since we ran the pipeline on a cluster shared with other users, we did not
have all the hardware for ourselves. Strict limits imposed by the SLURM system
for fair sharing meant we needed to adapt the source code, which was not built
with this in mind. We faced limitations with the maximum size of job arrays
supported, often capped at 100. To address this, we extended CCNet to support
job batching. Initially, CCNet would submit a job array equal in size to the
number of shards; our modifications enabled it to break the job array into
multiple smaller submissions to the SLURM scheduler.

We kept only documents with a language score for Romanian exceeding 0.5,
meaning that there are documents that may include other languages alongside
Romanian.

### Deduplication

For filtering and deduplication, we employed code from RedPajama (Computer,
2023). Exact deduplication reduced the dataset size by 37%, while fuzzy
deduplication with a 0.8 threshold further reduced it by 50%. We updated the
deduplication pipeline to not only identify duplicates but also remove them
from the dataset.

### Content Filtering

Next, we introduced a content filtering step to our pipeline. Using a regex-
based approach, we filtered HTML text extraction artifacts, such as navbar
text, from documents. To filter potentially controversial content, we utilized
a dictionary, removing documents containing specific words in either the
content or URL. For Personally Identifiable Information (PII), we replaced
phone numbers, email addresses, and links with special tokens.

### Quality Filtering

As with deduplication, we utilized existing code from RedPajama to compute a
set of quality signals. We then extended the code to filter documents based on
the quality signals introduced by Gopher (Rae et al., 2021), reducing the
dataset size by 50% to 156B tokens using the OLMo tokenizer trained on
Romanian. (589GB). With a different tokenizer such as Llama 3, which was
trained on many other languages, we obtain around 220B tokens; we expect much
worse compression for English-only tokenizers, such as Llama 2. The thresholds
and rules we employed for filtering are as follows:

  * •

Fraction of characters in the most common n-grams exceeded: 0.2 for 2-grams,
0.18 for 3-grams, 0.16 for 4-grams, 0.15 for 5-grams, 0.14 for 6-grams, 0.13
for 7-grams, 0.12 for 8-grams, 0.11 for 9-grams, 0.10 for 10-grams

  * •

Fewer than 50 words or more than 100,000 words

  * •

Median word length less than 3 or greater than 10 characters

  * •

More than 90% of lines start with bullet points

  * •

More than 30% of lines end with ellipses

  * •

Less than 30% of lines end with punctuation

The data acquisition process spanned several months, while deduplication and
filtering were completed in less than 18 hours on a single large-memory node.

##  4\. FuLG in action

To evaluate FuLG we conducted ablation studies using a 1B decoder-only model
based on OLMo (Groeneveld et al., 2024), with a sequence length of 2048 and
the OLMo tokenizer trained on the OSCAR-Ro corpus. We used a global batch size
of 256. We made several changes to the hyperparameters to adjust to the 1B
model size: weight decay of 0.001 and a max learning rate of 4e-4 annealed by
a cosine schedule, with an initial warmup of 1000 steps.

We trained three identical models on different pretraining datasets: Oscar,
mC4 and FuLG. We leveraged the Hugging Face transformers (Wolf et al., 2020)
library for training. We trained to completion on each dataset (290k steps for
FuLG, 75k for mC4 and 20k for OSCAR). Since FuLG was much bigger than both
OSCAR and mC4 (about 4 times bigger than mC4), we also included an earlier
checkpoint from FuLG at 70k steps. We do note that it slightly underperforms,
but we believe this naturally happens because of the cosine decay learning
rate schedule.

Dataset | Perplexity
---|---
FuLG (290k) | 16.06
FuLG (70k) | 21
mC4 | 15.38
OSCAR | 23.57
Table 1. Perplexities for datasets.

A commonly employed method to assess the quality of a dataset is to fix a
model, train it with different datasets, and measure perplexity against a
curated evaluation set. While perplexity is not a definitive measure of
dataset quality (Soldaini et al., 2024), we use it as a sanity check, with
values similar to existing datasets confirming our approach. We constructed a
perplexity dataset covering multiple domains, sourced from Wikipedia, news
articles, textbooks, research papers, and books, totaling 74M tokens. Although
we did not specifically perform a separate decontamination step, as this is
only a preliminary evaluation, we do note that the collection methods of the
evaluation dataset were manual, with care in picking high-quality sources. We
do plan to apply a decontamination step in further experiments, to make sure
our results are fair. Before calculating perplexity, the evaluation dataset
was cleaned using the clean-text Python package
333https://github.com/jfilter/clean-text. The results are shown in Table 1.
Both ablations of FuLG show perplexity values similar to those of existing
datasets.

Dataset | Grammar | Creativity | Complexity
---|---|---|---
FuLG (290k) | 8.5 | 7.1 | 4.6
FuLG (70k) | 7.25 | 5.875 | 4
mC4 | 7.5 | 5.2 | 3.1
OSCAR | 6.3 | 4.2 | 2.8
Table 2. Results for the story generation task.

We further qualitatively evaluated the trained models. Inspired by the
TinyStories (Eldan and Li, 2023) work, we used the models to generate stories
from given prompts and asked GPT-4 to rate the creativity, grammar, and
overall complexity of each story. We only considered responses that were
coherent stories, discarding anomalous outputs. Both OSCAR and FuLG (70k)
generated significantly more anomalies than the other ablations. In Table 2,
we present our findings, which suggest hat FuLG may enable better performance
for specific tasks.

##  5\. Discussion and Future Work

The availability of high-quality datasets for less commonly spoken languages
is crucial for the democratization of LLMs. While proprietary models often
demonstrate proficiency across a wide range of languages, open models
frequently underperform in this aspect. By developing large, high-quality open
datasets for diverse languages, we can foster the creation of superior open-
source models. These improved models could potentially be utilized by
governments worldwide to meet their digitalization needs.

FuLG represents an initial effort to improve dataset size and quality for the
Romanian language. We’re already underway to implementing a few other
improvements which will further increase dataset quality:

  * •

Data processing: We currently work with Common Crawl snapshots in WET format.
However, as previous research [citation] indicates, employing a more
sophisticated HTML parser could significantly improve both the quality of
extracted text and the overall volume of usable data.

  * •

Language-specific optimization: Romanian language particularities could be
leveraged to adapt and refine quality filters. Currently, we use thresholds
designed for the English language, but developing language-specific criteria
could yield better results.

  * •

Novel quality filters: Identifying and implementing new quality filters
tailored to the Romanian language could further enhance the dataset’s overall
quality and relevance.

## Acknowledgements

We acknowledge the EuroHPC Joint Undertaking for awarding this project access
to the EuroHPC supercomputer LUMI, hosted by CSC (Finland) and the LUMI
consortium through a EuroHPC Regular Access call. Some of the experiments were
performed on the Luxembourg national supercomputer MeluXina. The authors
gratefully acknowledge the LuxProvide teams for their expert support. We
acknowledge the computational resources provided by the PRACE award granting
access to Discoverer in SofiaTech, Bulgaria. The authors gratefully
acknowledge the HPC RIVR consortium and EuroHPC JU for funding this research
by providing computing resources of the HPC system Vega at the Institute of
Information Science. We acknowledge VSB – Technical University of Ostrava,
IT4Innovations National Supercomputing Center, Czech Republic, for awarding
this project access to the LUMI supercomputer, owned by the EuroHPC Joint
Undertaking, hosted by CSC (Finland) and the LUMI consortium through the
Ministry of Education, Youth and Sports of the Czech Republic through the
e-INFRA CZ (grant ID: 90254). This work was funded by a VMWare gift.

## References

  * (1)
  * Computer (2023) Together Computer. 2023\.  _RedPajama: an Open Dataset for Training Large Language Models_.  https://github.com/togethercomputer/RedPajama-Data
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021\.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  _arXiv preprint arXiv:2104.08758_ (2021).
  * Eldan and Li (2023) Ronen Eldan and Yuanzhi Li. 2023.  Tinystories: How small can language models be and still speak coherent english?  _arXiv preprint arXiv:2305.07759_ (2023).
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020\.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ (2020).
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024\.  Olmo: Accelerating the science of language models.  _arXiv preprint arXiv:2402.00838_ (2024).
  * Ion et al. (2012) Radu Ion, Elena Irimia, Dan Stefanescu, and Dan Tufis. 2012\.  ROMBAC: The Romanian Balanced Annotated Corpus.. In _LREC_. 339–344.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext. zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Laurençon et al. (2022) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022\.  The bigscience roots corpus: A 1.6 tb composite multilingual dataset.  _Advances in Neural Information Processing Systems_ 35 (2022), 31809–31826.
  * Liu et al. (2023) Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. 2023\.  Llm360: Towards fully transparent open-source llms.  _arXiv preprint arXiv:2312.06550_ (2023).
  * Manolescu and Çöltekin (2021) Mihai Manolescu and Çağrı Çöltekin. 2021\.  Roff-a romanian twitter dataset for offensive language. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_. 895–900.
  * Midrigan-Ciochina et al. (2020) Ludmila Midrigan-Ciochina, Victoria Boyd, Lucila Sanchez-Ortega, Diana Malancea_Malac, Doina Midrigan, and David P Corina. 2020.  Resources in Underrepresented Languages: Building a Representative Romanian Corpus. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_. 3291–3296.
  * Mititelu et al. (2018) Verginica Barbu Mititelu, Dan Tufiş, and Elena Irimia. 2018\.  The reference corpus of the contemporary Romanian language (CoRoLa). In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019\.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures _(Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019)_ , Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi (Eds.). Leibniz-Institut für Deutsche Sprache, Mannheim, 9 – 16.  https://doi.org/10.14618/ids-pub-9021
  * Penedo et al. (2024) Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024.  The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.  arXiv:2406.17557 https://arxiv.org/abs/2406.17557
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023\.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_ (2023).
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021\.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_ (2021).
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _arXiv e-prints_ (2019).  arXiv:1910.10683
  * Schöch et al. (2021) Christof Schöch, Tomaž Erjavec, Roxana Patras, and Diana Santos. 2021.  Creating the european literary text collection (eltec): Challenges and perspectives.  _Modern Languages Open_ (2021).
  * Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. 2024\.  Dolma: An open corpus of three trillion tokens for language model pretraining research.  _arXiv preprint arXiv:2402.00159_ (2024).
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024\.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_ (2024).
  * Tokpanov et al. (2024) Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, and Quentin Anthony. 2024\.  Zyda: A 1.3 T Dataset for Open Language Modeling.  _arXiv preprint arXiv:2406.01981_ (2024).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Váradi et al. (2020) Tamás Váradi, Svetla Koeva, Martin Yamalov, Marko Tadić, Bálint Sass, Bartłomiej Nitoń, Maciej Ogrodniczuk, Piotr Pezik, Verginica Barbu Mititelu, Radu Ion, et al. 2020\.  The MARCELL legislative corpus. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_. 3761–3768.
  * Wenzek et al. (2019) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019\.  CCNet: Extracting high quality monolingual datasets from web crawl data.  _arXiv preprint arXiv:1911.00359_ (2019).
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Zhang et al. (2024) Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. 2024\.  Map-neo: Highly capable and transparent bilingual large language model series.  _arXiv preprint arXiv:2405.19327_ (2024).