#  Turna: A Turkish Encoder-Decoder Language Model for Enhanced Understanding
and Generation

Gökçe Uludoğan1 Zeynep Yirmibeşoğlu Balal1 Furkan Akkurt1
Melikşah Türker1,2 Onur Güngör1 Susan Üsküdarlı1
1Department of Computer Engineering, Bogazici University
2 VNGRS-AI
{gokce.uludogan,furkan.akkurt,zeynep.yirmibesoglu,
<EMAIL_ADDRESS>
###### Abstract

The recent advances in natural language processing have predominantly favored
well-resourced English-centric models, resulting in a significant gap with
low-resource languages. In this work, we introduce the language model Turna,
which is developed for the low-resource language Turkish and is capable of
both natural language understanding and generation tasks. Turna is pretrained
with an encoder-decoder architecture based on the unified framework UL2 with a
diverse corpus that we specifically curated for this purpose. We evaluated
Turna with three generation tasks and five understanding tasks for Turkish.
The results show that Turna outperforms several multilingual models in both
understanding and generation tasks, and competes with monolingual Turkish
models in understanding tasks. Turna is made available at hf.co/boun-tabi-
LMG/turna.

##  1 Introduction

Recent advances in natural language processing (NLP) have predominantly
produced English-centric models Devlin et al. (2019); Clark et al. (2020);
Radford et al. (2019); Brown et al. (2020); Touvron et al. (2023); Jiang et
al. (2023). English-centric models have benefited from the vast amount of
training data gathered from an abundance of English resources present on the
web. As such, these models become utilized in applications and fuel an
abundance of further research leading to the state-of-the-art performances
across various tasks Touvron et al. (2023); Jiang et al. (2023). On the other
hand, low-resource languages face challenges due to the lack of data and
limited computational resources, leading to a significant gap between models
trained on well-resourced languages and those focusing on low-resource
languages. Multilingual models have been proposed that attempt to bridge this
gap Devlin et al. (2019); Conneau et al. (2020); Xue et al. (2021); Liu et al.
(2020). However, they often do not perform well in tasks requiring a deep
understanding of language-specific nuances, such as dependency parsing and
named entity recognition Virtanen et al. (2019); Baumann (2019); Tanvir et al.
(2021) and lag behind monolingual models of the same scale Rust et al. (2021);
Nozza et al. (2020).

Recently, pretrained language models built upon transformers Vaswani et al.
(2017) have dominated NLP. These models vary in terms of their architectures
and objectives (i.e., causal language modeling and denoising objectives). The
architectures are commonly classified as encoder-only, decoder-only, or
encoder-decoder models. Encoder-only models are typically trained with
denoising objectives and focus on understanding tasks Devlin et al. (2019);
Clark et al. (2020). Decoder-only models are designed for generation tasks
with causal language modeling Radford et al. (2019); Brown et al. (2020);
Touvron et al. (2023). Finally, encoder-decoder models deal with NLP tasks
that require both understanding and generation of texts Dong et al. (2019);
Tay et al. (2023). Towards this end, the Text-to-Text Transformer (T5) Raffel
et al. (2020) employs an encoder-decoder architecture that was pretrained with
a denoising objective known as span corruption. The Unifying Language Learning
(UL2) framework Tay et al. (2023) proposes the Mixture-of-Denoisers (MoD)
pretraining objective that combines several denoising objectives. By coupling
the MoD objective with an encoder-decoder architecture, they have achieved
state-of-the-art results in a range of NLP tasks.

For Turkish, while encoder-only models exist Schweter (2020), there is a need
for large-scale pretrained models that can perform both natural language
understanding (NLU) and generation (NLG). This work aims to develop such a
model for Turkish that performs well across a variety of tasks. Towards this
end, we first compile a diverse range of corpora for pretraining, including
web data, scientific articles, graduate theses, books, creative writing, and
transcriptions of parliamentary speech. Subsequently, we pretrain Turna, which
adopts an encoder-decoder architecture following the UL2 framework Tay et al.
(2023). Furthermore, we collect several publicly available Turkish datasets
from a variety of tasks for benchmarking purposes. Our contributions are as
follows:

  * •

The release of Turna111hf.co/boun-tabi-LMG/turna, the first unified language
model capable of both understanding and generation tasks in Turkish. Thus far,
this model is the largest of its kind, which has 1.1B parameters and is
trained on a diverse range of corpora consisting of 43B tokens from various
domains.

  * •

The evaluation of Turna on 13 datasets across eight tasks, showing that it
surpasses multilingual models across many tasks and it either outperforms or
is on par with BERTurk Schweter (2020) (a Turkish monolingual encoder-only
model) in understanding tasks.

  * •

The release of a publicly available open source code for data collection,
filtering222github.com/boun-tabi-LMG/turkish-academic-
text-harvest, model training333github.com/boun-tabi-LMG/turna, and fine-
tuning444github.com/boun-tabi-LMG/turkish-lm-tuner.

##  2 Related Work

###  2.1 Multilingual Language Models

Multilingual language models are used for tasks that involve multiple
languages and resources with limited data. Turkish, considered as a low-
resource language, is moderately represented in these models, such as mBERT
Devlin et al. (2019), XLM-R Conneau et al. (2020), mBART Liu et al. (2020),
mT5 Xue et al. (2021), XGLM Lin et al. (2022), mGPT Shliazhko et al. (2022),
and mDeBERTa He et al. (2023). However, these models’ scale-to-performance
ratios are often not as efficient as that of monolingual models for language-
specific applications with abundant data Rust et al. (2021); Nozza et al.
(2020).

###  2.2 Turkish Language Models

A series of bert models for Turkish known as berturk have already been trained
Schweter (2020). This collection includes several variations of bert Devlin et
al. (2019), Distilbert Sanh et al. (2019), Convberturk Jiang et al. (2020),
and electra Clark et al. (2020), with varying cases, vocabulary, and model
sizes. Most models were trained on a 35GB corpus with 4.4B tokens, drawn from
the Turkish oscar corpus Abadji et al. (2022), a Wikipedia dump, and various
OPUS corpora Tiedemann (2012). Some models, like Convberturk and ELECTRA, were
also trained on the Turkish portion of the mC4 corpus. The models have been
evaluated on various downstream tasks, such as part-of-speech tagging, named
entity recognition, and question answering. They often outperform their
multilingual counterparts, mBERT Devlin et al. (2019) and XLM-R Conneau et al.
(2020). However, these models are all encoder-only models, most suitable for
language understanding tasks. There is a clear need for Turkish models that
excel in both language understanding and generation.

###  2.3 Pretraining objectives

Recently, pretrained language models based on transformers have been dominant
in the NLP field, exhibiting variations in both components and objectives.
Models that exclusively employ encoders, typically trained with denoising
objectives, are geared toward understanding tasks, as exemplified in works
such as Devlin et al. (2019) and Clark et al. (2020). Conversely, models that
exclusively use decoders are designed for generation tasks, employing causal
language modeling, as demonstrated in various studies Radford et al. (2019);
Brown et al. (2020); Touvron et al. (2023). The Text-to-Text Transformer (T5)
Raffel et al. (2020), on the other hand, employs an encoder-decoder
architecture and undergoes pretraining with a denoising objective referred to
as span corruption. UniLM Dong et al. (2019) is also an encoder-decoder model,
but pre-trained using unidirectional, bidirectional, and sequence-to-sequence
language modeling. This can be seen as a combination of causal and denoising
objectives. Recently, Tay et al., 2023 proposed that various pretraining
objectives can be recast as each other. They introduced the UL2 framework
based on a pretraining objective called Mixture-of-Denoisers (MoD), which
combines different pretraining paradigms. They compared decoder-only and
encoder-decoder models trained with the MoD objective and found that encoder-
decoder models often perform better. Notably, by using the MoD objective and
moderately scaling up the model, they achieved state-of-the-art performance on
a diverse set of NLP tasks including understanding and generation tasks.

##  3 Data

We compiled a diverse Turkish monolingual dataset to pretrain our model. Our
dataset comprises of a web corpus, scientific corpora gathered from Turkish
articles and graduate theses, Turkish books, a corpus of specially collected
writings from Bilkent University, and transcriptions of parliamentary debates.
The details of each corpus are explained in the following subsections, and the
training corpora statistics are summarized in Table 1.

During data splitting, we ensured that the validation set of each dataset
contain a minimum of $100$K tokens. The resulting train-validation splits are
reported under each subsection.

###  3.1 Web Corpora

mC4 Raffel et al. (2020) and OSCAR-2201 Abadji et al. (2022) are two large
multilingual web corpora. Their Turkish sections contain 87.7M and 10.8M web
pages, respectively, and result in 98.5M web pages when combined. Although
significant in quantity, web data is full of noise that is not considered part
of the natural language, such as titles and keywords. Therefore, such corpora
should be cleaned before being used for training. The OSCAR and mC4 corpora
used in this work were cleaned by the VNGRS-AI team using a set of heuristics
and rules. The cleaned version of the combined web corpus contains 50.3M
pages.

###  3.2 Scientific Corpora

To create a corpus in the scientific domain characterized by its formal and
informative language style, we collected articles and theses written in
Turkish. We downloaded the articles from DergiPark555dergipark.org.tr, a major
platform for Turkish academic journals. Our initial collection included
$407,146$ articles, all in PDF format and labeled as Turkish. These articles
were sourced from $1,857$ distinct journals, comprising a diverse range of
topics. These articles form our Dergipark scientific corpus.

In addition to articles, we also collected scientific texts in the form of
theses. These theses, products of higher education in Turkey, were accessed
from Turkey’s National Thesis Center666tez.yok.gov.tr/UlusalTezMerkezi. From
this repository, we downloaded $486,166$ theses marked as Turkish, which
composes our YökTez scientific corpus.

The collected documents were in PDF format. For text extraction, we utilized
the Apache Tika parser777github.com/apache/tika. We applied a rigorous
cleaning and filtering strategy to remove undesired content like page numbers,
equations, table entries, and similar unnecessary tokens introduced by the
extraction process, as detailed in Section A.1.

We used $99.99$% of the cleaned Dergipark documents for training and the rest
for validation, to avoid over-inflation of the validation set due to the high
number of documents. For YökTez, $99.999$% of the documents were used for
training. The final number of documents and the number of tokens after line
and document-wise filtering of our scientific training corpora are listed in
Table 1.

###  3.3 Book Corpus

The Book Corpus is a compilation of $5,080$ Turkish fiction and non-fiction
books. We cleaned the Book Corpus in a similar, but simpler heuristic. We
first standardized the punctuation and removed invalid characters. The initial
$100$ lines of each book have been filtered out if they contain author,
translator, or publishing information. We dropped any line in the book that is
all numeric or contains a URL or an e-mail. After the initial $70$% lines, we
truncated the lines after a keyword indicating a bibliography, notes, or a
list of works of the author or the publishing house. $99.97$% of the books
were used for training ($5,078$ books), and the remaining two books for
validation.

###  3.4 Bilkent Creative Writings

The Bilkent Creative Writings corpus comprises $8,630$ documents produced by
Bilkent University students while taking creative writing courses in
Turkish888github.com/selimfirat/bilkent-turkish-
writings-dataset. We cleaned this data like the book corpus by removing lines
containing special keywords (such as the word for assignment in Turkish) and
truncating after bibliographies. $8,457$ of them were used for training and
the rest was used for validation.

###  3.5 ParlaMintTR

The ParlaMintTR corpus is assembled from the CLARIN Flagship
project999clarin.eu/parlamint and consists of the Turkish portion of
parliamentary debates in Europe ($1,335$ documents). We used the original form
of the debates without applying special cleaning or filtering. $1,333$ of the
debates were used for training, and two debates for validation.

Corpus | Type | # Docs | # Tokens (B)
---|---|---|---
OSCAR & mC4 | Web | $50,336,214$ | $25.33$
Dergipark | Scientific | $334,429$ | $1.78$
Yöktez | Scientific | $475,817$ | $15.24$
Books | Literary | $5,078$ | $0.61$
Bilkent Creative Writings | Creative Text | $8,457$ | $0.01$
ParlaMintTR | Dialogue | $1,333$ | $0.07$

Table 1: Training Datasets

##  4 Methodology

###  4.1 Model

We used an encoder-decoder Transformer model101010Specifically, we used the
version 1.1 of the official T5 implementation described at github.com/google-
research/text-to-text-transfer-
transformer/blob/main/released_checkpoints.md#t511 Raffel et al. (2020) for
Turna. This choice was based on the finding that encoder-decoder models
surpass decoder-only models when the UL2 objective is used, as demonstrated in
Tay et al., 2023. Furthermore, the encoder component can still be employed
effectively for understanding tasks when coupled with task-specific
classification heads, thus reducing the model parameters by half. Due to our
limited computational resources, we opted for the Large36L configuration Tay
et al. (2021) for our model. This configuration requires only $37$% of the
parameters of a model configuration of comparable size, yet still outperforms
it.

Turna has 36 encoder and decoder layers, each with 16 attention heads. The
model’s token embeddings are 1024 dimensional. The multi-layer perceptron
layers have 2816 hidden dimensions and employ Gated GeLu activations Shazeer
(2020). The parameters of the input and classification layers are not shared.
These architectural choices result in a model with 1.1B parameters.

For tokenization, we used a unigram subword tokenizer Kudo (2018) trained on
10GB of text that consists of random subsets of OSCAR Abadji et al. (2022),
OPUS Zhang et al. (2020) and Wikipedia dump dated September 17, 2021, using
the SentencePiece implementation111111github.com/google/sentencepiece Kudo and
Richardson (2018). This tokenizer121212github.com/vngrs-ai/vnlp/tree/main/
vnlp/turkish_word_embeddings is provided by the VNGRS-AI Team. The initial
vocabulary size of $32,000$ was expanded to $32,128$ with the addition of 128
sentinel tokens used by pretraining objectives.

###  4.2 Pretraining Objectives

The pretraining was performed with Mixture-of-Denoisers (MoD), consisting of
several denoising objectives, which were shown to achieve better downstream
performance Tay et al. (2023). These objectives are R-denoising (regular
denoising), S-denoising (sequential denoising), and X-denoising (extreme
denoising), each characterized by the mean length of the corrupted spans, the
ratio of corrupted tokens, and the number of corrupted spans. R-denoising
follows the standard span corruption method of T5, selecting spans of 2 to 5
tokens, covering about $15$% of the input. The task is then to predict the
corrupted tokens in the decoder output. S-denoising, on the other hand,
corrupts a continuous portion from a random point in the input, accounting for
approximately $25$% of the input. Similar to R-denoising, this objective aims
to predict a single corrupted span. However, it is similar to standard causal
language modeling in the modeling approach. X-denoising is designed as an
interpolation between R-denoising and S-denoising. It aims to corrupt $50$% of
the input on average. This is achieved through a varying mix of many short or
fewer long corrupted spans, exposing the model to both denoising and causal
language modeling-like objectives. During pretraining, these objectives are
randomly assigned to each input sequence, with a distribution of $40$% each
for R- and X-denoisers and $20$% for S-denoising.

The model differentiates between these denoisers by using specific sentinel
tokens at the beginning of samples: [NLG] for X-denoiser, [NLU] for
R-denoiser, and [S2S] for S-denoiser.

###  4.3 Implementation details

##### Pretraining.

We pretrained Turna for a total of $1,740,000$ steps with a batch size of 48
and a source and target sequence length of 512 using a single v3-8 type TPU
with the T5X131313github.com/google-research/t5x library. This configuration
results in Turna being exposed to 42.7B tokens at the end of its training. We
disabled the dropout during pretraining but enabled it during fine-tuning.

The pretraining data is a mixture of samples from the collected datasets. To
ensure a fair representation of different language characteristics, we
randomly selected samples from each dataset according to their proportions:
Web Corpora ($50$%), YökTez ($25$%), DergiPark ($10$%), Book Corpus ($10$%),
ParlaMintTR ($3$%), and Bilkent Creative Writings ($2$%).

##### Baselines.

We compared our model with multilingual models: mT5, specifically
mT5-large141414hf.co/google/mt5-large Xue et al. (2021), and
mBART151515hf.co/facebook/mbart-large-cc25 Liu et al. (2020), as well as a
monolingual encoder-only model, BERTurk161616hf.co/dbmdz/bert-base-turkish-
cased Schweter (2020), where applicable.

##### Fine-tuning.

We fine-tuned the models using Hugging Face’s transformers
library171717github.com/huggingface/transformers Wolf et al. (2020) on NVIDIA
A40 GPUs. The standard text-to-text formulation is used for fine-tuning the
encoder-decoder models, i.e., Turna, mT5 and mBART. Additionally, we fine-
tuned Turna’s encoder with a task-specific head for certain understanding
tasks, referring to it as Turna-Encoder. The models were optimized for 10
epochs with an early stopping patience of 3 epochs. We used the AdaFactor
optimizer Shazeer and Stern (2018) with a learning rate of
$1\text{\times}{10}^{-3}$ to tune Turna and mT5 models, without a scheduler.
However, our attempts at fine-tuning the mBART model with the AdaFactor
optimizer did not yield a satisfactory training loss curve. Consequently, we
opted for the AdamW optimizer Loshchilov and Hutter (2017) with a learning
rate of $5\text{\times}{10}^{-5}$ and a linear scheduler. The same optimizer
and scheduler settings were applied for fine-tuning the BERTurk and Turna-
Encoder models. Due to our limited computational resources, we could not
perform hyperparameter tuning and used the recommended fine-tuning settings
for AdaFactor181818hf.co/docs/transformers/main_classes/
optimizer_schedules#transformers.Adafactor and default trainer
settings191919hf.co/docs/transformers/main_classes/
trainer#trainer for AdamW. For each task and dataset, the batch size, and
maximum input and target length parameters were individually selected, and
their corresponding values can be found in Table 7.

We used beam decoding with a beam size of 4 and early stopping to generate
predictions. For summarization and title generation tasks, we also applied a
length penalty of 2 and enforced a no-repeat n-gram size of 3 to ensure the
diversity of the output and prevent repetition of sequences.

##  5 Experiments

###  5.1 Fine-tuning tasks

This section provides an overview of downstream tasks used to evaluate the
model. These tasks assess model capabilities across various domains, and
include both natural language understanding and generation tasks. The
understanding tasks include text classification, natural language inference,
semantic textual similarity, named entity recognition, and part-of-speech
tagging. The generation tasks comprise paraphrasing, summarization, and news
title generation.

##### Paraphrasing.

This task involves rephrasing a given text while retaining the original
meaning. It assesses the model’s understanding of semantics and its ability to
generate diverse texts. We utilized two paraphrasing datasets, constructed
from parallel corpora via machine translation and filtered based on semantic
similarity Alkurdi et al. (2022). These are TAT, which contains paraphrases
from Tatoeba202020tatoeba.org, and OST, which includes pairs from
OpenSubtitles2018 Lison et al. (2018).

##### Summarization.

Similar to paraphrasing, summarization also rephrases a text. However, it aims
to produce a condensed version that only includes key information.
Consequently, it imposes additional constraints on the model’s generative
capabilities. For evaluation, we used two datasets: TRNews Baykara and Güngör
(2022) and the Turkish subset of MLSUM Scialom et al. (2020).

##### News Title Generation.

Generating titles for news articles evaluates a model’s ability to capture the
most salient information in a concise manner and checks the model’s creativity
and understanding of key phrases in the news domain. We used the same two
summarization datasets: TRNews and MLSUM.

##### Named Entity Recognition.

Named entity recognition (NER) aims to locate named entities, and subsequently
classifies these entities into predefined categories, typically “person”,
“location” and “organization”. We employed two datasets for this task: WikiANN
Rahimi et al. (2019) and MilliyetNER Tür et al. (2003).

##### Part-of-speech Tagging.

Part-of-speech (POS) tagging involves categorizing each word in a sentence
according to its grammatical function. This task assigns a specific part of
speech, such as noun, pronoun, or verb, to every word, clarifying its role
within the sentence’s structure. We used two Turkish Universal Dependencies
Nivre et al. (2020) treebanks, IMST Türk et al. (2023) and BOUN Marşan et al.
(2023), to fine-tune and evaluate our model.

##### Semantic Textual Similarity.

Semantic textual similarity (STS) tests the model’s ability to contextually
compare two sentences by producing a similarity score. We used the STSb-TR
Beken Fikri et al. (2021) dataset to fine-tune and evaluate our model.

##### Natural Language Inference.

Natural language inference (NLI), also known as textual entailment, involves
examining a pair of sentences, the premise and the hypothesis, to determine
their relationship as “entailment”, “contradiction”, or “neutral”. This task
tests a model’s understanding of context by comprehending the premise and
assess if the hypothesis logically follows it. Therefore, NLI also measures a
model’s reasoning skills. For this task, we used the Natural Language
Inference in Turkish (NLI-TR) dataset Budur et al. (2020) for evaluation.

##### Text Classification.

Text classification involves categorizing texts into predefined groups based
on their contents. This task assesses the model’s contextual awareness and
robustness in extracting relevant features from the input text, allowing it to
discern important patterns and information crucial for accurate
classification. We used three different datasets for evaluating this task:
Product Reviews212121hf.co/datasets/turkish_product_reviews,
TTC4900222222kaggle.com/savasy/ttc4900 Yıldırım and Yıldız (2018), and Tweet
Sentiments Amasyali et al. (2018).

###  5.2 Evaluation Metrics

We evaluated the generation tasks with ROUGE Lin (2004), BLEU Papineni et al.
(2002) and METEOR Banerjee and Lavie (2005) metrics. For the understanding
tasks, we adopted standard classification metrics such as accuracy, precision,
recall, and F1. The only exception was semantic textual similarity, a
regression task, for which we used the Pearson correlation coefficient for
evaluation. For NLI and classification tasks, weighted precision, recall and
F1 were reported, leaving out accuracy due to its equality to weighted recall.

###  5.3 Results

####  5.3.1 Generation Tasks

We evaluated Turna’s generative capabilities on three tasks and compared the
results to mT5 and mBART. The results, as presented in Table 2, show that
Turna outperformed the baseline models in both paraphrasing and summarization,
with mT5 ranking second and mBART last. In title generation, Turna performed
the best on the TRNews dataset, followed by mBART. However, for the MLSUM
dataset, mBART outperformed both Turna and mT5.

Table 2: Downstream performance of models on generation tasks.

Task | Dataset | Model | Rouge1 | Rouge2 | RougeL | BLEU | METEOR
---|---|---|---|---|---|---|---
Paraphrasing | OST | mBART | $76.86$ | $61.34$ | $75.18$ | $48.85$ | $72.61$
mT5 | $77.49$ | $62.15$ | $75.87$ | $49.66$ | $73.61$
TURNA | $78.43$ | $63.58$ | $76.81$ | $51.47$ | $74.79$
TAT | mBART | $82.77$ | $68.68$ | $81.31$ | $55.57$ | $77.34$
mT5 | $88.76$ | $77.75$ | $87.51$ | $67.80$ | $85.58$
TURNA | $90.22$ | $80.23$ | $88.95$ | $71.14$ | $87.56$
Summarization | MLSUM | mBART | $41.39$ | $27.63$ | $35.61$ | $19.66$ | $32.30$
mT5 | $43.43$ | $29.95$ | $37.71$ | $21.58$ | $34.20$
TURNA | $44.33$ | $30.99$ | $38.62$ | $24.25$ | $36.47$
TRNews | mBART | $39.96$ | $25.53$ | $34.90$ | $16.69$ | $32.23$
mT5 | $41.46$ | $27.47$ | $36.60$ | $18.31$ | $34.48$
TURNA | $41.77$ | $27.81$ | $36.99$ | $19.05$ | $34.61$
Title Generation | MLSUM | mBART | $32.97$ | $19.71$ | $31.32$ | $7.41$ | $18.29$
mT5 | $32.60$ | $19.65$ | $30.93$ | $7.15$ | $17.75$
TURNA | $32.67$ | $19.60$ | $31.12$ | $7.08$ | $17.90$
TRNews | mBART | $35.40$ | $21.92$ | $34.32$ | $11.95$ | $23.26$
mT5 | $34.84$ | $21.62$ | $33.85$ | $11.96$ | $22.40$
TURNA | $36.47$ | $22.88$ | $35.47$ | $12.64$ | $23.62$

####  5.3.2 Understanding Tasks

In assessing understanding tasks, we compared both encoder-decoder models
fine-tuned with the standard text-to-text formulation and encoder-only models,
such as Turna-Encoder and BERTurk. Turna achieved results that surpass both
mT5 and mBART across various tasks and datasets, as detailed in Tables 3, 4,
and 5, reporting POS tagging & NER, NLI, and classification results,
respectively. Turna outperformed mBART and mT5 in all classification, NLI,
STS, POS tagging and NER tasks, except for the Milliyet (NER) dataset. While
Turna slightly lagged behind BERTurk on some tasks, this was not surprising as
encoder-decoder models often struggle with understanding tasks Lewis et al.
(2020); Kementchedjhieva and Chalkidis (2023). However, Turna-Encoder
surpassed berturk in NER, NLI and some classification tasks, and was
competitive in others. The notable exception was the semantic textual
similarity task (Table 6), where Turna-Encoder significantly lagged behind
BERTurk. This suggests that further hyperparameter tuning could improve
performance, as evidenced by an additional experiment where adjusting the
learning rate enabled Turna-Encoder to achieve a a significantly higher
Pearson correlation score in the STS task (refer to Table 11 in the Appendix).

Table 3: Downstream performance of models on POS tagging and NER.

Task | Dataset | Model | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---|---
POS | BOUN | mBART | $88.15$ | $87.75$ | $87.95$ | $87.75$
mT5 | $90.90$ | $90.74$ | $90.82$ | $90.74$
TURNA | $92.39$ | $92.35$ | $92.37$ | $92.35$
BERTurk | $90.60$ | $90.41$ | $90.50$ | $93.22$
TURNA-Encoder | $90.30$ | $90.31$ | $90.31$ | $93.05$
IMST | mBART | $77.68$ | $77.40$ | $77.54$ | $77.39$
mT5 | $93.17$ | $93.05$ | $93.11$ | $93.04$
TURNA | $94.66$ | $94.48$ | $94.57$ | $94.48$
BERTurk | $94.28$ | $94.14$ | $94.21$ | $95.62$
TURNA-Encoder | $93.34$ | $93.27$ | $93.31$ | $94.91$
NER | Milliyet | mBART | $87.62$ | $70.67$ | $78.23$ | $98.11$
mT5 | $84.73$ | $71.98$ | $77.83$ | $98.20$
TURNA | $91.36$ | $83.28$ | $87.13$ | $97.91$
BERTurk | $93.51$ | $94.84$ | $94.17$ | $99.24$
TURNA-Encoder | $95.16$ | $96.03$ | $95.59$ | $99.46$
WikiANN | mBART | $90.76$ | $89.12$ | $89.93$ | $95.84$
mT5 | $90.50$ | $89.90$ | $90.20$ | $95.93$
TURNA | $90.48$ | $90.20$ | $90.34$ | $96.18$
BERTurk | $89.83$ | $90.41$ | $90.12$ | $96.53$
TURNA-Encoder | $91.08$ | $92.01$ | $91.54$ | $97.08$

Table 4: Downstream performance of models on natural language inference (NLI).

Model | Precision | Recall | F1
---|---|---|---
mBART | $86.14$ | $86.06$ | $86.08$
mT5 | $83.67$ | $83.66$ | $83.66$
TURNA | $86.20$ | $86.19$ | $86.19$
BERTurk | $86.94$ | $86.88$ | $86.90$
TURNA-Encoder | $88.28$ | $88.30$ | $88.28$

Table 5: Downstream performance of models on text classification.

Dataset | Model | Precision | Recall | F1
---|---|---|---|---
Product Reviews | mBART | $87.67$ | $93.63$ | $90.55$
mT5 | $93.01$ | $94.17$ | $93.27$
TURNA | $94.67$ | $95.24$ | $94.81$
BERTurk | $94.90$ | $95.44$ | $94.70$
TURNA-Encoder | $95.57$ | $95.92$ | $95.67$
TTC4900 | mBART | $78.23$ | $71.81$ | $73.08$
mT5 | $67.52$ | $66.74$ | $66.80$
TURNA | $89.15$ | $88.11$ | $88.16$
BERTurk | $91.97$ | $91.85$ | $91.88$
TURNA-Encoder | $91.05$ | $90.53$ | $90.52$
Tweet Sentiment | mBART | $74.07$ | $71.85$ | $72.25$
mT5 | $68.20$ | $67.45$ | $66.71$
TURNA | $74.58$ | $73.78$ | $73.94$
BERTurk | $75.91$ | $75.20$ | $74.79$
TURNA-Encoder | $77.08$ | $76.82$ | $76.76$

Table 6: Downstream performance of models on semantic textual similarity
(STS).

Model | Pearson
---|---
mBART | $66.95$
mT5 | $59.40$
TURNA | $78.74$
BERTurk | $82.60$
TURNA-Encoder | $73.63$

##  6 Conclusion

In this study, we introduced Turna, a new Turkish language model that adopts
an encoder-decoder architecture following the UL2 framework. This model was
pretrained on a broad corpus covering web data, scientific articles, theses,
books, creative writing, and parliament corpora. Our comprehensive evaluations
across three generation and five understanding tasks on 13 different datasets
showed that Turna outperforms existing multilingual models, mT5 and mBART, and
performs better than or on par with the Turkish encoder-only model, berturk.
To encourage further research and facilitate benchmarking in Turkish NLP, we
have made our models and the entire source code for data collection,
filtering, model training, and fine-tuning publicly accessible.

## Limitations

Turna, with its 1.1B parameters, excels in a variety of NLP tasks, surpassing
similar-scale multilingual models like mT5 (1.2B) and mBART (610M) in both
generation and understanding. However, its efficiency, especially in
understanding tasks, is closely matched by the smaller, encoder-only model
berturk, which has only 110M parameters. This suggests that the scale-to-
performance ratio of Turna may not be as efficient as expected.

Addressing this, we modified Turna into Turna-Encoder by removing the decoder
and adding task-specific heads, which improved its efficiency. TURNA-Encoder,
having half the parameters of Turna, surpassed berturk in most tasks, thereby
improving its efficiency. However, the comparison with berturk indicates a
need for additional pretraining to fully leverage Turna’s larger parameter
count.

Current research on scaling laws indicates that training models for up to four
epochs can be beneficial Taylor et al. (2022); Muennighoff et al. (2023).
Despite having 1.1B parameters, Turna has been trained with approximately 43B
tokens, which is roughly equivalent to one epoch. This under-training might be
limiting its potential. Therefore, we suggest further pretraining of Turna to
enhance its performance.

In our downstream evaluations, we used the same optimization hyperparameters
across all tasks and datasets due to limited computational resources. This
approach may have influenced performance as datasets carry differing sizes and
tasks exhibit different difficulties. Hence, we suggest dataset and task-
specific hyperparameter tuning to thoroughly demonstrate the capabilities of
our model in downstream tasks.

## Acknowledgments

We thank the Google TPU Research Cloud program for providing us with credits
to pretrain our model on TPU v3-8 machines. We are grateful to TETAM and BOUN
CMPE for providing access to the GPU cluster used in fine-tuning and
evaluation experiments. We also thank VNGRS-AI team for providing the
tokenizer and the cleaned web corpus.

## References

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a Cleaner Document-Oriented Multilingual Crawled Corpus.  _arXiv e-prints_ , page arXiv:2201.06642.
  * Alkurdi et al. (2022) Besher Alkurdi, Hasan Yunus Sarioglu, and Mehmet Fatih Amasyali. 2022.  Semantic similarity based filtering for Turkish paraphrase dataset creation.  In _Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022)_ , pages 119–127, Trento, Italy. Association for Computational Linguistics.
  * Amasyali et al. (2018) Mehmet Fatih Amasyali, Hakan Tasköprü, and Kübra Çaliskan. 2018.  Words, meanings, characters in sentiment analysis.  In _2018 Innovations in Intelligent Systems and Applications Conference (ASYU)_ , pages 1–6. IEEE.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Baumann (2019) Antonia Baumann. 2019.  Multilingual language models for named entity recognition in German and English.  In _Proceedings of the Student Research Workshop Associated with RANLP 2019_ , pages 21–27, Varna, Bulgaria. INCOMA Ltd.
  * Baykara and Güngör (2022) Batuhan Baykara and Tunga Güngör. 2022.  Abstractive text summarization and new large-scale datasets for agglutinative languages turkish and hungarian.  _Language Resources and Evaluation_ , 56(3):973–1007.
  * Beken Fikri et al. (2021) Figen Beken Fikri, Kemal Oflazer, and Berrin Yanikoglu. 2021.  Semantic similarity based evaluation for abstractive news summarization.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 24–33, Online. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Budur et al. (2020) Emrah Budur, Rıza Özçelik, Tunga Gungor, and Christopher Potts. 2020.  Data and Representation for Turkish Natural Language Inference.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8253–8267, Online. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  _Advances in neural information processing systems_ , 32.
  * He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_.
  * Heafield (2011) Kenneth Heafield. 2011.  KenLM: Faster and smaller language model queries.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2020) Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. 2020.  Convbert: Improving bert with span-based dynamic convolution.  _Advances in Neural Information Processing Systems_ , 33:12837–12848.
  * Kementchedjhieva and Chalkidis (2023) Yova Kementchedjhieva and Ilias Chalkidis. 2023.  An exploration of encoder-decoder approaches to multi-label classification for legal and biomedical text.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5828–5843, Toronto, Canada. Association for Computational Linguistics.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lison et al. (2018) Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018.  OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Marşan et al. (2023) Büşra Marşan, Salih Furkan Akkurt, Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Gözde Berk, Seyyit Talha Bedir, Abdullatif Köksal, Balkız Öztürk Başaran, Tunga Güngör, and Arzucan Özgür. 2023.  Ud turkish boun.  Universal Dependencies.  Accessed: 2023-11-14.
  * Muennighoff et al. (2023) Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023.  Scaling data-constrained language models.  _arXiv preprint arXiv:2305.16264_.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Nozza et al. (2020) Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020.  What the [mask]? making sense of language-specific bert models.  _arXiv preprint arXiv:2003.02912_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  pages 311–318.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021.  How good is your tokenizer? on the monolingual performance of multilingual language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3118–3135, Online. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schweter (2020) Stefan Schweter. 2020.  Berturk - bert models for turkish.
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067, Online. Association for Computational Linguistics.
  * Shazeer (2020) Noam Shazeer. 2020.  Glu variants improve transformer.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4596–4604. PMLR.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mgpt: Few-shot learners go multilingual.  _arXiv preprint arXiv:2204.07580_.
  * Sulubacak and Eryiğit (2018) Umut Sulubacak and Gülşen Eryiğit. 2018.  Implementing universal dependency, morphology, and multiword expression annotation standards for turkish language processing.  _Turkish Journal of Electrical Engineering and Computer Sciences_ , 26(3):1662–1672.
  * Tanvir et al. (2021) Hasan Tanvir, Claudia Kittask, Sandra Eiche, and Kairit Sirts. 2021.  EstBERT: A pretrained language-specific BERT for Estonian.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 11–19, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Tay et al. (2021) Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021.  Scale efficiently: Insights from pre-training and fine-tuning transformers.  _arXiv preprint arXiv:2109.10686_.
  * Tay et al. (2023) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2023.  Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Tür et al. (2003) Gökhan Tür, Dilek Hakkani-Tür, and Kemal Oflazer. 2003.  A statistical information extraction system for turkish.  _Natural Language Engineering_ , 9(2):181–210.
  * Türk et al. (2023) Utku Türk, Şaziye Betül Özateş, Büşra Marşan, Salih Furkan Akkurt, Çağrı Çöltekin, Gülşen Cebiroğlu Eryiğit, Memduh Gökırmak, Hüner Kaşıkara, Umut Sulubacak, and Francis Tyers. 2023.  Ud turkish imst.  Universal Dependencies.  Accessed: 2023-11-14.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Virtanen et al. (2019) Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019.  Multilingual is not enough: Bert for finnish.  _arXiv preprint arXiv:1912.07076_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yıldırım and Yıldız (2018) Savaş Yıldırım and Tuğba Yıldız. 2018.  A comparative analysis of text classification for turkish language.  _Pamukkale Univ Muh Bilim Derg_ , 24(5):879–886.  Doi: 10.5505/pajes.2018.15931.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.

##  Appendix A Appendix

###  A.1 Cleaning Procedure for Scientific Corpus

Initially, we replaced invalid or misinterpreted characters resulting from
Optical Character Recognition (OCR) errors, employing a predefined dictionary.
Subsequently, we omitted preliminary text appearing before the abstract, which
typically contains non-essential information such as affiliations and article
metadata. This was achieved using regular expressions tailored for this
purpose. While this approach was sufficient for scientific articles, the
theses posed additional challenges, including sections like lists of figures,
tables, and customary declarations. To handle these sections, we relied on
regular expressions designed to identify and subsequently discard specific
titles and their accompanying content.

In our effort to maintain the quality of the extracted text from the PDF
articles, we also implemented a line-wise filtering procedure involving the
steps below:

  * •

Text Statistics: Each line from the articles was analyzed based on various
statistics. These included character count, token count, numeric content,
average token length, and metrics reflecting the prevalence of numbers,
specifically the proportion of numeric tokens to total tokens and frequency of
digit appearances. This stage ensured the removal of non-content elements,
such as headers, page numbers, and table items.

  * •

Language Identification and Correction: Given the potential presence of non-
Turkish lines within the articles, each line was checked for its Turkish
content using the langid library232323github.com/saffsd/langid.py. In cases of
potential anomalies or false detections, the surrounding lines were examined
to correct such anomalies, ensuring that the majority of our extracted content
is in Turkish.

  * •

Content Identification: Although article metadata typically appears at the
beginning of the documents, they may also appear elsewhere. To identify such
elements as dates, email addresses, and names, each line was checked using
specific regular expressions. Additionally, captions, identified by their
distinct patterns, were detected and subsequently removed.

  * •

Identification and Filtering of Special Sections: In scientific texts, certain
lines—like those in bibliographies and footnotes—may not contribute essential
content, or they may even disrupt the primary narrative. To address this, we
implemented strategies to detect and subsequently omit such lines. This step
ensured the retention of the text’s coherence and continuity.

  * •

Citation Filtering: Citations, while crucial to academic papers, can interrupt
text flow, especially when preparing data for language model training. We thus
used patterns to identify and remove inline citations, guaranteeing a smooth
textual flow.

After the line-wise filtering procedure was complete, we applied document-
based filtering with the help of a Statistical Language Model (LM) trained on
a compilation of May 2023 Turkish Wikipedia
articles242424hf.co/datasets/musabg/wikipedia-tr. A KenLM 5-gram language
model was trained Heafield (2011) on 6.8M sentences tokenized with a Turkish
SentencePiece tokenizer252525github.com/vngrs-ai/vnlp/tree/main/
vnlp/turkish_word_embeddings. The KenLM model was then used to discard
documents defined by separate thresholds for the Dergipark articles (less than
$5$% LM score) and the YökTez theses (less than $2$% LM score). The thresholds
have been selected by native Turkish speakers by analyzing the distribution of
documents and their qualities based on document-based average LM score.

###  A.2 Fine-tuning Datasets

##### OST Alkurdi et al. (2022)

OST is a paraphrasing dataset, constructed by translating English subtitles
from OpenSubtitles2018 Lison et al. (2018) into Turkish. The original
subtitles and their translations were preprocessed to create an unfiltered
version of the dataset with $1,944,955$ pairs. These pairs were then filtered
based on semantic similarity, resulting in a filtered version of the dataset
with $706,488$ pairs.

##### TAT Alkurdi et al. (2022)

TAT is another paraphrasing dataset created using the same methodology as OST.
The initial parallel corpus originates from Tatoeba262626tatoeba.org. The
unfiltered and filtered versions of the dataset include $265,203$ and $50,423$
pairs, respectively.

##### TR-News Baykara and Güngör (2022)

TR-News is a collection of news articles along with corresponding summaries
and titles covering a wide range of topics. It is compiled from three Turkish
national news outlets: Cumhuriyet, NTV, and HaberTürk. The dataset consists of
approximately 307K articles, split into $277,573$ train, $14,610$ validation,
and $15,379$ test documents.

##### MLSUM Scialom et al. (2020)

MLSUM is a large-scale, multilingual summarization dataset that includes
Turkish articles. The Turkish subset contains $273,617$ articles from
InternetHaber, further divided into $259,277$ train, $11,565$ validation, and
$12,755$ test documents.

##### WikiANN Rahimi et al. (2019)

WikiANN is a multilingual named entity recognition dataset containing
instances from Wikipedia articles annotated with tags of “location”, “person”,
and “organization”. The Turkish subset of the dataset includes $40,000$ rows,
split into $20,000$ for training, $10,000$ for validation, and $10,000$ for
testing.

##### MilliyetNER Tür et al. (2003)

Milliyet NER is a named entity recognition dataset that includes instances
from Turkish news articles annotated with tags of “location”, “person”, and
“organization”. The dataset comprises $515,123$ words, divided into a training
set of $419,996$, a validation set of $45,532$ and a test set of $49,595$
words.

##### UD Turkish IMST Türk et al. (2023)

The IMST-UD Treebank is a Turkish dependency treebank in the format of the
Universal Dependencies (UD) framework Sulubacak and Eryiğit (2018). The
treebank was annotated manually in a format other than UD, and then
automatically converted for the UD version v1.3 to be the first Turkish UD
treebank. It has since then received various updates and corrections. The
latest version, v2.13, has $56,422$ tokens in total, with $36,415$ tokens for
training, $10,257$ for validation, and $9,750$ for testing.

##### UD Turkish BOUN Marşan et al. (2023)

The BOUN treebank is another Turkish dependency treebank that has been a part
of the UD project since v2.7. Since then, it has received a few updates with
corrections. The latest version, v2.13, has $121,835$ tokens in total, with
$97,797$ tokens for training, $12,023$ for validation, and $12,015$ for
testing.

##### STSb-TR Beken Fikri et al. (2021)

STSb-TR is derived from the English Semantic Textual Similarity benchmark
(STSb) dataset Cer et al. (2017) by translating the English sentences into
Turkish using Google Translate, with no manual corrections. Each data element
has two sentences and a corresponding similarity score. The dataset contains
$5,749$ training, $1,500$ validation and $1,379$ test samples.

##### NLI-TR Budur et al. (2020)

The Natural Language Inference in Turkish (NLI-TR) dataset consists of two
large-scale datasets containing pairs of sentences labeled as “entailment”,
“contradiction”, or “neutral”. These sentence pairs were obtained by
translating the widely used NLI corpora, made up of SNLI Bowman et al. (2015)
and MultiNLI Williams et al. (2018). The SNLI dataset includes 570K samples,
with 550K for training, 10K for validation, and 10K for testing. The MultiNLI
dataset contains 413K samples, with 393K for training and 20K for validation,
evenly divided between matched and mismatched pairs.

##### Product Reviews

The Turkish Product Reviews is a sentiment classification dataset that
contains product reviews from various online sources, and is available on
Hugging Face272727hf.co/datasets/turkish_product_reviews. A total of $235,165$
reviews are categorized as positive or negative. We deduplicated the dataset
before usage, and split it with an 80-10-10 train-validation-test ratio. The
resulting dataset contains $186,806$ training, $23,351$ validation and
$23,351$ test samples.

##### TTC4900 Yıldırım and Yıldız (2018)

The dataset is made available by the Kemik NLP Group282828kemik.yildiz.edu.tr,
and contains $4,900$ news articles and texts classified with one of seven
categories: economy, culture-arts, health, politics, sports, technology and
world. The dataset is available on Kaggle292929kaggle.com/savasy/ttc4900 and
Hugging Face303030hf.co/datasets/ttc4900. The TTC4900 data was also
deduplicated before fine-tuning, and split with an 80-10-10 ratio, leaving
$3,631$ samples for training, and $454$ samples each for test and validation.

##### Tweet Sentiments Amasyali et al. (2018)

Tweet Sentiments is a sentiment classification dataset with three categories:
positive, negative and neutral. The dataset consists of $17,289$ tweets that
contain comments about a GSM operator, split into $13,832$ training and
$3,457$ test samples. Due to lack of a validation set, the training set was
split with a 90-10 train-validation ratio. After deduplication, the resulting
fine-tuning dataset contains $12,421$ training, $1,381$ validation and $3,456$
test samples.

###  A.3 Fine-tuning details

##### Data splits.

We used predefined splits for datasets, including training, validation, and
test sets. For datasets lacking both validation and test sets, we divided the
data into training, validation, and test sets with an 80-10-10 ratio. In the
absence of the validation set only, we utilized $10$% of the original training
data to generate a validation set, while the remaining $90$% was used for
training. We used the same approach for datasets that lacked a test set. For
the NLI task, we fine-tuned our model on the training set referred to as NLI-
TR Budur et al. (2020), which is the combination of the training sets of SNLI-
TR and MultiNLI-TR, and we used the already existing test and validation sets
of the SNLI-TR dataset.

##### Dataset-specific parameters.

Considering the varying lengths of dataset samples, we used dataset-specific
parameters. These parameters set the maximum input and target lengths, and
batch size to fit into the largest batch. In order to speed up the fine-tuning
process, we employed bf16 mixed precision in the summarization and title
generation experiments, allowing for a larger batch size. Table 7 shows the
hyperparameters used for fine-tuning.

Table 7: Dataset-specific hyperparameters for fine-tuning Task | Dataset | Max Input Length | Max Target Length | Batch Size
---|---|---|---|---
Summarization | TRNews | $768$ | $128$ | $4$
MLSUM | $768$ | $128$ | $4$
Title Generation | TRNews | $256$ | $64$ | $8$
MLSUM | $256$ | $64$ | $8$
Paraphrasing | Tatoeba | $20$ | $20$ | $128$
OpenSubtitles | $20$ | $20$ | $128$
NER | WikiANN | $60$ | $40$ | $64$
MilliyetNER | $380$ | $60$ | $8$
POS | BOUN | $90$ | $300$ | $8$
IMST | $60$ | $210$ | $16$
NLI | NLI-TR | $128$ | $8$ | $32$
Classification | Product Reviews | $20$ | $4$ | $32$
TTC4900 | $1,450$ | $8$ | $2$
Tweet Sentiment | $160$ | $4$ | $32$
STS | STSb-TR | $140$ | $10$ | $32$

###  A.4 Mode-Switching

In the UL2 framework, specific sentinel tokens are dedicated to different
pretraining objectives, enabling the model to adjust its mode for optimal task
performance. This approach is also applied to fine-tuning and few-shot
learning by using a token tailored to the needs of the downstream task, such
as [S2S] for generation tasks. This is known as mode switching.

We tested mode switching by fine-tuning TURNA on several tasks and datasets.
The results, detailed in Tables 8, 9, and 10, showed that TURNA models fine-
tuned without any sentinel token scored highest on paraphrasing evaluations.
However, a separate sentinel token achieved the best scores on different
classification datasets, with the scores being remarkably close. In the
semantic textual similarity task, the model trained with the [NLG] token
performed the best.

We found no consistent pattern in the performance of different tokens across
various tasks and datasets. This suggests that mode-switching might not always
enhance performance, and could potentially degrade it.

Table 8: Comparison of mode switching modes on the paraphrasing task.

Dataset | Mode | Rouge1 | Rouge2 | RougeL | BLEU | METEOR
---|---|---|---|---|---|---
OST | - | $78.43$ | $63.58$ | $76.81$ | $51.47$ | $74.79$
[NLG] | $76.20$ | $61.11$ | $74.50$ | $46.27$ | $73.76$
[NLU] | $77.18$ | $61.97$ | $75.33$ | $48.39$ | $74.02$
[S2S] | $77.20$ | $61.98$ | $75.44$ | $48.53$ | $74.05$
TAT | - | $90.22$ | $80.23$ | $88.95$ | $71.14$ | $87.56$
[NLG] | $89.66$ | $79.28$ | $88.41$ | $69.54$ | $87.18$
[NLU] | $89.08$ | $78.53$ | $87.90$ | $68.33$ | $86.82$
[S2S] | $89.71$ | $79.37$ | $88.45$ | $69.61$ | $87.26$

Table 9: Comparison of mode switching modes on the text classification task.

Dataset | Mode | Precision | Recall | F1
---|---|---|---|---
Product Reviews | - | $94.67$ | $95.24$ | $94.81$
[NLG] | $94.30$ | $95.03$ | $94.39$
[NLU] | $94.45$ | $95.10$ | $94.60$
[S2S] | $94.34$ | $95.04$ | $94.47$
TTC4900 | - | $89.15$ | $88.11$ | $88.16$
[NLG] | $89.50$ | $88.33$ | $88.39$
[NLU] | $86.18$ | $84.14$ | $84.31$
[S2S] | $90.83$ | $90.31$ | $90.24$
Tweet Sentiment | - | $74.58$ | $73.78$ | $73.94$
[NLG] | $76.01$ | $75.84$ | $75.56$
[NLU] | $75.45$ | $75.46$ | $75.45$
[S2S] | $75.55$ | $74.91$ | $74.86$

Table 10: Comparison of mode switching modes on semantic textual similarity
(STS).

Mode | Pearson
---|---
- | $78.74$
[NLG] | $79.71$
[NLU] | $78.45$
[S2S] | $78.30$

###  A.5 Hyperparameter Tuning

We conducted an additional experiment on the Semantic Textual Similarity task
due to the low Pearson correlation score obtained by TURNA-Encoder when
compared to TURNA and BERTurk. We fine-tuned TURNA-Encoder with different
learning rates on the regression task. The results are reported in Table 11.
The difference in Pearson correlation scores suggest that elaborate
hyperparameter tuning can significantly alter the downstream performance of
our model.

Table 11: Comparison of TURNA-Encoder performance with different learning
rates on semantic textual similarity (STS).

Learning Rate | Pearson
---|---
(Default) $5\text{\times}{10}^{-5}$ | $73.63$
$5\text{\times}{10}^{-4}$ | $77.13$
$5\text{\times}{10}^{-3}$ | $-3.56$
$5\text{\times}{10}^{-2}$ | $17.92$


For example if $C$ is horizontal then the elements of $G_C$ are of the form $\left (\begin{matrix} 1 & s \\ 0 & t\end{matrix} \right)$, with $t>0$. The cylinder surgery of such a matrix with $t=1$ consists of cylinder shears with shear parameter $s$. The cylinder surgery with $s=0$ consists of cylinder stretches with stretch parameter $t$. By an appropriate conjugation, the definition of cylinder shears and stretches is extended to non-horizontal cylinders.

We have:

For $\M$, any $q \in \M$, and an $\M$-parallel equivalence class of cylinders $C_1, \ldots, C_r$ on $M_q$, if $g \in G_{C_i}$ then the surface  obtained from $M_q$ by cylinder surgery corresponding to $g, C_1, \ldots, C_r$ is also in $\M$.

Suppose $q \in \M$ and $C_1, \ldots, C_r $ are $\M$-parallel cylinders on $M_q$, which are not necessarily a full equivalence class of $\M$-parallel cylinders.
Let $L$ be a lift of $\M$, let $\q \in L \cap \pi^{-1}(q)$ and let $V \subset H^1 \left( S, \Sigma; \C \right)$ such that $\dev(L)=V.$
Varying $g \in G_{C_1}$ gives rise to a two dimensional collection (in the previous example, corresponding to possible choices of the parameters $s,t$) of surfaces, obtained from $M_q$
by cylinder surgery corresponding to $g, C_1, \ldots, C_r$. This collection corresponds to a complex affine line in period coordinates. A generator for this complex line is

\begin{equation}\label{eq: sigma defn}
\sigma_{\{C_i, h_i\}}
\df \sum_{i=1}^r h_i \gamma_i^*,
\end{equation}

where $h_i$ is the height of $C_i$, $\gamma_i$ is the core curve of $C_i$  and $\gamma_i^*$ is the dual class to $\gamma_i$ in $H^1(S, \Sigma).$ Moving along the line tangent to $\sigma_{\{C_i, h_i\}}$ in $\M $ amounts to performing cylinder shears in each of the $C_i$, and moving along the line tangent to $\mathbf{i} \cdot \sigma_{\{C_i, h_i\}}$ in $\M $ amounts to performing cylinder stretches.

Below we will be interested in such one-parameter families of deformations that preserve $\M$ and are tangent to $\sigma_{\{C_i, h_i\}} $ as in equation (<ref>), in which the $\{C_i\}$ might not be a full equivalence class of $\M$-parallel cylinders, and the $h_i$ might not be their heights. Note that for cylinder shears, such surgeries are well-defined for any value of the shear parameter $s$, and for cylinder stretches, they are well-defined as long as $t>0$.

When $\sigma_{\{C_i, h_i\}} \in V$ for $\q, L, V $ as above we simply say that $\sigma_{\{C_i, h_i\}}$ is contained in the tangent space to $\M$ at $q$.

If $\M$ is an invariant subvariety and $q \in \M$ is not horizontally periodic, then there is a nonempty collection of $\M$-parallel cylinders $C_1, \ldots, C_r$ on $M_q$ which consists of cylinders disjoint from all horizontal saddle connections on $M_q$,
and positive $h_1, \ldots, h_r$ such that the class $\sigma_{\{C_i, h_i\}}$ as in equation (<ref>) is contained in the tangent space to $\M$ at $q.$
Furthermore, there is constant $A_0>0$, depending only on $\M$, so that if $q \in \M$ has no horizontal cylinders then one can choose a collection of cylinders with these properties so that, in addition, the sum of the areas of the cylinders is at least $A_0$.

We note that the $h_i$ in Proposition <ref> might not be the heights of the $C_i.$ The vector $\sigma_{\{C_i, h_i\}}$ is tangent to the line in $\M$ obtained by applying different cylinders shears to each $C_i$. In order to formalize this,
for $g_i \in G_{C_i}$ for each $i$, we define the cylinder surgery corresponding to $\{C_i, g_i\}$ to be the modification of $M_q$ obtained by applying $g_i$ to each $C_i$, leaving the complement $M \sm \bigcup_{i=1}^r C_i$ untouched. With this terminology,
the line tangent to $\sigma_{\{C_i, h_i\}}$ is the collection of surfaces obtained by cylinder surgery corresponding to $\{C_i, g_i\},$ where $s \in \R$ and  $g_i $ performs a cylinder shear with parameter $sh_i$ in $C_i$. Similarly, the line tangent to $\mathbf{i} \cdot \sigma_{\{C_i, h_i\}}$ is the collection of surfaces obtained by cylinder surgery corresponding to $\{C_i, g_i\},$ where $s \in \R$ and  $g_i $ performs a cylinder stretch with parameter $sh_i$ in $C_i$.

The validity of both statements is unchanged if we replace the surface $q$ with some surface $q'$  in its horocycle orbit $Uq$. This follows from the facts that the $U$-action is linear in charts,  preserves horizontal saddle connections and maps $\M$-equivalent cylinders to $\M$-equivalent cylinders preserving their area. According to
<cit.>, there is $q_\infty$ in the closure of $Uq$ which is horizontally periodic, and we will see that the required properties hold for all $q'$ sufficiently close to $q_\infty$. Let $M_\infty$ be the underlying surface of ${q_\infty}$, let $C_1, \ldots, C_r$ be the horizontal cylinders on $M_\infty$, and for each $i$, $A_i, c_i, h_i, \gamma_i$ denote respectively the area, circumference, height, and core curve of $C_i$. Here we consider $\gamma_i$ as an element of $H_1(S, \Sigma) $ by using a marking $f: S \to M_\infty$ corresponding to $\q_\infty \in \pi^{-1}(q_\infty).$ By Proposition <ref>, $\sigma_{\{C_i, h_i\}}$ belongs to the tangent space of $\M$ at $q.$

For any $\theta_0>0 $ there is a neighborhood $\mathcal{U} = \mathcal{U}(\theta_0)$ of $q_\infty$ in $\M$ such that  if $q' \in \mathcal{U}$ then the underlying surface has $r$ cylinders $C'_1, \ldots, C'_r$ of circumferences $c'_i$ and areas $A'_i$ satisfying

\begin{equation}\label{eq: the bounds}
c'_i < \bar c \df  %\frac12 \min_{i=1, \ldots, r}
2\max_{i=1, \ldots, r} c_i, \ \ \ \ A'_i
> \underline{A} \df \frac12 \min_{i=1, \ldots, r} A_i ,
\end{equation}

and with directions of core curves in $(-\theta_0, \theta_0)$.

If $C$ is a cylinder and $\sigma$ is a saddle connection on a translation surface, recall that we say that $\sigma$ crosses $C$ if it intersects all the core curves of $C$. Since a cylinder contains no singularities in its interior, if a saddle connection intersects the interior of a cylinder, then it must cross it. Let $s$  be the maximal length of a horizontal saddle connection on $M_q$ and let $\theta_0$ be small enough so that a horizontal segment of length $s$ cannot cross a cylinder of direction $\theta $ satisfying $0 < |\theta| < \theta_0$, with circumference at most $\bar c$ and area at least $\underline{A}$.

By making $\mathcal{U}$ smaller, so that it is an evenly covered neighborhood of $q_\infty$, we can ensure that
$\sigma_{\{C'_i, h_i\}}$ belongs to the tangent space of $\M$ at $q'.$ Indeed, if $\mathcal{V}$ is a connected component of $\pi^{-1}(\mathcal{U})$ and $\q', \q_\infty \in \mathcal{V}$ are preimages of $q', q_\infty$ respectively, then the core curves of the cylinders $C'_i, C_i$ map to the same elements $\gamma_i \in H_1(S, \Sigma)$ under the corresponding marking maps, and thus $\sigma_{\{C'_i, h_i\}} = \sigma_{\{C_i, h_i\}}$.

Now suppose that $q' \in Uq \cap \mathcal{U}$, and let $M'$ be the underlying surface. Since $q$ is not horizontally periodic, neither is $q'$. Therefore there is an equivalence class $C_1, \ldots, C_r$ of $\M$-parallel cylinders on $M_\infty$, so that the corresponding cylinders $C'_1, \ldots, C'_r$ are not horizontal cylinders on $M'$, and satisfy the bounds in equation (<ref>). Furthermore the maximal length of a horizontal saddle connection on $M'$ is $s$, since the horocycle flow maps horizontal saddle connections to horizontal saddle connections of the same length. By choice of $\theta_0$, the cylinders in this equivalence class are all disjoint from horizontal saddle connections on $M'$. This proves the first assertion.

Let $t$ be an upper bound on the number of horizontal cylinders for a surface in $\M$ and let $A_0 \df \frac{1}{2t}$. The argument above works for any collection of $\M$-parallel cylinders $C_1, \ldots, C_r$ which are horizontal on $q_\infty$ and are not horizontal on $q'$. If $q$ has no horizontal cylinders then neither does $q'$, and we can apply the argument with any equivalence class of $\M$-parallel horizontal cylinders $C_1, \ldots, C_r$ on $M_\infty$. One of these classes must have total area at least $\frac{1}{t}$, and thus for $\mathcal{U}$ sufficiently small, the sums of the areas of the corresponding cylinders $C'_1, \ldots, C'_r$ is at least $A_0$.

We first prove the first assertion. Let $C_1, \ldots, C_r$ be the $\M$-parallel cylinders on $M_q$, and $h_1, \ldots, h_r$ the positive numbers provided by Proposition <ref>.
We define $q'$ as $g \circ \varphi (q)$, where $g \in \GL$ and $\varphi$ is a cylinder surgery corresponding to $\{C_i, g_i\}$, and $g_i$ is the cylinder stretch with parameter $sh_i$ for some $s<0.$
Although the maps $g, \varphi$ do not preserve the area of the surface, we will choose parameters so that their composition does. Moreover neither of these maps changes the vertical component of the holonomy of any curve.
The $\GL$-action preserves $\M$, and by Proposition <ref>, so does $\varphi.$ The map $g$ will increase the length of all horizontal saddle connections on $M_q$, and the cylinder surgery $\varphi$ will not affect their length, since the cylinders $C_i$ are disjoint from the horizontal saddle connections on $q$.

The area of $\varphi (q)$ is smaller than the area of $q$ since all of the cylinders $C_i$ are stretched by a negative parameter. Let $A$  be the sum of the areas of the cylinders $C_1, \ldots, C_r$. Then by choosing the parameter $s$ appropriately, we can arrange so that the area of $\varphi(q)$ is $1-\frac{A}{2}.$ We now set

\begin{equation}\label{eq: def t}
t \df \left(1 -\frac{A}{2}\right)^{-1} \ \ \ \text{ and } g \df \left( \begin{matrix} t & 0 \\ 0 & 1 \end{matrix} \right).
\end{equation}

Then $g$ increases the lengths of horizontal saddle connections by a factor $t>1$, and multiplies the area of $\varphi(q)$ by $t$. This completes the proof of the first assertion.

For the second assertion, we use the second assertion in Proposition <ref> to choose the cylinders so the sum of their areas satisfies $A \geq A_0$. This ensures that the horizontal saddle connections on $q' \df g \circ \varphi (q)$ are longer than the horizontal saddle connections on $q$ by a factor of at least $t$, where $t>1$ is as in equation (<ref>) and $t-1$ is bounded away from $0$. In light of Proposition <ref>,  $q' \in W^{uu}(q)$ will not have horizontal cylinders either. So we can apply the above argument iteratively, at each stage obtaining surfaces in $W^{uu}(q)$ with longer and longer horizontal saddle connections. Since the lengths of these horizontal  saddle connections grows by a definite amount in each step, after finitely many steps they will all be longer than $T$.

§ MEASURES ON $\HH$ AND $\MOD(S,\SIGMA)$-INVARIANT MEASURES ON $\HHM$

The goal of this section is to prove a result on the correspondence between Radon measures on $\HH$ and $\Mod(S,\Sigma)$-invariant Radon measures on $\HHm$. This result is part of the folklore but we were not able to find a reference; see <cit.> for an analogous result in a restricted setting.

We state the result in a general setting.
Let $\tilde X$ be a paracompact manifold and $\Gamma$ a discrete group acting properly discontinuously on $\tilde X$. We will write the $\Gamma$-action as an action on the right.
Let $X=\tilde X/\Gamma$ be the quotient space and $\pi:\tilde X\to X$ the quotient map.
If $\Gamma$ acts freely then $X$ is a manifold and $\pi$ is a covering map. If $\Gamma$ does not act freely then we can view $X$ as an orbifold and $\pi$ as a regular orbifold covering map (although no knowledge of orbifolds is assumed in this section).
We do not assume that the action on $\Gamma$ is faithful but, since the action of $\Gamma$ is proper, the subgroup of $\Gamma$ that acts trivially on $\tilde X$ must be finite.

For $\tilde q\in\tilde X$ let $\Gamma(\tilde q)$ be the stabilizer of $\tilde q$ in $\Gamma$.
For any $q \in X$, we define a measure on $\tilde X$ by

\begin{equation}\label{eq: appendix counting measure}
\theta_q \df \sum_{\tilde q \in \pi^{-1}( q)} |\Gamma(\tilde q)|\cdot \delta_{\tilde q},
\end{equation}

where $\delta_{\tilde q}$ is the Dirac mass at $\tilde q$. The measure $\theta_{q}$ is supported on $\pi^{-1}(q)$.  For any $f \in C_c(\tilde X)$ and $\tilde q \in \tilde X$ we have

\begin{equation}\label{eq; other expression}
\int_{\tilde X} f \, d\theta_{\pi(\tilde q)} = \sum_{\gamma \in \Gamma}f(\tilde q \cdot \gamma).
\end{equation}

It follows from the fact that $\Gamma$ acts properly discontinuously on $\tilde X$ that the sum on the right-hand side is finite.

Given a Radon measure $\nu$ on $X$ we define a Radon measure $\tilde{\nu}$ on $\tilde X$, called the pre-image of $\nu$, by the formula

\begin{align}\label{liftdef1}
\int_{\tilde X} f \ d\tilde{\nu} = \int_{X} \left(\int_{\tilde X} f \ d\theta_q \right) \ d\nu(q) \ \ \text{ for any } f \in C_c(\tilde X).
\end{align}

Equation (<ref>) defines a unique Radon measure $\tilde{\nu}$ on $\tilde X$ in light of the Riesz Representation Theorem. To see that (<ref>) converges, note that the integrand $q \mapsto F(q)
\df \int_{\tilde{X}} f \ d\theta_q$ is a Borel function, which is supported on the compact set $\pi( \mathrm{supp} \, f),$ and is bounded by $D \, \| f \|_\infty,$ where $D \df \# \{\gamma \in \Gamma: (\mathrm{supp} \, f) \cdot \gamma \cap \mathrm{supp} \, f \neq \emptyset\}  $ is finite since the $\Gamma$-action is properly discontinuous.

By equation (<ref>) the measures $\theta_q$ are all $\Gamma$-invariant, and since $\tilde \nu$ is an average of the measures $\theta_q$, we have:

The measure $\tilde{\nu}$ is  $\Gamma$-invariant.

The following converse can be understood as a disintegration theorem for $\Gamma$-invariant Radon measures on $\tilde X$.

Let $m$ be a $\Gamma$-invariant Radon measure on $\tilde X$. There is a unique Radon measure $\mu$ on $X$ such that $m$ is the pre-image of $\mu$.

We call $\mu$ the image of $m$.

Let $m$ be given. We are claiming the existence of a Radon measure $\mu$ so that
equation (<ref>) holds (with $\tilde{\nu}, \nu$ replaced with $m, \mu$).

The idea of the proof is to build $\mu$ on small neighborhoods using the fact that $\pi$ is an orbifold cover. This will be made rigorous using a partition of unity. Let $\tilde q \in \tilde X$ and let $\Gamma(\tilde q)$ be the stabilizer of $\tilde q$ in $\Gamma$. Since $\Gamma$ acts properly discontinuously on $\tilde X$,  $\Gamma(\tilde q)$ is finite, and there is connected $\Gamma(\tilde q)$-invariant neighborhood $\V$ of $\tilde q$ and a neighborhood $\U$ of $\pi(\tilde q)$ such that $\pi$ induces a homeomorphism $\V / \Gamma(\tilde q) \to \U$, and

\begin{equation*}
\pi^{-1}(\U) =  \bigsqcup_{\gamma \in \Gamma(\tilde q) \backslash \Gamma} \V \cdot \gamma,
\end{equation*}

where $\gamma$ ranges over a set of coset representatives, and where the sets $\mathcal{V}\cdot \gamma$ are  disjoint.
We say that such a $\U\subset X$ is evenly covered (in the orbifold sense).

Let $(\U_i)_{i \in I}$ be a locally finite cover of $X$ by evenly covered neighborhoods.
Such a cover exists by the paracompactness of $\tilde X$ and the considerations above.
For each $i \in I$ choose a connected component $\V_i$ of $\pi^{-1}(\U_i)$. Denote by $\Gamma_i=\Gamma(\V_i)$ the stabilizer of $\V_i$ in $\Gamma$.
Let $(\rho_i)_{i\in I}$ be a partition of unity subordinate to the cover $(\U_i)_{i \in I}$ and define a Radon measure $\mu$ on $X$ (by using the Riesz Representation Theorem) such that for any $f \in C_c(X)$,

\begin{align}\label{disintegration}
\int_{X} f \ d\mu = \sum_{i \in I} \frac{1}{|\Gamma_i|} \int_{\V_i} (\rho_i f) \circ \pi \, dm.
\end{align}

We want to show that the measure $\mu$ satisfies equation (<ref>).
We claim first that for any $\tilde q \in \tilde X$, there is a neighborhood $\V$ around $\tilde q$ such that equation (<ref>) holds for any $f \in C_c(\tilde X)$ with support contained in $\V$.
Indeed, let $\tilde q \in \tilde X$ and let $\V$ be a neighborhood of $\tilde q$ small enough so that for any $i \in I$, $\V$ intersects at most one connected component of $\pi^{-1}(\U_i)$.
This is possible since the cover by the $\U_i$ is locally finite. Let $J = \{i \in I : \pi(\V) \cap \U_i \neq \emptyset \}$ and for $j \in J$, let $\gamma_j \in \Gamma$ be such that $\V_j \cdot \gamma_j \cap \V \neq \emptyset$.
By the assumption on $\mathcal{V}$, the coset  $\Gamma_j \cdot \gamma_j$ is uniquely determined.
Let $f \in C_c(\tilde X)$ with support contained in $\V$. We compute:

\begin{equation}
\begin{split}\label{computation}
\int_{X} \Big( \int_{\tilde X} f \ d\theta_q \Big) \ d\mu(q)
&= \sum_{i \in I} \frac{1}{|\Gamma_i|} \int_{\V_i} \sum_{\gamma \in \Gamma} \rho_i(\pi(\tilde q) )f(\tilde q \cdot \gamma) \ dm(\tilde q) \\
&= \sum_{j \in J} \frac{1}{|\Gamma_j|} \int_{\V_j \cdot \gamma_j} \sum_{\gamma \in \Gamma_j} \rho_j(\pi(\tilde q)) f(\tilde q \cdot \gamma) \ dm(\tilde q) \\
&= \sum_{j \in J} \frac{1}{|\Gamma_j|} \int_{\tilde X} \sum_{\gamma \in       \Gamma_j} \rho_j(\pi(\tilde q)) f(\tilde q) \ dm(\tilde q) \\
&= \sum_{j \in J} \int_{\tilde X} \rho_j(\pi(\tilde q)) f(\tilde q) \ dm(\tilde q) = \int_{\tilde{X}} f \, dm.
\end{split}
\end{equation}

Now let  $f$ be an arbitrary compactly supported continuous function and let $K$ denote its support.
Using a covering argument and the computation above, we can find finitely many $(\W_i)_i$ that cover $K$ and such that equation (<ref>) holds for continuous functions with support contained in $\W_i$. Let $(\psi_i)_i$ be a partition of unity associated with this cover.
We can write $f = \sum_i \psi_if$. By construction, each of the $\psi_if$ has support contained in $\W_i$ and the result follows by equation (<ref>) and the linearity of the integral.

To prove uniqueness of the measure $\mu$, we proceed as follows. Let $\mu_1$ and $\mu_2$ be two Radon measures on $X$ that satisfy equation (<ref>) (with $m,\mu_i $ instead of $\tilde{\nu}, \nu $).

Let $f \in C_c(X)$ be a compactly supported continuous function whose support is contained in an evenly covered neighborhood $\U$ and let $\V$ be a connected component of $\pi^{-1}(\U)$.
We denote by $\Gamma(\V)$ the stabilizer in $\Gamma$ of $\V$. Let $h$ be the function on $\tilde X$ that is equal to  $f \circ \pi$ on $\V$ and vanishes outside of $\V$.
Since the support of $f$ is contained in $\U$, the function $h$ is continuous and has compact support. Furthermore, it is easy to see that for any $q \in X$, $\int_{\tilde X} h \ d\theta_q = |\Gamma(\V)| f(q)$. This implies

\begin{align*}
\int_{X} f \ d\mu_1 = \frac{1}{|\Gamma(\V)|}  \int_{\tilde X} h \ dm =\int_{X} f \ d\mu_2.
\end{align*}

To deal with the case when $f$ is an arbitrary function of compact support we appeal once more to existence of partitions of unity.

Let $G$ be a group acting on $\tilde{X}$ so that the action commutes with the action of $\Gamma$.
The group $G$ induces an action on $X$ so that $\pi$ is $G$-equivariant.

A Radon measure $\mu$ on $X$ is invariant under $g\in G$ if and only if its pre-image $\tilde\mu$ is invariant under the action of $g$ on $\tilde X$.

We start by proving two formulas showing the naturality of the pre-image construction.

\begin{align}
g_*(\theta_q)& %=\sum_{\pi(\tilde q)=q}|\Gamma(\tilde q)|\cdot g_*(\delta_{\tilde q})\\
%           &
=\sum_{\pi(\tilde q)=q}|\Gamma(\tilde q)|\cdot \delta_{g(\tilde q)}\nonumber\\
& =\sum_{\pi(g(\tilde q))=g(q)}|\Gamma(\tilde q)|\cdot \delta_{g(\tilde q)}\label{cleq6}\\
&=\sum_{\pi(g(\tilde q))=g(q)}|\Gamma(g(\tilde q))|\cdot \delta_{g(\tilde q)}=\theta_{g(q)}. \label{cc5}
% & =\sum_{\pi(\bar q)=g(q)}|\Gamma(\bar q)|\cdot \delta_{\bar q}\label{cleq4}
\end{align}

In line (<ref>) we used the fact that $\pi(g(\tilde q))=g(\pi(\tilde q))=g(q)$. In line (<ref>) we used the fact

that the $\Gamma$ action commutes with $g$, which implies that $|\Gamma(g(\tilde q))|=|\Gamma(\tilde q)|$.

It suffices to show that both measures assign the same integrals to continuous functions of compact support on $\tilde X$. Let $f$ be such a function.

\begin{align}\nonumber
\int_{\tilde X}f\, d\widetilde{g_*(\nu)}& =\int_{X} \Big(\int_{\tilde X} f \ d\theta_q \Big) \ dg_*(\nu)(q)\\
& =\int_{X} \Big(\int_{\tilde X} f \ d\theta_{g(q)} \Big) d\nu(q) =\int_{X} \Big(\int_{\tilde X} f \ dg_*(\theta_q) \Big) d\nu(q) \label{c2eq2}\\
& =\int_{X} \Big(\int_{\tilde X} f\circ g\,  d\theta_q \Big) d\nu(q) \nonumber\\
& =\int_{\tilde X}f\circ g\, d\tilde\nu =\int_{\tilde X}f\, dg_*(\tilde\nu).  \label{c2eq5}
\end{align}

In line (<ref>) we used Claim <ref>. In line (<ref>) we used the definition of the pre-image applied to the function of compact support $f\circ g$.

Using these formulas we now prove the Proposition.
If $g_*(\mu)=\mu$ then $g_*(\tilde\mu)=\widetilde{g_*(\mu)}=\tilde\mu$ by Claim <ref>. So $\tilde\mu$  is invariant under the action of $g$ on $\tilde X$.
If $g_*(\tilde\mu)=\tilde\mu$ then $\widetilde{g_*(\mu)}=g_*(\tilde\mu)=\tilde\mu$ so the pre-images of $g_*(\mu)$ and $\mu$ are equal.
It follows from the uniqueness assertion in Proposition <ref> that $g_*(\mu)=\mu$.