RigoBERTa: A State-of-the-Art Language Model For Spanish

Alejandro Vaca Serrano1 | Guillem García Subies1 | Helena Montoro Zamorano1
---|---|---
Nuria Aldama García1 | Doaa Samy1 | David Betancur Sánchez1
Antonio Moreno Sandoval1,2 | Marta Guerrero Nieto1 | Álvaro Barbero Jiménez1,2

1Instituto de Ingeniería del Conocimiento

2Universidad Autónoma de Madrid

<EMAIL_ADDRESS>
Abstract
This paper presents RigoBERTa, a State-of-the-Art Language Model for Spanish.
RigoBERTa is trained over a well-curated corpus formed up from different
subcorpora with key features. It follows the DeBERTa architecture, which has
several advantages over other architectures of similar size as BERT or
RoBERTa. RigoBERTa performance is assessed over 13 NLU tasks in comparison
with other available Spanish language models, namely, MarIA, BERTIN and BETO.
RigoBERTa outperformed the three models in 10 out of the 13 tasks, achieving
new ”State-of-the-Art” results.

##  1 Introduction

The field of Natural Language Processing (NLP) has witnessed significant
advances in the last years, specially with the use of the Transformer
architecture, an encoder-decoder model based on the Attention Mechanism [38].
Several models, such as BERT [16], an encoder-based model useful for language
understanding tasks, or GPT-2 [31], a decoder-based model specialized on
language generation tasks, are based on this architecture. More recently,
extremely large language models such as GPT-3 [6] have pushed the State-of-
the-Art of NLP to previously unknown limits.

These language models introduce unsupervised pre-training methods that make
use of huge amounts of raw text corpora to generate not just word embeddings,
but a complete contextualization pipeline for creating general, effective and
meaningful representations of language, which is useful for solving different
NLP tasks. Due to these recent developments, the leading strategy for good
performance is to first pre-train a language model with huge amounts of
unlabeled data, and then fine-tune the language model for specific tasks with
the use of labeled data.

The most remarkable improvements in this field have been carried out in the
English language, for which there are many pre-trained, freely available
models. More recently, there has been a growing effort for applying these
techniques to other languages, such as French [25] [21] [39], Dutch [14],
German [35] [13] [11] and many others.

In spite of being one of the most spoken languages worldwide, there are not
many language models available for the Spanish language. However, in the last
year, more models have been released. Unfortunately, they are not as effective
as English language models, due to the following reasons:

  1. 1.

Scarcity of Spanish corpora of the same quality and volume as those used by
English models.

  2. 2.

Costs for pre-training large language models are substantial, in the order of
hundreds of thousands of dollars, affordable to large multinational companies.
For this reason the majority of large language models are in English, as it is
the language in which the most economic benefit can be obtained. Most language
models in other languages have been developed within the academia.

We aim to overcome those challenges by developing RigoBERTa, a State-of-the-
Art language model in Spanish. To that end, a big quality corpus was developed
and the selected model architecture achieves State-of-the-Art results faster
than preceding architectures such as BERT [16] or RoBERTa [24]. Other
improvements to previous methods were also implemented, which will be
described throughout the article, carried out by a multidisciplinary team. We
will describe the techniques used for overcoming the aforementioned
difficulties further.

The article is structured in sections. Section 2 reviews the main advances in
the field regarding the Spanish language. The methodology is described in
Sections 3 and 4. In section 5, fine-tuning experiments on 13 NLU tasks are
described. Finally, results obtained are reported and analyzed highlighting
the progress achieved by RigoBERTa over the three Spanish models (MarIA, BETO,
BERTIN).

##  2 Related Work

In this section, recent advances in language models are reviewed. The scope of
this review is limited to the Spanish language models. The first publicly
available monolingual model in Spanish was BETO [8], a Spanish BERT model. The
authors implement improvements from RoBERTa [24], such as dynamic masking and
whole-word masking, which were also applied to BERT as an update to the
original paper [16]. Only base versions of the model are released, with 110M
parameters, and a vocabulary size of 32k. Two models were published, cased and
uncased.

Both models are able to outperform multilingual BERT [16]111Multilingual BERT
[16] was trained on the top 104 languages with the largest Wikipedia, using
the same architecture and training procedure as the original BERT model for
English. on different benchmarks, thus showing that a monolingual model with
enough training can outperform a multilingual model, even when more resources
and training are used for the latter. The corpus used for training this
language model was SUC (Spanish Unnanotated Corpora) [7], which is about 18GB
of raw texts from different sources such as Spanish Wikipedia or the Spanish
portion of OpenSubtitles2018.

Training is carried out in two steps [8], the first 900k steps with batch size
of 2048 and sequence length of 128. The second consist of 1.1M steps with
batch size of 256 and maximum sequence length of 512. The authors also
contribute to the establishment of a baseline for evaluating language models
in Spanish, by selecting a variety of Natural Language Understanding (NLU)
tasks as a benchmark for Spanish models [8] similar to the English GLUE
Benchmark [40].

More recently, two more language models for Spanish have been developed. Both
of them are versions of the RoBERTa [24] architecture.

The first one, called BERTIN [33], was developed as part of the Flax/JAX
Community Event. Although they faced time and resources limitations, they were
able to develop the project without incurring in hardware costs. For BERTIN,
as mentioned, a RoBERTa architecture was chosen, with a vocabulary size of
50,265 (same as [24]).

A filtered version of MC4 is used, introduced in MT5 paper [44], a
multilingual version of T5 [4]. As the original Spanish portion of MC4 was too
big to handle for their computing resources and time, around 50M documents
were sampled from the original 400M documents forming the corpus, with a
technique called perplexity-sampling [33].

The model was trained for 250K iterations. For the first 230K steps a sequence
length of 128 was used and the last 20K steps sequence length was increased to
512, similarly to BETO [8].

MarIA [17], the latest language model released in Spanish, was developed in
the context of the _Plan de Impulso de Tecnologías del Lenguaje_
222https://plantl.mineco.gob.es/Paginas/index.aspx, funded by the Spanish
Government, in collaboration with the Barcelona SuperComputing
Center333https://www.bsc.es/ and _Biblioteca Nacional de España_
444http://www.bne.es/es/Inicio/index.html.

The corpus was 570GB of cleaned texts (see [17] for more details on their
cleaning process), bigger than any of the corpora used by the rest of the
language models in Spanish. A RoBERTa model [17] was developed, with a
vocabulary size of 50,262. The number of training steps is not specified,
albeit the model is trained for one complete epoch. The training process
lasted 48 hours with 16 computing nodes, each one with 4 NVIDIA V100 GPUs of
16GB VRAM.

Hyperparameters such as batch size or learning rate have not been reported,
and the training corpus has not been published. Although a large version of
the model was also developed, it is not considered in this work, as similar
results are reported compared to the base one, and the objective of this work
is to compare RigoBERTa against models of similar size.

Another contribution of [17] was to further extend the benchmark created first
by [8], by adding more language understanding tasks. Crucially, SQAC (Spanish
Question Answering Corpus)555https://huggingface.co/datasets/PlanTL-GOB-
ES/SQAC was introduced, the first Question Answering corpus developed
originally in Spanish666the other existing datasets of this type are mainly
built by automatic translations. This is very useful not only because we can
have a more complete overview on the models’ performance, but also because it
enables developing high quality Question Answering models in Spanish.

BETO [8], BERTIN [33], and the models presented in the paper, called MarIA
[17], were evaluated against the presented benchmark. An effective
hyperparameters space was published, easing the future task of evaluating
these models against new Spanish language models. The reported results show
that MarIA is the best performing model on a majority of tasks, while BETO [8]
performs equal or better on some tasks, and BERTIN [33] would be in third
position.

##  3 Pre-training data

As mentioned above, one of the key aspects for training a proper language
model is the corpus quality, as it contains all the information or knowledge
the model will be able to store and learn. An ideal corpus for training a
general-domain language model should be large, representative, varied and
clean, as in [24] [16] [44] [4] [18] [31].

With these basic principles in mind, several different Spanish corpora were
used, using a variety of cleaning techniques. Both of these steps are
carefully explained in the following lines.

###  3.1 OSCAR: Open Super-large Crawled ALMAnaCH coRpus

OSCAR [28] [29] is a very large multilingual corpus, obtained by language
classification and filtering of the CommonCrawl777https://commoncrawl.org/. It
has a portion of Spanish, of about 149GB of fuzzily and superficially
deduplicated texts888Obtained from https://huggingface.co/datasets/oscar.

As with the rest of the corpora, the linguists’ team analyzed some random
portions of Spanish OSCAR. This preliminary analysis releaved that some parts
of the corpus were corrupt and do not represent the actual use of Spanish
language. For example, javascript code, automatic writings, poor automatic
translations or malformed sentences, as expected from CommonCrawl corpora.

The linguistics team developed a set of quality guidelines to determine
whether a text resembles any possible human use of the language, to only
include texts for training that meet these criteria. Following these
guidelines, a 5,000 texts dataset with valid/non-valid labels was created for
later use.

###  3.2 Internal Sources

We had access to a news source through an official provider. This provided us
with 70 GB of raw texts of varying quality, according to the linguistics team.
Although most news articles seemed of good quality, other articles looked more
likely to be corrupt or exhibited a bad use of language (repetitive
meaningless texts, texts full of punctuation marks, etc.). However, being
these a minority, the corpus passed the quality check for being part of the
training data for RigoBERTa.

###  3.3 SUC: Spanish Unnanotated Corpora

SUC [7] is a corpus comprised of many different small subcorpora, taken from
the Huggingface Datasets Library [22]
999https://huggingface.co/datasets/large_spanish_corpus. It is a relatively
small corpus, as compared to the rest of the corpora used for pre-training
RigoBERTa, with only 18GB of raw texts. Moreover, most of its texts are too
short and documents are splitted in paragraphs and shuffled (this makes it
impossible to rebuild the sentences into the original documents, thus the
model could not see complete contexts.). Because of this, its already small
size was significantly diminished after the cleaning process.

###  3.4 mC4

MC4 [44] was included as part of the training data, applying the same approach
for filtering as [33], using ”preplexity-sampling” technique. This resulted in
1TB of raw texts, in full documents, around 60% of the original Spanish
portion of MC4. The analysis performed by our team resulted in a similar
conclusion to that of OSCAR, that is, around 50% or more of the texts are of
good quality, but a rigorous cleaning must be done in order to obtain a clean
version of the corpus. Moreover, the mC4 is formed from crawls from the
Internet, as OSCAR and a portion of SUC, therefore we could expect to have
many duplicate texts in our dataset, which is another feature to avoid in
quality checks.

###  3.5 Cleaning Process

For each of these corpora, we performed the same cleaning process, which
consisted on language, length and punctuation filters. First of all, we used a
language detection model for filtering out those texts with very low
probability of being written in Spanish. With this we emulate the filtering
carried out in [4] and [44]. Among model languages in Spanish, in [17] a
similar cleaning process was carried out.

Second, documents with less than 200 characters were filtered out following
[4] [44]. By applying this filter, most of SUC texts remain excluded.

Third, to fix encoding issues and similar problems
ftfy101010https://pypi.org/project/ftfy/ was used, so that texts originally in
latin-1, for example, which had been corrupted when transformed into utf-8,
were fixed to their original form.

Since MC4 was still too big, a punctuation filter was carried out. For this,
linguistic rules were developed, aimed at removing low quality texts not
filtered by the previous processes.

The cleaning process of GPT-3 [6] was replicated. It starts with a
deduplicating process with the use of PySpark
MinHashLSH111111https://spark.apache.org/docs/2.2.3/ml-features.html#minhash-
for-jaccard-distance. Secondly, a set of high quality texts was used to train
a logistic regression model to distinguish between good and bad quality texts,
then a filtering rule based on the Pareto distribution is applied, as in GPT-3
[6].

Two experiments were conducted. First, the samples manually annotated as
good/bad quality by our team described in 3.1 were used. Secondly, we used the
Wikipedia and the good quality part of Internal Sources as positive examples
and the rest of the corpus as negative examples.

However, after analyzing the outcomes of each of these experiments, it was
decided to keep only the first part of the cleaning process of GPT-3 [6], that
is, the fuzzy deduplication of documents. After applying all the cleaning
processes, we end up with a total volume of Spanish texts for training of
similar size, although slightly smaller, to the corpus used by MarIA [17].
These training data are of high quality given the strict cleaning filters
applied.

##  4 RigoBERTa Model and Training

In this section we describe the model architecture, as well as the training
process used for obtaining a State-of-the-Art language model in Spanish. We
start by explaining the procedure followed for training the tokenizer. Before
that, it is important to mention that we use DeBERTa [18], which improves over
Bert [16] and Roberta [24], which has an impact on the following subsections.

###  4.1 RigoTokenizer

For the training process ByteLevelBPETokenizer from Huggingface Tokenizers
library121212https://huggingface.co/docs/tokenizers/python/latest/ was used.
It is the tokenization algorithm used by DeBERTa [18], the model architecture
of RigoBERTa, and very similar to the tokenizer used by MarIA [17] and BERTIN
[33], with a vocabulary size of 50,265, as stated in [18].

###  4.2 Model Architecture

We chose DeBERTa [18] as the model architecture, as the English version
clearly outperforms English BERT and RoBERTa _base_ models. The authors of the
model also show that it converges faster, so it needs less training time and
resources than the English RoBERTa model. Therefore, due to its similar size,
increased training efficiency and specially, its enhanced effectiveness when
fine-tuning, we concluded that this was the best possible model of _base_ size
for developing RigoBERTa. For more details about the improvements of this
model with respect to BERT or RoBERTa, see the original article [18].

###  4.3 Training Procedure

For hyperparameters and training configuration, experiments were carried out
on a small subset of the corpus. The final training was then launched with the
best configuration from those experiments. The training was carried out by a
limited budget of 250k steps, about the same as BERTIN [33], less than one
third of the steps used in BETO 131313We perform an estimation of the number
of steps of BETO with 512 sequence length; as they perform an initial training
phase using 128 sequence length, we count each of these steps as 1/4 of a
step. [8] and between a half and a third of the steps trained by MarIA [17].
This is specially relevant when we look at the experimental results on
different benchmark tasks. With less training, RigoBERTa was able to achive
better results in many NLU tasks, as will be later shown and discussed in the
paper.

Regarding the hardware used for training the model, a single AWS p4d.24xlarge
instance, with 8 NVIDIA A100 GPUs with 40GB of memory each, was used.

##  5 Experimental Setting: Spanish GLUES

Building on the work by [8] and [17], a number of NLU tasks were selected from
a business-oriented perspective. With this set of tasks, performance of
RigboBERTA could be demostrated statistically and experimentally. The tasks
that form the benchmark for evaluating those models are described next.

Natural Language Inference: XNLI [12] Natural language inference is the task
of determining whether a ”hypothesis” is true (entailment), false
(contradiction), or undetermined (neutral) given a ”premise”. MNLI [41] is a
dataset for such task in English. It was automatically translated to 15
languages, forming the XNLI [12]. We use the Spanish portion of this dataset,
from Huggingface Datasets library [22]
141414https://huggingface.co/datasets/xnli.

Paraphrasing: PAWS-X [45] is a multilingual version of PAWS [46] with pairs of
sentences which have to be predicted as semantically equivalent or not. We
used the Spanish version of the
dataset151515https://huggingface.co/datasets/paws-x.

Document Classification: MLDoc [36] is a subset of the Reuters
corpus161616https://trec.nist.gov/data/reuters/reuters.html, which we
extracted using Facebook’s library
MLDoc171717https://github.com/facebookresearch/MLDoc. We decided to use both
the European Spanish and Latin American Spanish subsets, for training,
development and test, getting in each case the biggest version of the dataset
available (versions of different sizes are available). The task for this
dataset consists on classifying each document inside the following categories:
CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and
MCAT (Markets).

Workshop on Sentiment Analysis at SEPLN: TASS [34] is a small sentiment
analysis dataset of different varieties of Spanish which was part of the SEPLN
2020181818http://sepln2020.sepln.org/. We used a compilation of all versions
of Spanish for this task.

Spanish Question Answering Corpus: SQAC [17] was introduced in section 2 as
part of the contributions by the authors of MarIA. It has a similar format to
SQUAD v1.1 [32], with no adversarial questions, and it is formed with 18,817
questions and 6,247 textual contexts [17]. It is freely
accessible191919https://huggingface.co/datasets/PlanTL-GOB-ES/SQAC.

Cross-Lingual Question Answering Dataset: XQUAD + SQUAD-ES [5] [10]. XQUAD [5]
is a collection of 1190 question-answer pairs. These pairs were translated by
professional human translators to ten languages from the evaluation set of
SQUAD v1.1. As it is a very small dataset of well-known quality, we used the
Spanish portion of it as the test
set202020https://huggingface.co/datasets/xquad. A machine-translated version
of SQUAD v1.1, SQUAD-ES212121https://huggingface.co/datasets/squad_es [10],
was used for training and development sets. As there is only one split in
Huggingface Datasets [22] version, we split 10% of it as the development set.

CAPITEL-NER222222https://sites.google.com/view/capitel2020 [30] is a NER task
which was part of IberLEF
2020232323https://sites.google.com/view/iberlef2020/home, focusing on
identifying each token as Person (PER), Location (LOC), Organization (ORG) or
Other (OTH).

ConLL2002 [37] is a NER task including the tags PER, LOC, ORG and
MISC242424https://github.com/teropa/nlp/tree/master/resources/corpora/conll2002.
Although it is an old dataset, it is still widely used for benchmarking models
in the NER task in Spanish [8] [17].

CANcer TExt Mining Shared Task – tumor named entity recognition: CANTEMISTNER
[26] is a track from IberLEF 2020 formed by 3 tasks, of which we just use the
NER task, focused on detecting tumor morphology mentions in medical texts on a
collection of 3000 clinical cases.252525It can be accessed on the competition
page: https://temu.bsc.es/cantemist/.

Medical Document Anonymization Track: MEDDOCAN [3] is a track of IberLEF2019
that aimed at anonimizing clinical records. Only the NER task is used,
consisting on identifying several labels in clinical records262626See [3] for
more information. However, many labels had vey low frequency, so they were
discarded. Concretely, the 15 labels with lowest frequecies were removed. As
with the rest of the corpora released by BSC for IberLEF Conferences, it is an
open-access dataset272727https://temu.bsc.es/meddocan/.

MEDical DOcuments PROFessions recognition shared task: MEDDOPROF [23] is a
track focusing on clinical case reports, part of IberLEF2021. Only NER tasks 1
and 2 were used, so they will be referenced as MEDDOPROF1 and MEDDOPROF2,
respectively. MEDDOPROF1 consists on identifying professions (PROFESION) and
employment status (SITUACION_LABORAL), while MEDDOPROF2 has labels: patients
(PACIENTE), family member (FAMILIAR), health professional (SANITARIO) or
someone else related to the patient (OTROS). It is published in the BSC
page282828https://temu.bsc.es/meddoprof/. The dataset is originally splitted
in train and test, so for the development set we choose 10% of the train set.

Pharmacological Substances, Compounds and proteins and Named Entity
Recognition track: PHARMACONER292929https://temu.bsc.es/pharmaconer/ [2] was
part of the BioNLP Open Shared Tasks 2019, and the NER task (track 1, the one
we use) consists on identifying the entities NORMALIZABLES, NO_NORMALIZABLES,
PROTEINAS (proteins) and UNCLEAR on medical texts.

##  6 Evaluation

The four Spanish language models (RigoBERTa, BETO, BERTIN and MarIA) were
evaluated on the tasks introduced in the previous section. For that, the
original models authors’ recommendations were followed. For MarIA, BERTIN and
BETO, we used the hyperparameter space in [17], as it is the most extensive
research that has been carried out with these models, and these are the
hyperparameters they used for training their own model.

It does make sense that MarIA, BERTIN and BETO use the same hyperparameter
setup, as RoBERTa and BERT architectures are fundamentally the same, with the
only exception of the embeddings layer, as RoBERTa architecture has a
vocabulary size of 50,265 while BERT has 32,000.

On the other hand, we used a different architecture, DeBERTa, which has
several variations in the architecture with respect to RoBERTa and BERT [18].
These are: disentangled attention, relative positional embeddings on each
layer, contrasting with the absolute positional embeddings just in the first
layer of BERT and RoBERTa, and the Enhanced Mask Decoder. The latter accounts
for absolute word positions before applying the softmax layer for masked token
prediction.

For this reason, for RigoBERTa the hyperparameter space proposed in its
original paper [18] was used, following the principle of using the author’s
recommended settings for best optimization of each model for the proposed
tasks.

There are some notes on the experiments that should be taken into account.
First of all, regarding the Question Answering tasks, much higher results for
MarIA were obtained compared to the original authors [17]. This was due to a
bug we encountered on Huggingface Tokenizers library. This caused that, when
relying on Huggingface Transformers library’s [42] published and standardized
methodology for processing Question Answering datasets, some functionality of
BPE tokenizers does not work as intended, causing some examples to be
incorrectly processed. This affects MarIA, BERTIN and RigoBERTa, but not BETO,
which uses WordPiece[43]. This, in our opinion, was distorting the reported
results of MarIA [17] and BERTIN [33] on this dataset [17].

In order to solve this bug we developed a novel model-agnostic approach for
processing Question Answering datasets, explained in detail in Appendix A.
Question-Answering Processing Method, which does not depend much on
tokenizers’ functionality303030There are essential requirements, such as
including the CLS token at the beggining of texts when inputing them to the
model., and applied several tests to ensure the answers text spans inputed to
the model are exactly the same as the original answers in the dataset.

However, even with this methodology, we could not process Question Answering
datasets with BERTIN’s tokenizer, as it does not add a CLS token313131CLS
token is a special token included as the first token in each sentence[16] at
the beginning of the input ids when tokenizing texts, which is an unavoidable
condition for processing Question Answering datasets323232Moreover, the CLS
token is of crucial importance, as it is the token we pass to the task layer.
When absent, the first token in the sentence is used as the CLS token for
Classification tasks. We believe BERTIN’s modified tokenizer can be, at least
partially, the reason for its underperforming results compared to other models
in these type of tasks.. For that reason, the results for BERTIN on datasets
SQAC and SQUAD-ES+XQUAD-ES are set to values slightly lower than the minimum
score of the rest of the models in each case. By doing this, it is possible to
later perform statistical tests on performance for all models including
BERTIN.

Adam optimizer was used for all models [20], with parameters
$\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=1e-6$, following recommendations in
[8] [16] [24] [18]. The rest of the optimizer’s parameters as the weight decay
or the warm-up steps are optimized based on each model’s parameter search
space. We used a maximum sequence length of 512 for all models on all tasks.

Dataset | Metric | MarIA | BERTIN | BETO | RigoBERTa
---|---|---|---|---|---
CANTEMISTNER | F1-Macro | 0.923 | 0.795 | 0.899 | 0.933*
CAPITEL | F1-Macro | 0.878* | 0.865 | 0.870 | 0.874
ConLL2002 | F1-Macro | 0.899 | 0.901* | 0.896 | 0.895
MEDDOCAN | F1-Macro | 0.841 | 0.722 | 0.847 | 0.850*
MEDDOPROF1 | F1-Macro | 0.807 | 0.710 | 0.805 | 0.831*
MEDDOPROF2 | F1-Macro | 0.785 | 0.442 | 0.818 | 0.864*
MLDoc | F1-Macro | 0.956* | 0.944 | 0.954 | 0.956*
PAWS-x | F1-Macro | 0.889 | 0.901 | 0.897 | 0.910*
PHARMACONER | F1-Macro | 0.571 | 0.471 | 0.614 | 0.700*
SQAC | F1-Macro | 0.866 | — | 0.762 | 0.897*
SQUAD-ES + XQUAD-ES | F1-Macro | 0.818 | — | 0.756 | 0.854*
TASS2020 | F1-Macro | 0.473* | 0.461 | 0.461 | 0.467
XNLI | F1-Macro | 0.816 | 0.794 | 0.817 | 0.834*
Table 1: Evaluation table of models on test set.

For each model and dataset, we ran experiments with Optuna [1], for the
minimum between the total number of possible combinations in the
hyperparameter space and 20. Each of these runs is evaluated on the
development set. Then, the best performing model is evaluated on the test set,
from which the final metrics for that model in that dataset are taken. We use
F1-Macro [27] for all tasks. In the case of NER tasks, we compute it by
comparing the real and predicted label of each token, with no grouping
strategy for subtokens forming a word, as this would have introduced
uncertainty on whether the performance differences come from models’
performances or from the convenience of chosen aggregation strategies for
certain models. These final metrics are reported on table 1, with an asterisk
indicating the best performing model for each dataset.

Figure 1: Nemenyi Test for the Language Models in Spanish. This figure
represents each model’s average rank and the horizontal thick lines group
together models whose average ranks are closer to each other than the critical
distance, meaning that there is no statistical difference between those
models’ performances.

As part of the evaluation, we conducted the Nemenyi Test [19] [15] to
determine whether there is a statistical difference between models in terms of
performance. To do so, the test uses the average rank of each model and the
critical
distance333333https://edisciplinas.usp.br/pluginfile.php/4129451/mod_resource/content/1/model_selection_evaluation.pdf.
We have observed a lack of statistical tests on performance for all the rest
of the Spanish language models, so we include it to get deeper insights into
the effectiveness of each model.

The results from Nemenyi test presented in figure 1 show that there is a
statistically significant difference between RigoBERTa, with an average rank
of 1.42, and both BERTIN, ranking at 3.58, and BETO, with 2.73. Although our
model is not proven to be statistically better than MarIA, with an average
rank of 2.27, MarIA is not statistically better than BETO, while the latter
does not significantly outperform BERTIN.

###  6.1 Domain Adaptation: Benchmarking for the Biomedical Domain.

As an additional experiment, it was decided to compare RigoBERTa against
domain-specific models, to assess its domain-transfer ability. The biomedical
domain was selected for this, due to the availability of domain-specific
models and datasets of reported good quality.

In [9] two domain-specific models are presented. One is for the biomedical
domain, while the other is for the biomedical-clinical domain (which is
arguably a sub-domain of the former). These two models will be referred to as
MarIA-BioMedical and MarIA-BioClinical respectively, as they were developed as
part of the MarIA project [17] introduced previously. Domain specific corpora
were used for pre-training each model, described in detail in [9]. Both use
RoBERTa architecture [24] [9].

In [9] both models are evaluated against BETO [8] on a small set of 3
biomedical and clinical datasets, showing better results than the latter. In
this work, MarIA-BioMedical and MarIA-BioClinical are evaluated against MarIA
[17] and RigoBERTa, being these the two best-performing models in general, as
shown in table 1.

Table 2 shows the results of each model in terms of f1-score on a set of tasks
for the biomedical domain. RigoBERTa achieves the highest f1-score on all
tasks, outperforming MarIA, MarIA-BioMedical and MarIA-BioClinical, thus
showing a great domain adaptation ability. This is specially relevant because
RigoBERTa has not been trained on biomedical texts specifically, contrasting
with MarIA-BioMedical and MarIA-BioClinical.

Dataset | Metric | MarIA | MarIA-BioMedical | MarIA-BioClinical | RigoBERTa
---|---|---|---|---|---
CANTEMISTNER | F1-Macro | 0.923 | 0.924 | 0.928 | 0.933*
MEDDOCAN | F1-Macro | 0.841 | 0.796 | 0.642 | 0.850*
MEDDOPROF1 | F1-Macro | 0.807 | 0.797 | 0.746 | 0.831*
MEDDOPROF2 | F1-Macro | 0.785 | 0.754 | 0.764 | 0.864*
PHARMACONER | F1-Macro | 0.571 | 0.570 | 0.322 | 0.700*
Table 2: Evaluation table of models on test set of Biomedical Tasks.

##  7 Conclusions & Future Work

In this work we presented RigoBERTa, a State-of-the-Art model outperforming
the rest of the Spanish language models, as shown in section 6. In fact, to
the best of our knowledge this is the first work reporting statistical tests
on performance for language models in Spanish, showing that RigoBERTa is
statistically better than BETO [8] and BERTIN [33]. Moreover, RigoBERTa
outperforms MarIA [17] in 9 out of 13 benchmarks (and obtain similar results
in another one).

These results confirm that the main objective of this work was reached: to
build the best performing _base_ model up to date in Spanish by improving the
pre-training corpus and by choosing a better model architecture. These two
main contributions together can account for the observed difference in
performance. This is most remarkable in Question Answering tasks, which was
expected since the DeBERTa architecture [18] makes better use of spatial
information in texts, very useful for this concrete task.

As for future work, RigoBERTa was only trained for 250k steps, which is a
fourth of the total training steps planned. Looking at the figures of fine-
tuning performance per training steps in [18], it seems that there is room for
much improvement for our model, therefore we will continue training it.
Moreover, we plan to keep on training language models as the State-of-the-Art
evolves and new improved architectures are published, making use of everything
learned during the development of RigoBERTa. Additionally, taking advantage of
the great ability of RigoBERTa to learn specific domain tasks, domain-specific
models will be developed, by adapting the general-domain model.

We would like to acknowledge the support of this project by Instituto de
Ingeniería del Conocimiento, as without their faith on us and their
investment, this project would not have been possible. Also, we would like to
mention the Chair of Computational Linguistics (Cátedra de Lingüística
Computacional)343434http://catedras.iic.uam.es/catedra-linguistica-
computacional/, for letting us use their machines for the benchmarking
experiments.

## Appendix A. Question-Answering Processing Method

Tokenizers from Huggingface had some issues related to BPE Tokenizers, which
cause that the correspondence between generated tokens and original characters
(offset mapping) in the text do not match exactly. Therefore when this
functionality is used to determine the start and end tokens from start and end
characters in the original text, the resulting span of text (between start and
end token) is not the actual answer. For that reason, when relying on the
official methodology for processing question-answering datasets
353535https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb,
some answers are incorrectly processed, thus having some corrupt examples for
training and evaluating, therefore the results obtained by models using BPE
tokenizers are far from optimal. This affects, in the context of language
models in Spanish, RigoBERTa (presented in this paper), MarIA [17] and BERTIN
[33].

For solving this, a model-agnostic method was developed, which ensures that
the models’ inputs between the selected start and end tokens correspond to the
exact text answer in the original dataset. By doing this, models do not learn
on corrupted examples, therefore their performance on Question Answering tasks
is optimized. Then, tests are applied to check the aforementioned condition.

## References

  * [1] Takuya Akiba et al.  “Optuna: A Next-generation Hyperparameter Optimization Framework”, 2019  arXiv:1907.10902 [cs.LG]
  * [2] Aitor Gonzalez Agirre al.  “Pharmaconer: Pharmacological substances, compounds and proteins named entity recognition track”  In _Proceedings of The 5th Workshop on BioNLP Open Shared Tasks_ , 2019, pp. 1–10
  * [3] Montserrat Marimon al.  “Automatic De-identification of Medical Texts in Spanish: the MEDDOCAN Track, Corpus, Guidelines, Methods and Evaluation of Results.”  In _Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings_ , 2019, pp. 618–638
  * [4] Raffel C. al.  “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, 2020  URL: https://arxiv.org/pdf/1910.10683.pdf
  * [5] Mikel Artetxe, Sebastian Ruder and Dani Yogatama  “On the Cross-lingual Transferability of Monolingual Representations”, 2020  arXiv:1910.11856 [cs.CL]
  * [6] Tom B. Brown et al.  “Language Models are Few-Shot Learners”, 2020  arXiv:2005.14165 [cs.CL]
  * [7] J. al Cañete  “Large Spanish Corpus”, 2021  URL: https://huggingface.co/datasets/large_spanish_corpus
  * [8] J. al. Cañete  “SPANISH PRE-TRAINED BERT MODEL AND EVALUATION DATA”, 2020  URL: https://users.dcc.uchile.cl/~jperez/papers/pml4dc2020.pdf
  * [9] Casimiro Pio Carrino et al.  “Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario”  In _CoRR_ abs/2109.03570, 2021  arXiv: https://arxiv.org/abs/2109.03570
  * [10] Carrino Casimiro Pio, Costa-jussa Marta R. and Fonollosa Jose A..  “Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering”  In _arXiv e-prints_ , 2019, pp. arXiv:1912.05200v1  arXiv:1912.05200v2
  * [11] Branden Chan, Stefan Schweter and Timo Möller  “German’s Next Language Model”  In _Proceedings of the 28th International Conference on Computational Linguistics_ Barcelona, Spain (Online): International Committee on Computational Linguistics, 2020, pp. 6788–6796  DOI: 10.18653/v1/2020.coling-main.598
  * [12] Alexis Conneau et al.  “XNLI: Evaluating Cross-lingual Sentence Representations”  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ Brussels, Belgium: Association for Computational Linguistics, 2018
  * [13] Deepset.AI  “Open Sourcing German BERT Model”, 2020  URL: https://www.deepset.ai/german-bert
  * [14] P. al. Delobelle  “RobBERT: a Dutch RoBERTa-based Language Model”, 2020  URL: https://arxiv.org/pdf/2001.06286.pdf
  * [15] Janez Demšar  “Statistical Comparisons of Classifiers over Multiple Data Sets”  In _Journal of Machine Learning Research_ 7.1, 2006, pp. 1–30  URL: http://jmlr.org/papers/v7/demsar06a.html
  * [16] J. al. Devlin  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019  URL: https://arxiv.org/pdf/1810.04805.pdf
  * [17] Asier Gutiérrez-Fandiño et al.  “Spanish Language Models”, 2021  arXiv:2107.07253 [cs.CL]
  * [18] P.. He  “DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION”, 2021  URL: https://arxiv.org/pdf/2006.03654.pdf
  * [19] Steffen Herbold, Alexander Trautsch and Jens Grabowski  “Correction of ”A Comparative Study to Benchmark Cross-project Defect Prediction Approaches””, 2017  arXiv:1707.09281 [cs.SE]
  * [20] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”, 2017  arXiv:1412.6980 [cs.LG]
  * [21] H. al. Le  “FlauBERT: Unsupervised Language Model Pre-training for French”, 2020  URL: https://arxiv.org/pdf/1912.05372.pdf
  * [22] Quentin Lhoest et al.  “Datasets: A Community Library for Natural Language Processing”  In _CoRR_ abs/2109.02846, 2021  arXiv: https://arxiv.org/abs/2109.02846
  * [23] Salvador Lima-López et al.  “NLP applied to occupational health: MEDDOPROF shared task at IberLEF 2021 on automatic recognition, classification and normalization of professions and occupations from medical texts”  In _Procesamiento del Lenguaje Natural_ 67, 2021, pp. 243–256  URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6393
  * [24] Y. al. Liu  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019  URL: https://arxiv.org/pdf/1907.11692.pdf
  * [25] L. al. Martin  “CamemBERT: a Tasty French Language Model”, 2020  URL: https://arxiv.org/pdf/1911.03894.pdf
  * [26] A Miranda-Escalada, E Farré and M Krallinger  “Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results”  In _Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings_ , 2020
  * [27] Juri Opitz and Sebastian Burst  “Macro F1 and Macro F1”, 2021  arXiv:1911.03347 [cs.LG]
  * [28] Pedro Javier Ortiz Su’arez, Laurent Romary and Benoit Sagot  “A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages”  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ Online: Association for Computational Linguistics, 2020, pp. 1703–1714  URL: https://www.aclweb.org/anthology/2020.acl-main.156
  * [29] Pedro Javier Ortiz Su’arez, Benoit Sagot and Laurent Romary  “Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures”, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019  Mannheim: Leibniz-Institut f”ur Deutsche Sprache, 2019, pp. 9–16  DOI: 10.14618/ids-pub-9021
  * [30] Jordi Porta-Zamorano and Luis Espinosa-Anke  “Overview of CAPITEL Shared Tasks at IberLEF 2020: Named Entity Recognition and Universal Dependencies Parsing”, 2020  arXiv:2011.05932 [cs.CL]
  * [31] A. al. Radford  “Language Models are Unsupervised Multitask Learners”, 2019  URL: https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
  * [32] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev and Percy Liang  “SQuAD: 100,000+ Questions for Machine Comprehension of Text”, 2016  arXiv:1606.05250 [cs.CL]
  * [33] Javier de la Rosa et al.  “BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling”, 2022-03
  * [34] Julio Villena-Román Sara Lana-Serrano y Eugenio Martínez-Cámara y José Carlos González-Cristóbal  “TASS - Workshop on Sentiment Analysis at SEPLN”  In _Procesamiento del Lenguaje Natural_ 50.0, 2013, pp. 37–44  URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/4657
  * [35] R. al. Scheible  “GottBERT: a pure German Language Model”, 2020  URL: https://arxiv.org/pdf/2012.02110.pdf
  * [36] Holger Schwenk and Xian Li  “A Corpus for Multilingual Document Classification in Eight Languages”  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ Miyazaki, Japan: European Language Resources Association (ELRA), 2018
  * [37] E.. Tjong Kim Sang  “Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition”, 2002  URL: https://arxiv.org/abs/cs/0209010
  * [38] A. al. Vaswani  “Attention is All You Need”, 2017  URL: https://arxiv.org/pdf/1706.03762.pdf
  * [39] M. Vincent  “LePetit: A pre-training efficient and lightning fast French Language Model”, 2020  URL: https://medium.com/illuin/lepetit-a-pre-training-efficient-and-lightning-fast-french-language-model-96495ad726b3
  * [40] Alex Wang et al.  “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”, 2019  arXiv:1804.07461 [cs.CL]
  * [41] Adina Williams, Nikita Nangia and Samuel Bowman  “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference”  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ New Orleans, Louisiana: Association for Computational Linguistics, 2018, pp. 1112–1122  URL: http://aclweb.org/anthology/N18-1101
  * [42] Thomas Wolf et al.  “HuggingFace’s Transformers: State-of-the-art Natural Language Processing”  In _CoRR_ abs/1910.03771, 2019  arXiv: http://arxiv.org/abs/1910.03771
  * [43] Yonghui Wu et al.  “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation”, 2016  arXiv:1609.08144 [cs.CL]
  * [44] L. al. Xue  “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”, 2021  URL: https://arxiv.org/pdf/2010.11934v3.pdf
  * [45] Yinfei Yang, Yuan Zhang, Chris Tar and Jason Baldridge  “PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification”, 2019  arXiv:1908.11828 [cs.CL]
  * [46] Yuan Zhang, Jason Baldridge and Luheng He  “PAWS: Paraphrase Adversaries from Word Scrambling”, 2019  arXiv:1904.01130 [cs.CL]


[a,b]Tamas G. Kovacs

# Localization at the quenched SU(3) phase transition

###### Abstract

It is known that the deconfining transition of QCD is accompanied by the
appearance of localized eigenmodes at the low end of the Dirac spectrum. In
the quenched case localization appears exactly at the critical temperature of
deconfinement. In the present work, using quenched simulations exactly at the
critical temperature, we show that the localization properties of low Dirac
modes change abruptly between the confined and deconfined phase. This means
that in the real Polyakov loop sector, the mobility edge has a discontinuity
at the critical temperature. In contrast, in the complex sector, there is no
such discontinuity at $T_{c}$, even the lowest Dirac modes remain localized at
the critical temperature in the deconfined phase.

##  1 Introduction

The spectrum of the QCD Dirac operator is well known to encode many important
features of the underlying gauge theory. In particular, it undergoes dramatic
changes at the finite temperature transition into the quark-gluon plasma
state. In the present work we study how certain properties of the Dirac
spectrum change at the finite temperature phase transition of the pure $SU(3)$
gauge theory. The pure gauge theory provides a good testing ground for this
study, since – in contrast to QCD with physical quark masses – this system has
a genuine first order phase transition, not just a crossover. As the character
of the gluon fields changes dramatically at the phase transition, the
corresponding changes in the Dirac spectrum can be easily detected.

The phase transition of the quenched theory is signaled by the spontaneous
breaking of the $Z(3)$ symmetry of the Polyakov loop. While in the low
temperature, confining (symmetric) phase the expectation of the Polyakov loop
is zero, above the phase transition temperature, in the deconfined phase, the
symmetry is spontaneously broken and the Polyakov loop develops a nonzero
expectation value, belonging to one of the three $Z(3)$ sectors. In the
present work we perform quenched simulations exactly at the phase transition
temperature, where in a finite volume the system can tunnel between the
confined and deconfined phase. Therefore, in a simulation performed at the
transition temperature we can sample both phases, and in fact, in the
deconfined phase all three Polyakov loop sectors as well.

Figure 1:  Scatter plot of the Polyakov loop in the complex plane. The
encircled regions are the different sectors, the confined in the center (red
circle), the deconfined real sector on the right (blue circle) and the
deconfined complex sectors to the left (green circles).

Concerning the Dirac spectrum we have to distinguish three different sectors
in terms of the Polyakov loop, and the Dirac spectrum is expected to behave
differently in these sectors. These are the confined phase, the real sector in
the deconfined phase and the two complex sectors in the deconfined phase. Due
to the complex conjugation symmetry, Dirac spectra in the two complex sectors
are the same and we do not have to treat these sectors separately [1]. In Fig.
1 we depict the different sectors by showing a scatter plot of the complex
Polyakov loop on an ensemble of $48^{3}\times 8$ lattice configurations
exactly at the critical temperature. We can see that although the probability
of finding the Polyakov loop in the four above mentioned sectors is enhanced,
there is still considerable tunneling among the different sectors.

##  2 Details of the simulation

Let us first summarize the technical details of the simulations. We generated
quenched gauge field configurations of spatial linear size $L=48$ and temporal
size $N_{t}=8$ using the Wilson gauge action at the critical value of the
coupling $\beta=6.063$. The configurations were separated into three different
groups (sectors) according to the value of the average Polyakov loop.
Configurations with $|P|<0.03$ were assigned to the confined phase, the ones
with $|P|>0.06$ were considered to be in the deconfined phase. The latter
configurations were further sorted into the real and complex Polyakov loop
sector according to the sign of the real part of the Polyakov loop . This set
of criteria was found to include the peaks of the Polyakov loop distribution
into the respective sectors (see Fig. 1). We computed the lowest 100
eigenvalues (with positive imaginary parts) of the overlap Dirac operator
separately on the three ensembles of configurations belonging to the above
three sectors.

##  3 Results

Figure 2:  The spectral density of the overlap Dirac operator on the three
ensembles of $48^{3}\times 8$ lattice configurations at Wilson $\beta=6.063$.

The simplest quantity characterizing the Dirac spectrum is the spectral
density. In Fig. 2 we compare the spectral density in the three different
sectors. The confined phase and the deconfined complex sectors exhibit
qualitatively similar spectral densities. In contrast, the spectral density in
the deconfined real sector is markedly different. It has a high, but narrow
spike at the very low end, above which the spectral density drops
considerably. The narrow spike in the spectral density can be attributed to
mixing instanton-antiinstanton zero modes that produce several pairs of
complex eigenvalues of rather small magnitude [2]. Although in the other two
sectors there is also a slight accumulation of eigenvalues near zero, this is
by far less pronounced than in the real sector. This is caused by a stronger
mixing of topological zero modes, which in the confined phase is due to a
higher density of topological fluctuations. In the deconfined complex Polyakov
sectors this stronger mixing can be attributed to the more extended nature of
the topological zero modes [3]-[5].

The difference in the bulk spectral density between the real and complex
sector in the confined phase can be qualitatively understood by considering
the lowest Matsubara modes in the free theory with different boundary
conditions. Indeed, the effective boundary condition for the Dirac equation is
a combination of the phase $\pi$, coming from the antiperiodic boundary
condition for fermions, and the phase of the Polyakov loop. In the real sector
the Polyakov loop phase is zero, in the complex sectors, it is $\pm 2\pi/3$,
making the magnitude of the overall effective phase $\pi$ and $\pi/3$,
respectively. Thus the lowest free Matsubara mode is much higher in the real
sector, and this makes the density of low-lying bulk modes considerably lower
also in the interacting case. In fact, this is how the breaking of the $Z(3)$
symmetry by dynamical quarks can be understood, since the quark determinant
favors the real sector that has fewer small eigenvalues.

Figure 3:  Scatter plot of the real part of the average Polyakov loop versus
the $100^{th}$ smallest Dirac eigenvalue on a set of lattice configurations.

The direct connection between the Polyakov loop and the Dirac spectrum is
further demonstrated in Fig. 3, where we plot the real part of the average
Polyakov versus the $100^{th}$ eigenvalue of the Dirac operator. For positive
real part there is a strong correlation between the two quantities. It is
clear that the more ordered the Polyakov loop is in the real sector, the fewer
low eigenvalues the Dirac operator has.

##  4 Localization at $T_{c}$

It is known that in the high temperature phase the lowest part of the Dirac
spectrum consists of localized eigenmodes (for a recent review see [6]).
Approaching the transition temperature from above, the mobility edge,
separating the low localized modes from the bulk of the spectrum, moves toward
zero, and when it reaches zero, all modes become delocalized, as expected in
the low temperature phase. In the quenched case, by extrapolating the mobility
edge, it was found to vanish exactly at the critical temperature [7, 8]. A
naturally arising question is whether the lowest part of the spectrum is
localized or delocalized exactly at the critical point. More precisely, the
extrapolations presented in Refs. [7, 8] are compatible both with a
continuously vanishing mobility edge and one that has a discontinuity at the
phase transition. Our present simulation, done exactly at the critical point,
gives us an opportunity to distinguish these two possibilities. This can be
done by looking at the localization properties of the lowest Dirac modes
separately in the confining and the deconfining phase and comparing the values
of the mobility edge in the two phases.

Figure 4:  The unfolded level spacing distribution, corresponding to localized
and delocalized modes. The red shaded region depicts the integral, defining
$I_{0.5}$ for the localized case, the blue one the difference of the integrals
for the localized and the delocalized case.

The simplest way to decide whether eigenmodes in a certain region of the
spectrum are localized or delocalized is to compute the unfolded level spacing
distribution (ULSD). If the modes are localized, the corresponding eigenvalues
are statistically independent, they are described by a Poisson distribution,
and the unfolded level spacing distribution is exponential. If the eigenmodes
are delocalized, the ULSD is well approximated by the Wigner surmise of the
corresponding random matrix universality class, which in the case of the
overlap Dirac operator, is the unitary class. In Fig. 4 these two
distributions are shown. A convenient way to monitor how the distribution
changes across the spectrum is to consider instead of the whole distribution,
just a single parameter that can distinguish the two extreme cases (localized
and delocalized). For this purpose we use the parameter

$I_{0.5}=\int_{0}^{0.5}p(x)d\\!x,$ (1)

where the upper limit of the integration was chosen to be the crossing point
of the two distributions (approximately 0.5) to maximize the difference
between the two limiting cases (see Fig. 4).

Figure 5:  The parameter $I_{0.5}$ across the spectrum computed separately in
the three different sectors, the confined phase and in the deconfined phase in
the real and the complex sector on the $48^{3}\times 8$ lattices.

To monitor how the ULSD changes across the spectrum and to determine the
mobility edge, we split the spectrum into narrow bins and in each bin
separately compute the parameter $I_{0.5}$. For details of the unfolding and
how we assign the eigenvalue pairs to bins, see the Appendix of Ref. [8]. We
repeated this procedure for all three sectors, i.e. the confined phase and for
the deconfined phase in the real and complex sector. The results are shown in
Fig. 5. It is clear from the figure that both in the confined phase and in the
deconfined complex sector, already the lowest Dirac modes are delocalized,
thus the mobility edge is effectively at zero. However, in the deconfined real
sector, in the one that would be selected if dynamical fermions were present,
there is a finite range in the spectrum above zero where the eigenmodes seem
to be localized. In this region $I_{0.5}$ almost reaches the value expected
for localized modes. The slight deviation might be due to the finite system
size or some more subtle effects (see Refs. [9]-[11]). Notwithstanding these
details, it is obvious from the figure that the localization properties of the
lowest Dirac eigenmodes change abruptly between the confined phase and the
deconfined real Polyakov sector (the physical sector). This is consistent with
the fact that the transition in the $SU(3)$ pure gauge theory is first order.
It is, however, surprising that the deconfined complex Polyakov sector does
not exhibit such an abrupt change, the lowest modes there are delocalized,
like in the confined phase.

##  5 Conclusions

In the present paper we studied how the spectral properties of the Dirac
operator change at the finite temperature first order phase transition of the
pure $SU(3)$ gauge theory. We showed that the phase transition is accompanied
by discontinuous changes in the spectral properties. Namely, both the spectral
density around zero and the localization properties of the lowest modes change
abruptly at the transition. In particular, there is a band around zero in the
spectrum, where the corresponding eigenmodes become localized at the
transition in the real Polyakov loop sector. This also implies that the
mobility edge is discontinuous at the transition if we restrict our attention
to the real (physical) Polyakov loop sector. In contrast, in the complex
sector even the lowest eigenmodes remain delocalized at the transition. This
property can be qualitatively understood by the more extended nature in this
sector of zero modes carried by Kraan-van Baal calorons.

## References

  * [1] M. Cardinali, M. D’Elia, F. Garosi and M. Giordano, Localization properties of Dirac modes at the Roberge-Weiss phase transition [arXiv:2110.10029 [hep-lat]].
  * [2] R. A. Vig and T. G. Kovacs, Ideal topological gas in the high temperature phase of SU(3) gauge theory, Phys. Rev. D 103, no.11, 114510 (2021) doi:10.1103/PhysRevD.103.114510 [arXiv:2101.01498 [hep-lat]].
  * [3] T. C. Kraan and P. van Baal, Periodic instantons with nontrivial holonomy, Nucl. Phys. B 533, 627-659 (1998) doi:10.1016/S0550-3213(98)00590-2 [arXiv:hep-th/9805168 [hep-th]].
  * [4] C. Gattringer, M. Gockeler, P. E. L. Rakow, S. Schaefer and A. Schaefer, A Comprehensive picture of topological excitations in finite temperature lattice QCD, Nucl. Phys. B 618, 205-240 (2001) doi:10.1016/S0550-3213(01)00509-0 [arXiv:hep-lat/0105023 [hep-lat]].
  * [5] C. Gattringer and S. Schaefer, New findings for topological excitations in SU(3) lattice gauge theory, Nucl. Phys. B 654, 30-60 (2003) doi:10.1016/S0550-3213(03)00083-X [arXiv:hep-lat/0212029 [hep-lat]].
  * [6] M. Giordano and T. G. Kovacs, Localization of Dirac Fermions in Finite-Temperature Gauge Theory, Universe 7, no.6, 194 (2021) doi:10.3390/universe7060194 [arXiv:2104.14388 [hep-lat]].
  * [7] T. G. Kovacs and R. A. Vig, Localization transition in SU(3) gauge theory, Phys. Rev. D 97, no.1, 014502 (2018) doi:10.1103/PhysRevD.97.014502 [arXiv:1706.03562 [hep-lat]].
  * [8] R. A. Vig and T. G. Kovacs, Localization with overlap fermions, Phys. Rev. D 101, no.9, 094511 (2020) doi:10.1103/PhysRevD.101.094511 [arXiv:2001.06872 [hep-lat]].
  * [9] A. Alexandru and I. Horváth, Possible New Phase of Thermal QCD, Phys. Rev. D 100, no.9, 094507 (2019) doi:10.1103/PhysRevD.100.094507 [arXiv:1906.08047 [hep-lat]].
  * [10] A. Alexandru and I. Horváth, Unusual Features of QCD Low-Energy Modes in the Infrared Phase, Phys. Rev. Lett. 127, no.5, 052303 (2021) doi:10.1103/PhysRevLett.127.052303 [arXiv:2103.05607 [hep-lat]].
  * [11] A. Alexandru and I. Horváth, Anderson Metal-to-Critical Transition in QCD, [arXiv:2110.04833 [hep-lat]].