# A Survey of Spanish Clinical Language Models

Guillem García Subies
Universidad Carlos III de Madrid
Instituto de Ingeniería del Conocimiento
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Álvaro Barbero Jiménez
Instituto de Ingeniería del Conocimiento
<EMAIL_ADDRESS>
Paloma Martínez Fernández
Universidad Carlos III de Madrid
<EMAIL_ADDRESS>

###### Abstract

This survey focuses in encoder Language Models for solving tasks in the
clinical domain in the Spanish language. We review the contributions of 17
corpora focused mainly in clinical tasks, then list the most relevant Spanish
Language Models and Spanish Clinical Language models. We perform a thorough
comparison of these models by benchmarking them over a curated subset of the
available corpora, in order to find the best-performing ones; in total more
than 3000 models were fine-tuned for this study. All the tested corpora and
the best models are made publically available in an accessible way, so that
the results can be reproduced by independent teams or challenged in the future
when new Spanish Clinical Language models are created.

##  1 Introduction

The enormous wealth of data present in the Electronic Health Records (EHR)
opens up multiple possibilities for its use not only in biomedical research
(secondary use) but also in clinical practice (primary use). Transforming
unstructured data (as clinical notes) into structured data of EHR offers
researchers and healthcare professionals the possibility of working with
clinical data of higher quality and accuracy because redundancy of information
has been eliminated, data is validated, and also integrated in databases, thus
allowing accessing using structured queries. The most frequent uses of EHR in
research, in addition to coding systems for billing purposes (e.g., ICD-10
coding of diagnoses in discharge reports), range from exploring new diagnostic
and therapeutic solutions, to evaluating patient outcomes, identifying
populations at risk, developing databases and repositories, among many others.

Processing data coming from EHR has been one the most important challenges in
natural language processing (NLP). The application of NLP techniques in
analysing information contained in the unstructured fields (free text) of the
EHR makes it possible to extract the relevant information (expressed in
concepts, relationships and events) and its transformation into a structured
format. In this way, the structured information obtained can be stored in
repositories integrated within the EHR itself, which would open up the
possibility for its automatic exploitation, facilitating the development of
decision support tools, clinical practice guidelines, support tools for the
development of epidemiological studies, among many other applications.

Apart from the inherent complexities of understanding text, clinical text has
additional peculiarities. While documents published in scientific journals and
books have undergone careful editing, clinical narrative texts are usually
hastily written in a telegraphic style, with numerous ellipsis and spelling
errors and with incorrect syntax in many cases, abuse of negations, jargon,
abbreviations among others. For instance, incomplete sentences (predominance
of phrases instead of complete sentences), new terms arising every day, others
due to misspellings or even new terms that medical professionals invent to
generate their notes. From the semantic point of view, there are words with
uses in medical language that differ from the language of general use. From
the contextual point of view, the abuse of ellipsis makes interpretation
difficult (e.g., the paragraph: "Complaints of chest pain, increased
frequency, especially with exertion. Usually associated with short respiration
and nausea and vomiting" represents the entity "chest pain due to chest
angina").

###  1.1 Spanish language in NLP

With $559$ million speakers, Spanish is the fourth most spoken language in the
world [1]. However, and despite its relevance, there are few relevant NLP
resources in Spanish. For instance, at the date of the writing of this article
there are $16520$ English models and $3466$ corpora in the Hugging Face Hub,
but only $137$9 and $351$ respectively for Spanish [2] (a lot of those being
multilingual models). This means that there are $11.35$ English models and
2.38 corpora for every million English speakers while there are only $2.30$
and $0.59$ respectively for Spanish. Another example can be the French
language, with 309 million speakers, which has $4.64$ and $1.24$ French
corpora for every million speakers.

Compared to English, Spanish is a highly inflectional language with a richer
morphology; morphemes signify many syntactic, semantic, and grammatical
aspects of words (such as gender, number, etc.). From a syntactic perspective,
Spanish texts feature more subordinate clauses and lengthy sentences with a
high degree of word order flexibility; for example, the subject is not
restricted to appearing solely before the verb in a sentence. Spanish clinical
texts have particularities that are explained below.

There is clearly a lack of resources for non-English language models, which
has become even more pressing since the revolution brought about by BERT [3]
that has culminated with ChatGPT and GPT-4 [4, 5, 6]. This lack is also
quantified with the big amount of biomedical and clinical language models in
other languages. Just to name a few, in English we can find BioBERT [7],
PubMedBERT [8], BioGPT [9] or ClinicalGPT [10] and DrBERT [11] or CamemBERT-
bio [12] for French.

Therefore, a significant amount of research efforts are still required in the
development of corpora and language models for the Spanish language, even more
in the clinical domain [13].

####  1.1.1 Spanish language in the Clinical domain

Compared to the general Spanish language, in the Clinical domain there are
more anglicisations as a result of the translation of English words used in
biomedicine. Some of them are freely altered, while others are perfect
replicas of the originals; for example, "adipocytokine" is also rendered as
"adipocitocina", "adipocitoquina", and "adipocitokina". Additionally, the
Spanish language uses accent marks that do not exist in English language, and
whether or not these accent marks are preferred results in lexical variants.
For example, the word "ebola" may become "ébola" or "ebola."

When translated, adjectives ending in "-al" sometimes retain their original
form and other times adopt Spanish morphological conventions. For instance,
"retinal" may become "retinal" or "retiniano". Greco-latin prefixes exhibit
variations such as "psi-" ("psiquiatra" vs. "siquiatra"). In contrast to
Spanish, where there are many variations, English uses hyphens between words
more consistently. For example, "beta-alanine" is converted into "beta-
alanina" or "beta alanina" or "alanina beta". The names of active ingredients
are sometimes kept the same as they are in English and other times they are
changed. For example, the name "acetaminophen" is changed to "acetaminofeno,"
"acetaminofén". Some terminology that refer to gender (male/female) are
ambiguous ("el enzima"/"la enzima") or both are acceptable, for example, "el
tiroides"(male)/ "la tiroides" (female) for the hormone "thyroid".

Abbreviations are frequently used in clinical notes, and typically Spanish and
English abbreviations coexist. The term "PSA" stands for "prostate-specific
antigen," which is preferable to the term "APE" (antígeno prostático
específico). However, both languages use polysemic abbreviations frequently.
Sentences in the two languages are syntactically highly similar (short
sentences or phrases, abuse of negation particles and unconventional
abbreviations, misspellings, speculation, and grammatical sentence anomalies,
among other things). In conclusion, due to term replication or partial
adaptation, there are more lexical variants of medical terms in Spanish than
in English. Because of these factors, studying these texts requires more
resources and normalization tools.

###  1.2 Purpuse of the study

The main objective of this study is to gather all the resources available to
deal with clinical textual data in Spanish and benchmark the best models in
order to obtain a leaderboard. Thus, we fine-tune more than $3000$
combinations for the best available models on $12$ public corpora to achieve
our goal.

In the rest of this paper we present a brief summary on the evolution of NLP
in the latest years in Section 2. In section 3 we make an overview of some of
the latest corpora released in the clinical domain for the Spanish language.
Then, in Section 4, we list the best Spanish language models (both general and
domain specific). In Section 5 we explain what corpora, models and metrics we
used for the benchmark. Finally, in Section 6, we present a detailed benchmark
of a selection of the models and corpora in order to get insights of the
situation of this research field.

Contributions We make the following contributions:

  * •

We create the first public benchmark for clinical Spanish language models.

  * •

We prove that there is a lack of good quality clinical Spanish language
models.

  * •

We make a survey with the most relevant corpora and models released in the
recent years.

##  2 Previous work

This last decade has experienced an exponential growth in NLP. We went from
word embedding models like Word2vec [14] and GloVe [15], to gigantic language
models with billions of parameters like PaLM [16].

This growth can be explained with the, also exponential, development of
specific hardware dedicated to speed up the matrix calculations made in neural
networks like NVIDIA’s H100 GPUs [17], Google’s TPUs [18] or AWS’s Trainium
[19]. With such an evolution in hardware, the training of neural architectures
for the processing of sequences such as Recurrent Neural Networks (RNNs) [20,
21], bidirectional RRNs [22] and Long Short-term Memory (LSTM) [23] (and
modifications like GRUs [24]) became practical.

These networks were used along word embeddings in order to train models for
specific downstream tasks such as translation, token classification (or NER),
text generation, text classification, etc. Every vector model encoded just a
word and did not take into account context. Then, some models started to
improve the encoding with subword information like FastText [25]. The biggest
breakthrough here came in 2017 with the Transformer architecture and the
contextualized embeddings [26]. After that a lot of new architectures appeared
such as ELMo [27] and FLAIR [28].

However, the one that definitely changed everything after the Transformer was
BERT [3] in 2018, a bidirectional encoder model based in the transformer that
obtained much better results than the state of the art by that time. After
that, hundreds of BERT-inspired models (like RoBERTa [29] or DeBERTa [30],
just to name a few) were published [31]. Huggingface’s Transformers library
[32] and subsequently their Datasets library [33] and Hugginface Hub [34]
helped to democratize these new architectures.

In parallel other models based in the Transformer were being created like GPT
[35] which was, opposite to BERT, just the generative part of the
Transformers. After that, the architecture of GPT-2 [36] stacked more and more
transformers layers one on the top of another. GPT-3 [4] proved a really big
change also, although the main difference with the architecture of their
predecessors was just the size. However, it obtained outstanding results even
in few-show scenarios without the need of fine-tuning in demanding benchmarks
like WinoGrande [37]. This set a precedent, after all the decade of Open
Sourced models, GPT-3 would not be made available to the public, companies
understood their real potential.

After GPT-3, in 2022 ChatGPT was trained using Reinforcement Learning from
Human Feedback (RLHF) [38] and suddenly, the chatbot revolution began.
Companies announced their models but did not release them like GPT-4 [5] or
LLaMA [39].

In our specific context of domain specific clinical models, they tried to
follow the main enhancements of this ever-growing trend. For the clinical
domain in Spanish language, most of the improvements were made in shared tasks
such as IberLEF [40, 41] and BioNLP open shared tasks, [42], [43].

##  3 Corpora

NLP requires a great amount of data in order to obtain good results [44], so
it its very important to gather as much data as possible. When talking about
NLP data we can classify it into two big groups depending whether the data is
structured (labeled) or not.

###  3.1 Unannotated corpora

Corpora are very valuable in NLP because they are often very big and can be
used to pre-train large language models without the need of labels. Although
most of the times these corpora can be crawled online [45], this is not always
possible in the clinical domain because data such as Electronic Health Records
are not publicly available and contain personal data.

Note that this is not the case with biomedical data where PubMed papers and
public clinical trials can be used [46].

In this section we will review the most relevant corpora in Spanish for the
clinical domain.

####  3.1.1 SPACCC

The Spanish Clinical Case Corpus (SPACCC) [47] is a collection of $1000$
clinical cases from SciELO. A clinician classified the original dump of
clinical cases into realistic clinical cases and not suitable for this
specific corpus.

The corpus has $382470$ tokens, a cc-by-4.0 licence and is publicly available
111https://zenodo.org/record/2560316.

####  3.1.2 European Clinical Case Corpus

European Clinical Case Corpus (E3C) [48] is a multilingual collection of
clinical cases. The corpus contains clinical cases in English, French, Basque
and Spanish.

We will focus in the Spanish part, which is a collection of other well known
corpora such as SPACCC, NUBes (detailed down in Section 3.2.13), IULA+
(detailed down in Section 3.2.9) and a small amount of newly crawled SciELO
clinical cases.

Although the corpus is mainly unlabeled, it has some labeled sub-sets.
Specifically, the corpus has 3 different parts:

  * •

Gold Standard: Named entities of time and factuality in THYME standard and
medical entities in SNOMED-CT and ICD-10 standards. Annotated manually. This
corpus has $20k$ tokens.

  * •

Silver Standard: The same as above but with a bigger size and annotated
automatically. This corpus has $70k$ tokens.

  * •

Unnanotated data: All the data collected, about $1M$ tokens.

The corpus has a cc-by-nc-4.0 licence and is publicly available
222https://live.european-language-grid.eu/catalogue/corpus/7618.

Corpus | Tokens |
---|---|---
E3C | 1M |
SPACCC | 0.38M |
Table 1: Summary of Spanish Clinical Corpora

###  3.2 Labeled data

Labeled corpora are used to train and evaluate the models in downstream tasks
such as text classification or NER.

Given the nature of the clinical data, there are few corpora and most of them
have a relatively small size compared to corpora in other domains. Also,
almost all of these corpora come from shared tasks.

A summary of all the corpora is presented in Table 2.

####  3.2.1 BARR2

Biomedical Abbreviation Recognition and Resolution 2nd Edition (BARR2) [49] is
a NER corpus of clinical cases from SciELO where the entities are
abbreviations (Short Form, SF) and their respective definition (Long Form,
LF).

The corpus consists of two sub-tasks:

  * •

Detection of pairs SF/LF mentions in text. This is some kind of multi label
classification given that there is no need to classify the tokens
individually.

  * •

Detection of SF mentions and their characters offset (NER) and generating
their corresponding LF.

While this corpus has been annotated by different experts, there are no
annotation guidelines nor consistency analysis so it may not be as curated as
a proper Gold Standard corpus.

The corpus has a size of $318$, $146$ and $220$ train, validation and test
samples respectively. The test split comes from a bigger background
unnanotated split of $2879$ unnanotated samples. The corpus has a cc-by-4.0
licence and is publicly available 333https://temu.bsc.es/BARR2/datasets.html.

####  3.2.2 CANTEMIST

CANTEMIST (CANcer TExt Mining Shared Task – tumor named entity recognition)
[50] corpus is a Named Entity Recognition (NER) corpus that focuses in tumor
morphology concepts in clinical texts written in Spanish. Specifically.
CANTEMIST is focused in detecting mentions of tumor morphology terms and link
them to their corresponding eCIE-O-3.1 codes (the Spanish branch of
International Classification of Diseases for Oncology). The corpus includes
diverse clinical cases covering different cancer types, both common and rare.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus consists of three independent sub-tasks:

  * •

CANTEMIST-NER track: In this track, the goal is to find tumor morphology
mentions in medical documents.

  * •

CANTEMIST-NORM track: This track focuses on clinical concept normalization or
named entity normalization. The entities are tumor morphology entity mentions
along with their corresponding eCIE-O-3.1 codes.

  * •

CANTEMIST-CODING track: The objective of this track is to return a ranked list
of ICD-O-3 codes for each document.

The corpus has a size of $501$, $500$ and $300$ train, validation and test
samples respectively. The test split comes from a bigger background
unnanotated split of $4932$ unnanotated samples. The corpus has a cc-by-4.0
licence and is publicly available 444https://zenodo.org/record/3978041.

####  3.2.3 CARES

CARES (Corpus of Anonymised Radiological Evidences in Spanish) [51] is a
corpus composed of different radiology reports from several areas. These
reports are annotated to hierarchically classify their ICD-10 codes.
Specifically, each report has an annotation for the areas of the body that the
report is talking about, their corresponding IDC-10 chapters, the ICD-10 codes
and the sub-codes.

While this corpus has been annotated by different experts, there are no
annotation guidelines nor consistency analysis so it may not be as curated as
a proper Gold Standard corpus.

The corpus has a size of $2250$, and $966$ train and test samples
respectively. The corpus has a afl-3.0 licence and is publically available
555https://huggingface.co/datasets/chizhikchi/CARES.

####  3.2.4 Chilean Waiting List Corpus

The Chilean Waiting List Corpus (CWLC) [52] is a NER clinical corpus made up
of anonymized referrals from waiting lists in Chilean public hospitals. 48% of
the entities in the corpus are nested and the main entities are "Finding",
"Procedure", "Family member", "Disease", "Body Part", "Medication" and
"Abbreviation".

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $9000$ train samples. The corpus has a cc-by-nc-
sa-4.0 licence and is publically available
666https://zenodo.org/record/7555181.

####  3.2.5 CodiEsp

CodiEsp (Clinical Case Coding in Spanish Shared Task) [53] is a corpus of
clinical cases from a variety of medical topics, including oncology, urology,
cardiology, pneumology or infectious diseases. The corpus has 3 different
tasks:

  * •

CodiEsp-D: Multi-label classification of ICD10-CM (CIE10 Diagnóstico in
Spanish).

  * •

CodiEsp-P: Multi-label classification of ICD10-CM (CIE10 Procedimiento in
Spanish).

  * •

CodiEsp-X: NER task with the purpose of marking those segments in the text
where evidences for the CIE10 labels can be found.

The corpus provides a manually annotated Gold Standard generated by a
practicing physician and a clinical documentalist. There are also annotation
guidelines and consistency analysis to ensure the quality of the corpus.

The corpus has a size of $500$, $250$ and $250$ train, development and test
samples respectively. The test split comes from a bigger background
unnanotated split of $2751$ unnanotated samples. The corpus has a cc-by-4.0
licence and is publically available 777https://zenodo.org/record/3837305.

####  3.2.6 CT-EBM-SP

CT-EBM-SP (Clinical Trials for Evidence-based Medicine in Spanish) [54] is a
NER corpus made up of PubMed and SciELO abstracts and clinical trials
announcements published in the European Clinical Trials Register and
Repositorio Español de Estudios Clínicos. Although these are not strictly
clinical sources, abstracts are rather general and simple descriptions and,
given the size of the corpus, it can be very useful to enhance the
generalization of the models.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $12600$, $4510$ and $4550$ train, development and
test samples respectively. The corpus has a cc-by-4.0 licence and is
publically available 888https://huggingface.co/datasets/lcampillos/ctebmsp.

####  3.2.7 DisTEMIST

The DisTEMIST (Disease Text Mining Shared Task) corpus [55] is a collection of
clinical cases written in Spanish, covering various medical specialties. The
corpus is annotated with disease mentions (i.e. NER corpus) that have been
standardized using Snomed-CT terminology.

An expert filtered the gathered data so only relevant documents were included.
After that, the data was preprocessed to delete redundant information such as
figure references or citations. The corpus provides a manually annotated Gold
Standard generated by experts. There are also annotation guidelines and
consistency analysis to ensure the quality of the corpus.

The corpus consists of two independent sub-tasks:

  * •

DISTEMIST-entities subtrack: NER to find diseases, without classifying them
into any kind of taxonomy.

  * •

DISTEMIST-linking subtrack: NER to find disease mentions and assign a Snomed-
CT term.

There is also a multilingual version of the corpus containing data in English,
Portuguese, Catalan, Italian, French, and Romanian.

The corpus has a size of $750$ and $250$ train and test samples respectively.
The test split comes from a bigger background unnanotated split of $3000$
unnanotated samples. The corpus has a cc-by-4.0 licence and is publically
available 999https://zenodo.org/record/7614764.

####  3.2.8 eHealthKD

eHealthKD (eHealth Knowledge Discovery) [56] is a cross-domain NER corpus with
samples gathered from MedlinePlus and Wikinews. These sources create a very
heterogeneous corpus that can be very useful to prove the generalization of
the models. The corpus has 2 different tasks:

  * •

Subtask A: NER to recognize concepts, actions, predicates and references.

  * •

Subtask B: Relation extraction from the previous NER task.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $1500$, $50$ and $150$ train, development and test
samples respectively. There are also $50$ and $150$ development and test
samples in English, however they are not taken into account here because they
are out of scope. The corpus has a cc-by-nc-sa-4.0 licence and is publically
available 101010https://github.com/ehealthkd/corpora/tree/master.

####  3.2.9 IULA-SCRC

IULA Spanish Clinical Record Corpus (SCRC) [57] is a NER corpus with negation
entities extracted from various anonymized clinical records. Besides IULA-
SCRC, there is also another version of the corpus annotated with NUBes’
annotation guidelines called IULA+.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $3194$ train samples. The corpus has a cc-by-sa-3.0
licence and is publically available 111111https://github.com/Vicomtech/NUBes-
negation-uncertainty-biomedical-corpus.

####  3.2.10 IxaMed-GS

IxaMed-GS [58] is a NER corpus composed of real (anonymized) Electronic Health
Records. The entities annotated are related to diseases and drugs and there
are also annotations of relationships between entities indicating adverse drug
reaction events.

The corpus provides a manually annotated Gold Standard generated by experts in
pharmacology and pharmacovigilance. There are also annotation guidelines and
consistency analysis to ensure the quality of the corpus.

The corpus has a size of $1875$, $1097$ and $995$ train, development and test
samples respectively. The corpus is not publically available and it can only
be obtained through an agreement with its authors.

####  3.2.11 LivingNER

LivingNER [59] is a corpus focused on species, pathogens and food in clinical
case reports.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus consists of three independent sub-tasks:

  * •

subtask 1 (LivingNER – Species NER): NER to identify either a species of a
disease or human entities.

  * •

subtask 2 (LivingNER – Species Norm): NER to identify species of a disease and
classifying them into NCBITax taxonomy.

  * •

subtask 3 (LivingNER – Clinical IMPACT): Multi-label classification. Classes:
high impact (or not) and NCBI IDs.

The corpus has a size of $1000$, $500$ and $485$ train, development and test
samples respectively. The test split comes from a bigger background
unnanotated split of $~{}13000$ unnanotated samples. The corpus has a cc-
by-4.0 licence and is publically available
121212https://zenodo.org/record/7614764.

####  3.2.12 MEDDOCAN

MEDDOCAN (Medical Document Anonymization Track) [60] is a corpus of clinical
cases sampled from the SPACCC corpus and enriched with synthetic personal
information. The NER entities of the corpus range from the personal data of a
patient to even the personal data of the doctors.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $500$, $250$ and $250$ train, development and test
samples respectively. The test split comes from a bigger background
unnanotated split of $2000$ unnanotated samples. The corpus has a cc-by-4.0
licence and is publically available
131313https://huggingface.co/datasets/bigbio/meddocan.

####  3.2.13 NUBes

The NUBes (Negation and Uncertainty annotations in Biomedical texts in
Spanish) corpus [61] is a collection of $29,682$ sentences from anonymised
documents from a Spanish private hospital annotated with negation and
uncertainty. The sentences come from documents from the following kinds: Chief
Complaint, Present Illness, Physical Examination, Diagnostic Tests, Surgical
History, Progress Notes, and Therapeutic Recommendations.

This is the biggest negation corpus for Spanish clinical domain. Samples have
been annotated for a NER task with all the negations and the uncertainty that
is reflected in the text. The corpus provides a manually annotated Gold
Standard generated by expert linguists. There are also annotation guidelines
and consistency analysis to ensure the quality of the corpus.

The corpus has no explicit testing splits so we can consider that the train
split has $29,682$ samples. The corpus has a cc-by-sa-3.0 licence and is
publically available 141414https://github.com/Vicomtech/NUBes-negation-
uncertainty-biomedical-corpus.

####  3.2.14 PharmaCoNER

The PharmaCoNER (Pharmacological Substances, Compounds and proteins and Named
Entity Recognition) corpus [62] is a random sample of the SPACCC corpus that
has been annotated manually for two different tasks.

  * •

NER: The entities are drugs and chemicals that appear in the text. Depending
on their nature, they can be "normalizable" using SNOMED-CT, "non
normalizable", "proteins" or "unclear".

  * •

Concept indexing: Task to identify all the SNOMED-CT IDs corresponding to each
of the entities from the first task.

The corpus provides a manually annotated Gold Standard generated by medicinal
chemistry experts. There are also annotation guidelines and consistency
analysis to ensure the quality of the corpus.

The corpus has a size of $500$, $250$ and $250$ train, development and test
samples respectively. The test split comes from a bigger background
unnanotated split of $2751$ unnanotated samples. The corpus has a cc-by-4.0
licence and is publically available 151515https://zenodo.org/record/4270158.

####  3.2.15 SocialDisNER

SocialDisNER (Mining social media content for disease mentions) [63] is a
collection of health-related tweets focused on disease mentions. Although it
is not strictly a clinical corpus, its unique nature of informal language
talking about health can be very useful to check the generalization of the
models.

The corpus provides a manually annotated Gold Standard generated by experts.
There are also annotation guidelines and consistency analysis to ensure the
quality of the corpus.

The corpus has a size of $6000$, $2000$ and $2000$ train, development and test
samples respectively. The corpus has a cc-by-4.0 licence and is publically
available 161616https://zenodo.org/record/6803567.

corpus | Train samples | Val samples | Test Samples | Problem type
---|---|---|---|---
BARR2 | 318 | 146 | 220 | NER
CANTEMIST | 501 | 500 | 300 | NER
CARES | 2250 | 0 | 996 | Classification
CWLC | 9000 | 0 | 0 | NER
CodiEsp | 500 | 250 | 250 | Classification
CT-EBM-SP | 12600 | 4510 | 4550 | NER
DisTEMIST | 750 | 0 | 250 | NER
eHealthKD | 1500 | 50 | 150 | NER
IULA-SCRC | 3194 | 0 | 0 | NER
IxaMed-GS | 1875 | 1097 | 995 | NER
LivingNER | 1000 | 500 | 485 | NER
MEDDOCAN | 500 | 250 | 250 | NER
NUBes | 29,682 | 0 | 0 | NER
PharmaCoNER | 500 | 250 | 250 | NER
SocialDisNER | 6000 | 2000 | 2000 | NER
Table 2: Summary of Spanish Clinical Corpora

##  4 Models

Despite some recent attempts on creating static word embeddings like Spanish
Clinical Embeddings [64], the Word embeddings for the Spanish clinical
language [65] and even contextual embeddings like the Spanish Clinical Flair
[66] we will not emphasize on those because it has been proven that
transformer-based language models outperform them in most of the situations
[67].

However, given that there are not that many Spanish clinical language models,
we will also list general models with good results in the Spanish language.

###  4.1 BETO

BETO [68] was the first Spanish language model with BERT architecture. The
model was trained with the Spanish Unnanotated Corpora (SUC) [69] (4GB corpus)
and has 110M parameters (the same as BERT). Although it was released in
$2020$, the model is still useful as a baseline given its simplicity and
stability.

The model has cc-by-4.0 licence and is publically available
171717https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased.

###  4.2 MarIA

The next general Spanish language model is MarIA [70]. Released in $2021$,
MarIA is based in the RoBERTa [29] architecture. The base version of the model
has 117M parameters and the large one has 355M, with the large version having
the exact same architecture and size than the original RoBERTa-large. The
corpus used to train this model is a web crawl performed by the Spanish
National Library with a size of 570GB once clean.

Although both models have good results, this paper will focus on the large
model given that the results are better and the model is still small enough to
fit in a single GPU.

The model has apache-2.0 licence and is publicly available
181818https://huggingface.co/PlanTL-GOB-ES/roberta-large-bne.

###  4.3 RigoBERTa

RigoBERTa is a DeBERTa-based model released in $2022$ by the Instituto de
Ingeniería del Conocimiento (IIC) [71]. The model has 134M parameters and was
trained on a mix of various crawl sources (such as OSCAR [45] and mC4 [72]),
Spanish news articles and the Spanish Unnanotated Corpora (SUC) [69].

Later, in $2023$, the second version of the model has been released, RigoBERTa
2. This is the model that will be tested in the benchmark.

The model has all rights reserved and is not publically available, although it
can be obtained through its autors.

###  4.4 XLM-RoBERTa

Before jumping into the specific clinical models, it is important to note that
there are some very good performing multilingual models that we should take
into account given the small amount of purely Spanish language models publicly
available. Otherwise we could leave very good models out [73].

XLM-RoBERTa [74] is a multilingual version of RoBERTa trained in 2019. The
model was trained with the CC100 corpus [74]. This corpus has 2.5TB in size
and has data of $100$ different languages.

There are several models trained with this architecture and corpus: base,
large, xl and xxl sizes with 250M, 560M, 3.5B and 10.7B parameters
respectively. For this paper we will stick with the large model given that it
fits in a GPU and is stable enough given the sizes of the corpora used in the
benchmark.

The model has MIT licence and is publically available
191919https://huggingface.co/xlm-roberta-large.

###  4.5 DeBERTaV3

DeBERTaV3 [75] is the latest DeBERTa [30] based model and it was released in
2021. Although DeBERTa is a family of English language models, DeBERTaV3
includes a multilingual model too, mDeBERTaV3.

The model was trained with the CC100 corpus, the same as XLM-RoBERTa. The
resulting multilingual model has 276M parameters which starts to be in the big
side for a single GPU.

The model has MIT licence and is publically available
202020https://huggingface.co/microsoft/mdeberta-v3-base.

###  4.6 GPT family

After the release of ChatGPT in late 2022 and GPT-4 in early 2023, this family
of models brought a revolution both to the NLP research community and to the
society in general [76]. Although these are generative models, they shine for
their ability to tackle any kind of problem in a zero or few shot learning
approach.

These models are not open source and can only be accessed via the OpenAI API
212121https://platform.openai.com/docs/api-reference. Therefore we can only
make educated guesses about how they have been trained and with which corpora.
There are also some ethical concerns regarding the closed nature of the models
[6], even bigger in this medical domain where a lot of medical institutions
have strict security protocols that forbid them to upload personal data to the
internet.

For instance, ChatGPT is an evolution of GPT-3 [4] fine-tuned with some kind
of RLHF (Reinforcement Learning from Human Feedback) [77]. In the case of
GPT-4 [5] we can assume that is an evolution of GPT-3 with a bigger corpora
and more parameters.

There is evidence that GPT-4 has improved a lot the results in some clinical
NLP benchmarks in English [78], however it has not been tried in Spanish or in
more encoder-focused tasks like Named Entity Recognition. In the Section 6.1
we will try to use these models in some clinical corpora to check their
viability in every aspect.

###  4.7 Galén family

The Galén [79] models are domain adaptations [80] of preexisting Spanish
capable language models like mBERT, BETO and XLM-RoBERTa published in 2021.
This specific domain adaptation has been performed by further pretraining the
above mentioned models with a clinical corpus extracted from the Galén
Oncology Information System [81].

All models are publically available under a MIT licence
222222https://github.com/guilopgar/ClinicalCodingTransformerES, however they
are not published in the Model Hub, only their weights are published which
makes it harder to reuse them.

###  4.8 bsc-bio-ehr-es

bsc-bio-ehr-es [82] is a biomedical model based in the RoBERTa architecture
published in 2022. The model has been trained from scratch (instead of a
domain adaptation) using a large biomedical corpora and also a smaller corpus
of electronic health records.

There are two models in these series, bsc-bio-es and bsc-bio-ehr-es where the
first is trained only with the first corpora and the later is trained with all
the gathered data. Both models have 125M parameters. In this paper we will
only compare the bsc-bio-ehr-es model given that the authors report better
results for this use case.

Most of the data sources for these models are not publically available so we
cannot assert the claims of the authors.

The model has apache-2.0 licence and is publically available
232323https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es.

###  4.9 Other Generative Large Language Models

Since the release of ChatGPT, a lot of new generative Large Language Models
have been created. Although they are way out of scope for this study and most
of them are in English, we will mention the most relevant ones that may be
featured in future work. Also we carry out a preliminary study on chat models
(OpenAI models) to check their feasibility in Section 6.1.

Most of the models have a focus in English language, but due to their size,
they can all perform well in Spanish. Most notoriously we can find Llama [39]
and Llama2 [83] with some of their biggest versions being multilingual, Falcon
[84] with a 40B multilingual version, Vicuna [85], BLOOM [86] and PaLM [16].

##  5 Method

Formally, we will evaluate results using the F1 score for the NER corpora and
F1 with micro average classification tasks. However we will also take into
account the availability of the models and the computational cost of running
them.

The main evaluation has been designed for the encoder models and they are
fine-tuned using the chosen corpora. For the metaparameters optimization we
designed a grid of parameters for all the models, described in the Appendix A.

###  5.1 Corpora

Most of the clinical corpora in Spanish are either NER corpora or non-labeled
corpora so there is a relatively low variety of corpora in order to make a
benchmark. Having this into account, we selected the following corpora for the
benchmark:

  * •

cantemist: Good example of well annotated NER corpus (3.2.2).

  * •

caresA: Multi-label classification of the body area mentioned on the
radiological report (3.2.3).

  * •

caresC: Multi-label classification of the ICD-10 chapters that correspond to
the radiological report (3.2.3).

  * •

ctebmsp: (3.2.6).

  * •

distemist: Good example of well annotated NER corpus (3.2.7).

  * •

ehealth_kd: Subtask A, NER with very different entities (3.2.8).

  * •

livingner1: LivingNER task 1. Good example of well annotated NER corpus
(3.2.11).

  * •

livingner3: LivingNER task 3. Multi-label classification corpus.(3.2.11).

  * •

meddocan: Anonymization corpus in clinical cases (3.2.12).

  * •

nubes: Big NER corpus and with negation entities (3.2.13).

  * •

pharmaconer: Good example of well annotated NER corpus (3.2.14).

  * •

socialdisner: Informal language talking about health (3.2.15).

###  5.2 Models

Due to the limited time and computational budget, not every available model
has been tested in the benchmark. Next we will list the chosen models:

  * •

BETO: We use BETO as a baseline for Spanish language models.

  * •

BETO_Galén: A domain adaptation of BETO, it should be better for these
experiments.

  * •

bsc-bio-ehr-es: Smaller and trained only with relevant data.

  * •

MarIA: The best open-sourced Spanish only encoder language model.

  * •

mDeBERTaV3: Second-best performing multilingual model.

  * •

RigoBERTa 2: The best Spanish only encoder language model.

  * •

XLM-R_Galén: A domain adaptation of the best model for Spanish.

  * •

XLM-RoBERTa-Large: The best results for Spanish language.

We will also show some experiments with GPT-3 and GPT-4 that we did not
include in the final benchmark due to some inherent limitations to generative
language models and the limited OpenAI API functionality.

###  5.3 Public benchmark

For the sake openness, all the final fine-tuned models, as well as all the
corpora used and the code for the benchmark will be publically available
242424https://github.com/iiconocimiento/survey-spanish-clinical-language-
models. Note that neither the RigoBERTa 2 fine-tuned models nor the base model
can be publically available due to license constrains.

Reuploading the corpora and gathering them all in the Hugging Face Hub allows
us to use the Hugging Face Leaderboards
252525https://huggingface.co/spaces/autoevaluate/leaderboards to create the
first benchmark for Spanish Clinical Language Models.

All the information about the resources will be displayed in Appendix B.

##  6 Evaluation and results

In Table 3 we can see the results of the benchmark. RigoBERTa 2 obtains the
best results by far, winning in $6$ out of the $12$ corpora. Overall it was
also the most stable model, consistently obtaining good results, not like
mDeBERTaV3 that, despite having very good results, obtained an exceptionally
bad result in the livingner3 corpus. XML-RoBERTa-Large obtains also very good
results and is the best open-sourced model available.

Paradoxically, the more specific the model is, the worse that its results are.
At a first glance it might look like the bigger model always wins, however we
must remember that both XLM-R_Galén and BETO_Galén are domain adaptations of
bigger models. Specifically XLM-R_Galén uses as base model XML-RoBERTa-Large,
the best performing model in the benchmark. Consequently this means that it is
not always worth to adapt a model to a certain domain if the corpus for the
adaptation is not big or good enough.

bsc-bio-ehr-es, the only model trained from scratch in this benchmark obtains
relatively good results given its size and the relatively small amount of data
used during the training process. However, this model does not obtain any
improvement against the general Spanish language models or the multilingual
ones.

| Spanish Only models | Multilingual
---|---|---
| Clinical models | General Models
Corpus | XLM-RG | BETO_G | bsc-ehr | BETO | MarIA | RigoB2 | mDeB3 | XLM-RL
cantemist | 0.898 | 0.802 | 0.864 | 0.898 | 0.902 |  0.903 | 0.890 |  0.904
caresA | 0.989 | 0.977 | 0.991 | 0.992 | 0.992 |  0.997 | 0.993 |  0.994
caresC | 0.823 | 0.551 |  0.862 | 0.835 | 0.840 | 0.854 | 0.756 |  0.847
ctebmsp | 0.881 | 0.726 | 0.876 | 0.880 | 0.877 |  0.907 | 0.902 |  0.906
distemist | 0.759 | 0.346 | 0.759 | 0.801 | 0.793 |  0.832 | 0.808 |  0.817
ehealth_kd | 0.830 | 0.658 | 0.836 | 0.843 | 0.836 |  0.865 | 0.844 |  0.871
livingner1 | 0.907 | 0.646 | 0.938 | 0.938 | 0.939 |  0.951 |  0.953 | 0.949
livingner3 | 0.500 | 0.000 | 0.604 | 0.626 |  0.644 | 0.621 | 0.153 |  0.606
meddocan | 0.947 | 0.682 | 0.967 | 0.957 | 0.977 |  0.979 | 0.974 |  0.978
nubes | 0.908 | 0.762 | 0.903 | 0.908 | 0.911 |  0.915 | 0.919 |  0.920
pharmaconer | 0.915 | 0.708 | 0.904 | 0.908 | 0.914 |  0.927 | 0.922 |  0.924
socialdisner | 0.919 | 0.777 | 0.921 | 0.915 | 0.920 |  0.943 | 0.935 |  0.941
Average | 0.853 | 0.621 | 0.869 | 0.873 | 0.877 |  0.890 | 0.833 |  0.887
Wins group | 0 | 0 | 1 | 0 | 1 |  10 | 1 |  11
Wins total | 0 | 0 | 1 | 0 | 1 |  6 | 1 |  3
Table 3: Results with models grouped into Spanish Only and Multilingual
models. Spanish Only models are splitted into Clinical models and General
models. Best results for every group are underlined and best results overall
are in bold. We also report the sum of the scores for every model, wins in own
group and wins overall. The metric is micro F1 for classification models and
F1 for NER models. XLM-RF = XLM-R_Galén, BETO_G = BETO_Galén, bsc-ehr = bsc-
bio-ehr-es, RigoB2 = RigoBERTa 2, mDeB3 = mDeBERTaV3, XLM-RL = XLM-RoBERTa-
Large

In the Figure 1 we can see a Nemenji plot with a classification of the models
given the results of the benchmark. We can see that, although the first models
are within the critical distance, both RigoBERTa 2 and XLM-RoBERTa-Large are
statistically better than all the domain adaptations.

Figure 1: Nemenji plot of the results

###  6.1 GPT family

We made two attempts to use this technology, first with the NER corpus
CANTEMIST and then with the multi-label classification corpus CaresC.
Generative models are not prepared for any of these tasks so we had to tune
the prompts in order to achieve what we wanted. As we will see with the
results of these experiments, it is not easy to use these models properly, but
it is worth giving them a chance [87].

####  6.1.1 Named Entity Recognition

First, in the NER corpus we observed that the models (both GPT-3 and GPT-4)
are capable of finding some entities in the text. However there is no way to
obtain a comparable score if those entities do not have their respective
positions also labeled in the text. It is technically possible to perform
token classification tasks such as NER with generative language models: we
just need to add a classification head over each output embedding. However,
the OpenAI API does not allow to do it straightforwardly and transforms the
task into a text-to-text problem which turned out to be inefficient and error
prone.

While the models more or less understood our queries, there were a lot of
formatting errors and most of the times every entity was positioned wrong. In
Appendix C.1 a more detailed description of both the prompts and parameters
used can be found.

After a lot of attempts without a good solution we exhausted our budged for
this experiment and we decided to stop experimenting and not try any fine-
tuning or few shot evaluation due to the high risk of a bad result and the
high cost of the platform. This budget, similarly with the one showed in Table
4, was higher that what would have costed a GPU machine to fine-tune an
encoder model.

We also tried the edits endpoint of the API to insert the NER entity after its
appearance, but it did not work well either.

####  6.1.2 Classification

Next we tried with a multi-label classification corpus CaresC. Although it is
relatively simple to correctly prompt these models in order to perform binary
or multiclass classification, more precise prompting had to be done to achieve
consistent and good results. All the details of these process can be found in
the Appendix C.2.

Once we obtained good enough prompts, we evaluated the performance in a few-
shot approach. Also a fine-tunning was attempted, taking into account that the
fine-tuning API does not support multi-label classification, so the training
process was a general text generation fine-tuning. In the Table 4 we can see a
detailed summary of the results and costs.

model | F1 | Cost ($)
---|---|---
text-davinci-003 (few-shot) | 0.268 | 11
gpt-4 (few-shot) | 0.386 | 46
bsc-bio-ehr-es | 0.862 | 20
text-ada-001(fine-tune) | 0.506 | 5
text-davinci-003 (fine-tune) | 0.621 | 166
Table 4: Table showing the micro F1 obtained by the models and their cost in
the CaresC corpus. The cost for the bsc-bio-ehr-es was calculated taking into
account the duration of the training process and the cost of a p4d.24xlarge in
AWS. Note that gpt-4 is a chat-only model, not a completion model.

Surprisingly, we can see that when comparing GPT models against a far simpler
encoder-only model such as bsc-bio-ehr-es, the latter obtains the best results
by far. Within GPT models, there is a clear increase of quality when a fine-
tuning process is performed to the base models, up to a point that the few
shot models appear as a poor-quality but expensive alternative. Even when
using the most expensive GPT approach (fine-tune the Davinci model), we obtain
a significantly poor result in comparison to a classic encoder fine-tuning.
The only acceptable exception to all these bad results is the fine-tuning of
the Ada model, which is $4$ times cheaper than bsc-bio-ehr-es; however the
result in terms of quality is considerably worse.

There is also the concern that all the data used in the OpenAI API can be
privately used by OpenAI to improve their models, which can make these
solutions unusable for the clinical domain given the European data protection
laws.

All in all, these results show that this Large Generative Language Models
revolution does not apply to all fields, problem types and languages. There is
also more evidence of this in more thorough studies like the one by Chen et
al. [88]. Wang et al. [10] (similarly to Luo et al. [9] with BioGPT, instead
trained a ClinicalGPT in order to greatly improve the results in clinical
domain tasks). However it must be noted that all these efforts generated
English language models so they can be seen as future work for the Spanish
language.

##  7 Conclusion

Through this study we have observed that there is a fair amount of corpora and
models for the application of NLP to clinical problems in Spanish. However a
lot of that corpora is not publically available or does not have a proper Gold
Standard so models can easily learn from it and be evaluated.

We benchmarked the most promising models for the clinical domain in Spanish in
a variety of corpora to check their real performance, and we found out that
none of them is good enough to beat a big general multilingual model or a
closed-source commercial one. This holds true even if the clinical models are
domain adaptations of the best model open-source model. This means that there
is a lot of work to be done for the Spanish language in terms of publicly
available language models, and that we have room to improve.

Following those experiments, our main contribution was the release of a
standardized benchmark to test future Spanish language models for medical
applications and the gathering of a series of models and corpora in an easy to
access platform.

We also tried the revolutionary generative models and proved that they cannot
be used for all use cases: at this time there is no such thing as an
Artificial General Intelligence and there are more narrow and affordable
solutions that perform better for certain applications.

It is clear then, that we need better Spanish language models, both general
and domain specific. We also need a big corpora of Spanish clinical texts so
these models can be trained. We are aware that this is a big claim, but we
just cannot relegate ourselves to using multilingual models in this crucial
domain where people’s lives are at stake.

## Acknowledgments

This work was supported in part by the Instituto de Ingeniería del
Conocimiento and R&D&i ACCESS2MEET (PID2020-116527RB-I0) project supported by
MCIN AEI/10.13039/501100011033/. Additionally, this work has been supported by
"Intelligent and interactive home care system for the mitigation of the
COVID-19 pandemic" PRTR-REACT UE project.

## References

  * [1] David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors.  Ethnologue: Languages of the World.  SIL International, Dallas, Texas, twenty-sixth edition, 2023.
  * [2] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf.  Datasets: A community library for natural language processing.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [5] OpenAI.  Gpt-4 technical report, 2023.
  * [6] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models, 2023.
  * [7] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, sep 2019.
  * [8] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing, 2020.
  * [9] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.  BioGPT: generative pre-trained transformer for biomedical text generation and mining.  Briefings in Bioinformatics, 23(6), sep 2022.
  * [10] Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li.  Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation, 2023.
  * [11] Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud.  Drbert: A robust pre-trained model in french for biomedical and clinical domains, 2023.
  * [12] Rian Touchent, Laurent Romary, and Eric De La Clergerie.  CamemBERT-bio : Un modèle de langue français savoureux et meilleur pour la santé.  In Christophe Servan and Anne Vilnat, editors, 18e Conférence en Recherche d’Information et Applications
16e Rencontres Jeunes Chercheurs en RI
30e Conférence sur le Traitement Automatique des Langues Naturelles
25e Rencontre des Étudiants Chercheurs en Informatique pour le Traitement
Automatique des Langues, pages 323–334, Paris, France, 2023. ATALA.

  * [13] Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, and Rada Mihalcea.  A phd student’s perspective on research in nlp in the era of very large language models, 2023.
  * [14] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, 2013.
  * [15] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * [17] NVIDIA Corporation.  Nvidia h100 tensor core gpu.  Website, August 2023.
  * [18] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon.  In-datacenter performance analysis of a tensor processing unit, 2017.
  * [19] Amazon Web Services.  Amazon trainium.  Website, August 2023.
  * [20] John J. Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  Proceedings of the National Academy of Sciences of the United States of America, 79(8):2554–2558, 1982.
  * [21] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  Nature, 323(6088):533–536, 10 1986.
  * [22] M. Schuster and K.K. Paliwal.  Bidirectional recurrent neural networks.  IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [23] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [24] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.
  * [25] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information, 2017.
  * [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [27] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations, 2018.
  * [28] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf.  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [30] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  In International Conference on Learning Representations, 2021.
  * [31] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun.  A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, 2023.
  * [32] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Huggingface’s transformers: State-of-the-art natural language processing, 2020.
  * [33] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf.  Datasets: A community library for natural language processing, 2021.
  * [34] Huggingface.  Huggingface’s hub.  Website, August 2023.
  * [35] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training, 2018.
  * [36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [37] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale, 2019.
  * [38] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [40] Julio Gonzalo, Manuel Montes-y Gómez, and Paolo Rosso.  Iberlef 2021 overview: Natural language processing for iberian languages.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021), pages 1–15. CEUR Workshop, 2021.
  * [41] Julio Gonzalo, Manuel Montes-y Gómez, and Francisco Rangel.  Overview of iberlef 2022: Natural language processing challenges for spanish and other iberian languages.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop, 2022.
  * [42] Liliya Akhtyamova, Paloma Martínez, Karin Verspoor, and John Cardiff.  Testing contextualized word embeddings to improve ner in spanish clinical case narratives.  IEEE Access, 8:164717–164726, 2020.
  * [43] Renzo M Rivera-Zavala and Paloma Martínez.  Analyzing transfer learning impact in biomedical cross-lingual named entity recognition and normalization.  BMC bioinformatics, 22(1):1–23, 2021.
  * [44] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [45] Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot.  Towards a Cleaner Document-Oriented Multilingual Crawled Corpus.  arXiv e-prints, page arXiv:2201.06642, January 2022.
  * [46] Casimiro Pio Carrino, Jordi Armengol-Estapé, Ona de Gibert Bonet, Asier Gutiérrez-Fandiño, Aitor Gonzalez-Agirre, Martin Krallinger, and Marta Villegas.  Spanish biomedical crawled corpus: A large, diverse dataset for spanish biomedical language models, 2021.
  * [47] Ander Intxaurrondo.  Spaccc, November 2018.  Funded by the Plan de Impulso de las Tecnologías del Lenguaje (Plan TL).
  * [48] Anne-Lyse Minard, Roberto Zanoli, Begoña Altuna, Manuela Speranza, Bernardo Magnini, and Alberto Lavelli.  European clinical case corpus.  Dataset (Text corpus), August 09 2021.
  * [49] Ander Intxaurrondo, Montserrat Marimon, Aitor Gonzalez-Agirre, Jose Antonio Lopez-Martin, Heidy Rodriguez, Jesus Santamaria, Marta Villegas, and Martin Krallinger.  Finding mentions of abbreviations and their definitions in spanish clinical cases: The barr2 shared task evaluation results.  IberEval@ SEPLN, 2150:280–289, 2018.
  * [50] A Miranda-Escalada, E Farré, and M Krallinger.  Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020.
  * [51] Mariia Chizhikova, Pilar López-Úbeda, Jaime Collado-Montañez, Teodoro Martín-Noguerol, Manuel C. Díaz-Galiano, Antonio Luna, L. Alfonso Ureña-López, and M. Teresa Martín-Valdivia.  Cares: A corpus for classification of spanish radiological reports.  Computers in Biology and Medicine, 154:106581, 2023.
  * [52] Pablo Báez, Fabián Villena, Matías Rojas, Felipe Bravo-Marquez, and Jocelyn Dunstan.  The chilean waiting list corpus, November 2020.
  * [53] Antonio Miranda-Escalada, Aitor Gonzalez-Agirre, Jordi Armengol-Estapé, and Martin Krallinger.  Overview of automatic clinical coding: Annotations, guidelines, and solutions for non-english clinical cases at codiesp track of clef ehealth 2020\.  CLEF (Working Notes), 2020, 2020.
  * [54] Leonardo Campillos-Llanos, Ana Valverde-Mateos, Adrián Capllonch-Carrión, and Antonio Moreno-Sandoval.  A clinical trials corpus annotated with umls© entities to enhance the access to evidence-based medicine.  BMC Medical Informatics and Decision Making, 21(1):1–19, 2021.
  * [55] Antonio Miranda-Escalada, Luis Gascó, Salvador Lima-López, Eulàlia Farré-Maduell, Darryl Estrada, Anastasios Nentidis, Anastasia Krithara, Georgios Katsimpras, Georgios Paliouras, and Martin Krallinger.  Overview of distemist at bioasq: Automatic detection and normalization of diseases from clinical texts: results, methods, evaluation and multilingual resources.  In Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings, 2022.
  * [56] Alejandro Piad-Morfis y Suilan Estevez-Velarde y Yoan Gutierrez y Yudivian Almeida-Cruz y Andrés Montoyo y Rafael Muñoz.  Overview of the ehealth knowledge discovery challenge at iberlef 2021\.  Procesamiento del Lenguaje Natural, 67(0):233–242, 2021.
  * [57] Montserrat Marimon, Jorge Vivaldi, and Núria Bel Rafecas.  Annotation of negation in the IULA Spanish Clinical Record Corpus.  In Proceedings of the Workshop Computational Semantics Beyond Events and Roles (SemBEaR), pages 43–52, Valencia, Spain, Apr 2017. Association for Computational Linguistics.
  * [58] Maite Oronoz, Koldo Gojenola, Alicia Pérez, Arantza Díaz de Ilarraza, and Arantza Casillas.  On the creation of a clinical gold standard corpus in spanish: Mining adverse drug reactions.  Journal of Biomedical Informatics, 56:318–332, 2015.
  * [59] Antonio Miranda-Escalada, Eul‘alia Farré-Maduell, Salvador Lima-López, Darryl Estrada, Luis Gascó, and Martin Krallinger.  Mention detection, normalization & classification of species, pathogens, humans and food in clinical documents: Overview of livingner shared task and resources.  Procesamiento del Lenguaje Natural, 2022.
  * [60] Montserrat Marimon, Aitor Gonzalez-Agirre, Ander Intxaurrondo, Heidy Rodriguez, Jose Lopez Martin, Marta Villegas, and Martin Krallinger.  Automatic de-identification of medical texts in spanish: the meddocan track, corpus, guidelines, methods and evaluation of results.  In IberLEF@ SEPLN, pages 618–638, 2019.
  * [61] Salvador Lima, Naiara Perez, Montse Cuadros, and German Rigau.  Nubes: A corpus of negation and uncertainty in spanish clinical texts.  arXiv preprint arXiv:2004.01092, 2020.
  * [62] Aitor Gonzalez-Agirre, Montserrat Marimon, Ander Intxaurrondo, Obdulia Rabal, Marta Villegas, and Martin Krallinger.  Pharmaconer: Pharmacological substances, compounds and proteins named entity recognition track.  In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 1–10, 2019.
  * [63] Luis Gasco Sánchez, Darryl Estrada Zavala, Eulàlia Farré-Maduell, Salvador Lima-López, Antonio Miranda-Escalada, and Martin Krallinger.  The SocialDisNER shared task on detection of disease mentions in health-relevant content from social media: methods, evaluation, guidelines and corpora.  In Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task, pages 182–189, Gyeongju, Republic of Korea, October 2022. Association for Computational Linguistics.
  * [64] Almagro M., Martínez R., Fresno V., and Montalvo S.  Spanish clinical embeddings (sce), July 2021.
  * [65] Carolina Chiu, Fabián Villena, Kinan Martin, Fredy Núñez, Cecilia Besa, and Jocelyn Dunstan.  Word embeddings for the spanish clinical language, June 2022.
  * [66] Matías Rojas, Jocelyn Dunstan, and Fabián Villena.  Clinical flair: A pre-trained language model for Spanish clinical natural language processing.  In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 87–92, Seattle, WA, July 2022. Association for Computational Linguistics.
  * [67] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.  Mteb: Massive text embedding benchmark, 2023.
  * [68] José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez.  Spanish pre-trained bert model and evaluation data.  In PML4DC at ICLR 2020, 2020.
  * [69] José Cañete.  Compilation of large spanish unannotated corpora, May 2019.
  * [70] Asier Gutiérrez Fandiño, Jordi Armengol Estapé, Marc Pàmies, Joan Llop Palao, Joaquin Silveira Ocampo, Casimiro Pio Carrino, Carme Armentano Oller, Carlos Rodriguez Penagos, Aitor Gonzalez Agirre, and Marta Villegas.  Maria: Spanish language models.  Procesamiento del Lenguaje Natural, 68, 2022.
  * [71] Alejandro Vaca Serrano, Guillem Garcia Subies, Helena Montoro Zamorano, Nuria Aldama Garcia, Doaa Samy, David Betancur Sanchez, Antonio Moreno Sandoval, Marta Guerrero Nieto, and Alvaro Barbero Jimenez.  Rigoberta: A state-of-the-art language model for spanish, 2022.
  * [72] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer, 2021\.
  * [73] Rodrigo Agerri and Eneko Agirre.  Lessons learned from the evaluation of spanish language models, 2022.
  * [74] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July 2020. Association for Computational Linguistics.
  * [75] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021.
  * [76] Reuters.  Chatgpt sets record fastest growing user base, analyst note.  Reuters Technology, February 2023.
  * [77] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [78] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz.  Capabilities of gpt-4 on medical challenge problems, 2023.
  * [79] Guillermo López-García, José M. Jerez, Nuria Ribelles, Emilio Alba, and Francisco J. Veredas.  Transformers for clinical coding in spanish.  IEEE Access, 9:72387–72397, 2021.
  * [80] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith.  Don’t stop pretraining: Adapt language models to domains and tasks.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online, July 2020. Association for Computational Linguistics.
  * [81] N. Ribelles, J. M. Jerez, D. Urda, J. L. Subirats, A. Márquez, C. Quero, and L. A. Franco.  Galén: Sistema de información para la gestión y coordinación de procesos en un servicio de oncología.  RevistaeSalud, 6(21):1–12, 2010.
  * [82] Casimiro Pio Carrino, Joan Llop, Marc Pàmies, Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Joaquín Silveira-Ocampo, Alfonso Valencia, Aitor Gonzalez-Agirre, and Marta Villegas.  Pretrained biomedical language models for clinical NLP in Spanish.  In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 193–199, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [83] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [84] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  Falcon-40B: an open large language model with state-of-the-art performance, 2023.
  * [85] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * [86] BigScience Workshop.  BLOOM (revision 4ab0472), 2022.
  * [87] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
  * [88] Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu.  Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations, 2023.

##  Appendix A Metaparameters

All the models in the evaluation have been trained with all the configurations
from the Table 5. Each one of the $7$ models has been trained with every
combination of the parameters for all the $12$ corpora used for the
evaluation, in total $3024$ models were trained.

Metaparameter | Configurations
---|---
Batch Size | $\\{16,32,64\\}$
Classifier Dropout | $\\{0,0.1,0.2\\}$
Learning Rate | $\\{1e-5,2e-5,3e-5,4e-5\\}$
Warmup Ratio | $0$
Warmup Steps | $0$
Weight Decay | $0$
Optimizer | AdamW
Epochs | $10$
Early Stopping Patience | $3$
Table 5: Parameter grid used for the experiments

##  Appendix B Public Resources

###  B.1 Corpora

Some corpora did not have a labeled test split so some of the splits used do
not coincide with the original splits in the corpora. In the following list we
can see a list of the used corpora and their corresponding Huggingface Hub
repository. Note that the name of the repository is the one that was used for
the fine-tuned models.

  * •

cantemist: PlanTL-GOB-ES/cantemist-ner

  * •

caresA: chizhikchi/CARES

  * •

caresC: chizhikchi/CARES

  * •

ctebmsp: lcampillos/ctebmsp

  * •

distemist: bigbio/distemist

  * •

ehealth_kd: ehealth_kd

  * •

livingner1: IIC/livingner1

  * •

livingner3: IIC/livingner3

  * •

meddocan: bigbio/meddocan

  * •

nubes: plncmm/nubes

  * •

pharmaconer: PlanTL-GOB-ES/pharmaconer

  * •

socialdisner: IIC/socialdisner

###  B.2 Models

First of all, we uploaded both BETO Galén
262626https://huggingface.co/IIC/BETO_Galen and XLM-R Galén
272727https://huggingface.co/IIC/XLM-R_Galen to the Model Hub because they
were only available through they raw weights in GitHub.

Every fine-tuned model follows the structure model-name-corpus, so for
example, the model id for MarIA and the CANTEMIST corpus would be IIC/roberta-
large-bne-cantemist 282828https://huggingface.co/IIC/roberta-large-bne-
cantemist. So combining the models from the list below and the corpus names
from the list in Appendix B.1 lead to their corresponding Huggingface Hub
address.

  * •

BETO: IIC/bert-base-spanish-wwm-cased-{corpus}

  * •

BETO_Galen: IIC/BETO_Galen-{corpus}

  * •

MarIA: IIC/roberta-large-bne-{corpus}

  * •

XLM-RoBERTa-Large: IIC/xlm-roberta-large-{corpus}

  * •

XLM-R_Galen: IIC/XLM_R_Galen-{corpus}

  * •

bsc-bio-ehr-es: IIC/bsc-bio-ehr-es-{corpus}

  * •

mDeBERTaV3: IIC/mdeberta-v3-base-{corpus}

###  B.3 Code Repository

All the code used for the benchmark is publically available in GitHub
292929https://github.com/iiconocimiento/survey-spanish-clinical-language-
models. Note that the code uses some data and models from their raw origin
because the benchmark was done before uploading every model and corpus to the
model hub.

##  Appendix C GPT-3 experiments

Here we will explain in more detail the experiments with the OpenAI API.

###  C.1 NER experiments

In the following snippet we can see an example prompt introduced to the chat
model where example1 and example2 are few shot samples and text is the sample
that has to be predicted:

Recibes un texto y tu tarea es reconocer y extraer las entidades específicas
del tipo ‘ENFERMEDAD‘. Tu formato de salida es el siguiente: [[inicio, fin,
’ENFERMEDAD’], [inicio, fin, ’ENFERMEDAD’], …, [inicio, fin, ’ENFERMEDAD’]
Donde inicio y fin es la posición del texto que ocupa cada enfermedad.
Ejemplos: {example1} {example2} Texto: {text} Enfermedades:

When using the chat API we also used the system prompt to bias the model
towards our use case:

Eres un sistema de Named Entity Recognition (NER) extremadamente inteligente y
preciso en el dominio biomédico.

A simplified version of the above prompt was also tried, however the results
could not be compared to the Gold Standard because of the lack of the entity
positions:

Identifica y las enfermedades que aparecen en los siguientes textos. Puede
haber enfermedades repetidas. Ejemplos: {example1} {example2} Texto: {text}
Enfermedades:

Some parameter combinations were tried, however the best performing one was:
temperature=0, top_p=0.8, presence_penalty=-2 and frequency_penalty=0. Logit
biases were also introduced, but they made the model reach local minima very
fast resulting in nonsense predictions.

###  C.2 Classification experiments

Here we followed 2 different approaches, few-shot evaluation of the test data
and fine-tuning.

Like for the NER corpus, for the few-shot approach we tried with some prompts
with examples and system prompts:

Recibes un texto y tu tarea es elegir todos los capítulos del standard ICD-10
al que pertenece (números del 1 al 21). Ejemplos: {example0} {example1}
{example2} Entrada: {text} -> ###

Where an example could be the text and the chapters in a Python list syntax:

{text} -> [11, 14]

In this case, the system prompt was:

Eres un sistema experto en códigos ICD-10 extremadamente inteligente y preciso
en el dominio biomédico.

Some parameter combinations were tried, however the best performing one was:
temperature=0, top_p=0.8, presence_penalty=0 and frequency_penalty=0. Logit
biases were also introduced, the model is sensitive to the bias but we
achieved some extra consistency with a bias of $1$ for every possible class
(the tokens for the numbers of the chapters) and $0.2$ for the comma, opening
and closing bracket tokens.

For the fine-tuning approach we followed the steps and good practices
explained in the OpenAI API documentation. We used the same parameters and the
same prompts used for the few-shot evaluation.