# Multilingual Sequence-to-Sequence Models for Hebrew NLP

Matan Eyal Hila Noga Roee Aharoni
Idan Szpektor Reut Tsarfaty

Google Research
<EMAIL_ADDRESS>
###### Abstract

Recent work attributes progress in NLP to large language models (LMs) with
increased model size and large quantities of pretraining data. Despite this,
current state-of-the-art LMs for Hebrew are both under-parameterized and
under-trained compared to LMs in other languages. Additionally, previous work
on pretrained Hebrew LMs focused on encoder-only models. While the encoder-
only architecture is beneficial for classification tasks, it does not cater
well for sub-word prediction tasks, such as Named Entity Recognition, when
considering the morphologically rich nature of Hebrew. In this paper we argue
that sequence-to-sequence generative architectures are more suitable for LLMs
in the case of morphologically rich languages (MRLs) such as Hebrew. We
demonstrate that by casting tasks in the Hebrew NLP pipeline as text-to-text
tasks, we can leverage powerful multilingual, pretrained sequence-to-sequence
models as mT5, eliminating the need for a specialized, morpheme-based,
separately fine-tuned decoder. Using this approach, our experiments show
substantial improvements over previously published results on existing Hebrew
NLP benchmarks. These results suggest that multilingual sequence-to-sequence
models present a promising building block for NLP for MRLs.

##  1 Introduction

In recent years, large pretrained language models showed impressive results in
a variety of NLP tasks Devlin et al. (2019); Raffel et al. (2020), domains
Beltagy et al. (2019); Araci (2019), and languages Antoun et al. (2020); Chan
et al. (2020). These models were trained in a self-supervised fashion on large
corpora with language modeling objectives. By doing this, models are taking
advantage of information available in the training data without any access to
explicit labels Petroni et al. (2019). Recent works Kaplan et al. (2020);
Hoffmann et al. (2022) argue that language models capabilities scale as a
power-law with model size, dataset size, and the amount of compute used for
training.

Figure 1: Finetuning mT5 on Hebrew tasks in text-to-text settings

After the success such models achieved on English benchmarks, analogous
language-specific models were developed to improve benchmark results in a
variety of languages such as Arabic and French to name a few Antoun et al.
(2020); Martin et al. (2020). Hebrew NLP was no different with a number of
BERT variations Devlin et al. (2019) proposed: HeBERT Chriqui and Yahav
(2022), AlephBERT Seker et al. (2022), and recently AlephBERTGimmel (ABG) a
variation of AlephBERT with a larger vocabulary Guetta et al. (2022).

Contrary to the scaling laws proposed in Kaplan et al. (2020), all of the
Hebrew variations of BERT are both trained with a relatively small pretraining
data, and are under-parameterized. In terms of training data, HeBERT and
AlephBERT (and similarly ABG) were trained on 10.5GB and 16GB of Hebrew
pretraining data respectively. mT5 Xue et al. (2021) in comparison, was
trained on mC4, where its public
replica111https://github.com/allenai/allennlp/discussions/5265 is a 27TB
collection of natural text in 101 languages drawn from the public Common
Crawl, 66GB of which is Hebrew (4.125x more Hebrew training data compared to
AlephBERT). In terms of the model size, ABG, the largest Hebrew LM, has 60x
and 945x fewer parameters compared to English T5 XXL Raffel et al. (2020) and
GPT3 Brown et al. (2020), respectively, language models that were released two
years earlier. Although English T5 is a much larger model than available
Hebrew language models, it can still be fine-tuned on common GPUs. See
Appendix A for more details.

In addition to scale differences, all previous Hebrew LMs use an encoder-only
architecture, even though the morphological complexity of Hebrew and other
morphologically rich languages (MRLs)222MRLs, in contrast configurational
languages (e.g. English), express grammatical functions at the word level via
phenomena such as inflectional affixes, pronominal clitics, etc. pose
challenges for the effectiveness of this model. Consider the task of POS
tagging: Assigning POS tags for the phrase ‘‘babayit halavan"333Hebrew
transcribed to English. Translated as “In the White House”. The phrase is made
of the morphemes be-ha-bayit ha-lavan (in-the-house the-white), but written
and pronounced as “babayit halavan” without explicit boundaries. requires to
initially segment the phrase to its morphemes and only then assign each
morpheme its matching POS tag. Since the number of input tokens does not match
the number of output tags (2 input words and 5 output tags, one for each
morpheme), a one-to-one token-to-tag classification head, as commonly employed
in encoder-only models, is not feasible. The same problem appears in semantic
tasks like Question-Answering (QA) and Named Entity Recognition (NER). For
example, the named entity for ‘‘babayit halavan" is ‘‘habayit halavan". This
goes beyond what encoder-only models can do by requiring the model to label a
string that is not part of the input text.

To overcome this architectural obstacle in encoder-only models, AlephBERT
authors used Brusilovsky and Tsarfaty (2022) three-step segmentation and
tagging approach: Contextualize the input, pass resulting embeddings to LSTM
decoder that generates the segmentation separated by a space symbol, finally
pass the white space representation to a classification head. While effective
for morpho-syntactic tasks, these additional architecture components do not
enable full generartive capabilities and are not pretrained, therefore cannot
benefit from some of pretrained LMs advantages.

In contrast, sequence-to-sequence models can simply take the raw text as input
and, for POS tagging for example, generate the morphemes and tags in sequence,
for example: [1em]2embe>>ADP@@ha>>DET@@bayit>>NOUN ha>>DET@@lavan>>NOUN ‘‘@@"
acts as a morpheme delimiter within a word and “>>" is the morpheme-tag
delimiter. See Sec. 3 for more details. For NER and Question-Answering we can
simply generate the target word forms without explicitly going through a
segmentation phase. This change in approach to using sequence-to-sequence
modeling is relevant for all MRLs, and in this paper we demonstrate its
effectiveness specifically for Hebrew.

This work thus identifies the challenge of current Hebrew NLP as a three-
faceted problem: Under-parameterization, limited training data, as well as the
use of a suboptimal pre-training architecture.444So far we mentioned encoder-
only models as an example for suboptimal modeling choices for MRLs, but this
is also the case when using poor tokenization, small vocabularies etc. To
address these three challenges at once, we propose using mT5 Xue et al.
(2021), a large multilingual sequence-to-sequence model that was pretrained on
a vast amount of multilingual data including a significant amount of Hebrew
data.555It is beyond the scope of this paper to examine the factors that
contributed to its improved performance, see Sec. 7. To adapt classification,
span prediction, and token/morpheme classification tasks to mT5’s text-to-text
paradigm, we propose the text-only formulations illustrated in Figure 1.
Subsequently, we report here that this paradigm change produces empirical
improvements on all tasks evaluated compared to previous state-of-the-art,
some of which are dramatic, as 27.9 F1 increase in Hebrew Question-Answering.

##  2 Modeling

We use mT5 Xue et al. (2021), a multilingual generative text-to-text version
of T5 Raffel et al. (2020), trained simultaneously on 101 languages. We
evaluate mT5 on all its available sizes - Small, Base, Large, XL and XXL
ranging from 300M to 13B parameters. Subsequently, we propose casting all
Hebrew NLP tasks for which evaluation benchmarks exist as text-to-text tasks:
The input text is fed into the model and targets are produced in a generative
manner.

In contrast to text-to-text formulations of classification and span
prediction, token classification is not as common in the literature, and
specifically when the tokens consist of multiple morphemes, as is the case in
MRLs. For example, in POS tagging for MRLs, each morpheme is assigned a POS
tag, unlike in non-MRLs, where tagging is a word-level task. As a result, a
generative model cannot simply generate tag predictions one after the other
but it requires to first segment the text and only then label it accordingly.
e.g., An unsatisfactory generation for “habayit" is DET,NOUN as we cannot
recover which morpheme belongs to which tag. An acceptable model output, on
the other hand, is ha-DET,lavan-NOUN as we can recover that ha was tagged with
a DET and lavan with a NOUN. Throughout our experiments we tested a number of
different text-to-text formulations. The best candidates are depicted in
Figure 1.

Model | Param | ParaShoot | NEMO | BMC | Sentiment
---|---|---|---|---|---
Count | Token | Morph. | Analysis
| EM | F1 | F1 | F1 | F1 | F1
mBERT | 178M | 32 | 56.1 | 79.11 | 69.78 | 87.77 | 84.21
HeBERT | 110M | 18.2 | 36.7 | 81.13 | 77.29 | 89.41 | 87.13
AlephBERT | 126M | 26 | 49.6 | 83.62 | 77.55 | 91.12 | 89.02
ABG | 185M | - | - | 86.26 | 80.39 | - | 89.51
mT5 - Small | 300M | 24.52 | 48.71 | 66.74 | 62.08 | 77.7 | 87.55
mT5 - Base | 580M | 36.65 | 62.97 | 74.7 | 69.12 | 85.5 | 87.25
mT5 - Large | 1.2B | 42.6 | 70.13 | 84.33 | 81.85 | 90.77 | 88.73
mT5 - XL | 3.7B | 46.15 | 73.27 | 88.65 | 84.43 | 93.01 | 88.9
mT5 - XXL | 13B | 50.37 | 77.5 | 89.86 | 88.65 | 93.29 | 89.61

Table 1:  mT5 outperforms previous encoder-only LMs on a variety of semantic
Hebrew downstream tasks.

##  3 Experiments

##### Goal

The goal of this study is to assess the performance of a sequence-to-sequence
large language model, specifically mT5, that was trained on a large quantity
of multilingual data, compared to existing Hebrew language models, in order to
showcase the efficacy of larger, well trained and thoughtfully designed models
on Hebrew tasks.

##### Models

We fine-tuned different sizes of mT5 (Small-XXL) on all Hebrew tasks in a
single-task fashion for 4096 steps, with a constant learning rate of 1e-3. For
test set evaluation, we used the best-performing checkpoint from the
development set, as tasks usually converge earlier. We compared the mT5 models
against YAP More et al. (2019), mBERT Devlin et al. (2019), HeBERT Chriqui and
Yahav (2022), AlephBERT Seker et al. (2022) and ABG Guetta et al. (2022). We
compare against YAP as it’s the only available model trained on the
lemmatization task. YAP’s scores are produced by us. mBERT, HeBERT, AlephBERT
and ABG scores are all from Guetta et al. (2022). ParaShoot scores are from
Keren and Levy (2021). Summary of the results can be found in Tables 1, 2.

###  3.1 Tasks

We assembled an evaluation suite of Hebrew benchmarks composed of the
following tasks: QA Keren and Levy (2021), NER Bareket and Tsarfaty (2021);
Mordecai and Elhadad (2005), Sentiment Analysis Amram et al. (2018), and the
morpho-syntactic tasks of segmentation, POS tagging and lemmatization from
Sade et al. (2018), where we used the the latest dataset version, compatible
with ABG.

####  3.1.1 Question-Answering

Keren and Levy (2021) introduced ParaShoot, an Hebrew Question-Answering
dataset which was created using the format and crowdsourcing methodology of
SQuAD Rajpurkar et al. (2016). We report token-level F1 and Exact Match scores
as no morpheme boundaries are available. The input is constructed by
concatenating the context and question, with the output being the answer.

We conducted manual evaluation of different mT5 models on this dataset to
evaluate the impact of model sizes, see details in B.

####  3.1.2 Named Entity Recognition

Bareket and Tsarfaty (2021) created NEMO, an NER annotation for the Hebrew UD
corpus Sade et al. (2018). As entities in Hebrew can correspond to only a
subset of a word’s morphemes (See “habayit halavan" example in Sec. 1), the
authors proposed two dataset versions: token-level, where entities correspond
to white-space boundaries, similarly to BMC Mordecai and Elhadad (2005), and
morpheme-level, with morpheme-based boundaries. The authors additionally
revised the common NER evaluation procedure by comparing predicted and target
entities on the surface form, boundaries, and entity types, but not positions.
Thus, we train the sequence-to-sequence model to simply generate all of the
sentence entities and tags one after the other in the input text.

####  3.1.3 Sentiment Analysis

Correspondingly with previous work, we report F1 scores for Amram et al.
(2018), a sentiment analysis dataset curated by annotating Facebook user
comments with positive/negative/neutral tags.666We use Seker et al. (2022)
refined version which does not include leaks between split sets. Encoder input
is the raw text with the decoder generates either <extra_id_0>, <extra_id_1>
or <extra_id_2> which correspond to the positive, negative and neutral tags.

####  3.1.4 Word Segmentation, POS Tagging and Lemmatization

Sade et al. (2018) manually validated the UDv2 version of the Hebrew treebank
resulting in a set of morpho-syntactic tasks. Aligned to previous work we
report word segmentation and POS tagging. We also evaluate our model on the
lemmatization task and compare it to YAP More et al. (2019), an open-source
Hebrew parser. Correspondingly to previous work in Hebrew, we report aligned
MultiSet (mset) scores. To produce the output for all these tasks we use two
additional tokens: ‘‘@@" is the morpheme delimiter within a word and ‘‘>>" is
the morpheme-tag delimiter. e.g., segmenting, and POS tagging “habayit
halavan" should return be@@ha@@bayit ha@@lavan and
be>>ADP@@ha>>DET@@bayit>>NOUN ha>>DET@@lavan>>NOUN, respectively.

Model | Segmentation | POS Tagging | Lemmatization
---|---|---|---
YAP | 93.64 | 90.13 | 78.6
mBERT | 96.07 | 93.14 | -
HeBERT | 97.90 | 95.80 | -
AlephBERT | 97.88 | 95.81 | -
ABG | 98.09 | 96.22 | -
mT5 - Small | 94.83 | 94.55 | 89.96
mT5 - Base | 96.34 | 95.9 | 92.09
mT5 - Large | 96.76 | 95.58 | 92.21
mT5 - XL | 98.32 | 96.91 | 95.13
mT5 - XXL | 98.67 | 97.46 | 95.53

Table 2: Morpheme-Based Aligned MultiSet (mset) Results on the UD Corpus

##  4 Results

Our findings demonstrate a marked improvement over previously published
results on existing Hebrew NLP benchmarks and by that establishing the
advantage of using large multilingual sequence-to-sequence pretrained LMs such
as mT5.

mT5 produces the biggest performance boost for the Question-Answering task of
ParaShoot, with mT5-base already surpassing baseline models and mT5-XXL
outperforming AlephBERT by 27.9 F1 points. For NER, mT5 produces better
results than evaluated baselines on both datasets. The largest performance
boost comes in the morpheme-level NEMO version where mT5 learns to segment and
label entities in an end-to-end fashion.

For sentiment analysis, mT5 outperforms the baseline models by a small
fraction, however, manual analysis we performed on mT5’s best performing model
show that 34% of its errors are annotation errors and for further 30% our
annotators we not able to decide what is the correct label. We conclude that
we should work towards a cleaner, more challenging sentiment analysis dataset
in Hebrew. We report error reduction of 30.3% and 32.8% for the segmentation
and POS tagging tasks compared to previous state-of-the-art. For the
lemmatization task we report an increase of 16.93 mset F1 points compared to
YAP. All of these are an important step towards closing the gap in morpho-
syntactic tasks comparing to other languages.

##  5 Related work

HeBERT Chriqui and Yahav (2022) is the first pretrained transformer-based
language model trained on Hebrew Wikipedia and OSCAR Ortiz Suárez et al.
(2020) for the task of user-generated sentiment analysis. AlephBERT Seker et
al. (2022) was pretrained on the same copora in addition to a very large
number of Hebrew tweets. Guetta et al. (2022) tackled the extreme data
sparseness in MRLs lexica Tsarfaty et al. (2020) by pretraining with roughly
2.5x of AlephBERT vocabulary size, leading to performance improvements.
Orthogonally, Keren et al. (2022) proposed using char-level LMs to mitigate
the same sparseness problem, albeit results were inconclusive.

Xue et al. (2021) showed that mT5 outperforms baseline models on a number of
multilingual datasets but did not directly evaluate on Hebrew. Alternatively,
monolingual Hebrew LM papers only compared against mBERT Devlin et al. (2019)
as the sole multilingual baseline.

##  6 Conclusions

All Hebrew LMs to date are encoder-only models, which could not directly
generate morpheme sequences, and thus necessitate a specially-tuned decoder.
In this work we propose to take advantage of mT5, a publicly available
multilingual large language model that was trained on a considerable amount of
multilingual and Hebrew data. Additionally the generative approach of text-to-
text modeling is more aligned with the morphological challenges inherent in
Hebrew and by that dispense the need for specially tuned decoders. We fine-
tuned and evaluated mT5 on a set of Hebrew downstream tasks and report that
mT5 outperforms all previous baselines. Subsequently, we propose that
multilingual sequence-to-sequence models should be used for Hebrew and other
MRLs as an alternative for their smaller encoder-only counterparts.

##  7 Limitations

mT5, comparing to previous Hebrew LMs, is bigger, pretrained on more
multiligual data, and learning to segment and tag in an end-to-end manner.
While it was beyond the scope of this paper to pretrain new LMs and study
which factors contributed to the improved performance, identifying these
factors will be useful for determining the most effective approach for future
work. While larger mT5 models perform better than available LMs, they require
more powerful hardware accelerators and take longer to train and infer. Future
research should aim to achieve similar levels of performance with smaller mT5
models. Additionally, the inclusion of data from 101 languages in the training
of mT5 may have negatively impacted its performance on Hebrew, as some of the
data may not have been relevant or beneficial to this particular language.
Future work will need to address this issue by training a monolingual Hebrew
LM in order to further improve performance for Hebrew.

An inherent limitation in sequence-to-sequence models is that they can
generate inconsistent text with respect to the input text Lee et al. (2018);
Rohrbach et al. (2018). While potentially sensitive in different applications,
a number of evaluation frameworks have been suggested to reduce the number of
such “hallucinations" Honovich et al. (2021, 2022). Another limitation of our
evaluation framework is that, for lack of available datasets, we did not
evaluate mT5 on purely generative tasks such as summarization and paraphrasing

##  8 Acknowledgements

We thank Dan Bareket and Eylon Guetta from Bar Ilan University for their help
in sharing the UD and NEMO data.

## References

  * Amram et al. (2018) Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018.  Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2242–2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  AraBERT: Transformer-based model for Arabic language understanding.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 9–15, Marseille, France. European Language Resource Association.
  * Araci (2019) Dogu Araci. 2019.  Finbert: Financial sentiment analysis with pre-trained language models.  _arXiv preprint arXiv:1908.10063_.
  * Bareket and Tsarfaty (2021) Dan Bareket and Reut Tsarfaty. 2021.  Neural modeling for named entities and morphology (NEMO2).  _Transactions of the Association for Computational Linguistics_ , 9:909–928.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Brusilovsky and Tsarfaty (2022) Idan Brusilovsky and Reut Tsarfaty. 2022.  Neural token segmentation for high token-internal complexity.  _arXiv preprint arXiv:2203.10845_.
  * Chan et al. (2020) Branden Chan, Stefan Schweter, and Timo Möller. 2020.  German’s next language model.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Chriqui and Yahav (2022) Avihay Chriqui and Inbal Yahav. 2022.  Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition.  _INFORMS Journal on Data Science_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Guetta et al. (2022) Eylon Guetta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022.  Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_ , pages 161–175, Dublin, Ireland. Association for Computational Linguistics.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7856–7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Keren et al. (2022) Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022.  Breaking character: Are subwords good enough for mrls after all?  _arXiv preprint arXiv:2204.04748_.
  * Keren and Levy (2021) Omri Keren and Omer Levy. 2021.  ParaShoot: A Hebrew question answering dataset.  In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 106–112, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lee et al. (2018) Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018.  Hallucinations in neural machine translation.  _NeurIPS 2018 Workshop on Interpretability and Robustness for Audio, Speech, and Language_.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Mordecai and Elhadad (2005) Naama Ben Mordecai and Michael Elhadad. 2005.  Hebrew named entity recognition.  _MONEY_ , 81(83.93):82–49.
  * More et al. (2019) Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019.  Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew.  _Transactions of the Association for Computational Linguistics_ , 7:33–48.
  * Ortiz Suárez et al. (2020) Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1703–1714, Online. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018.  Object hallucination in image captioning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4035–4045, Brussels, Belgium. Association for Computational Linguistics.
  * Sade et al. (2018) Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018.  The Hebrew Universal Dependency treebank: Past present and future.  In _Proceedings of the Second Workshop on Universal Dependencies (UDW 2018)_ , pages 133–143, Brussels, Belgium. Association for Computational Linguistics.
  * Seker et al. (2022) Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022.  AlephBERT: Language model pre-training and evaluation from sub-word to sentence level.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 46–56, Dublin, Ireland. Association for Computational Linguistics.
  * Tsarfaty et al. (2020) Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020.  From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7396–7408, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.

##  Appendix A Running T5 on common GPUs

AlephBertGimmel Guetta et al. (2022), the largest Hebrew LM to date is roughly
the same size of BERT base Devlin et al. (2019), and even though mT5 is 60
times larger than AlephBertGimmel, we do not need to horizontally scale our
hardware accelerators by 60 to accommodate it. As models have grown in size,
hardware accelerators have also become more advanced. T5 Small, Base and Large
can all be fine-tuned in a 2016 Nvidia P100 or 2017 Nvidia V100 GPU
accelerators. T5 XL and XXL can be fine-tuned in the 2020 Nvidia A100 GPU, the
same accelerator used for pretraining AlephBERTGimmel.

Given the widespread availability of these GPU accelerators, we argue that the
mT5 models we evaluate in this work can be easily trained and deployed
nowadays.

##  Appendix B Manual Evaluation of mT5 on Question-Answering

mT5-small performs similarly to previous state-of-the-art models on on the
Question-Answering task of ParaShoot Keren and Levy (2021). We conducted
manual evaluation of mT5-XXL against mT5-small, both as a way to analyse the
impact of the model size, while holding other factors constant, and as a way
to compare to the performance of previous state-of-the-art models.

We ran our mT5 experiments using 3 seeds with the best performing model,
mT5-XXL, achieving 77.99 F1 and 50.63 EM scores. Our worst performing model,
mT5-small, reached 47.67 F1 and 24.39 EM scores. From the 519 exact match
prediction mT5-XXL model made, 167 of which mT5-small received F1 scored of 0.
Manually evaluating the mistakes mT5-small made we conclude that for a lot of
the examples the model did not manage to fully "understand the question", here
mixing when and where:

Context: l’.hr tbwst .srpt bmsgrt hm‘rkh ‘l .srpt h.hly.t mpqd h.sb’ h.srpty
bmwSbh pwl lw’y lz’n.tywM lhmSyK bl.hymh l.sd .srpt h.hwpSyt>…

Question: mty h.hly.t mpqd h.sb’ lhmSyK bl.hymh l.sd .srpt h.hwpSyt ? >

The gold and mT5-XXL prediction is l’.hr tbwst .srpt bmsgrt hm‘rkh ‘l .srpt>

The small model predicted bmwSbh pwl lw’y lz’n.tywM>.

mT5 additionally made a few hallucination errors, returning answers that were
not in the original context.

Interestingly, mT5-small was able to accurately answer 49 questions that
mT5-XXL was unable to, although for only 3 of these mT4-XXL received 0 F1
scored. Manually evaluating these 3 errors, 2 of them are alternative possible
answers.