# On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based
Multilingual Model

Nohil Park1 Joonsuk Park4,5,6 Kang Min Yoo3,4,5, Sungroh
Yoon1,2,11footnotemark: 1
1Department of Electrical and Computer Engineering, Seoul National University
2Interdisciplinary Program in Artificial Intelligence, Seoul National
University
3Artificial Intelligence Institute, Seoul National University
4NAVER Cloud 5NAVER AI Lab 6University of Richmond
{pnoil2588<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

An exciting advancement in the field of multilingual models is the emergence
of autoregressive models with zero- and few-shot capabilities, a phenomenon
widely reported in large-scale language models. To further improve model
adaptation to cross-lingual tasks, another trend is to further fine-tune the
language models with either full fine-tuning or parameter-efficient tuning.
However, the interaction between parameter-efficient fine-tuning (PEFT) and
cross-lingual tasks in multilingual autoregressive models has yet to be
studied. Specifically, we lack an understanding of the role of linguistic
distributions in multilingual models in the effectiveness of token-based
prompt tuning. To address this question, we conduct experiments comparing
prompt tuning and fine-tuning on the decoder-based multilingual model, XGLM,
with four cross-lingual tasks (XNLI, PAWS-X, POS, NER). According to our
study, prompt tuning achieves on par or better performance over fine-tuning
across all languages while updating at most 0.13% of the model parameters.
Moreover, we empirically show that prompt tuning is more effective in
enhancing the performance of low-resource languages than fine-tuning. Our
further analysis shows that the phenomenon is related to the tokenization
scheme of the multilingual model.

##  1 Introduction

Recent efforts to use encoder and encoder-decoder models on multilingual tasks
have shown that prompt tuning can be more effective than fine-tuning. At the
same time, a strand of research aims to tackle multilingual tasks leveraging
multilingual models while avoiding the cost of full fine-tuning (Pfeiffer et
al., 2020; Huang et al., 2022; Fu et al., 2022). In particular, to widely used
models such as XLM-R (Conneau et al., 2020), variations of prompt tuning have
been applied (Fu et al., 2022; Huang et al., 2022; Zhao and Schütze, 2021).
This method, when applied as-is, demonstrates better performance on cross-
lingual tasks compared to full fine-tuning (Tu et al., 2022).

In this way, previous research has predominantly concentrated on models
employing encoder-only (Conneau et al., 2020) or encoder-decoder (Xue et al.,
2021) architectures. Thus, there remains a necessity to explore the
effectiveness of the prompt tuning for cross-lingual transfer learning in the
decoder-based multilingual models, such as XGLM (Lin et al., 2022) and BLOOM
(Scao et al., 2022). This will not only fill the gap in the study of prompt
tuning on decoder-only models, but also has the potential to allow more
efficient use of such models.

In this paper, we compare the effectiveness of prompt tuning against that of
fine-tuning on a decoder-only multilingual model, XGLM. The goal is analogous
to that of an existing study on an encoder-based model, XLM-R (Tu et al.,
2022). We conduct experiments utilizing P-tuning v2 (Liu et al., 2022), which
entails adding fixed-length continuous prompts in front of the input sequences
and all layers of XGLM. During this process, we update only the prompt
embeddings and the classifier heads while keeping the remaining parameters
frozen. In the analysis, we verify our hypothesis on the influence of language
imbalance and excessively fine-grained tokenization on prompt tuning for low-
resource languages.

We find that models trained with prompt tuning perform comparable to or even
better than models trained with fine-tuning across the four cross-lingual
tasks, namely XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019), NER
(Rahimi et al., 2019), and POS (Liang et al., 2020). In the XNLI task, for
example, models trained with prompt tuning achieve accuracy improvements of up
to 2.9% over full fine-tuned models, while updating only 0.08%-0.13% of the
parameters. Also, we observe a correlation between the language distribution
and the cross-lingual performance. In particular, prompt tuning yields greater
performance gains compared to fine-tuning for low-resource languages.

This paper presents the following contributions.

  * •

As far as we know, this paper represents the first study on the effectiveness
of the parameter-efficient prompt tuning for the decoder-only multilingual
model XGLM.

  * •

We observe that prompt tuning shows similar or even better performance
compared to fine-tuning across the four cross-lingual tasks. Furthermore,
prompt tuning shows its effectiveness in enhancing performance gains for low-
resource languages.

  * •

We identify distinct patterns of prompt tuning and analyze that language
distribution and excessive tokenization on low-resource languages are related
to this phenomenon.

##  2 Approach

###  2.1 Model

XGLM adopts the identical Transformer decoder structure as GPT-3 (Brown et
al., 2020), but it incorporates a larger set of 250K joint vocabularies in its
embedding parameters to encompass 500B-token multilingual corpora consisting
of 30 languages. Given these characteristics, the XGLM models are structured
to align with the layer count and hidden dimension size of GPT-3 models,
resulting in slightly larger model sizes due to the inclusion of additional
embedding parameters.

Unlike conventional multilingual models such as mBERT (Pires et al., 2019),
XLM-R (Conneau et al., 2020), and mT5 (Xue et al., 2021), XGLM employs a
decoder-only architecture. While causal language modeling of the decoder-only
model is advantageous for generation tasks, it is known to perform as well as
masked language modeling models in natural language understanding tasks (Liu
et al., 2021).

In this paper, we utilize the XGLM-564M, the smallest among the publicly
available pre-trained XGLM models provided on
HuggingFace111https://huggingface.co/facebook/xglm-564M, as the backbone for
our models.

###  2.2 Prompt Tuning

Unlike the initial approach of appending continuous tokens only at the
beginning of the input sequence (Liu et al., 2021), P-tuning v2 goes a step
further by adding such tokens in front of each model layer. This expands the
tunable parameters to some extent but enhances the representation learning
capabilities for each task.

In this paper, we adopt the fixed-length continuous prompts attached to both
the input sequence and the front of each layer of the XGLM model, following
the settings of P-tuning v2. For each task during prompt tuning, we update the
classification head attached to the model along with the continuous prompts,
while freezing the rest of the pre-trained model weights.

Previous work on prompt tuning reparameterizes the continuous prompts to
improve performance, instead of directly optimizing them (Li and Liang, 2021;
Liu et al., 2021). We also incorporate a two-layer MLP prompt encoder as the
reparameterization trick in the experiments.

###  2.3 Cross-Lingual Transfer Tasks

We experiment with four cross-lingual tasks: XNLI (Conneau et al., 2018),
PAWS-X (Yang et al., 2019), NER (Rahimi et al., 2019), and POS (Liang et al.,
2020). These tasks can be grouped into two categories: sequence-pair
classification and structured prediction tasks, with two datasets each in
sequential order. We use the datasets provided by
HuggingFace222https://huggingface.co/datasets.

##  3 Experiment

###  3.1 Experimental Settings

Referring to the previous studies (Li and Liang, 2021; Liu et al., 2021; Liu
et al., 2022), we conduct the experiments on various sets of hyperparameters
to find the best-performing model in each task and tuning method, as shown in
Table 1. Furthermore, we conduct multi-seed training to introduce randomness
to weight initialization on the continuous prompts and the classification
heads, and the ordering of training data (Chen et al., 2022). We apply early
stopping with respect to the evaluation loss of the validation set to obtain
the best checkpoint for each task. For each setup, we use the average and
standard deviation of results from the three models as the final performance
metrics.

Hyperparamters Search Space Learning Rate 5e-5, 1e-4, 5e-4, 1e-3, 5e-3, 1e-2,
5e-2 Prompt Length 8, 10, 15, 20, 25, 30, 35, 40, 45, 50 Batch Size 64, 128

Table 1:  Hyperparameter search space for each model.

Task Sentence-pair Classification Structured Prediction XNLI PAWS-X NER POS
Metric Acc. $\uparrow$ Acc. $\uparrow$ F1 $\uparrow$ Acc. $\uparrow$ Language
FT PT FT PT FT PT FT PT en 81.92$\pm$0.92 82.15$\pm$0.92 92.38$\pm$0.32
93.42$\pm$0.51 74.29$\pm$0.55 74.47$\pm$0.36 92.97$\pm$0.06 92.79$\pm$0.04 ru
69.55$\pm$2.04 73.24$\pm$0.29 - - - - 72.36$\pm$2.20 75.25$\pm$0.57 zh
69.11$\pm$0.87 70.72$\pm$0.97 76.05$\pm$0.68 76.68$\pm$1.08 - - 30.48$\pm$2.48
27.33$\pm$1.88 de 74.26$\pm$0.99 74.75$\pm$0.17 83.98$\pm$0.69 83.87$\pm$0.46
32.61$\pm$0.34 39.59$\pm$0.87 77.55$\pm$1.78 77.45$\pm$0.57 es 75.65$\pm$0.51
76.26$\pm$0.07 85.62$\pm$0.58 87.03$\pm$0.63 54.19$\pm$0.71 56.19$\pm$0.18
77.89$\pm$2.07 78.01$\pm$0.77 fr 75.87$\pm$1.18 75.90$\pm$0.24 87.05$\pm$0.71
86.32$\pm$1.18 - - 75.28$\pm$0.76 74.34$\pm$1.05 ja - - 72.23$\pm$1.58
71.13$\pm$2.52 - - - - it - - - - - - 71.97$\pm$1.57 72.29$\pm$0.69 el
71.38$\pm$2.23 74.54$\pm$0.16 - - - - 69.95$\pm$3.16 79.41$\pm$3.09 ko - -
71.15$\pm$1.60 70.38$\pm$0.98 - - - - pl - - - - - - 29.29$\pm$3.98
28.83$\pm$3.89 nl - - - - 32.84$\pm$0.13 34.92$\pm$0.51 43.02$\pm$5.10
44.17$\pm$4.71 tr 64.51$\pm$3.44 69.18$\pm$1.27 - - - - 57.31$\pm$4.47
61.72$\pm$0.91 ar 64.59$\pm$2.51 69.58$\pm$0.28 - - - - 59.88$\pm$2.50
56.76$\pm$1.37 vi 70.41$\pm$1.36 71.59$\pm$0.46 - - - - 57.82$\pm$3.07
57.19$\pm$0.92 th 59.35$\pm$5.40 67.50$\pm$0.96 - - - - 44.62$\pm$3.38
44.85$\pm$0.93 bg 73.99$\pm$1.55 76.10$\pm$0.37 - - - - 77.62$\pm$1.58
80.41$\pm$0.96 hi 64.60$\pm$1.57 67.28$\pm$1.01 - - - - 51.49$\pm$5.90
55.39$\pm$0.85 ur 56.87$\pm$6.08 62.83$\pm$1.05 - - - - 49.65$\pm$4.19
53.30$\pm$0.56 sw 61.03$\pm$4.46 64.93$\pm$0.76 - - - - - - Avg.
68.87$\pm$1.02 71.77$\pm$0.36 81.21$\pm$0.31 81.26$\pm$1.01 48.48$\pm$0.16
51.29$\pm$0.29 61.13$\pm$2.46 62.32$\pm$0.65

Table 2:  Cross-lingual transfer evaluation results of XGLM-564M models with
prompt tuning (PT) and fine-tuning (FT) on XNLI, PAWS-X, NER, and POS tasks.
Each model is trained on the English training set and evaluated on the test
set of all target languages to measure performance. The final performance of
the models is presented as the average and standard deviation of the results
obtained from three models trained with three different seeds. Languages are
sorted in descending order based on their proportion in CC100-XL (Lin et al.,
2022), XGLM’s pre-training dataset. The performance of languages not covered
in the test sets is indicated with a hyphen (-).

###  3.2 Results

The evaluation results of the XGLM-564M model tuned with prompt tuning and
fine-tuning, respectively, can be found in Table 2. On the four cross-lingual
tasks, prompt tuning generally performs on par with or outperforms fine-
tuning. Even considering the performance variance, prompt tuning exhibits
better performance than fine-tuning especially in the XNLI and NER tasks.

We also note that high-resource languages, such as English, Russian, Spanish,
and French, consistently exhibit relatively better performance across all
tasks, and vice versa. Among many possible causes, we speculate the language
imbalance in the pre-training dataset of the XGLM model.

Lin et al. (2022) also point out that low-resource languages exhibit a
relatively lower overlap of subwords with other languages, and the tokenizer
trained on imbalanced corpora further breaks down words into smaller subword
units, leading to a higher level of granularity. We share a similar
perspective that the fine-grained tokenization may restrict the model’s
capacity to capture more comprehensive contextual information, thereby
potentially contributing to the performance decline in low-resource languages
as shown in Table 2.

###  3.3 Analysis

By comparing the performance differences between the two models for each
language in Table 2, we can evaluate the performance gains obtained through
each tuning technique. For the XNLI task, we can observe that languages such
as English, German, Spanish, and French have performance differences of less
than 1% in terms of performance gains. Languages like Turkish, Arabic, Thai,
and Urdu, on the other hand, show larger differences in performance gains,
ranging from 4.67% to as high as 8.15%. In summary, excluding a few
exceptional languages such as Chinese and Vietnamese, we can conclude that
low-resource languages generally exhibit larger performance improvements
through prompt tuning compared to high-resource languages, surpassing the
magnitude of performance gains achieved through fine-tuning.

In the case of low-resource languages, where word granularity is high as
mentioned in Lin et al. (2022), although it may be disadvantageous for
capturing high-level contextual information, it can be advantageous for
learning language identification information instead. In other words, we
presume the finely tokenized subwords may act as a form of discrete prompts
(Liu et al., 2021; Han et al., 2021), leading to the observation that prompt
tuning is more effective for low-resource languages compared to fine-tuning.
Liu et al. (2021) report that using discrete tokens as anchor prompts along
with continuous prompts improve the model performance on various NLU tasks. In
this paper, although such anchor prompts are not used explicitly, we speculate
that the small subword units of low-resource languages might implicitly aid
the performance gains of prompt tuning.

##  4 Related Work

###  4.1 Multilingual Models

Multilingual models are based on the widely used Transformers architecture
(Vaswani et al., 2017) and the language modeling objectives commonly employed
in monolingual models (Devlin et al., 2019; Brown et al., 2020; Raffel et al.,
2020).

Significantly improved performance in these models has been achieved by
combining various techniques such as tokenizers designed to accommodate
numerous languages (Sennrich et al., 2016; Kudo, 2018; Song et al., 2021),
considerably larger vocabularies (Pires et al., 2019; Conneau et al., 2020;
Lin et al., 2022), special tokens for language indication and training
objectives that are helpful for learning language alignment information
(CONNEAU and Lample, 2019). Similar to the recent advancement in the large-
scale language models (Brown et al., 2020; Smith et al., 2022; Chowdhery et
al., 2022), multilingual models are also maximizing the zero-shot and few-shot
performance on various downstream tasks through larger parameters and a
greater amount of multilingual training corpora (Lin et al., 2022; Scao et
al., 2022).

###  4.2 Prompt Tuning

Prompt tuning (Lester et al., 2021) is a parameter-efficient tuning technique
that aims to achieve performance similar to fine-tuning, while updating only a
small portion of a large language model. It primarily involves appending
fixed-length tunable continuous prompts called prefix to the input sentence,
while freezing the remaining parameters, to maximize computational efficiency
(Li and Liang, 2021; Liu et al., 2021). Liu et al. (2022) proposed P-tuning v2
with additional trainable parameters attached to every model layer to bring
performance boosting across various tasks.

Prompt tuning was found to be effective in the encoder-based multilingual
model and the language-independent decision boundary was stated as the primary
factor that contributes to the superiority of it (Tu et al., 2022). Although
Winata et al. (2022) reports prompt tuning performance on the XGLM model, it
does not fall under the parameter-efficient tuning as both the discrete prompt
templates attached to the input sentence and the remaining pre-trained weights
are tuned. Additionally, the evaluation dataset (Winata et al., 2023) is
specialized in the Indonesian language family, while the cross-lingual
datasets we employed cover a broader range of language groups.

##  5 Conclusion

In this paper, we reconfirm the performance of prompt tuning observed in the
encoder-based multilingual models in the context of the decoder-based
generative multilingual model, XGLM. We also demonstrate that performance
tends to improve overall as the language becomes dominant in the training set,
with prompt tuning particularly yielding significant performance gains
compared to fine-tuning in low-resource languages. We suspect that the
fundamental cause of this phenomenon lies in the language imbalance which
leads to fine-grained tokenization of low-resource languages. As a result, in
low-resource languages, finely segmented subword units function similarly to
discrete prompts and lead to the effectiveness of prompt tuning.

In future work, we intend to expand our experiments to cross-lingual natural
language generation tasks (Liang et al., 2020) to observe the potential
strengths of the generative language model. Furthermore, investigating whether
prompt tuning can maintain competitiveness over fine-tuning when evaluated on
other large-scale decoder-only models such as BLOOM (Scao et al., 2022) is
also an interesting topic. We also aim to study whether other PEFT methods
such as LoRA (Hu et al., 2022) or Adapters (Houlsby et al., 2019) work well on
the decoder-based multilingual models.

## Limitations

Our experiments are based on the pre-trained XGLM-564M model, the smallest
among the publicly released versions, due to limited computation resources
with a single A40 GPU. Previous studies claim that as the model size
increases, prompt tuning becomes even more parameter-efficient, maximizing its
effectiveness (Lester et al., 2021; Kaplan et al., 2020). Therefore, expanding
our experiments to publicly available larger models, such as XGLM-7.5B, in
order to demonstrate competitiveness can become a promising future work.

In this paper, we achieved performance comparable to fine-tuning across
various cross-lingual tasks, despite the most basic form of prompt tuning
method. We look forward to expand our experiments using prompt tuning
techniques specifically designed for multilingual models (Fu et al., 2022; Vu
et al., 2022; Huang et al., 2022), as well as exploring other branches of
parameter-efficient tuning techniques (Hu et al., 2022; Houlsby et al., 2019;
Pfeiffer et al., 2020).

## Ethics Statement

By adopting the parameter-efficient prompt tuning, we expect increased energy
efficiency and reduced carbon footprint by utilizing fewer GPU resources. Due
to the inherent language imbalance in the training corpus, there is a
performance degradation in low-resource languages. We show that prompt tuning
can compensate for this performance decline by leveraging the interaction with
tokens specific to low-resource languages through our analysis.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2022) Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022.  Revisiting parameter-efficient tuning: Are we really there yet? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2612–2626, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * CONNEAU and Lample (2019) Alexis CONNEAU and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fu et al. (2022) Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022.  Polyglot prompt: Multilingual multitask prompt training.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9919–9935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Han et al. (2021) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021.  Ptr: Prompt tuning with rules for text classification.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Huang et al. (2022) Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. 2022.  Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11488–11497, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021.  Gpt understands, too.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.
  * Song et al. (2021) Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021.  Fast WordPiece tokenization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tu et al. (2022) Lifu Tu, Caiming Xiong, and Yingbo Zhou. 2022.  Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5478–5485, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022.  Overcoming catastrophic forgetting in zero-shot cross-lingual generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9279–9300, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Winata et al. (2022) Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. 2022.  Cross-lingual few-shot learning on unseen languages.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 777–791, Online only. Association for Computational Linguistics.
  * Winata et al. (2023) Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023.  NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 815–834, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Zhao and Schütze (2021) Mengjie Zhao and Hinrich Schütze. 2021.  Discrete and soft prompting for multilingual models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8547–8555, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

##  Appendix A Model Details

Task Model Prompt Length Learning Rate Batch Size XNLI FT - 1e-5 64 PT 15 1e-2
128 PAWS-X FT - 5e-5 128 PT 15 5e-3 128 NER FT - 5e-5 64 PT 15 1e-2 128 POS FT
- 5e-5 64 PT 10 1e-3 64

Table 3:  Hyperparameters for the best checkpoints of XGLM-564M trained with
prompt tuning (PT) and fine-tuning (FT) on each task.

Table 3 is a summary of the setups that achieve the best performance across
the four cross-lingual tasks through multi-seed experiments. The hidden
dimension size of the MLP prompt encoder is fixed at 512, thus the proportion
of tunable parameters varies depending on the length of the continuous
prompts. Out of the total 564M parameters in the model, approximately 0.087%,
0.1304%, and 0.1738% are tunable when the prompt length is 10, 15, and 20,
respectively. All experiments are done on a single A40 GPU.

##  Appendix B Subword Fertility

Language XNLI PAWS-X NER POS English (en) 1.24 1.31 1.35 1.26 Russian (ru)
1.51 - - 1.71 Chinese (zh) 2.09 14.69 - 2.13 German (de) 1.41 1.59 1.55 1.4
Spanish (es) 1.25 1.41 1.34 1.31 French (fr) 1.31 1.51 - 1.4 Japanese (ja) -
16.46 - - Italian (it) - - - 1.35 Greek (el) 1.57 - - 1.62 Korean (ko) - 2.33
- - Polish (pl) - - - 1.71 Dutch (nl) - - 1.42 1.46 Turkish (tr) 1.5 - - 1.73
Arabian (ar) 1.62 - - 1.45 Vietnamese (vi) 1.36 - - 1.55 Thai (th) 1.61 - -
1.64 Bulgarian (bg) 1.5 - - 1.55 Hindi (hi) 1.46 - - 1.38 Urdu (ur) 1.21 - -
1.41 Swahili (sw) 1.43 - - -

Table 4:  Language-specific subword fertility. Languages are sorted in
descending order based on their proportion in CC100-XL, the pre-training
dataset of XGLM models).

To support the hypothesis in Section 3.3, we include language-specific subword
fertility results (average number of subwords per word) as shown in Table 4.
Since the pre-training dataset for the XGLM model, CC100-XL, is proprietary,
direct calculation of subword fertility for the languages comprising the pre-
trained model’s vocabulary is challenging. Instead, we aim to use tokenization
results obtained with the pre-trained XGLM tokenizer on samples from the four
cross-lingual transfer datasets employed in our experiments for assessing
subword fertility.

For the XNLI task, low-resource languages (Turkish, Arabic, Thai, and Urdu)
with significant performance gains have an average fertility of 1.485, whereas
high-resource languages (English, German, Spanish, and French) with smaller
performance gains have an average fertility of 1.3025.

For the POS task, low-resource languages (Turkish, Hindi, and Urdu) with
notable performance gains have an average fertility of 1.5067, while high-
resource languages (Spanish and Italian) with modest performance gains exhibit
an average fertility of 1.33.

It’s worth noting that Chinese and Japanese have higher fertility compared to
other languages, mainly because they often convey meaning at the character
level, resulting in fine-grained tokenization regardless of their proportion
in the pre-training dataset (Lin et al., 2022). Excluding these exceptional
cases, we can infer a correlation between the extent of tokenization,
represented by fertility, and the magnitude of performance gains achieved
through prompt tuning.