# A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs

Vaibhav Singh♠ξ, Amrith Krishna♢λ, Karthika N J♠ξ, Ganesh Ramakrishnan♠ξ
♠Indian Institute of Technology Bombay, ♢SML
ξ{singhvaibhav, karthika<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Low-resource languages, by its very definition, tend to be under represented
in the pre-training corpora of Large Language Models. In this work, we
investigate three low-resource cross-lingual approaches that enable an LLM
adapt to tasks in previously unseen languages. Llama-2 is an LLM where Indic
languages, among many other language families, contribute to less than
$0.005\%$ of the total $2$ trillion token pre-training corpora. In this work,
we experiment with the English-dominated Llama-2 for cross-lingual transfer to
three Indic languages, Bengali, Hindi, and Tamil as target languages. We study
three approaches for cross-lingual transfer, under ICL and fine-tuning. One,
we find that adding additional supervisory signals via a dominant language in
the LLM, leads to improvements, both under in-context learning and fine-
tuning. Two, adapting the target languages to word reordering may be
beneficial under ICL, but its impact diminishes with fine tuning. Finally,
continued pre-training in one low-resource language can improve model
performance for other related low-resource languages.

##  1 Introduction

Figure 1: Improved natural language understanding (NLU) and generation (NLG)
of Llama-2-7b in Bengali and Tamil through continued pre-training in Hindi
(Bridging) and leveraging English for cross-lingual transfer (Handholding).
Figure 2: Task of slot filling, using the cross-lingual transfer objective
from English to Hindi, using an LLM. In this example, the word ‘sun’
translates to ‘sūraja’ in Hindi and ‘sunday’ translates to ‘ravivāra’. Thus,
in the output. the LLM assigns the label
$\underline{\textit{weather\\_descriptor}}$ to the word ‘sun’ in Hindi, and
the label $\underline{\textit{date}}$ to ‘sunday’ in Hindi. Refer to Table 11
and Table 12 for details on the prompt.

Large language models (LLM; Brown et al., 2020; Touvron et al., 2023;
Chowdhery et al., 2022; Mesnard et al., 2024) are known to generalise well
across several tasks, including in few shot and zero-shot setups. However,
there is limited evidence that shows the ability of these models to generalise
to tasks in new languages out of the box, especially to those with which the
model has limited exposure to. In this work, we investigate how effectively we
can leverage the LLMs for cross lingual transfer, especially for adapting it
to low-resource languages.

LLMs typically require tens of billions, if not trillions, of tokens for its
pre-training. Now, that is a challenge for majority of the languages in the
world. More than $80\%$ of languages in the world are ‘left behind’ (Joshi et
al., 2020), and barely have enough digitised data that matches the
requirements for pre-training an LLM from scratch. For instance, the most
populous country in the world, India, speaks more than $400$
languages111https://en.wikipedia.org/wiki/Languages_of_India, with $22$ of
them recognised as scheduled languages by the Government of India. However,
none of these languages contribute to more than $0.005\%$ of the pre-training
data of an open-source LLM like Llama-2 (Touvron et al., 2023). In fact, more
than 95% of these languages lack enough digital resources to incorporate them
into an LLM. These resource-poor languages tend to get poorer in
representation with the progress in the field (Joshi et al., 2020; Ojo et al.,
2024).

Some of the recent works, explore various techniques to adapt an LLM to new
languages, especially with limited target language resources Rathore et al.
(2023). Tanwar et al. (2023) exploit cross-lingual transfer to improve in-
context learning (ICL) for binary sequence classification tasks in low-
resource languages by utilizing in-context exemplars from a high-resource
language semantically similar to the input in the target language. Husain et
al. (2024) employ continual pre-training on Llama-2 with romanized pre-
training corpora of non-roman script languages, to exploit cross-lingual
transfer using the script of English. Awasthi et al. (2023) use $540$b PaLM
(Chowdhery et al., 2022) to generate training data in low-resource languages
using labelled instances in English. Razumovskaia et al. (2024) provide
analyses of multilingual capabilities of LLMs on NLU tasks under the settings
of in-context learning (ICL), supervised fine-tuning (SFT), and supervised
instruction-tuning (SIT).

Our investigation primarily involves the following three questions, centered
around information extraction (IE) tasks in a low-resource language using an
instruction-tuned LLM. Q1. Handholding: For an IE task in a low-resource
target language, would providing a parallel, annotated sentence in the
predominant language of the LLM, help to exploit cross-lingual transfer,
resulting in improved performance for the target language. By predominant
language, we imply the language that forms the majority of the pre-training
corpora. Q2. Masquerading: Would adapting the target language to resemble the
predominant language enable in cross-lingual transfer, benefiting the target
language. Finally, Q3. Bridging: Whether model adaptation in one of the low-
resource languages can benefit other related low-resource languages. More
clarity on these questions, is presented in Section 2.

We focus on three Indic languages, namely, Bengali, Hindi, and Tamil. These
languages are culturally diverse within the Indic context, with Bengali and
Hindi belonging to the Indo-Aryan family and Tamil to the Dravidian family. To
evaluate our hypotheses Q1, Q2, and Q3, we focus on two information extraction
tasks: slot filling and named entity recognition (NER). Further, we use a $7$
billion parameter English-centric LLM Llama-2 as our base LLM, unless
otherwise stated. The slot filling and named entity recognition tasks possess
label-set size of $55$ and $3$, respectively. Additionally, none of Bengali,
Hindi, and Tamil contribute to more than $0.005\%$ of the pre-training corpora
of Llama-2. Moreover, English is the predominant language, contributing to
roughly $90\%$ of the pre-training corpora.

In our experiments, we simlulate a low-resource scenario where we do not
expect the target language to have more than roughly $10,000$ instances. In
Bridging, when Llama-2 is adapted with Hindi through continued pre-training,
we use more than 10,000 sentences in Hindi. However, in this case, Hindi is
referred to as the bridge language. The evaluation is solely performed on
Bengali and Tamil, both of which satisfy aforementioned criteria for the low-
resource setting. Our investigation includes exploiting few-shot in-context
learning (ICL) ability of Llama-2 as well as model adaptation with parameter-
efficient supervised fine-tuning (PEFT). To evaluate Llama-2, or any auto-
regressive LLM in general, we frame the tasks of slot filling and named entity
recognition as text-to-text generation tasks. Figure 2 showcases slot filling
as a text-to-text generation task.

Extensive experiments on Llama-2 show that Handholding improves NLU and NLG in
Bengali, Hindi and Tamil by exploiting cross-lingual transfer from English,
under both few-shot ICL and PEFT. Further, Bridging with Hindi, improves
monolingual task performance in related languages of Bengali and Tamil under
PEFT. Ultimately, Handholding + Bridging turns out the most beneficial
combination, yielding best task performance for both low-resource languages of
Bengali and Tamil. A quantitative overview has been presented in Figure 1.

Our major contributions can be summarized as follows:

  * •

We demonstrate that the predominant language of an LLM can be leveraged to aid
low-resource languages. Specifically, leveraging English via Handholding,
improves the overall performance of Llama-2 for information extraction tasks
in Hindi, Bengali, and Tamil under both few-shot in-context learning (ICL) and
parameter-efficient fine-tuning (PEFT).

  * •

Improved natural language understanding and generation in Bengali and Tamil,
as shown by our experiments with Llama-2 adapted with Hindi (Bridging),
demonstrates that adapting a model in one low-resource language can benefit
other related languages.

  * •

Modifying target language via (Masquerading) to resemble the predominant
language, English, gives superficial benefits in few-shot ICL and diminishes
further in PEFT.

##  2 Preliminaries

###  2.1 Task Definition

Given a finite label-set $\mathcal{L}$, let
$\mathbf{X}^{S}=(X_{1}^{S},X_{2}^{S},\ldots,X_{n}^{S})$ denote a sentence in
source language and $\mathbf{A}^{S}=(A_{1}^{S},A_{2}^{S},\ldots,A_{n}^{S})$
represent the corresponding word-level label sequence, where
$A_{i}^{S}\in\mathcal{L}\cup\\{\phi\\}$ and $\phi$ indicates the absence of a
label. A labelled source sequence is given by
$\mathbf{Z}^{S}=((X_{1}^{S},A_{1}^{S}),(X_{2}^{S},A_{2}^{S}),\ldots,(X_{n}^{S},A_{n}^{S}))$.
In Handholding, our goal is to transfer these annotations to a parallel,
unannotated sentence in target language
$\mathbf{X}^{T}=(X_{1}^{T},X_{2}^{T},\ldots,X_{m}^{T})$, producing an labelled
target sentence $\mathbf{Z}^{T}$. Figure 2 demonstrates the defined text-to-
text cross-lingual setup. Formally,

$\mathbf{Z}^{T}=\arg\max_{\mathbf{Y}}P_{\text{LLM}}(\mathbf{Y}\mid\mathbf{Z}^{S},\mathbf{X}^{T})$

where $\mathbf{Y}=((Y_{1},B_{1}),(Y_{2},B_{2}),\ldots,(Y_{m},B_{m}))$ is a
potential annotated target sentence, with $Y_{i}$ being elements of
$\mathbf{X}^{T}$ and $B_{i}$ being elements of $\mathcal{L}\cup\\{\phi\\}$. In
our context, the conditional probability can be decomposed following the auto-
regressive nature of LLM generation:

$\displaystyle P_{\text{LLM}}(\mathbf{Y}\mid\mathbf{Z}^{S},\mathbf{X}^{T})$
$\displaystyle=$
$\displaystyle\prod_{i}P((Y_{i},B_{i})\mid(Y_{j},B_{j})_{<i},\mathbf{Z}^{S},\mathbf{X}^{T})$

In a similar manner, as shown in Figure 2, a monolingual objective with no
Handholding, can be formulated in the following manner:

$\mathbf{Z}^{T}=\arg\max_{\mathbf{Y}}P_{\text{LLM}}(\mathbf{Y}\mid\mathbf{X}^{T})$
$\displaystyle
P_{\text{LLM}}(\mathbf{Y}\mid\mathbf{X}^{T})=\prod_{i}P((Y_{i},B_{i})\mid(Y_{j},B_{j})_{<i},\mathbf{X}^{T})$

###  2.2 Handholding, Masquerading, and Bridging

#### Predominant Language as a Point of Supervision:

In our work, with Llama-2, English is the predominant language with $89.70\%$
presence in the pre-training corpora of Llama-2. On the contrary, low-resource
languages like Bengali, Hindi, and Tamil, cover less than $0.005\%$, and can
be regarded as ‘unseen’ when compared to English. To leverage the
understanding of Llama-2 in English for an IE task in a low-resource ‘target’
language, we include annotated parallel sentence in English as a part of the
task-specific prompt to the LLM. As shown in Figure 2, referred to as
Handholding, we utilize annotated English sentence $(\mathbf{Z^{S}})$ to
facilitate cross-lingual transfer to the target language.

#### Adaptation of Target Language:

To further aid cross-lingual transfer, we look at ways in which the target
language can resemble English. First, we look at word order. Word order refers
to the arrangement of words in a sentence. Word order is one of the syntactic
features that varies across languages. English follows subject-verb-object
order. On the contrary, Indic languages largely follow subject-object-verb
word order where the verb appears at the tail part of a sentence. Second, we
look at the script of English, to aid cross-lingual transfer. As English
follows the Latin script, we employ transliteration schemes to transform the
sentence in the target language to Latin. We refer to this adaptation of the
target to resemble English as Masquerading. Figure 2 gives an overview of
target sentence $(\mathbf{X^{T}})$ masqueraded to resemble English.

#### Related Language as a Bridge:

Continual pre-training (Cui et al., 2024; Gupta et al., 2023), vocabulary
extension (Zhao et al., 2024), instruction-tuning(Gala et al., 2024; Li et
al., 2023; Husain et al., 2024) are some of the ways to increase
representation of language(s) into an LLM. As Hindi is one of the most
represented languages in India, we investigate the effect of adapting an LLM
in Hindi through continual pre-training, on related low-resource languages of
Bengali and Tamil. We refer to this as Bridging. Hindi in this scenario,
becomes the bridge language, while Bengali and Tamil become the target
languages for evaulation.

##  3 Experiments

###  3.1 Datasets

#### Slot Filling:

We use Amazon Massive (FitzGerald et al., 2022). The dataset includes slot
annotated virtual assistant utterances parallel across $51$ languages. We
choose sentences from [utt] and [annot_utt] fields of the dataset to represent
unannotated sequence $\mathbf{X}$ and ground-truth annotated sequence
$\mathbf{Z}$ respectively for cross-lingual transfer among languages: English,
Bengali, Hindi, and Tamil. This dataset includes $55$ label types, including
place_name, business_name, music_genre, among others. Refer to Table 9 for all
label types and Table 8 for the train-test split.

#### Named Entity Recognition:

We work with with AI4Bharat Naamapadam (Mhaske et al., 2023), the largest
publicly available NER dataset for 11 Indic languages, sampled and annotated
from Samanantar (Ramesh et al., 2022). For the languages in focus, Bengali,
Hindi, and Tamil, Naamapadam has $961.7\text{k}$, $985.8\text{k}$, and
$497.9\text{k}$ instances in their train split, respectively. We sample
$16\text{k}$ instances for each of the languages. Due to the absence of
ground-truth annotated parallel sequences in English for each of Hindi,
Bengali, and Tamil, we leverage the same strategy as (Mhaske et al., 2023) and
pick the corresponding set of English sentences from Samanantar and annotate
them using a bert-base token-classification reference model. List of all label
types and train-test split can be found in Table 9 and Table 8, respectively.

###  3.2 Implementation Details

To evaluate all the hypotheses presented in Section 2, we use English-centric
Llama-2-7b Touvron et al. (2023). By ‘English-centric’, we mean to point that
English is the predominant language of the LLM. Particularly, we use
Llama-2-7b-chat, the instruction-tuned variant of pre-trained base Llama-2-7b.
The need for the instruction-tuned variant is mainly attributed to the nature
of a prompt-based generation task where we expect an LLM to be prompted with
an instruction followed by an input instance.

For Handholding, we use English as the labelled point of supervision to enable
cross-lingual transfer. Further, we do not use ground-truth English labels
during task-specific model inference; instead, we label the English sentence
using a token classification model before the cross-lingual transfer step. We
refer to these predicted labels for English as pseudo labels and the ground-
truth labels for English as oracle labels. For slot filling, we use $84.05$ F1
score xlm-roberta-base222https://huggingface.co/cartesinus/xlm-r-base-amazon-
massive-slot token classification model proposed in (Kubis et al., 2023).
Whereas, for named entity recognition, we use $91.3$ F1 score bert-
base333https://huggingface.co/dslim/bert-base-NER token classifier, as
discussed in Section 3.1. Figure 4 shows the difference between an oracle and
pseudo labelled sentence in English for the task of slot filling.

In Masquerading with word order, we use GIZA++ (Och and Ney, 2003), a word
alignment model based on the statistical models by IBM (Brown et al., 1993)
and pre-trained LM-based SimAlign (Sabet et al., 2021) to generate word re-
ordered target sentences. Specifically, we use SimAlign for Hindi and GIZA++
for Bengali and Tamil based on qualitative assessment. In the latter setting
of Masquerading, we follow ISO 15919:2001 to transliterate the sentences in
Bengali, Hindi, and Tamil to Latin script. Refer Figure 3 for an example of
adapting Hindi to resemble English.

Figure 3: English follows  subject verb object word order in contrast to
Hindi. Hindi follows the word order of  subject object verb As shown,
$\mathbf{X^{T}}$ is presented in SOV order and $\text{re-ordered
}\mathbf{X^{T}}$ is presented in SVO order. $\text{transliterated
}\mathbf{X^{T}}$ is $\mathbf{X^{T}}$ in Latin script using ISO 15919:2001.
Here, only the script of $\mathbf{X^{T}}$ is changed, keeping the word order
of Hindi.

For Bridging, we utilize Airavata-7b (Gala et al., 2024), a continually pre-
trained and instruction-tuned version of pre-trained base Llama-2-7b model in
code-mixed Hindi and English. To ensure that the effect of Bridging in Hindi
on Bengali and Tamil can be solely attributed to the increased representation
of Hindi, we highlight the key differences between Llama-2-7b-chat and
Airavata-7b.

According to Touvron et al. (2023), Llama-2-7b-chat builds on Llama-2-7b base
pre-trained model through supervised fine-tuning with publicly available SFT
datasets (Chung et al., 2022) and $27,540$ high-quality in-house vendor-based
SFT annotations followed by reinforcement learning through human feedback
(RLHF) (Ouyang et al., 2022) with over $1$ million human annotated instances.
Whereas, to train Airavata-7b, Gala et al. (2024) employ LoRA fine-tuning on a
continually pre-trained Llama-2-7b with publicly available English SFT
datasets, with their translations in Hindi, amounting to a total of
$385\text{K}$ SFT instances.

We note two observations: (1) the utilized SFT datasets do not cover either of
the two datasets used in our evaluation, eliminating any case of labelled data
leakage and (2) the quality of the SFT instances used for training Airavata-7b
does not match that of Llama-2-7b-chat, mainly due to absence of high quality
in-house annotations and the Hindi subset being translations of publicly
available English SFT instances, which generally possess insufficient
diversity and insufficient quality (Touvron et al., 2023). Hereafter, we refer
to Llama-2-7b-chat and Airavata-7b, simply as $\texttt{Llama}_{\texttt{chat}}$
and Airavata, respectively.

Figure 4: Here, $\text{oracle }\mathbf{Z^{S}}$ refers to the ground-truth
annotation of $\mathbf{X^{S}}$. $\text{pseudo }\mathbf{Z^{S}}$ is obtained
after passing $\mathbf{X^{S}}$ through an xlm-roberta-base token
classification model.

We use HuggingFace
transformers444https://huggingface.co/docs/transformers/index (Wolf et al.,
2020) for task and language adaptation with PEFT and ICL experiments. For ICL,
we employ openICL (Wu et al., 2023) and use $k$-nearest neighbour based
retrieval for few-shot demonstrations, following Liu et al. (2022). For
retrieval, we compute sentence level representation of the inference time
input and the training data using Reimers and Gurevych (2019). We specifically
use xlm-roberta-base (Conneau et al., 2020) as the base pre-trained model. We
choose $8$ input-output pairs as for the few-shot demonstrations. These
demonstrations for both tasks are mutually exclusive. For instance, in
Masquerading with word order, we keep all demonstrations to have re-ordered
sentences in the target language. It ensures that the few-shot examples are
directly relevant to the task variation with high specificity.

For PEFT, we utilize HuggingFace PEFT555https://github.com/huggingface/peft
with LoRA (Hu et al., 2021) on top of 4-bit quantization, to fine-tune
$\texttt{Llama}_{\texttt{chat}}$ and Airavata on a single 80GB NVIDIA A100
Tensor Core GPU. With PEFT-LoRA, trainable parameters amount to only 0.5% of
the total parameters of the aforementioned LLMs. We train our models with
32-bit paged AdamW (Loshchilov and Hutter, 2019) optimizer, with an initial
learning rate of $1\times 10^{-3}$ coupled with a cosine scheduler. Refer to
Appendix D for detailed model configuration.

During inference, we switch to Contrastive
Search666https://huggingface.co/blog/introducing-csearch (Su and Collier,
2023) with $\alpha=0.6$ to penalize token repetitions and control model
behavior to generate human-level coherent outputs.

#### Metrics:

We use micro-F1 as our primary evaluation metric for slot filling and named
entity recognition, both being NLU tasks. Given that both tasks are framed as
text-to-text tasks via an LLM, we also include Exact Match to capture
correctness, and chrF++ (Popović, 2017) to assess the lexical overlap between
the LLM-generated prediction and the ground-truth reference. Additionally, we
measure the naturalness of the generated output on $500$ randomly sampled test
instances using MAUVE (Pillutla et al., 2021).

##  4 Results

In this section, we present our findings with comparative analysis for the
approaches of Handholding, Masquerading, and Bridging on Llama-2 with few-shot
ICL and PEFT. For consolidated quantitative figures with PEFT refer to Table
7.

#### Monolingual ICL Results:

We report near zero performance with $\texttt{Llama}_{\texttt{chat}}$ in the
monolingual ICL settings. We follow few-shot prompt demonstration under 3
different ICL settings. Here, we provide the input in the target language as
is, or masquerade it by either transliterating or re-ordering the input.
Nevertheless, we observe near-zero micro-F1, exact match (EM) scores, and poor
lexical overlap with reference outputs in all three languages for both the
tasks. These observations align with the observations made in (Razumovskaia et
al., 2024) and demonstrate the challenges in adapting a new unseen language in
ICL settings to an LLM like Llama-2.

Language Metric |  $\texttt{Llama}_{\texttt{chat}}$ (monolingual)
---|---
F1 | EM | chrF++ | MAUVE
Slot Filling
Bengali | $54.72$ | $22.37$ | $71.40$ | $89.07$
Hindi | $51.89$ | $23.15$ | $70.90$ | $59.82$
Tamil | $44.29$ | $14.37$ | $70.65$ | $49.04$
Named Entity Recognition
Bengali | $59.98$ | $24.69$ | $85.91$ | $95.28$
Hindi | $71.58$ | $38.25$ | $90.00$ | $98.70$
Tamil | $39.92$ | $12.25$ | $68.72$ | $33.06$

Table 1: Monolingual performance of $\texttt{Llama}_{\texttt{chat}}$ under
PEFT.

#### Monolingual PEFT Results:

As shown in Table 1, we observe performance improvements under monolingual
settings, when the model parameters are updated with task-specific PEFT.
Averaged over both tasks, the exact match (EM) scores of labelled output
generations in Bengali, Hindi, and Tamil stand at $23.53\%$, $30.7\%$, and
$13.31\%$, respectively. Whereas, the lexical overlap of the generated outputs
with the ground-truth outputs are $78.65\%$, $80.45\%$, and $69.68\%$,
respectively. These Indic languages are morphologically rich, in general,
leading to lower EM scores, though report higher chrF++ (lexical overlap) and
MAUVE (naturalness) scores, comparatively.

Language Metric |  $\texttt{Llama}_{\texttt{chat}}$ (Handholding)
---|---
F1 | EM | chrF++ | MAUVE
Slot Filling
Bengali | $64.32$ | $36.82$ | $79.27$ | $90.39$
Hindi | $60.60$ | $36.70$ | $77.95$ | $89.72$
Tamil | $61.48$ | $33.79$ | $80.67$ | $76.51$
Named Entity Recognition
Bengali | $80.35$ | $45.44$ | $91.00$ | $93.36$
Hindi | $78.03$ | $47.50$ | $90.38$ | $97.09$
Tamil | $74.18$ | $42.69$ | $88.75$ | $81.34$

Table 2: Effect of Handholding on $\texttt{Llama}_{\texttt{chat}}$ under PEFT.

#### Handholding PEFT Results:

Table 2 shows the performance for the target language under PEFT with
Handholding. We observe that Handholding can help further improve the
performance in the target language, with task-specific PEFT. Bengali, Hindi
and Tamil benefit from labelled sentence in English under PEFT by $9.6\%$,
$8.71\%$, and $17.19\%$ micro-F1 score for slot filling, and $20.37\%$,
$6.45\%$, and $34.26\%$ micro-F1 score for named entity recognition. EM scores
also improve by an average of $17.6\%$, $11.4\%$, and $24.93\%$ for Bengali.
Hindi and Tamil, respectively. Similarly, lexical overlap improves in $6$ out
of $6$ cases. However, we observe a drop of $1.92\%$ and $1.61\%$ in
naturalness scores of Bengali and Hindi for the NER task.

Language Change | H | $\stackrel{{\scriptstyle\textit{H + M}}}{{\textit{(re-ordered)}}}$ | $\stackrel{{\scriptstyle\textit{H + M}}}{{\textit{(transliterated)}}}$
---|---|---|---
Slot Filling
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $28.02$ | $\underline{30.12^{*}}$ | $18.01$
$\textit{en}_{(source)}\rightarrow\textit{hi}_{(target)}$ | $38.97$ | $\underline{40.82^{*}}$ | $16.57$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $22.09$ | $\underline{24.38^{*}}$ | $12.61$
Named Entity Recognition
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $13.89$ | $\underline{27.88^{*}}$ | $17.78$
$\textit{en}_{(source)}\rightarrow\textit{hi}_{(target)}$ | $47.61$ | $\underline{49.82^{*}}$ | $19.61$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $19.07$ | $\underline{30.08^{*}}$ | $18.84$

Table 3: Micro-F1 scores for the combination of Handholding (H) and
Masquerading (M) under few-shot ICL.The symbol, ∗ represents statistically
significant gains based on pairwise t-tests with just Handholding ($p<0.05$).

#### Handholding ICL Results:

Similarly, Table 3 reports significant improvements in cross-lingual transfer
to the target language when using Handholding under ICL settings as well. With
few-shot ICL using Handholding, we see significant gains, as compared to the
near-zero performances with few-shot ICL in monolingual settings. Moreover, we
are getting non-zero EM scores in $4$ out of 6 cases with Handholding under
ICL. Nevertheless, as expected, the performance improvements in absolute terms
is much higher in Handholding with task-specific PEFT (Table 2).

#### Handholding and Masquerading ICL Results:

Further, Handholding, along with Masquerading via word re-ordering, leads to
statistically significant results under ICL. Table 3 shows the results for
both Masquerading via re-ordering and transliteration. For both the tasks, re-
ordering the sentences in all the three languages to resemble the word order
in English leads to statistically significant results. However, Handholding +
Masquerading via transliterated target sentences under ICL results in
performance drops. As shown in Table 3, the use of transliterated sentences
generally results in worse performance than using Handholding alone, except
for Bengali in NER.

Language Change | H | $\stackrel{{\scriptstyle\textit{H + M}}}{{\textit{(re-ordered)}}}$
---|---|---
Slot Filling
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $\underline{64.32}$ | $63.19$
$\textit{en}_{(source)}\rightarrow\textit{hi}_{(target)}$ | $60.60$ | $\underline{61.11}$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $61.48$ | $\underline{63.30}$
Named Entity Recognition
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $\underline{80.35}$ | $55.23$
$\textit{en}_{(source)}\rightarrow\textit{hi}_{(target)}$ | $\underline{78.03}$ | $54.01$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $\underline{74.18}$ | $43.96$
Table 4: Micro-F1 scores for the combination of Handholding (H) and
Masquerading (M) under PEFT.

#### Handholding and Masquerading PEFT Results:

As shown in Table 2 and Table 3, Handholding benefits the target language,
both under ICL and PEFT settings. Similarly, combining Handholding with
Masquerading via word re-ordering has shown to be beneficial under ICL. Table
4 presents the results for the combination of Handholding and Masquerading
with task-specific PEFT. However, the benefits from Masquerading appear to
diminish or be counterproductive during PEFT, especially for NER tasks.
Nevertheless we see statistically significant gains for Slot Filling in Tamil,
though not for Hindi. Within Masquerading, we do not explore the setting of
transliteration of target sentence due to its consistent poor performance
under few-shot ICL. For slot filling, Bengali sees a reduction of $1.13\%$
micro-F1 whereas Hindi and Tamil observe increase in micro-F1 scores by
$0.51\%$ and $1.82\%$, respectively.

Language Model | $\texttt{Llama}_{\texttt{chat}}$ | Airavata
---|---|---
Slot Filling
$\textit{bn}_{(target)}$ | $54.72$ | $\underline{64.28}^{*}$
$\textit{ta}_{(target)}$ | $44.29$ | $\underline{46.03}^{*}$
Named Entity Recognition
$\textit{bn}_{(target)}$ | $59.98$ | $\underline{66.62}^{*}$
$\textit{ta}_{(target)}$ | $39.92$ | $\underline{66.14}^{*}$
Table 5: Micro-F1 scores for the effect of Bridging on monolingual performance
in Bengali and Tamil. The symbol, ∗ represents statistically significant gains
for Airavata based on pairwise t-tests with $\texttt{Llama}_{\texttt{chat}}$
($p<0.05$).

#### Bridging:

In Bridging, Hindi serves as the bridge language, while English still remains
the predominant language. In this case, we evaluate model performance on
Bengali and Tamil as the target languages. As discussed in Section 3.2, we use
Airavata to evaluate the effect of increased representation of Hindi on the
related languages of Bengali and Tamil. Our first observation follows that
Bridging improves monolingual performance in both Bengali and Tamil with task-
specific PEFT. As shown in Table 5, Airavata outperforms
$\texttt{Llama}_{\texttt{chat}}$ in both Bengali and Tamil for both tasks of
slot filling and named entity recognition. For slot filling, Bengali observes
an increase of $9.56\%$ micro-F1, $21.37\%$ increase in EM score, $10.17\%$
increase in lexical overlap and an improved output naturalness by $9.63\%$.
Whereas, Tamil benefits with an increased micro-F1, and EM of $1.74\%$, and
$7.03\%$. respectively. However, lexical overlap and naturalness of generated
outputs with reference outputs falls by $9.31\%$ and $12.52\%$ in Airavata as
compared to $\texttt{Llama}_{\texttt{chat}}$. For named entity recognition, we
see similar improvements under all metrics, for both languages post Bridging
except the fall in naturalness for Bengali by $2.47\%$.

Language Model | $\texttt{Llama}_{\texttt{chat}}$ | Airavata
---|---|---
Slot Filling
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $64.32$ | $\underline{67.21}$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $61.48$ | $\underline{65.24}$
Named Entity Recognition
$\textit{en}_{(source)}\rightarrow\textit{bn}_{(target)}$ | $80.35$ | $\underline{84.80}$
$\textit{en}_{(source)}\rightarrow\textit{ta}_{(target)}$ | $74.18$ | $\underline{82.09}$
Table 6: Micro-F1 scores for the combination of Handholding (H) + Bridging (B)
under PEFT.

#### Handholding and Bridging:

Table 6 presents the best performing combination, in terms of model
performance for slot filling and named entity recognition. This is achieved by
Bridging Llama-2 with Hindi, followed by task-specific model adaptation
through PEFT with Handholding. In this case, Bengali benefits by $2.89\%$
micro-F1, $11.72\%$ EM score, $1.54\%$ lexical overlap and $4.98\%$ in
naturalness as compared to Handholding with $\texttt{Llama}_{\texttt{chat}}$
for the task of slot-filling and $4.45\%$ in micro-F1, $13.81\%$ in EM score,
$2.86\%$ in lexical overlap and $6.49\%$ in naturalness for named entity
recognition. Similarly, for slot filling, Tamil observes increase of $3.84\%$
micro-F1, $10.37\%$ EM score, but a drop in $0.26\%$ lexical overlap and
$2.69\%$ naturalness of generated output. Whereas, for named entity
recognition, model performance in Tamil increases by $7.91\%$ micro-F1,
$19.87\%$ EM score, $5.89\%$ lexical overlap, and $18.12\%$ naturalness score.

##  5 Conclusion

In this work, through extensive experiments on English-centric Llama-2-7b-chat
under both ICL and PEFT, we show that Handholding improves NLU and NLG in low-
resource languages: Bengali, Hindi and Tamil by exploiting cross-lingual
transfer from English, demonstrating that the predominant language of an LLM
can be leveraged to aid low-resource languages. Further, Bridging with a low-
resource related language Hindi, results to improved monolingual task
performance in related languages of Bengali and Tamil. Ultimately, through
Handholding + Bridging, we show that incorporating both the predominant
language of the LLM and adapting the LLM in a related language results to
better cross-lingual transfer, leading to significantly improved understanding
and generation in other related low-resource languages. However, adapting the
target language to resemble the predominant language in terms of syntax and
script (Masquerading), only leads to superficial performance improvements in
the low-resource language.

## Limitations

The very notion of the cross-lingual transfer objective from an labelled
sentence in source language to an unannotated sentence in target language
requires parallel data. High-quality parallel data is not uniformly available
for all language pairs, specifically for underrepresented language families
like the Indic family. The requirement of an annotated source during training
and/or inference adds up as a bottleneck. As shown in Section 3.2, it can be
subdued if we have a reference model to label the source, before cross-lingual
transfer. However, the likelihood of a high-accuracy reference model is
minimal when considering the case of cross-lingual transfer of annotations
between two underrepresented languages.

## References

  * Awasthi et al. (2023) Abhijeet Awasthi, Nitish Gupta, Bidisha Samanta, Shachi Dave, Sunita Sarawagi, and Partha Talukdar. 2023.  Bootstrapping multilingual semantic parsers using large language models.
  * Brown et al. (1993) Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993.  The mathematics of statistical machine translation: Parameter estimation.  _Computational Linguistics_ , 19(2):263–311.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Cui et al. (2024) Yiming Cui, Ziqing Yang, and Xin Yao. 2024.  Efficient and effective text encoding for chinese llama and alpaca.
  * FitzGerald et al. (2022) Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022.  Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  * Gala et al. (2024) Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Aswanth Kumar M, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M. Khapra, Raj Dabre, Rudra Murthy, and Anoop Kunchukuttan. 2024.  Airavata: Introducing hindi instruction-tuned llm.  _arXiv preprint arXiv: 2401.15006_.
  * Gupta et al. (2023) Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023.  Continual pre-training of large language models: How to (re)warm your model?
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Husain et al. (2024) Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. 2024.  Romansetu: Efficiently unlocking multilingual capabilities of large language models models via romanization.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kubis et al. (2023) Marek Kubis, Paweł Skórzewski, Marcin Sowański, and Tomasz Ziętkiewicz. 2023.  Back transcription as a method for evaluating robustness of natural language understanding models to speech recognition errors.  _arXiv preprint arXiv:2310.16609_.
  * Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023.  Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.
  * Mesnard et al. (2024) Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024.  Gemma: Open models based on gemini research and technology.
  * Mhaske et al. (2023) Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh M. Khapra, Pratyush Kumar, Rudra Murthy, and Anoop Kunchukuttan. 2023.  Naamapadam: A large-scale named entity annotated data for Indic languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10441–10456, Toronto, Canada. Association for Computational Linguistics.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational Linguistics_ , 29(1):19–51.
  * Ojo et al. (2024) Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David Ifeoluwa Adelani. 2024.  How good are large language models on african languages?
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  In _NeurIPS_.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Ramesh et al. (2022) Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022.  Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages.  _Transactions of the Association for Computational Linguistics_ , 10:145–162.
  * Rathore et al. (2023) Vipul Rathore, Rajdeep Dhingra, Parag Singla, and Mausam. 2023.  ZGUL: Zero-shot generalization to unseen languages using multi-source ensembling of language adapters.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6969–6987, Singapore. Association for Computational Linguistics.
  * Razumovskaia et al. (2024) Evgeniia Razumovskaia, Ivan Vulić, and Anna Korhonen. 2024.  Analyzing and adapting large language models for few-shot multilingual nlu: Are we there yet?
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Sabet et al. (2021) Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2021.  Simalign: High quality word alignments without parallel training data using static and contextualized embeddings.
  * Su and Collier (2023) Yixuan Su and Nigel Collier. 2023.  Contrastive search is what you need for neural text generation.
  * Tanwar et al. (2023) Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023.  Multilingual llms are better cross-lingual in-context learners with alignment.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Wu et al. (2023) Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. 2023.  OpenICL: An open-source framework for in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 489–498, Toronto, Canada. Association for Computational Linguistics.
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.

##  Appendix A Evaluation Results

Refer to Table 7 for micro-F1, EM and lexical overlap scores for all
experiments with Handholding, Masquerading and Bridging under PEFT.

Language Configuration | Llama-2 | Airavata
---|---|---
monolingual | H | H + M | B (monolingual) | H + B
F1 | EM | chrF++ | MAUVE | F1 | EM | chrF++ | MAUVE | F1 | EM | chrF++ | MAUVE | F1 | EM | chrF++ | MAUVE | F1 | EM | chrF++ | MAUVE
Slot Filling
Bengali | $54.72$ | $22.37$ | $71.40$ | $89.07$ | $64.32$ | $36.82$ | $79.27$ | $90.39$ | $63.19$ | $0.96$ | $71.81$ | $37.6$ | $64.28$ | $43.74$ | $\underline{81.57}$ | $\underline{98.70}$ | $\underline{67.21}$ | $\underline{48.54}$ | $80.81$ | $95.37$
Hindi | $51.89$ | $23.15$ | $70.90$ | $59.82$ | $60.60$ | $\underline{36.70}$ | $\underline{77.95}$ | $\underline{89.72}$ | $\underline{61.11}$ | $17.29$ | $73.49$ | $24.18$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Tamil | $44.29$ | $14.37$ | $70.65$ | $49.04$ | $61.48$ | $33.79$ | $\underline{80.67}$ | $\underline{76.51}$ | $63.30$ | $17.80$ | $74.96$ | $19.67$ | $46.03$ | $21.40$ | $61.34$ | $36.52$ | $\underline{65.24}$ | $\underline{44.16}$ | $80.41$ | $73.82$
Named Entity Recognition
Bengali | $59.98$ | $24.69$ | $85.91$ | $95.28$ | $80.35$ | $45.44$ | $91.00$ | $93.36$ | $55.23$ | $0.37$ | $54.43$ | $15.14$ | $66.42$ | $34.63$ | $89.45$ | $92.81$ | $\underline{84.80}$ | $\underline{59.25}$ | $\underline{93.86}$ | $\underline{99.85}$
Hindi | $71.58$ | $38.25$ | $90.00$ | $\underline{98.70}$ | $\underline{78.03}$ | $\underline{47.50}$ | $\underline{90.38}$ | $97.09$ | $54.01$ | $0.63$ | $46.18$ | $18.62$ | $-$ | $-$ | $-$ |  | $-$ | $-$ | $-$ | $-$
Tamil | $39.92$ | $12.25$ | $68.72$ | $33.06$ | $74.18$ | $42.69$ | $88.75$ | $81.34$ | $43.96$ | $1.31$ | $49.93$ | $45.28$ | $66.14$ | $42.81$ | $91.42$ | $99.22$ | $\underline{82.09}$ | $\underline{62.56}$ | $\underline{94.64}$ | $\underline{99.46}$

Table 7: micro-F1, EM, chrF++, and MAUVE scores under PEFT with the model
configurations of H: Handholding, M: Masquerading, and B: Bridging. Here,
MAUVE is computed on $500$ randomly sampled test instances.

##  Appendix B Dataset Splits

The dataset split for both tasks is presented in Table 8. For Massive, we use
the train, validation, and test split as on HuggingFace
datasets777https://huggingface.co/datasets/MASSIVE. For evaluation, we
restrict the test set to only contain utterances that have at least $1$ token
with a slot label. For Naamapadam, we split the $16$k sampled instances in a
$8$:$1$:$1$ ratio to create train, validation, and test subsets.

Task Dataset Split | Train | Test
---|---|---
Slot Filling | $11.5\text{k}$ | $1.9\text{k}$
Named Entity Recognition | $12.8\text{k}$ | $1.6\text{k}$
Table 8: Dataset split for slot filling and named entity recognition tasks.

##  Appendix C List of Label Types

Complete list of label types within Massive and Naamapadam is showcased in
Table 9.

##  Appendix D Training and Inference Configuration

We present our PEFT and ICL hyperparameter settings in Table 10. These
hyperparameters remain the same across both Llama-2-7b-chat and Airavata-7b.

##  Appendix E Prompt Details

Refer to Tables 11, 12 and 13 for prompts used in our experiments.

date | time | color_type
---|---|---
house_place | place_name | time_zone
artist_name | timeofday | meal_type
food_type | order_type | news_topic
music_genre | weather_descriptor | playlist_name
device_type | player_setting | song_name
media_type | joke_type | alarm_type
music_descriptor | business_name | business_type
general_frequency | change_amount | event_name
ingredient | person | coffee_type
drink_type | music_album | relation
radio_name | app_name | podcast_descriptor
audiobook_author | audiobook_name | cooking_type
list_name | game_name | podcast_name
movie_type | movie_name | transport_type
transport_name | transport_agency | transport_descriptor
definition_word | currency_name | personal_info
email_address | email_folder | game_type
change_amount |  |
person (PER) | organization (ORG) | location (LOC)
Table 9: List of all label types in Massive and Naamapadam, in that order. | Massive | Naamapadam
---|---|---
LoRA rank | 8 | 8
LoRA alpha | 16 | 16
Batch size (Training) | 32 | 16
Batch size (Inference) | 4 | 4
Gradient checkpointing | True | True
Gradient accumulation steps | 4 | 4
Max. gradient norm | 0.3 | 0.3
Epochs | 2, 3 | 3
Learning rate | 1e-3 | 1e-3
Optimizer | 32-bit AdamW (paged) | 32-bit Adam (paged)
Precision | bf16 | bf16
LR scheduler | cosine | cosine
Train batch size | 32 | 16
Warm-up ratio | 0.05 | 0.05
Max. sequence length (Training) | 512 | 1024
Stopping Criteria (Inference) | 512 | 768
Penalty alpha (Inference) | 0.6 | 0.6
top_k (Inference) | 4 | 4
Table 10: Complete set of hyperparameters for PEFT and ICL. For ICL, we use
the same inference-time hyperparameters as mentioned above. Reinsert the slot
annotations into the following Hindi sentence using the information in the
English sentence.
---
### Hindi: [Unannotated target]
### English: [Annotated source]
### Output:
Table 11: Example prompt format for PEFT with the cross-lingual annotation
transfer objective. Reinsert the slot annotations into the following Hindi
sentence.
---
### Hindi: [Unannotated target]
### Output:
Table 12: Prompt format for PEFT with the monolingual annotation objective.
<<SYS>> Add annotations for the corresponding tokens in Tamil sentences using
the annotation information given in the English sentence. The annotations are
marked in the format [annotation_type : token/value]
---
Input will be provided in the following format
### Tamil: Tamil sentence
### English: English sentence
Output should be printed after the string ‘‘### Output:"
The final output should be the Tamil sentence with annotations inserted
corresponding to the annotations of the English sentence. Do not add any extra
annotations to the Tamil sentence, which are not present in the English
sentence input.<</SYS>>
Add annotations for the given tokens <list of tokens present in annotated
source> in Tamil sentence using the annotation information given in the
English sentence
### Tamil: [Unannotated target]
### English: [Annotated source]
### Output: [Annotated target]
.
.
.
$\times$ n few-shot examples
Add annotations for the given tokens <list of tokens present in annotated
source> in Tamil sentence using the annotation information given in the
English sentence
### Tamil: <An unannotated Tamil sentence>
### English: <An annotated English sentence>
### Output:
Table 13: Example prompt format for few-shot ICL with the cross-lingual
annotation transfer objective.


# Robustness of Majorana edge states of short-length Kitaev chains coupled
with environment

Motohiko Ezawa  Department of Applied Physics, University of Tokyo, Hongo
7-3-1, 113-8656, Japan

###### Abstract

Recently, the two-site Kitaev model hosting Majorana edge states was
experimentally realized based on double quantum dots. In this context, we
construct two-band effective models describing Majorana edge states of a
finite-length Kitaev chain by using the isospectral matrix reduction method.
We analytically estimate the robustness of Majorana edge states as a function
of model parameters. We also study effects of coupling to an environment based
on non-Hermitian Hamiltonians derived from the Lindblad equation. We study
three types of dissipation, a local one, an adjacent one and a global one. It
is found that the Majorana zero-energy edge states acquire nonzero energy such
as $E\propto\pm\left(i\gamma\right)^{L}$ for the local dissipation, where
$\gamma$ is the magnitude of the dissipation and $L$ is the length of the
chain. On the other hand, the Majorana zero-energy edge states acquire nonzero
energy such as $E\propto\pm i\gamma$ irrespective of the length $L$ for the
global dissipation. Hence, the Majorana edge states are robust against the
local dissipation but not against the global one. Our results will be useful
for future studies on Majorana edge states based on quantum dots.

##  I Introduction

Majorana fermions are key for topological quantum computation[1, 2, 3, 4, 5,
6]. The Majorana edge states form a nonlocal qubit, which is robust for local
perturbation. Thus, the qubit based on Majorana fermions will resolve the
problem of decoherence in quantum computation. Majorana fermions are
materialized in topological superconductors[7, 8, 9, 10]. A simplest model of
a topological superconductor hosting Majorana fermions is the Kitaev
chain[11]. Despite the simpleness of the model, it is hard to materialize it
because it is hard to realize the $p$-wave superconducting order on the
lattice. Recently, the two-site Kitaev chain was experimentally realized in
double quantum dots[12]. In addition, the three-site Kitaev chain was also
experimentally realized in a nanowire device[13] It evokes studies on the
Minimal Kitaev chain based on double quantum dots[14, 15, 16, 17, 18, 19] and
short-length quantum dots[20, 21, 22]. However, the Majorana edge states in
the two-site Kitaev chain is not topologically protected. Indeed, we need a
precise tuning of the model parameters so that the Majorana edge states exist
exactly at the zero energy. On the other hand, the Majorana edge states are
robust if the length of the Kitaev chain is long enough. It is hard to
increase the number of quantum dots in the current technology. There are
studies on the condition for the existence of the Majorana edge states to have
exact zero energy[23, 24, 25, 26, 27]. However, there is no construction of
the two-band effective model based on Majorana edge states for a finite-length
chain. It is intriguing if we can estimate the robustness of the Majorana edge
states for a finite-length Kitaev chain. It is discussed[11] that the
robustness of the Majorana zero-energy states increases exponentially as a
function of the length of the Kitaev chain.

The platform of the Majorana fermions such as a quantum dot system has an
interaction with another system such as a substrate. In general, the coupling
to bath makes the system an open quantum system. It is commonly analyzed based
on the Lindblad equation[28]. The short-time dynamics is well described by a
non-Hermitian Hamiltonian derived from the Lindblad equation[28]. A Kitaev
chain with loss and gain is studied in the context of non-Hermitian
Hamiltonian[29, 30, 31, 32, 33, 34, 35, 36, 37].

In this paper, we construct effective two-band models for the Majorana edge
states by using the isospectral matrix reduction method[38, 39]. In addition,
we study the effect of the coupling between an environment based on the non-
Hermitian Hamiltonian formalism. We study three cases of dissipation. The
first one is a local dissipation, where there are hoppings within a single
site via an environment as in Fig.1(a). The second one is an adjacent
dissipation, where there are hopping between nearest-neighbor sites via an
environment as in Fig.1(b). The last one is a global dissipation, where the
every site is coherently coupled via an environment as in Fig.1(c). We find
that the Majorana edge states are robust against the local dissipation because
$E\propto\pm\left(i\gamma\right)^{L}$ but not against the global dissipation
because $E\propto\pm i\gamma$ irrespective of the length $L$, when the
amplitude of the dissipation $\gamma$ is small enough.

Figure 1: Illustration of (a) local dissipation, where a quantum dot and an
environment interact independently in each single quantum dot, (b) adjacent
dissipation, where two quantum dots and an environment interact in nearest-
neighbor sites, and (c) global dissipation, where every quantum dots and an
environment interact coherently.

Figure 2: (a1)$\sim$(a4) The energy $E/t$ as a function of $\Delta/t$. We have
set $\mu=0$. (b1)$\sim$(b4) The energy as a function of $\mu/t$. We have set
$\Delta=t$. Blue dotted curves are the energy derived from the effective two-
band model. (a1) and (b1) $L=2$, (a2) and (b2) $L=3$, (a3) and (b3) $L=4$,
(a4) and (b4) $L=5$ and (a5) and (b5) $L=6$. (c) Color palette indicating the
amplitude at the edge sites, where red color indicates the edge states and the
green color indicates the bulk states.

##  II Kitaev chain

The Kitaev p-wave superconductor model is defined on the 1D lattice as[11, 8]

$\displaystyle\hat{H}$
$\displaystyle=-\mu\sum_{x=1}^{L-1}c_{x}^{\dagger}c_{x}-t\sum_{x=1}^{L-1}\left(c_{x}^{\dagger}c_{x+1}+c_{x+1}^{\dagger}c_{x}\right)$
$\displaystyle-\sum_{x=1}^{L-1}\left(\Delta c_{x}c_{x+1}+\Delta
c_{x+1}^{\dagger}c_{x}^{\dagger}\right),$ (1)

where $\mu$ is the chemical potential, $t>0$ is the nearest-neighbor hopping
strength, $\Delta>0$ is the $p$-wave pairing amplitude of the superconductor,
and $L$ is the length of the chain.

###  II.1 Majorana representation

The system is topological for $\left|\mu\right|<2t$, where Majorana edge
states emerge at both the edges of the chain. We rewrite the fermion operator
in terms of two Majorana operators as

$c_{x}=\left(\frac{\gamma_{B,x}+i\gamma_{A,x}}{2}\right),\quad
c_{x}^{\dagger}=\frac{\gamma_{B,x}-i\gamma_{A,x}}{2},$ (2)

where these Majorana operators satisfy

$\gamma_{\alpha,x}=\gamma_{\alpha,x}^{\dagger},\quad\left\\{\gamma_{\alpha,x},\gamma_{\alpha^{\prime},x^{\prime}}\right\\}=2\delta_{\alpha\alpha^{\prime}}\delta_{xx^{\prime}}$
(3)

with $\alpha=A,B$. The Hamiltonian is rewritten in terms of Majorana operators
as

$\displaystyle\hat{H}$
$\displaystyle=-\frac{\mu}{2}\sum_{x=1}^{L-1}\left(1+i\gamma_{B,x}\gamma_{A,x}\right)$
$\displaystyle-i\sum_{x=1}^{L-1}\left[\left(\Delta+t\right)\gamma_{B,x}\gamma_{A,x+1}+\left(\Delta-t\right)\gamma_{A,x}\gamma_{B,x+1}\right].$
(4)

When $\mu=0$ and $t=\Delta\neq 0$, where the system is topological, the
Hamiltonian is simplified as

$\hat{H}=-2it\sum_{x=1}^{L-1}\gamma_{B,x}\gamma_{A,x+1}=4t\sum_{x=1}^{L-1}\left(d_{x}^{\dagger}d_{x}-\frac{1}{2}\right),$
(5)

where

$d_{x}=\frac{1}{2}\left(\gamma_{A,x+1}+i\gamma_{B,x}\right),\quad
d_{x}^{\dagger}=\frac{1}{2}\left(\gamma_{A,x+1}-i\gamma_{B,x}\right).$ (6)

The system is exactly solvable. The ground states are given by
$d_{x}|0\rangle_{d}=0$ with the energy $-2t$, whose excited states are
$|1\rangle_{d}=d_{x}^{\dagger}|0\rangle_{d}$ with the energy $2t$. They
constitute the bulk band. Apart from them, because this Hamiltonian does not
contain $\gamma_{A,1}$ and $\gamma_{B,N}$, there exist two Majorana states
perfectly localized at the two edge sites and exactly at the zero energy. A
non-local fermion operator is defined from them as

$f=\frac{1}{2}\left(\gamma_{A,1}+i\gamma_{B,N}\right).$ (7)

A qubit
($\left|0\right\rangle_{\text{qubit}},\left|1\right\rangle_{\text{qubit}}$) is
constructed such as $f\left|0\right\rangle_{\text{qubit}}=0$ and
$\left|1\right\rangle_{\text{qubit}}=f^{\dagger}\left|0\right\rangle_{\text{qubit}}$.
It is interesting to note that the Majorana chain even with $L=2$ is possible
to support a qubit.

We show the energy spectrum of (1) as a function of $\Delta/t$ for
$L=2,3,4,5,6$ in Fig.2(a1)$\sim$(a5) and that as a function of $\mu/t$ in
Fig.2(b1)$\sim$(b5). Especially, the energy has a linear dependence
$E=\left|\Delta-t\right|$ for the two-site Kitaev chain with $\mu=0$ as shown
in Fig.2(a1). On the other hand, the energy has a parabolic dependence
$E\propto\mu^{2}$ for the two-site Kitaev chain with $\Delta=t$ as shown in
Fig.2(b1).

The real part of the energy is exactly zero for odd length model as shown in
Fig.2(a2) and (a4). We will analytically verify these results by deriving an
effective two-band model in the following.

Figure 3: (a1)$\sim$(a4) real part and (b1)$\sim$(b4) imaginary part of energy
as a function of the local dissipation $\hbar\gamma/2t$. (a1) and (b1) $L=2$,
(a2) and (b2) $L=3$, (a3) and (b3) $L=4$, (a4) and (b4) $L=5$ and (a5) and
(b5) $L=6$. We have set $\Delta=t$ and $\mu=0$. See the color palette in
Fig.2.

Figure 4: (a1)$\sim$(a4) real part and (b1)$\sim$(b4) imaginary part of energy
as a function of the adjacent dissipation $\hbar\gamma/2t$. The horizontal
axis is $\eta_{\text{adj}}$. (a1) and (b1) $L=2$, (a2) and (b2) $L=3$, (a3)
and (b3) $L=4$, (a4) and (b4) $L=5$ and (a5) and (b5) $L=6$. We have set
$\Delta=t$ and $\mu=0$. See the color palette in Fig.2.

##  III Open quantum system

Effects of the coupling between the system and an environment are described by
the Lindblad equation[28] for the density matrix $\rho$ as

$\frac{d\rho}{dt}=-\frac{i}{\hbar}\left[\hat{H},\rho\right]+\left(\sum_{\alpha}L_{\alpha}\rho
L_{\alpha}^{\dagger}-\frac{1}{2}\left\\{L_{\alpha}^{\dagger}L_{\alpha},\rho\right\\}\right),$
(8)

where $L$ is the Lindblad operator describing the dissipation. This equation
is rewritten in the form of

$\frac{d\rho}{dt}=-\frac{i}{\hbar}\left(\hat{H}_{\text{eff}}\rho-\rho\hat{H}_{\text{eff}}^{\dagger}\right)+\sum_{\alpha}L_{\alpha}\rho
L_{\alpha}^{\dagger},$ (9)

where $H_{\text{eff}}$ is a non-Hermitian effective Hamiltonian defined by
$H_{\text{eff}}\equiv H+H_{\text{dissipation}}$ with the dissipation
Hamiltonian

$\hat{H}_{\text{dissipation}}\equiv-\frac{i\hbar}{2}L^{\dagger}L.$ (10)

It describes a short-time dynamics[28]. We consider three types of
dissipations as illustrated in Fig.1.

Figure 5: (a1)$\sim$(a4) real part and (b1)$\sim$(b4) imaginary part of energy
as a function of the global dissipation $\hbar\gamma/2t$. The horizontal axis
is $\eta_{\text{global}}$. (a1) and (b1) $L=2$, (a2) and (b2) $L=3$, (a3) and
(b3) $L=4$, (a4) and (b4) $L=5$ and (a5) and (b5) $L=6$. We have set
$\Delta=t$ and $\mu=0$. See the color palette in Fig.2.

###  III.1 Local dissipation

First, we study the local dissipation[37], where the Lindblad operators are
given by

$L_{x}^{-}=\sqrt{\gamma_{-}}c_{x},\quad
L_{x}^{+}=\sqrt{\gamma_{+}}c_{x}^{\dagger},$ (11)

where $\gamma_{\pm}$ represent the dissipation. They describes the effect that
the particle is coming in and out of a single site. The corresponding
dissipation Hamiltonian reads

$\hat{H}_{\text{dissipation}}=-\frac{i\hbar}{2}\sum_{x=1}^{L}\left[\gamma
c_{x}^{\dagger}c_{x}+\gamma_{+}\right],$ (12)

where $\gamma\equiv\gamma_{-}-\gamma_{+}$. By introducing a complex chemical
potential

$\tilde{\mu}=\mu+\frac{i\hbar}{2}\gamma,$ (13)

the effect of the local dissipation is fully taken into account. We show the
energy spectrum as a function of $\gamma$ in Fig.3. The real part of the
energy for odd length is exactly zero as shown in Fig.3(a2) and (a4). On the
other hand, the imaginary part of the energy for even length is exactly zero
if the dissipation is smaller than a certain critical value
$\left|\gamma\right|<\left|\gamma_{\text{critical}}\right|$. The flat region
of the zero-energy Majorana edge states is expanded for longer chains. We
derive these properties based on an effective model with the use of the
isospectral matrix reduction method in Sec.V.2.

###  III.2 Adjacent dissipation

Next, we study the adjacent dissipation[40], where the Lindblad operators are
given by

$L_{x}^{-}=\sqrt{\gamma_{-}/2}\left(c_{x}+c_{x+1}\right),\quad
L_{j}^{+}=\sqrt{\gamma_{+}/2}\left(c_{x}^{\dagger}+c_{x+1}^{\dagger}\right).$
(14)

They describe the effect that the particle can hop between nearest-neighbor
sites via an environment. The corresponding dissipation Hamiltonian reads

$\displaystyle\hat{H}_{\text{dissipation}}-\frac{\gamma_{+}}{2}$
$\displaystyle=-\frac{i\hbar}{2}\sum_{x=1}^{L-1}\frac{\gamma}{2}\left(c_{x}^{\dagger}c_{x}+c_{x}^{\dagger}c_{x+1}+c_{x+1}^{\dagger}c_{x}+c_{x+1}^{\dagger}c_{x+1}\right).$
(15)

The effect of the adjacent dissipation is taken into account by considering
the complex chemical potential and the complex hopping defined by

$\tilde{\mu}=\mu+\frac{i\hbar}{2}\gamma,\qquad\tilde{t}=t+\frac{i\hbar}{2}\gamma.$
(16)

We show the energy spectrum as a function of $\gamma$ in Fig.4. The imaginary
part of the energy acquire nonzero energy as a linear function of $\gamma$ for
$L=2$ as shown in Fig.4(b1). This is because the two Majorana edge states
couple directly via the adjacent dissipation term. It follows from Fig.4 that
deviations from the zero energy of the Majorana edge states becomes smaller
for longer chains. Comparing Figs.3 and 4, the Majorana edge states are found
to be more fragile than the local dissipation. We derive these properties
based on an effective model with the use of the isospectral matrix reduction
method in Sec.V.3.

###  III.3 Global dissipation

Finally, we study the global dissipation, where the Lindblad operators are
given by

$L_{x}^{-}=\sqrt{\gamma_{-}}\sum_{x=1}^{L}c_{x},\quad
L_{x}^{+}=\sqrt{\gamma_{+}}\sum_{x=1}^{L}c_{x}^{\dagger}.$ (17)

They describe the effect that all particles are equally coupled via an
environment. The corresponding dissipation Hamiltonian reads

$\displaystyle\hat{H}_{\text{dissipation}}$ $\displaystyle=$
$\displaystyle-\frac{i\hbar}{2}\gamma\sum_{x,x^{\prime}=1}^{L}\left(c_{x}^{\dagger}c_{x}+c_{x}^{\dagger}c_{x^{\prime}}+c_{x^{\prime}}^{\dagger}c_{x}+c_{x^{\prime}}^{\dagger}c_{x^{\prime}}\right)$
(18) $\displaystyle+\gamma_{+},$

which is highly nonlocal. We show the energy spectrum as a function of
$\gamma$ in Fig.5 for $L=2,3,4,5$. It is found from Fig.5(b1)~(b5) that the
imaginary part of the energy is linear as a function of $\gamma$ irrespective
of the length of the chain although the slope becomes smaller for a longer
chain. Hence, the Majorana edge states are not robust in the presence of the
global dissipation even for a long Kitaev chain. It is natural because the
global dissipation term couples the Majorana edge states directly. We derive
these properties based on an effective model with the use of the isospectral
matrix reduction method in Sec.V.4.

##  IV Minimal Kitaev chain

In the presence of the local dissipation or the adjacent dissipation, the
Hamiltonian of the two-site Kitaev chain reads

$\hat{H}=\left(\begin{array}[]{cccc}c_{1}^{\dagger}&c_{2}^{\dagger}&c_{1}&c_{2}\end{array}\right)H\left(\begin{array}[]{c}c_{1}\\\
c_{2}\\\ c_{1}^{\dagger}\\\ c_{2}^{\dagger}\end{array}\right),$ (19)

with

$H=\left(\begin{array}[]{cccc}-\tilde{\mu}&-\tilde{t}&0&\Delta\\\
-\tilde{t}&-\tilde{\mu}&-\Delta&0\\\ 0&-\Delta&\tilde{\mu}&\tilde{t}\\\
\Delta&0&\tilde{t}&\tilde{\mu}\end{array}\right)$ (20)

where we use the complex chemical potential $\tilde{\mu}$ in (13) for the
local dissipation and the complex chemical potential $\tilde{\mu}$ and the
complex hopping $\tilde{t}$ in (16) for the adjacent dissipation. The energy
spectrum is exactly obtained as

$E=\pm\tilde{t}\pm\sqrt{\Delta^{2}+\tilde{\mu}^{2}}.$ (21)

We note that the adjacent dissipation and the global dissipation is identical
for $L=2$.

##  V Isospectral matrix reduction method

It is impossible to exactly diagonalize the Hamiltonian matrix hosting
Majorana edge states except for the exactly solvable parameter $\mu=0$,
$\Delta=\pm t$ for the Kitaev chain for $L\geq 3$.

We derive an effective two-band model near the zero-energy describing the
Majorana edge states based on the isospectral matrix reduction method[38, 39].
We first diagonalize the Hamiltonian matrix for $\mu=0$, $\Delta=t$ as

$\displaystyle H^{\prime}$ $\displaystyle\equiv$ $\displaystyle UHU^{-1}$ (22)
$\displaystyle=$
$\displaystyle\text{diag.}\left\\{0,0,-2t,-2t,\cdots,-2t,2t,2t,\cdots,2t\right\\}.$

The first two zero-energy states correspond to the Majorana edge states. Then
we divide $H^{\prime}$ into the form of

$H^{\prime}=\left(\begin{array}[]{cc}H_{1}&V\\\
V^{\dagger}&H_{2}\end{array}\right),$ (23)

where $H_{1}$ is the $2\times 2$ matrix and $H_{2}$ is the
$(2L-2)\times(2L-2)$ matrix. The eigenequation reads

$\left(\begin{array}[]{cc}H_{1}&V\\\
V^{\dagger}&H_{2}\end{array}\right)\left(\begin{array}[]{c}\psi_{1}\\\
\psi_{2}\end{array}\right)=E\left(\begin{array}[]{c}\psi_{1}\\\
\psi_{2}\end{array}\right),$ (24)

from which we derive

$\psi_{2}=\left(E-H_{2}\right)^{-1}V^{\dagger}\psi_{1}.$ (25)

Then, we obtain a single nonlinear eigen equation for $\psi_{1}$ as

$\tilde{H}\left(E\right)\psi_{1}=E\psi_{1},$ (26)

where

$\tilde{H}\left(E\right)=H_{1}+V\left(E-H_{2}\right)^{-1}V^{\dagger}.$ (27)

Exact solutions may be obtained by solving the nonlinear equation. However, it
is practically impossible to solve it because it is an algebraic equation of
the order of $E^{2L}$. Instead, we seek a solution in the vicinity of the zero
energy, where the Hamiltonian is well approximated by

$H_{\text{eff}}\equiv H_{1}-VH_{2}^{-1}V^{\dagger}.$ (28)

The second term is written in the form of

$-VH_{2}^{-1}V^{\dagger}=F\sigma_{x}.$ (29)

We explicitly determine $H_{\text{eff}}$ in what follows.

###  V.1 Hermitian model

First, we derive an effective two-band model for the Hermitian system. We find

$H_{1}=\left(t-\Delta\right)\sigma_{x}$ (30)

for $L=2$ and $H_{1}=0$ for $L\geq 3$. On the other hand, we find

$F=\frac{1}{\left(\Delta+t\right)^{L-1}}\sum_{m=0}^{\left\lfloor
L/2\right\rfloor}\left(\begin{array}[]{c}L-m\\\
m\end{array}\right)\mu^{L-2m}\left(\Delta^{2}-t^{2}\right)^{2m}$ (31)

for $L\geq 2$. The energy is given by

$E=\pm F.$ (32)

If $\mu=0$, we have $F=0$ for odd $L$, which well describes the energy near
the zero energy as shown in Fig.2(a2) and (a4). On the other hand, we find

$F=-\frac{\left(\Delta-t\right)^{L/2}}{\left(\Delta+t\right)^{L/2+1}}\propto\left(\Delta-t\right)^{L/2}$
(33)

for even $L$, which well describes the energy near the zero energy as shown in
Fig.2(a1), (a3) and (a5). It is almost zero for small $\left|\Delta-t\right|$
and large $L$, where the Majorana edge states are robust.

If $\Delta=t$, we have

$F=-\frac{\mu^{L}}{\left(2t\right)^{L-1}}\propto\mu^{L}.$ (34)

It well fits the energy near the zero energy as shown in Fig.2(b1)$\sim$(b5).
It is almost zero for small $\mu$ and large $L$, where the Majorana edge
states are robust.

###  V.2 Local dissipation

Next, we derive effective two-band models for the local dissipation with
$\Delta=t$ and $\mu=0$. We find

$H_{\text{eff}}=-\frac{\left(i\gamma\right)^{L}}{\left(2t\right)^{L-1}}$ (35)

for $L\geq 2$. This formula well explains the fact that the real [imaginary]
part of the energy is zero for even [odd] $L$ as shown in Fig.3(a2) and (a4),
[(b1) and (b3)]. Hence the Majorana edge states becomes robust for a long
chain.

###  V.3 Adjacent dissipation

We derive effective two-band models for the adjacent dissipation with $\mu=0$.
We find

$H_{1}=\left(t+i\gamma-\Delta\right)\sigma_{x}$ (36)

for $L=2$ and $H_{1}=0$ for $L\geq 3$. On the othe hand,

$F=-\sum_{m=0}^{\left\lfloor L/2\right\rfloor}\left(\begin{array}[]{c}L-m\\\
m\end{array}\right)\frac{\left(i\gamma\right)^{L-2m}}{\left(\Delta+t\right)^{L-1}}\left(\Delta^{2}-\left(t+i\gamma\right)^{2}\right)^{m}$
(37)

for $L\geq 2$. For small $\gamma$ ($\left|\gamma/t\right|\ll 1$), they are
explicitly given by

$\begin{array}[]{ll}F=-i\gamma-\gamma^{2}/2t+\cdots&\text{for }L=2,\\\
F=-3i\gamma^{3}/4t^{2}+\gamma^{2}/2t+\cdots&\text{for }L=3,\\\
F=-i\gamma^{3}/t^{2}-\gamma^{2}/2t+\cdots&\text{for }L=4,\\\
F=-i\gamma^{3}/4t^{2}-\gamma^{4}/4t^{3}+\cdots&\text{for }L=5,\\\
F=i\gamma^{3}/4t^{2}+\gamma^{4}/t^{3}+\cdots&\text{for }L=6,\end{array}$ (38)

In general,

$F=ia\gamma^{A}+b\gamma^{B}+\cdots,$ (39)

with certain real numbers $a$ and $b$, where

$A=2\left\lfloor\frac{L+1}{4}\right\rfloor+1,\qquad
B=2\left\lfloor\frac{L-1}{4}\right\rfloor+2.$ (40)

The robustness of the Majorana edge modes is enhanced when the chain length
becomes longer. However, it is more fragile than the local dissipation.

###  V.4 Global dissipation

Finally, we derive effective two-band models for the global dissipation with
$\Delta=t$ and $\mu=0$. We find

$H_{1}=i\gamma\sigma_{x},\quad
F=\frac{\left(L-1\right)\gamma^{2}}{2t+i\left(L-1\right)\gamma}\sigma_{x}$
(41)

for $L\geq 2$. For small $\gamma$ ($\left|\gamma/t\right|\ll 1$), the
Hamiltonian is approximated as

$H_{\text{eff}}\simeq\left(i\gamma+\frac{\left(L-1\right)\gamma^{2}}{2t}\right)\sigma_{x}.$
(42)

The real part of the energy is parabolic as a function of $\gamma$
irrespective of the length $L$ as shown in Fig.5(a1)$\sim$(a5), while the
imaginary part of the energy is linear as a function of $\gamma$ irrespective
of the length $L$ as shown in Fig.5(b1)$\sim$(b5). Hence, the robustness of
the Majorana edge states is not enhanced even if we use a chain with long
length $L$ in the presence of the global dissipation.

Actually, the energy spectrum in the vicinity of the zero energy is exactly
obtained without using the isospectral matrix reduction method at the point
$\mu=0$ and $\Delta=t$, whose results reads

$F=i\frac{L}{2}\gamma+2t+i\sqrt{\left(\frac{L}{2}\gamma\right)^{2}-2i\left(L-2\right)t\gamma-4t^{2}}.$
(43)

It gives the same result as in (42) for small $\gamma$
($\left|\gamma/t\right|\ll 1$).

##  VI Discussion

We have constructed two-band effective models describing the Majorana edge
states in the presence of three types of dissipation. We have found the even-
odd effect on the stability of the Majorana edge states. We have also found
that the robustness of the Majorana edge states is affected by the type of
dissipation, where the global dissipation is detrimental for the stability of
the Majorana edge states. The local dissipation may be realized when the
system couples with the substrate where the real coordinate is a good quantum
number. On the other hand, the global dissipation may be realized when the
system couples with the substrate where the momentum coordinate is a good
quantum number. Our results will be useful for experimental realization of the
Kitaev chain based on quantum dots.

This work is supported by CREST, JST (Grants No. JPMJCR20T2) and Grants-in-Aid
for Scientific Research from MEXT KAKENHI (Grant No. 23H00171).

## References

  * [1] S. B. Bravyi and A. Yu. Kitaev, Fermionic Quantum Computation, Annals of Physics 298, 210 (2002)
  * [2] D. A. Ivanov, Non-Abelian statistics of half-quantum vortices in p-wave superconductors, Phys. Rev. Lett. 86, 268 (2001).
  * [3] A. Kitaev, Fault-tolerant quantum computation by anyons, Ann. Phys. 303, 2 (2003).
  * [4] S. Das Sarma, M. Freedman, and C. Nayak, Topologically protected qubits from a possible non-Abelian fractional quantum Hall state, Phys. Rev. Lett. 94, 166802 (2005).
  * [5] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Non-Abelian anyons and topological quantum computation, Rev. Mod. Phys. 80, 1083 (2008).
  * [6] Motohiko Ezawa, Systematic construction of topological-nontopological hybrid universal quantum gates based on many-body Majorana fermion interactions arXiv:2304.06260
  * [7] X.-L. Qi, S.-C. Zhang, Topological insulators and superconductors, Rev. Mod. Phys. 83, 1057 (2011).
  * [8] J. Alicea, New directions in the pursuit of Majorana fermions in solid state systems, Rep. Prog. Phys. 75, 076501 (2012)
  * [9] M. Sato and Y. Ando, Topological superconductors: a review, Rep. Prog. Phys. 80, 076501 (2017).
  * [10] J. Alicea, Y. Oreg, G. Refael, F. von Oppen and M.P.A. Fisher, Non-Abelian statistics and topological quantum information processing in 1D wire networks, Nat. Phys. 7, 412 (2011).
  * [11] A. Yu Kitaev, Unpaired Majorana fermions in quantum wiresm Phys.-Usp. 44 131 (2001)
  * [12] Tom Dvir, Guanzhong Wang, Nick van Loo, Chun-Xiao Liu, Grzegorz P. Mazur, Alberto Bordin, Sebastiaan L. D. ten Haaf, Ji-Yin Wang, David van Driel, Francesco Zatelli, Xiang Li, Filip K. Malinowski, Sasa Gazibegovic, Ghada Badawy, Erik P. A. M. Bakkers, Michael Wimmer and Leo P. Kouwenhoven, Realization of a minimal Kitaev chain in coupled quantum dots, Nature 614, 445 (2023)
  * [13] Alberto Bordin, Xiang Li, David van Driel, Jan Cornelis Wolff, Qingzhen Wang, Crossed Andreev reflection and elastic co-tunneling in a three-site Kitaev chain nanowire device, arXiv:2306.07696
  * [14] Athanasios Tsintzis, Ruben Seoane Souto and Martin Leijnse, Creating and detecting poor man’s Majorana bound states in interacting quantum dots Phys. Rev. B 106, L201404 (2022)
  * [15] Chun-Xiao Liu, Haining Pan, F. Setiawan, Michael Wimmer and Jay D. Sau, Fusion protocol for Majorana modes in coupled quantum dotsm Phys. Rev. B 108, 085437 (2023)
  * [16] Rouven Koch, David van Driel, Alberto Bordin, Jose L. Lado and Eliska Greplova, Adversarial Hamiltonian learning of quantum dots in a minimal Kitaev chain, arXiv:2304.10852
  * [17] Athanasios Tsintzis, Ruben Seoane Souto, Karsten Flensberg, Jeroen Danon and Martin Leijnse, Roadmap towards Majorana qubits and nonabelian physics in quantum dot-based minimal Kitaev chains, arXiv:2306.16289
  * [18] D. Michel Pino, Ruben Seoane Souto and Ramon Aguado, Minimal Kitaev–transmon qubit based on double quantum dots, arXiv:2309.12313
  * [19] William Samuelson, Viktor Svensson and Martin Leijnse, A minimal quantum dot-based Kitaev chain with only local superconducting proximity effect, arXiv:2310.03536
  * [20] Mahan Mohseni, Hassan Allami, Daniel Miravet, David J. Gayowsky, Marek Korkusinski, Pawel Hawrylak, Majorana excitons in a Kitaev chain of semiconductor quantum dots in a nanowire, arXiv:2307.00100
  * [21] Ruben Seoane Souto, Athanasios Tsintzis, Martin Leijnse and Jeroen Danon , Probing Majorana localization in minimal Kitaev chains through a quantum dot, arXiv:2308.14751
  * [22] Sebastian Miles, David van Driel, Michael Wimmer and Chun-Xiao Liu, Kitaev chain in an alternating quantum dot-Andreev bound state array, arXiv:2309.15777
  * [23] Hsien-chung Kao, Phys. Rev. B 90, 245435, Chiral zero modes in superconducting nanowires with Dresselhaus spin-orbit coupling, Phys. Rev. B 90, 245435 (2014)
  * [24] Suraj Hegde, Vasudha Shivamoggi, Smitha Vishveshwara and Diptiman Sen, Quench dynamics and parity blocking in Majorana wires New J. Phys. 17 053036 (2015)
  * [25] A. A. Zvyagin, Majorana bound states in the finite-length chain Low Temp. Phys. 41 625 (2015)
  * [26] Chuanchang Zeng, Christopher Moore, Apparao M. Rao, Tudor D. Stanescu, and Sumanta Tewari, Analytical solution of the finite-length Kitaev chain coupled to a quantum dot Phys. Rev. B 99, 094523 (2019)
  * [27] Nico Leumer, Magdalena Marganska, Bhaskaran Muralidharan and Milena Grifoni, Exact eigenvectors and eigenvalues of the finite Kitaev chain and its topological properties J. Phys.: Condens. Matter 32 445502 (2020)
  * [28] G. Lindblad, On the generators of quantum dynamical semigroups, Commun. Math. Phys. 48, 119 (1976)
  * [29] C. Yuce, Majorana edge modes with gain and loss, Phys. Rev. A 93, 062130 (2016)
  * [30] Qi-Bo Zeng, Baogang Zhu, Shu Chen, L. You, and Rong Lu, Phys. Rev. A 94, 022119 (2016)
  * [31] Kohei Kawabata, Yuto Ashida, Hosho Katsura, Masahito Ueda, Parity-time-symmetric topological superconductor, Phys. Rev. B 98, 085116 (2018)
  * [32] Kohei Kawabata, Sho Higashikawa, Zongping Gong, Yuto Ashida and Masahito Ueda, Topological unification of time-reversal and particle-hole symmetries in non-Hermitian physics Nat. Com. 10, 297 (2019)
  * [33] Kohei Kawabata, Ken Shiozaki, Masahito Ueda and Masatoshi Sato, Symmetry and Topology in Non-Hermitian Physics, Phys. Rev. X 9, 041015 (2019)
  * [34] Naoyuki Shibata and Hosho Katsura, Dissipative spin chain as a non-Hermitian Kitaev ladder, Phys. Rev. B B 99, 174303 (2019)
  * [35] Motohiko Ezawa, Braiding of Majorana-like corner states in electric circuits and its non-Hermitian generalization, Phys. Rev. B 100, 045407 (2019)
  * [36] Xiao-Ming Zhao, Cui-Xian Guo, Su-Peng Kou, Lin Zhuang, and Wu-Ming Liu Phys. Rev. B 104, 205131, Defective Majorana zero modes in a non-Hermitian Kitaev chain (2021)
  * [37] Simon Lieu, Non-Hermitian Majorana modes protect degenerate steady states, Phys. Rev. B 100, 085110 (2019)
  * [38] Lumen Eek, Anouar Moustaj, Malte Rontgen, Vincent Pagneux, Vassos Achilleos, and Cristiane Morais Smith, Emergent non-Hermitian models, arXiv:2310.11988
  * [39] Malte Rontgen, Xuelong Chen, Wenlong Gao, Maxim Pyzh, Peter Schmelcher, Vincent Pagneux, Vassos Achilleos and Antonin Coutant, Latent Su–Schrieffer–Heeger models, arXiv:2310.07619
  * [40] Sebastian Diehl, Enrique Rico, Mikhail A. Baranov and Peter Zoller, Topology by dissipation in atomic quantum wires, Nature Physics 7, 971 (2011)