# Enhancing Multilingual Capabilities of Large Language Models through Self-
Distillation from Resource-Rich Languages

Yuanchi Zhang1, Yile Wang2, Zijun Liu1, Shuo Wang∗1, Xiaolong Wang1,
Peng Li 2, Maosong Sun1, Yang Liu1,2
1 Department of Computer Science and Technology, Tsinghua University, Beijing,
China
2 Institute for AI Industry Research (AIR), Tsinghua University, Beijing,
China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding
authors.

###### Abstract

While large language models (LLMs) have been pre-trained on multilingual
corpora, their performance still lags behind in most languages compared to a
few resource-rich languages. One common approach to mitigate this issue is to
translate training data from resource-rich languages into other languages and
then continue training. However, using the data obtained solely relying on
translation while ignoring the original capabilities of LLMs across languages
is not always effective, which we show will limit the performance of cross-
lingual knowledge transfer. In this work, we propose SDRRL, a method based on
Self-Distillation from Resource-Rich Languages that effectively improve
multilingual performance by leveraging the internal capabilities of LLMs on
resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM)
and source languages (English and French) across various comprehension and
generation tasks, experimental results demonstrate that SDRRL can
significantly enhance multilingual capabilities while minimizing the impact on
original performance in resource-rich languages.111The source code is
available at https://github.com/THUNLP-MT/SDRRL.

##  1 Introduction

Contemporary large language models (LLMs; OpenAI, 2022, 2023; Touvron et al.,
2023a, b; Jiang et al., 2023; Google et al., 2023) are predominantly trained
on multilingual corpora. However, the language distribution in the data is
highly imbalanced. For instance, LLMs like LLaMA-2 Touvron et al. (2023b),
with English as the primary language, have also been trained on Japanese text,
yet the quantity of English tokens used during pre-training exceeds that of
Japanese by a factor of 897.

Figure 1: Comparison between vanilla supervised fine-tuning (SFT), translate-
then-SFT, and our proposed method. Besides using the translated question-
answer pairs in the target language (e.g., Japanese), SDRRL further leverages
the generated answer $A^{\star}_{\rm EN}$ by LLMs in the resource-rich
language (e.g., English) and collects self-distillated data (in green box) to
help enhance its multilingual capabilities.

The imbalanced data distribution above has led to significant limitations in
the capabilities of LLMs across most languages. To enhance the multilingual
capabilities, a common approach follows the translating and then supervised
fine-tuning (SFT; Ouyang et al., 2022) paradigm, as shown in Figure 1(b).
Specifically, training data is translated into the target language using
either the model itself or an external machine translation (MT) system before
continuing the training process, thereby offering more data in the target
language and improving multilingual capabilities.

However, the translate-then-SFT method encounters several challenges: First,
the multilingual enhancement gained from translated ‘‘question-answer’’ pairs
is limited and may sometimes even degrade the capabilities in the original
primary language Zhu et al. (2024). Second, constrained by the accuracy of
machine translation (especially for the low-resource languages), the
translated texts used for training can be highly noisy, containing numerous
awkward sentences and incorrect content, adversely affecting the quality of
the generated text and the multilingual abilities of the LLMs. Therefore, we
explore a new question along this trajectory: Besides translating the training
pairs, can we enhance the abilities in other languages by leveraging the
original relatively strong capabilities of LLMs in resource-rich language?

In this paper, we introduce SDRRL, a method that uses Self-Distillation from
Resource-Rich Languages) to achieve the goal mentioned above. Specifically, as
illustrated in Figure 1(c), SDRRL comprises two parts: (1) Self-Distillation:
Instead of the ground-truth answer, responses from LLMs in resource-rich
languages are collected to construct a transfer set. These are then translated
into other languages using machine translation systems and code-switching
tools, forming ‘‘question-answer’’ pairs that are semantically identical but
linguistically varied, and conducting sentence-level knowledge self-
distillation within the same batch. (2) Incorporating External Parallel
Corpus: We further involve a small amount of machine translation data in the
distillation, aiming to align the linguistic representation spaces better and
mitigate the negative impact of the noise in machine translation systems on
the generative capabilities of LLMs.

Our experiments, based on LLaMA-2-7B Touvron et al. (2023b) and SeaLLM-7B
Nguyen et al. (2023) with English as the resource-rich language, demonstrate
that even with a smaller set of English instruction data as the transfer set,
SDRRL can effectively distill English capabilities into 14 other languages,
showing effectiveness in both multilingual comprehension and generation tasks.
Further analysis indicates that SDRRL helps preserve the original capabilities
in high-resource languages and improves the quality of generated responses.

##  2 Related Work

Multilingual Language Models. Using multilingual data during the pre-training
is a common approach to enhance the multilingual capabilities of LLMs (Li et
al., 2022; Lample and Conneau, 2019; Workshop et al., 2022; Lin et al., 2022;
Xue et al., 2021). Despite being pre-trained and fine-tuned targeting a few
resource-rich languages, recent instruction-following LLMs (Touvron et al.,
2023b; Jiang et al., 2023; Wang et al., 2023a) have been found to still
possess significant multilingual understanding and generation capabilities
Bandarkar et al. (2023); Niklaus et al. (2023). However, limited by the
imbalanced training data distribution (Yang et al., 2023), the multilingual
capabilities of these popular LLMs lag behind those of languages with abundant
resources (Pahune and Chandrasekharan, 2023).

Cross-Lingual Transfer. To enhance the capabilities in languages with scarce
resources, one line of work is cross-lingual transfer, where skills learned
from one source language can be readily transferred to other languages
(Etxaniz et al., 2023; Huang et al., 2023; Ranaldi and Zanzotto, 2023). This
has been approached by designing prompts that leverage LLMs to self-translate
questions into resource-rich languages (Qin et al., 2023), or by utilizing
external machine translation systems for assistance (Zhao et al., 2024).
Efforts have also been made to distill synthetic data from high-resource
languages to low-resource ones (Chai et al., 2024). Shaham et al. (2024) and
Kew et al. (2023) leverage similarities between languages to stimulate
capabilities in others. Compared to their work, we focus on proficiency in the
resource-rich language and leverage it to improve performance in other
languages.

Cross-Lingual Alignment. Another line of work is cross-lingual alignment
(Schuster et al., 2019a). Given the scarcity of multilingual data, the
construction of alignment data or loss functions of varying granularity can
align mid- and low-resource languages with those that are resource-rich. This
includes the construction of pre-training tasks using multilingual aligned
lexicons (Chi et al., 2021), alignment of word embeddings (Wen-Yi and Mimno,
2023; Schuster et al., 2019b), using aligned data on one side of a problem to
improve mathematical reasoning processes (Zhu et al., 2024), and encouraging
language switching in chain-of-thought (CoT; Wei et al., 2022) reasoning (Chai
et al., 2024). Mao and Yu (2024a) have leveraged the LLM’s own capabilities to
generate aligned data, while others have constructed it with the aid of
external systems (Ranaldi and Pucci, 2023; Chen et al., 2023). Deriving and
constructing multilingual supervision signals from existing datasets overlooks
the fact that the model’s own responses in high-resource languages can also
serve as effective supervision signals. We show in our experiments that self-
distillation not only improves the LLM’s multilingual performance but also
helps maintain the performance in the original resource-rich languages.

Knowledge Distillation. Knowledge distillation (Hinton et al., 2015) is a
widely used method for transferring knowledge (Gou et al., 2021). In the text
generation domain, sequence-level knowledge distillation (Kim and Rush, 2016)
has been used as a means of data augmentation in areas such as machine
translation (Gordon and Duh, 2019). In particular, self-distillation (Zhang et
al., 2019, 2022b; Pham et al., 2022) is often utilized to distill knowledge
from one component of a model to another (Zhang et al., 2022a), or from one
stage of a model to another (Yang et al., 2019). In this work, we apply
distilling knowledge between the different linguistic representation spaces
within the same LLM to enhance multilingual capabilities.

##  3 Method

In this section, we first revisit the supervised fine-tuning (SFT) and
translate-then-SFT paradigm, subsequently dividing the discussions into two
parts of our proposed SDRRL. In the first part, we construct a transfer set
using responses in the resource-rich language from LLMs through sentence-level
self-distillation. In the second part, we employ parallel translation-based
instruction data to further improve multilingual generation capabilities.

###  3.1 SFT and Translate-then-SFT Paradigm

We consider the given instruction dataset comprised of $N$ entries
$\mathcal{D}=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i=1}^{N}$, where
$\mathbf{x}_{i}$ symbolizes the input sentence (question) for the $i$-th data
point, and $\mathbf{y}_{i}$ signifies the corresponding ground-truth response
(answer).

Supervised Fine-Tuning. For a LLM $\mathcal{M}_{\theta}$ parameterized by a
set of parameters $\theta$, which produces a response denoted as
$\mathbf{\hat{y}}=\mathcal{M}_{\theta}(\mathbf{x})$ for the given input
question $\mathbf{x}$, the objective of SFT is to align the output sentence
$\mathbf{\hat{y}}$ as closely as possible with the ground-truth response
$\mathbf{y}$. Specifically, the cross-entropy (CE) loss is employed to assess
the discrepancy between the model output $\mathbf{\hat{y}}$ and the ground-
truth output $\mathbf{y}$ for a single sample $(\mathbf{x},\mathbf{y})$,
defined as:

$\ell_{\rm
CE}(\mathbf{y},\mathbf{\hat{y}})=-\sum_{j=1}^{|\mathcal{V}|}y_{j}\log(\hat{y}_{j})$
(1)

where $y_{j}$ is the one-hot encoding of the ground truth output $\mathbf{y}$
at position $j$, $\hat{y}_{j}$ is the probability of the model output
$\mathbf{\hat{y}}$ at position $j$, and $|\mathcal{V}|$ is the size of the
vocabulary in the LLM.

For the entire dataset $\mathcal{D}$, the total loss is calculated as the
average of all sample losses:

$\mathcal{L}_{\rm SFT}=\frac{1}{N}\sum_{i=1}^{N}\ell_{\rm
CE}(\mathbf{y}_{i},\mathcal{M}_{\theta}(\mathbf{x}_{i}))$ (2)

Translate-then-SFT. For the translation-then-SFT paradigm, we define the
machine translation system as a function $\mathcal{T}$, which accepts text in
one language as the source language (Src) and outputs equivalent text in the
target language (Tgt). Using the machine translation system $\mathcal{T}$,
each pair $(\mathbf{x}_{i},\mathbf{y}_{i})$ is translated into the target
language, resulting in the translated dataset $\mathcal{D}^{\rm
MT}=\\{(\mathbf{x}^{\rm MT}_{i},\mathbf{y}^{\rm
MT}_{i})\\}_{i=1}^{N}=\\{\mathcal{T}(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i=1}^{N}$.

Similar to Eq. 1, the LLM $\mathcal{M}_{\theta}$ is then trained on the
translated dataset $\mathcal{D}^{\prime}$, where the loss for a single sample
$(\mathbf{x}^{\rm MT},\mathbf{y}^{\rm MT})$ is computed as:

$\ell_{\rm CE}(\mathbf{y}^{\rm MT},\mathbf{\hat{y}}^{\rm
MT})=-\sum_{j=1}^{|\mathcal{V}|}y^{\rm MT}_{j}\log(\hat{y}^{\rm MT}_{j})$ (3)

where $\mathbf{\hat{y}}^{\rm MT}=\mathcal{M}_{\theta}(\mathbf{x^{\rm MT}})$ is
the response of models to the question $\mathbf{x^{\rm MT}}$ in target
language.

###  3.2 Self-Distillation from Resource-Rich Languages (SDRRL)

LLMs exhibit superior comprehension and generation capabilities in resource-
rich languages, which we suppose can be a learning reference for other
languages to enhance the multilingual capabilities of LLMs. To achieve this,
we propose sentence-level knowledge distillation from resource-rich language
responses. The core motivation is that the responses of LLMs in the resource-
rich language serve as samples from the resource-rich language representation
space. By adding these responses and their translations to the transfer set,
the gap for cross-linguistic learning is reduced, facilitating the improvement
of multilingual capabilities.

####  3.2.1 Transfer Set Construction

We construct a transfer set for sentence-level distillation by collecting LLM
responses in the resource-rich language. For the original instruction dataset
$\mathcal{D}=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i=1}^{N}$, LLM
$\mathcal{M}_{\theta}$ generates responses for each question $\mathbf{x}_{i}$,
yielding $\mathbf{\hat{y}}_{i}=\mathcal{M}_{\theta}(\mathbf{x}_{i})$, then we
get the generated dataset
$\mathcal{G}=\\{(\mathbf{x}_{i},\mathbf{\hat{y}}_{i})\\}_{i=1}^{N}=\\{(\mathbf{x}_{i},\mathcal{M}_{\theta}(\mathbf{x}_{i}))\\}_{i=1}^{N}$.
The synthesized transfer set $\mathcal{D}_{\text{synth}}$ is obtained by
equally probable random sampling from both datasets $\mathcal{D}$ and
$\mathcal{G}$:

$\mathcal{D}_{\text{synth}}=\text{Sample}(\mathcal{D})\cup\text{Sample}(\mathcal{G})$
(4)

####  3.2.2 Transfer Set Translation

The above constructed transfer set $\mathcal{D}_{\text{synth}}$ contains
question $\mathbf{x}_{i}$, ground-truth answer $\mathbf{y}_{i}$, and response
$\mathbf{\hat{y}}_{i}$ by LLM $\mathcal{M}_{\theta}$. We consider translating
them into the target language using the machine translation system
$\mathcal{T}$, resulting in $\mathbf{x}^{\rm
MT}_{i}=\mathcal{T}(\mathbf{x}_{i}),\mathbf{y}^{\rm
MT}_{i}=\mathcal{T}(\mathbf{y}_{i})$, and $\mathbf{\hat{y}}^{\rm
MT}_{i}=\mathcal{T}(\mathbf{\hat{y}}_{i})$. Moreover, we use WMT22-cometkiwi-
da Rei et al. (2022b) as a reference-free metric to assess the translation
quality where the translation quality with scores below a threshold $\tau=0.8$
is rejected.

In particular, four sub-datasets are generated, each containing different
language combinations of questions and responses:

  * •

$\mathcal{D}_{\rm LL}$: Both the questions and responses remain in the
resource-rich language, i.e., $\\{\mathbf{x}_{i},\mathbf{y}_{i}\\}$ or
$\\{\mathbf{x}_{i},\hat{\mathbf{y}}_{i}\\}$.

  * •

$\mathcal{D}_{\rm TL}$: The questions are translated into the target language,
while responses remain in the resource-rich language, i.e.,
$\\{\mathcal{T}(\mathbf{x}_{i}),\mathbf{y}_{i}\\}$ or
$\\{\mathcal{T}({\mathbf{x}}_{i}),\hat{\mathbf{y}}_{i}\\}$.

  * •

$\mathcal{D}_{\rm LT}$: The questions remain in the resource-rich language,
while responses are translated into the target language, i.e.,
$\\{\mathbf{x}_{i},\mathcal{T}(\mathbf{y}_{i})\\}$ or
$\\{\mathbf{x}_{i},\mathcal{T}(\hat{\mathbf{y}}_{i})\\}$.

  * •

$\mathcal{D}_{\rm TT}$: Both the questions and responses are translated into
the target language, i.e.,
$\\{\mathcal{T}(\mathbf{x}_{i}),\mathcal{T}(\mathbf{y}_{i})\\}$ or
$\\{\mathcal{T}(\mathbf{x}_{i}),\mathcal{T}(\hat{\mathbf{y}}_{i})\\}$.

This approach, by providing semantically identical but linguistically diverse
samples, aids in the implicit alignment of language representation spaces,
enhancing unified multilingual performance. Furthermore, $\mathcal{D}_{\rm
TL}$ and $\mathcal{D}_{\rm LT}$ enhance LLM’s cross-linguistic generative
capabilities, helping mitigate off-target issues in target language
generation.

####  3.2.3 Applying Code-Switching

Through the aforementioned machine translation process, we achieve alignment
in sentence level (i.e., the sentence of question-answer pairs). Additionally,
token-level alignment is introduced using a code-switching tool, applied only
to the question components $\mathbf{x}_{i}$ of $\mathcal{D}_{\rm LL}$,
$\mathcal{D}_{\rm TL}$, $\mathcal{D}_{\rm LT}$, and $\mathcal{D}_{\rm TT}$ to
increase language diversity without compromising generative capabilities.

Specifically, given $\mathbf{x}_{i}$ composed of a sequence of tokens
$\mathbf{x}_{i}={x}_{i,1},{x}_{i,2},\ldots,{x}_{i,n}$, where ${x}_{i,k}$
denotes the $k$-th token in question $\mathbf{x}_{i}$ (similarly for
$\mathbf{\hat{x}}^{\rm MT}_{i}$), the code-switched version ${x}_{i,k}$ for
each token is generated by applying the rule:

${x}_{i,k}=\begin{cases}\text{Dict}({x}_{i,k})&\text{with probability}\ p;\\\
{x}_{i,k}&\text{with probability}\ 1-p,\end{cases}$ (5)

where each token ${x}_{i,k}$ in $\mathbf{x}_{i}$ is replaced by its
corresponding token in the bilingual dictionary for code-switching
$\text{Dict}({x}_{i,k})$ with a $p=0.15$ probability if ${x}_{i,k}$ is found
in the bilingual dictionary. Responses, either in the source language
$\mathbf{y}_{i}$ (similarly for $\mathbf{\hat{y}}_{i}$) or the target language
$\mathbf{y}_{i}^{\rm MT}$ (similarly for $\mathbf{\hat{y}}_{i}^{\rm MT}$),
remain unchanged.

####  3.2.4 Incorporating External Parallel Corpus

The Template for Constructing $\mathcal{D}_{\text{mt}}$ and
$\mathcal{D}_{\text{comp}}$
---
# Construct Data for Machine Translation
Question: Translate the following sentence from English to Indonesian.
The quick brown fox jumps over the lazy dog.
Answer: Sang rubah cokelat cepat melompati anjing malas.
# Construct Data for Sentence Completion
Question: Complete the following sentence in Indonesian according to its
context.
Sang rubah cokelat cepat
Answer: Sang rubah cokelat cepat melompati anjing malas.

Table 1: The template for constructing $\mathcal{D}_{\text{mt}}$ and
$\mathcal{D}_{\text{comp}}$ with Indonesian-English as an example.
$\mathcal{D}_{\text{mt}}$ includes bidirectional translations.
$\mathcal{D}_{\text{comp}}$ contains only the target language sentences, which
are split at random positions.

The target language sequences $\mathbf{\hat{y}}_{i}$ synthesized by the
external machine translation system $\mathcal{T}$ may contain low-quality
translations, thereby introducing a significant amount of noise into the
knowledge distillation transfer dataset $\mathcal{D}_{\text{synth}}$.To
mitigate the impact of noise on the multilingual generative capabilities of
LLMs, we leverage a tiny external parallel corpus
$\mathcal{P}=\\{(\mathbf{s}_{i},\mathbf{t}_{i})^{L}_{i=1}\\}$ between the
resource-rich language Src and the target language Tgt. Based on the templates
in Table 1, we can construct two parts of instruction data: machine
translation task instructions ($\mathcal{D}_{\text{mt}}$) and sentence
completion task instructions ($\mathcal{D}_{\text{comp}}$). By incorporating
these two parts, the transfer set includes non-synthetic natural target
language texts, which helps improve the generative quality of LLMs in the
target language.

####  3.2.5 Training Objective

The final training dataset $\mathcal{D}_{\text{train}}$ includes
$\mathcal{D}_{\rm LL}$, $\mathcal{D}_{\rm TL}$, $\mathcal{D}_{\rm LT}$,
$\mathcal{D}_{\rm TT}$, $\mathcal{D}_{\text{mt}}$, and
$\mathcal{D}_{\text{comp}}$. The total loss function is defined as:

$\mathcal{L}_{\rm
SDRRL}=\sum_{\begin{subarray}{c}d\in\mathcal{U}\end{subarray}}\frac{1}{|\mathcal{D}_{d}|}\sum_{\\{\mathbf{x},\mathbf{y}\\}\in\mathcal{D}_{d}}\ell_{\rm
CE}(\mathcal{M}_{\theta}(\mathbf{x}),\mathbf{y}),$ (6)

where $\mathcal{U}=\\{\rm LL,TL,LT,TT,mt,comp\\}$ and $\mathcal{D}_{d}$
corresponds to the respective data subset (e.g., $\mathcal{D}_{\rm LL}$,
$\mathcal{D}_{\rm TL}$, etc.).

##  4 Experiments

###  4.1 Setup

We use LLaMA-2-7B Touvron et al. (2023b) as the base model. Drawing from the
distribution of language in pre-training corpus, we use English (Eng) as a
resource-rich language and conduct experiments on 14 languages: Czech (Ces),
Danish (Dan), Ukrainian (Ukr), Bulgarian (Bul), Finnish (Fin), Hungarian
(Hun), Norwegian (Nob), Indonesian (Ind), Japanese (Jpn), Korean (Kor),
Portuguese (Por), Slovenian (Slv), Vietnamese (Vie), and Polish (Pol).
Stanford Alpaca instruction data Taori et al. (2023) serve as the base of the
transfer set $\mathcal{D}$, providing questions and ground-truth answers in
English. For machine translation, we utilize open-source NLLB-200-3.3B Costa-
jussà et al. (2022) model. To improve translation quality, we follow Zeng et
al. (2021) to filter low-quality translations and use CLD3 Ooms (2024) model
to remove off-target translations. We also follow Lin et al. (2021) to
construct bilingual dictionaries for code-switching. See appendix A for more
details.

##### Implementation Details

Our code is implemented using DeepSpeed Rasley et al. (2020) on eight NVIDIA
A800-SXM4-80GB GPUs. Following Wang et al. (2023a), we set the training
duration to four epochs with an automatically calculated learning rate and
employ early stopping. Other hyperparameters are set according to Hiyouga
(2023).

##### Baselines.

For comparison, we consider the following baseline systems that enhance
LLaMA-2’s multilingual capabilities using different instruction fine-tuning
methods:

  * •

SFT Ouyang et al. (2022): It only involves English instruction datasets in the
process of fine-tuning, which is not multilingual-oriented.

  * •

Translate-then-SFT (Chen et al., 2023, T-SFT): It uses an external machine
translation system to translate English instruction data into non-English
languages and construct multilingual data for instruction fine-tuning.

  * •

Cross-Lingual Instruction Tuning (CIT; Li et al., 2023a): It constructs cross-
lingual instructions for fine-tuning, imposing models to respond in the target
language given the source language as context.

  * •

Cross-Lingual Chain-of-Thought Reasoning (XCOT; Chai et al., 2024): It applies
code-switching to multilingual instruction training data, using high-resource
instruction data to supervise the training of low-resource languages with
cross-lingual distillation.

##### Datasets.

We evaluate the multilingual capabilities of LLMs on four representative
datasets:

  * •

BELEBELE Bandarkar et al. (2023): A widely used language understanding dataset
covering 122 languages, where each question, linked to a short passage, has
four multiple-choice answers. This dataset proves challenging for state-of-
the-art LLMs. Accuracy is reported in our experiments.

  * •

FLORES Goyal et al. (2022): A benchmark for machine translation with parallel
text from Wikipedia for 204 languages, making up over 40,000 directions. We
evaluate the bidirectional translation results between the target language and
English, reporting scores using SacreBLEU Post (2018) and COMET score using
WMT22-comet-da model Rei et al. (2022a).

  * •

XL-SUM Hasan et al. (2021): A multilingual abstractive summarization benchmark
for 44 languages, comprising multiple long news texts requiring summarization
into a single sentence. ROUGE-1 and ROUGE-L F1 scores are reported.

  * •

MKQA Longpre et al. (2020): An open-domain question-answering dataset across
26 diverse languages, providing multiple possible short answers as ground
truth for each question. We use the official evaluation script and report
token overlapped F1 scores.

| Ces | Dan | Ukr | Bul | Fin | Hun | Nob | Ind | Jpn | Kor | Por | Slv | Vie | Pol | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Performance on Target Language
SFT | 49.33 | 48.33 | 46.67 | 49.11 | 39.78 | 43.22 | 49.22 | 46.15 | 42.01 | 37.99 | 55.98 | 42.79 | 42.91 | 44.69 | 45.58
T-SFT | 48.22 | 51.67 | 47.11 | 51.22 | 47.11 | 45.67 | 51.33 | 49.72 | 41.56 | 43.69 | 56.20 | 46.03 | 47.60 | 48.72 | 48.28
CIT | 50.11 | 53.44 | 47.22 | 51.44 | 48.00 | 45.67 | 53.33 | 49.94 | 43.24 | 46.26 | 56.65 | 46.70 | 45.59 | 49.72 | 49.09
XCOT | 51.56 | 54.22 | 47.83 | 52.78 | 47.00 | 45.67 | 51.33 | 49.16 | 43.02 | 46.15 | 56.42 | 46.48 | 46.82 | 48.49 | 49.07
SDRRL | 52.11 | 55.00 | 48.33 | 54.00 | 49.56 | 46.44 | 53.89 | 52.40 | 45.81 | 46.82 | 57.88 | 47.26 | 48.38 | 50.17 | 50.58
Performance on English Language
SFT | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39 | 65.39
T-SFT | 63.91 | 65.25 | 66.03 | 65.25 | 65.70 | 65.36 | 65.25 | 65.70 | 61.01 | 60.45 | 63.80 | 65.47 | 65.47 | 65.92 | 64.61
CIT | 63.46 | 65.47 | 65.59 | 64.02 | 61.23 | 63.46 | 64.13 | 65.92 | 62.01 | 63.46 | 64.02 | 63.24 | 62.91 | 62.91 | 63.70
XCOT | 65.70 | 65.47 | 66.15 | 66.48 | 65.81 | 65.70 | 66.55 | 64.92 | 63.24 | 65.43 | 62.46 | 66.50 | 63.91 | 66.37 | 65.34
SDRRL | 66.26 | 65.70 | 67.15 | 66.53 | 65.92 | 66.70 | 66.59 | 67.15 | 65.13 | 65.45 | 66.48 | 66.59 | 65.57 | 66.82 | 66.29

Table 2: Results of baselines and our SDRRL on BELEBELE benchmark. In each
column, the best result is in bold and the second best result is underlined.

| Ces | Dan | Ukr | Bul | Fin | Hun | Nob | Ind | Jpn | Kor | Por | Slv | Vie | Pol | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
BLEU scores on X-to-English Tasks
SFT | 34.66 | 42.57 | 34.17 | 33.91 | 26.76 | 28.15 | 38.34 | 20.78 |  07.56 |  03.15 | 33.25 | 11.94 | 16.01 | 15.31 | 24.75
T-SFT | 32.63 | 32.21 | 31.13 | 31.05 | 23.53 | 24.18 | 27.44 | 23.38 |  07.82 |  07.68 | 33.03 | 14.36 | 19.63 | 19.43 | 23.39
CIT | 26.54 | 29.88 | 24.25 | 26.66 | 21.51 | 21.24 | 30.21 | 29.02 |  06.00 |  07.58 | 34.46 | 16.57 | 25.84 | 19.19 | 22.78
XCOT | 31.52 | 31.26 | 29.90 | 31.05 | 24.37 | 23.60 | 32.50 | 27.33 |  08.29 |  09.23 | 35.86 | 17.82 | 25.46 | 19.40 | 24.83
SDRRL | 36.38 | 45.71 | 35.33 | 37.49 | 30.80 | 31.62 | 40.88 | 30.93 | 15.42 | 12.20 | 39.81 | 21.15 | 28.68 | 22.52 | 30.64
BLEU scores on English-to-X Tasks
SFT | 13.00 | 21.91 | 11.18 | 12.98 |  08.39 |  09.07 | 18.53 | 34.54 | 17.03 | 18.15 | 43.06 | 28.46 | 25.06 | 27.65 | 20.64
T-SFT | 22.68 | 27.78 | 23.11 | 27.59 | 15.31 | 16.96 | 25.60 | 31.79 | 19.52 | 18.11 | 39.75 | 26.17 | 25.09 | 26.04 | 24.68
CIT | 22.03 | 28.57 | 19.92 | 26.85 | 14.54 | 17.46 | 25.97 | 29.46 | 13.81 | 15.33 | 35.24 | 22.60 | 22.33 | 22.84 | 22.64
XCOT | 23.11 | 32.20 | 21.97 | 27.33 | 15.80 | 17.38 | 25.96 | 30.33 |  09.31 | 15.13 | 38.04 | 26.56 | 25.43 | 26.03 | 23.90
SDRRL | 27.91 | 39.00 | 27.25 | 33.93 | 20.88 | 22.09 | 29.64 | 35.32 | 20.51 | 20.47 | 43.36 | 30.09 | 29.87 | 27.86 | 29.16
COMET scores on X-to-English Tasks
SFT | 85.35 | 87.60 | 84.58 | 84.97 | 85.69 | 84.40 | 86.35 | 73.54 | 63.41 | 45.44 | 78.91 | 80.98 | 63.43 | 68.46 | 76.65
T-SFT | 84.71 | 84.26 | 83.33 | 83.82 | 83.78 | 82.02 | 83.31 | 78.94 | 78.39 | 72.95 | 80.38 | 81.82 | 73.81 | 79.06 | 80.76
CIT | 81.71 | 82.84 | 80.06 | 81.72 | 82.14 | 82.29 | 83.37 | 84.62 | 76.16 | 73.88 | 83.71 | 76.38 | 80.60 | 78.97 | 80.60
XCOT | 84.40 | 84.47 | 83.11 | 83.90 | 84.67 | 81.96 | 84.68 | 83.50 | 78.83 | 75.66 | 83.23 | 76.48 | 79.46 | 78.75 | 81.65
SDRRL | 86.04 | 88.51 | 84.82 | 86.08 | 86.98 | 85.70 | 87.15 | 89.46 | 83.33 | 79.02 | 85.15 | 84.02 | 81.43 | 83.08 | 85.06
COMET scores on English-to-X Tasks
SFT | 57.19 | 70.93 | 55.25 | 54.99 | 60.29 | 53.94 | 71.97 | 83.82 | 82.46 | 82.14 | 84.57 | 55.96 | 80.78 | 82.14 | 69.75
T-SFT | 78.94 | 81.34 | 79.92 | 81.43 | 78.53 | 76.01 | 82.69 | 84.90 | 82.76 | 80.58 | 86.42 | 69.06 | 81.62 | 82.86 | 80.50
CIT | 79.87 | 82.47 | 78.63 | 81.70 | 78.39 | 76.18 | 83.19 | 84.18 | 78.15 | 78.45 | 85.12 | 80.12 | 80.77 | 80.17 | 80.53
XCOT | 79.22 | 83.29 | 79.16 | 80.86 | 78.63 | 75.51 | 83.30 | 84.68 | 74.27 | 77.31 | 86.01 | 78.65 | 82.24 | 82.72 | 80.42
SDRRL | 84.29 | 86.91 | 83.51 | 85.40 | 84.62 | 81.06 | 85.55 | 86.00 | 83.65 | 82.66 | 87.64 | 82.63 | 83.93 | 83.61 | 84.39

Table 3: Results of baselines and our SDRRL on FLORES benchmark.

###  4.2 Main Results

Table 2 shows the experimental results of the multilingual understanding task.
Tables 3, Table 4 and Table 5 show the results on multilingual generation
tasks. From the experimental results, we can observe that:

(1) SDRRL effectively enhances performance in the target languages.
Specifically, in every comprehension and generation task, our method surpasses
the baselines in almost all target languages. As shown in Table 2, SDRRL
improves [erformane in comprehension tasks by approximately +1.5 BLEU score.
On the Flores dataset, SDRRL yields up to about +6.0 BLEU score improvement in
both directions and about +4.0 COMET score improvement (Table 3). This
demonstrates that using proficient responses in resource-rich languages as
supervisory signals for knowledge distillation significantly enhances
performance in other target languages.

(2) SDRRL exhibits stronger robustness in generation tasks. For example, on
the XL-SUM dataset (Table 4), which requires the generation of longer texts,
the average performance of CIT and XCOT decreased due to the quality of
machine-translated texts and pipeline noise, yet SDRRL still achieved about
+0.55 ROUGE-L F1 score improvement. On the FLORES dataset (Table 3), which
requires cross-lingual text generation, T-SFT and CIT lead to decrease of
-1.36 and -2.08 BLEU scores, respectively, while our method improves by +5.88
BLEU scores. This suggests that adding machine-translated data constructed
instructions to the self-distillation process effectively improves
multilingual generation and mitigates the negative impact of low-quality
translated texts.

| Ind | Jpn | Kor | Por | Vie | Ukr | Avg.
---|---|---|---|---|---|---|---
Performance on Target Language (ROUGE-1)
SFT | 20.82 |  06.17 | 0.66 | 23.38 |  09.30 |  08.10 | 11.41
T-SFT | 25.61 | 32.11 | 7.67 | 26.68 | 20.59 | 14.19 | 21.14
CIT | 24.64 | 16.11 | 5.80 | 26.33 | 20.55 | 11.21 | 17.44
XCOT | 22.55 | 32.39 | 7.26 | 26.21 | 19.84 | 13.38 | 20.27
SDRRL | 26.08 | 33.15 | 8.18 | 27.40 | 20.98 | 14.35 | 21.69
Performance on Target Language (ROUGE-L)
SFT | 16.03 |  04.13 | 0.61 | 15.84 |  07.21 |  06.72 |  08.42
T-SFT | 20.15 | 22.83 | 6.93 | 18.41 | 15.18 | 11.73 | 15.87
CIT | 19.02 | 11.22 | 5.14 | 18.06 | 14.91 | 9.00 | 12.89
XCOT | 17.19 | 22.32 | 6.52 | 18.05 | 14.57 | 10.78 | 14.91
SDRRL | 20.47 | 22.81 | 7.35 | 19.09 | 15.52 | 11.84 | 16.18
Performance on English Language (ROUGE-1)
SFT | 26.35 | 26.35 | 26.35 | 26.35 | 26.35 | 26.35 | 26.35
T-SFT | 27.49 | 26.89 | 26.68 | 27.28 | 26.42 | 26.75 | 26.92
CIT | 27.84 | 27.40 | 26.57 | 27.39 | 27.17 | 24.83 | 26.87
XCOT | 26.44 | 25.45 | 25.43 | 26.78 | 26.00 | 25.90 | 26.00
SDRRL | 28.18 | 27.73 | 27.44 | 27.57 | 27.52 | 27.23 | 27.61
Performance on English Language (ROUGE-L)
SFT | 18.68 | 18.68 | 18.68 | 18.68 | 18.68 | 18.68 | 18.68
T-SFT | 19.64 | 19.11 | 18.94 | 19.54 | 18.73 | 19.01 | 19.16
CIT | 19.93 | 19.56 | 18.81 | 19.56 | 19.34 | 17.43 | 19.11
XCOT | 18.63 | 17.83 | 17.86 | 18.91 | 18.29 | 18.15 | 18.28
SDRRL | 20.25 | 19.88 | 19.56 | 19.69 | 19.66 | 19.44 | 19.75

Table 4: Results of baselines and our SDRRL on XL-SUM benchmark on the target
language and English.

| Nob | Dan | Fin | Hun | Jpn | Kor | Por | Vie | Pol | Avg.
---|---|---|---|---|---|---|---|---|---|---
Performance on Target Language
SFT | 37.30 | 38.28 | 37.30 | 35.21 | 32.80 | 33.18 | 39.29 | 37.50 | 37.50 | 36.48
T-SFT | 39.73 | 39.59 | 38.95 | 38.60 | 33.96 | 33.90 | 39.93 | 38.71 | 38.14 | 37.95
CIT | 40.18 | 39.94 | 37.94 | 38.40 | 33.50 | 34.24 | 39.86 | 39.94 | 38.84 | 38.09
XCOT | 39.03 | 39.28 | 38.12 | 35.60 | 33.07 | 33.69 | 39.96 | 39.49 | 38.49 | 37.41
SDRRL | 40.64 | 40.92 | 39.71 | 39.02 | 39.51 | 34.06 | 41.12 | 40.02 | 39.45 | 39.38
Performance on English Language
SFT | 41.62 | 41.62 | 41.62 | 41.62 | 41.62 | 41.62 | 41.62 | 41.62 | 41.62 | 41.62
T-SFT | 44.92 | 42.63 | 44.24 | 44.21 | 41.65 | 42.11 | 42.63 | 42.65 | 42.81 | 43.09
CIT | 44.09 | 43.86 | 43.55 | 44.12 | 42.83 | 43.29 | 42.51 | 42.52 | 43.41 | 43.35
XCOT | 43.23 | 43.16 | 43.53 | 43.06 | 42.59 | 42.58 | 43.39 | 42.58 | 43.29 | 43.05
SDRRL | 45.42 | 45.33 | 45.47 | 44.78 | 43.26 | 43.58 | 43.99 | 45.77 | 44.71 | 44.70

Table 5: Results of baselines and our SDRRL on MKQA dataset on the target
language and English.

(3) SDRRL can maintain the original strong capabilities in English. The
results show that it is more challenging to retain the original English
capabilities for languages with unique alphabets (e.g., Japanese and Korean).
For example, in the Japanese comprehension task (Table 2), all baseline
methods lead to a performance drop in English compared to SFT, while only our
method successfully preserving the original English capabilities.

###  4.3 Ablation Study

|  | NLU Avg. | NLG Avg.
---|---|---|---
|  | Tar. | Eng | Tar. | Eng
1 | Full Method | 50.58 | 66.29 | 28.24 | 31.69
2 |  \- $\mathcal{D}_{\rm TL}$ and $\mathcal{D}_{\rm LT}$ | 49.56 | 65.93 | 26.15 | 30.55
3 |  \- $\mathcal{D}_{\rm synth}$ \+ $\mathcal{D}$ | 48.59 | 65.10 | 25.16 | 30.10
4 |  \- $\mathcal{D}_{\rm mt}$ and $\mathcal{D}_{\rm comp}$ | 50.41 | 66.01 | 26.61 | 30.19
5 |  \- Code Switching | 50.37 | 65.94 | 27.13 | 30.69
6 | Only $\mathcal{D}_{\rm mt}$ and $\mathcal{D}_{\rm comp}$ | 41.25 | 61.61 | 17.89 | 22.28

Table 6: Ablation study. Average scores of target language (Tar.) and English
(Eng) on natural language understanding task (NLU, including BELEBELE) and
natural language generation tasks (NLG, including FLORES, XL-SUM ROUGE-L, and
MKQA) are reported.

We further investigate the effectiveness of each component of our proposed
SDRRL. The results are shown in Table 6, where average scores on natural
language understanding and generation tasks are reported. Our observations
include:

(1) Rows 1 to 5 demonstrate that removing any single component results in
performance degradation, affirming the necessity and efficacy of each
component in SDRRL.

(2) Insights from row 3 suggest a significant performance decline in both
target languages and English when model-generated responses
($\hat{\textbf{y}}_{i}$) are removed from $\mathcal{D}_{\rm synth}$,
highlighting the effectiveness of utilizing responses in resource-rich
languages as additional supervision signals for improving multilingual
capabilities. Moreover, row 2 indicates that substituting sentences with their
semantic counterparts in different languages also contributes to multilingual
performance improvement.

(3) Row 4 and 5 reveal that $\mathcal{D}_{\rm mt}$, $\mathcal{D}_{\rm comp}$,
and code-switching provide a limited amount of ground truth. This additional
supervisory signal is beneficial for generative tasks and helps improve the
quality of responses.

(4) Despite introducing a small amount of parallel data through
$\mathcal{D}_{\rm mt}$ and $\mathcal{D}_{\rm comp}$, as shown in row 6,
relying solely on these additional data for LLM training supervision leads to
severe performance degradation. Compared to row 4, this indicates that these
data do not inherently bring positive performance gains but are used to
mitigate the deterioration of the LLM’s multilingual generative representation
space caused by noisy machine-translated text, serving as a regularization
mechanism in knowledge distillation.

###  4.4 Visualization of Representation Space for Source and Target Langauges

Figure 2: t-SNE visualizations of output representations by LLaMA-2 before and
after applying SDRRL. The markers in red and blue represent semantically
equivalent instructions in different languages.

We visualize the sentence representations of input instructions to investigate
the effect of SDRRL on the multilingual representation space. Following common
practices in sequence classification Li et al. (2023c), we input instructions
into the LLaMA-2 and use the last hidden states of the last token as the
vector representation of the sentence. We then apply t-SNE Van der Maaten and
Hinton (2008) to reduce the 4096-dim representations to 2-dim for
visualization.

As shown in Figure 2, after applying SDRRL, the representations of
semantically equivalent instructions in the source and target languages are
drawn closer together. This implies that SDRRL has improved the multilingual
representation space by aligning the representation space of the target
language closer to that of the resource-rich, better-modeled source language,
thereby enhancing the performance in target languages.

###  4.5 SeaLLM as Different Backbone Model

By using the responses of LLMs in high-resource languages as the supervisory
signal for knowledge distillation, SDRRL is applicable to various LLMs, not
limited to LLaMA-2. In this part, we conduct experiments on SeaLLM-7B Nguyen
et al. (2023), a specialized language model optimized for Southeast Asian
languages.

As shown in Table 7, SDRRL results in an improvement of +2.39 average scores
on the target languages. In English, SDRRL maintains its original performance,
while the baselines exhibit a performance drop of at least -2.02 average
scores compared to vanilla SFT. This demonstrates the generalizability of
SDRRL in different LLMs. See appendix B for detailed results on more datasets.

###  4.6 Further Analysis

##### Non-English Source Languages.

SDRRL is also capable of transferring multilingual performance using other
source languages in high-resource. In appendix C, we opt for experiments with
French instead of English. Experimental results reveal that, despite the LLM
and the machine translation system exhibiting stronger performance in English,
SDRRL still achieves positive distillation gains with French as the source
language.

##### Case Study.

In appendix D, we provide several case studies to offer deeper insights into
the impact of SDRRL on the generation capabilities of LLMs. It is observed
that the SDRRL process is able to alleviate off-target issues in the target
language, reduce grammatical errors and hallucinations, and enhance the
fluency of the output text.

| BELE. | XL-SUM | FLORES | MKQA | Avg.
---|---|---|---|---|---
Performance on Target Language
SFT | 42.24 | 16.48 | 18.45 | 38.86 | 29.01
T-SFT | 42.77 | 15.32 | 16.59 | 43.40 | 29.52
CIT | 42.53 | 15.75 | 20.49 | 43.70 | 30.62
XCOT | 41.19 | 15.79 | 17.21 | 42.04 | 29.06
SDRRL | 43.67 | 17.89 | 25.86 | 44.63 | 33.01
Performance on English Language
SFT | 60.19 | 15.25 | 28.49 | 39.62 | 35.89
T-SFT | 58.70 | 15.63 | 23.72 | 37.43 | 33.87
CIT | 58.66 | 15.42 | 18.31 | 36.67 | 32.27
XCOT | 57.73 | 14.90 | 23.96 | 37.94 | 33.63
SDRRL | 60.67 | 16.24 | 29.47 | 40.32 | 36.68

Table 7: Results of baselines and our SDRRL on SeaLLM. The average scores
across various datasets are reported, and full results are available in
appendix B.

##  5 Conclusion and Future Work

We introduce Self-Distillation from Resource-Rich Languages (SDRRL) to enhance
the multilingual capabilities of LLMs. SDRRL uses the model itself to generate
high-quality responses in resource-rich source languages and their target
language counterparts as supervision signals for knowledge distillation,
aiming to align additional target languages with resource-rich languages. We
conduct comprehensive experiments across 16 languages on LLaMA-2 and SeaLLM.
The results demonstrate that, compared to various baselines, our method
significantly enhances the performance of target languages while preserving
the capabilities of source languages. This highlights the multilingual
potential of LLMs and illuminates paths for further research into multilingual
LLMs.

## Limitations

Firstly, within our method pipeline, some components are interchangeable. For
example, our approach relies on external machine translation systems to
provide translations in the target language, while future research could
explore self-translation with LLMs that achieve great low-resource translation
capabilities, thereby simplifying the process. Additionally, our method uses a
small amount of machine-translated parallel corpus to construct the transfer
set, but employing monolingual texts in the target language represents a
promising research direction. Secondly, our experiments are conducted with
only a single source language and target language. Subsequent research could
investigate using a mix of multiple languages as both source and target
languages and explore the mutual influences between different languages to
further enhance the multilingual capabilities of LLMs. Thirdly, our method
does not involve engineering on the architecture of LLMs. For specific
extremely low-resource languages, modifying the architecture and introducing
additional data, such as vocabulary expansion or continuing pre-training,
might be beneficial in enhancing multilingual performance.

## References

  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.
  * Chai et al. (2024) Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. 2024.  xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning.
  * Chen et al. (2023) Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. 2023.  Breaking language barriers in multilingual mathematical reasoning: Insights and observations.
  * Chen et al. (2020) Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020.  Distilling knowledge learned in bert for text generation.
  * Chi et al. (2021) Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2021.  Improving pretrained cross-lingual language models via self-labeled word alignment.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Etxaniz et al. (2023) Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023.  Do multilingual language models think better in english?
  * Google et al. (2023) Gemini Team Google, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Gordon and Duh (2019) Mitchell A. Gordon and Kevin Duh. 2019.  Explaining sequence-level knowledge distillation as data-augmentation for neural machine translation.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ , 129:1789–1819.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.
  * Hiyouga (2023) Hiyouga. 2023.  Llama factory.  https://github.com/hiyouga/LLaMA-Factory.
  * Huang et al. (2023) Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.
  * Huang et al. (2022) Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. 2022.  Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.
  * Kew et al. (2023) Tannon Kew, Florian Schottmann, and Rico Sennrich. 2023.  Turning english-centric llms into polyglots: How much multilinguality is needed?
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-level knowledge distillation.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.
  * Li et al. (2023a) Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2023a.  Align after pre-train: Improving multilingual generative models with cross-lingual alignment.
  * Li et al. (2023b) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023b.  Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation.
  * Li et al. (2022) Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2022.  Pretrained language models for text generation: A survey.
  * Li et al. (2023c) Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu lee Wang, Qing Li, and Xiaoqin Zhong. 2023c.  Label supervised llama finetuning.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual language models.
  * Lin et al. (2021) Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2021.  Pre-training multilingual neural machine translation by leveraging alignment information.
  * Longpre et al. (2020) Shayne Longpre, Yi Lu, and Joachim Daiber. 2020.  Mkqa: A linguistically diverse benchmark for multilingual open domain question answering.
  * Ma et al. (2022) Yukun Ma, Trung Hieu Nguyen, and Bin Ma. 2022.  Cpt: Cross-modal prefix-tuning for speech-to-text translation.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6217–6221.
  * Mao and Yu (2024a) Zhuoyuan Mao and Yen Yu. 2024a.  Tuning llms with contrastive alignment instructions for machine translation in unseen, low-resource languages.
  * Mao and Yu (2024b) Zhuoyuan Mao and Yen Yu. 2024b.  Tuning llms with contrastive alignment instructions for machine translation in unseen, low-resource languages.
  * Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2023.  Seallms – large language models for southeast asia.
  * Niklaus et al. (2023) Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023.  Lextreme: A multi-lingual and multi-task benchmark for the legal domain.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_. Association for Computational Linguistics.
  * Ooms (2024) Jeroen Ooms. 2024.  _cld3: Google’s Compact Language Detector 3_.  R package version 1.6.0.
  * OpenAI (2022) OpenAI. 2022.  ChatGPT.  https://openai.com/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pahune and Chandrasekharan (2023) Saurabh Pahune and Manoj Chandrasekharan. 2023.  Several categories of large language models (llms): A short survey.  _International Journal for Research in Applied Science and Engineering Technology_ , 11(7):615–633.
  * Pham et al. (2022) Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. 2022.  Revisiting self-distillation.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Qin et al. (2023) Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023.  Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.
  * Ranaldi and Pucci (2023) Leonardo Ranaldi and Giulia Pucci. 2023.  Does the english matter? elicit cross-lingual abilities of large language models.  In _Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)_ , pages 173–183.
  * Ranaldi and Zanzotto (2023) Leonardo Ranaldi and Fabio Massimo Zanzotto. 2023.  Empowering multi-step reasoning across languages via tree-of-thoughts.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery.
  * Rei et al. (2022a) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rei et al. (2022b) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b.  CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Saberi et al. (2024) Iman Saberi, Fatemeh Fard, and Fuxiang Chen. 2024.  Advfusion: Multilingual adapter-based knowledge transfer for code summarization.
  * Schuster et al. (2019a) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019a.  Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing.
  * Schuster et al. (2019b) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019b.  Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing.
  * Shaham et al. (2024) Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024.  Multilingual instruction tuning with just a pinch of multilinguality.
  * Sun et al. (2020) Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020.  Knowledge distillation for multilingual unsupervised neural machine translation.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11).
  * Wang et al. (2023a) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023a.  Openchat: Advancing open-source language models with mixed-quality data.
  * Wang et al. (2023b) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023b.  Openchat: Advancing open-source language models with mixed-quality data.  _arXiv preprint arXiv:2309.11235_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wen-Yi and Mimno (2023) Andrea Wen-Yi and David Mimno. 2023.  Hyperpolyglot llms: Cross-lingual interpretability in token embeddings.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.
  * Yang et al. (2019) Chenglin Yang, Lingxi Xie, Chi Su, and Alan L. Yuille. 2019.  Snapshot distillation: Teacher-student optimization in one generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yang et al. (2023) Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023.  Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages.
  * Yoon et al. (2024) Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024.  Langbridge: Multilingual reasoning without multilingual supervision.
  * Zeng et al. (2021) Xianfeng Zeng, Yijin Liu, Ernan Li, Qiu Ran, Fandong Meng, Peng Li, Jinan Xu, and Jie Zhou. 2021.  WeChat neural machine translation systems for WMT21.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 243–254, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.
  * Zhang et al. (2022a) L. Zhang, C. Bao, and K. Ma. 2022a.  Self-distillation: Towards efficient and compact neural networks.  _IEEE Transactions on Pattern Analysis; Machine Intelligence_ , 44(08):4388–4403.
  * Zhang et al. (2022b) Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. 2022b.  Self-distillation: Towards efficient and compact neural networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(8):4388–4403.
  * Zhang et al. (2019) Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.
  * Zhang et al. (2023) Yuanchi Zhang, Peng Li, Maosong Sun, and Yang Liu. 2023.  Continual knowledge distillation for neural machine translation.
  * Zhang and Liu (2021) Yuanchi Zhang and Yang Liu. 2021.  Directquote: A dataset for direct quotation extraction and attribution in news articles.  _arXiv preprint arXiv:2110.07827_.
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.
  * Zhu et al. (2024) Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024.  Question translation training for better multilingual reasoning.

##  Appendix A Implementation Details

The signature of SacreBLEU we use in this work is ‘‘nrefs:1 | case:mixed | eff:no | tok:flores200 | smooth:exp | version:2.0.0’’. The Stanford Alpaca dataset comprises 52,002 entries, licensed under the CC BY-NC 4.0 agreement. For each target language, machine translation parallel corpora are sampled from Opus100 Zhang et al. (2020), consisting of 1,000 entries. When utilizing NLLB for machine translation, we set the beam size to 4, with the remaining configurations adopting the default parameters from Huggingface Transformers Wolf et al. (2020). In the reimplementation of baselines, the same machine translation system is employed to provide multilingual alignment data. For the 16 languages involved in our experiments, XL-SUM and MKQA datasets have not covered all of them. During the evaluation of MKQA, questions lacking ground truth are skipped.

##  Appendix B More Detailed Results on SeaLLM

We conduct experiments on three common Southeast Asian languages: Indonesian
(Ind), Thai (Tha), and Khmer (Khm). As shown in Table 8, 9, 10, and 11, SDRRL
still outperforms the baselines, demonstrating the generalizability of SDRRL
in different LLMs.

| Ind | Khm | Tha | Avg.
---|---|---|---|---
Performance on Target Language
SFT | 47.71 | 32.56 | 46.44 | 42.24
T-SFT | 48.31 | 32.89 | 46.77 | 42.77
CIT | 48.83 | 32.56 | 46.20 | 42.53
XCOT | 45.81 | 32.22 | 45.55 | 41.19
SDRRL | 50.39 | 33.67 | 46.96 | 43.67
Performance on English Language
SFT | 60.56 | 60.56 | 59.46 | 60.19
T-SFT | 60.78 | 57.89 | 57.43 | 58.70
CIT | 58.55 | 58.10 | 59.33 | 58.66
XCOT | 57.77 | 57.99 | 57.43 | 57.73
SDRRL | 61.68 | 60.89 | 59.44 | 60.67

Table 8: Results of baselines and our SDRRL on BELEBELE benchmark using
SeaLLM.

|  Ind |  Tha |  Avg.
---|---|---|---
Performance on Target Language (ROUGE-1)
SFT |  21.91 |  24.65 |  23.28
T-SFT |  21.07 |  21.26 |  21.17
CIT |  21.19 |  23.93 |  22.56
XCOT |  22.20 |  21.85 |  22.02
SDRRL |  23.62 |  25.78 |  24.70
Performance on Target Language (ROUGE-L)
SFT |  16.46 |  16.50 |  16.48
T-SFT |  16.23 |  14.40 |  15.32
CIT |  15.84 |  15.66 |  15.75
XCOT |  16.93 |  14.65 |  15.79
SDRRL |  18.06 |  17.73 |  17.89
Performance on English Language (ROUGE-1)
SFT |  21.39 |  22.85 |  22.12
T-SFT |  21.93 |  23.17 |  22.55
CIT |  21.65 |  23.07 |  22.36
XCOT |  21.27 |  21.99 |  21.63
SDRRL |  23.47 |  23.55 |  23.01
Performance on English Language (ROUGE-L)
SFT |  14.79 |  15.71 |  15.25
T-SFT |  15.19 |  16.07 |  15.63
CIT |  14.91 |  15.92 |  15.42
XCOT |  14.66 |  15.14 |  14.90
SDRRL |  16.34 |  16.15 |  16.24

Table 9: Results of baselines and our SDRRL on XL-SUM benchmark on the target
language using SeaLLM.

| Ind | Tha | Thm | Avg.
---|---|---|---|---
xx$\rightarrow$en (BLEU)
SFT | 36.75 | 20.93 | 20.22 | 28.49
T-SFT | 32.23 | 14.41 | 15.21 | 23.72
CIT | 22.52 | 15.84 | 14.10 | 18.31
XCOT | 33.20 | 16.48 | 14.71 | 23.96
SDRRL | 38.30 | 21.76 | 20.64 | 29.47
en$\rightarrow$xx (BLEU)
SFT | 30.26 | 16.53 | 6.64 | 18.45
T-SFT | 28.29 | 13.10 | 4.88 | 16.59
CIT | 31.21 | 18.15 | 9.76 | 20.49
XCOT | 29.15 | 14.28 | 5.26 | 17.21
SDRRL | 36.28 | 24.02 | 15.43 | 25.86
xx$\rightarrow$en (COMET)
SFT | 86.94 | 82.89 | 80.07 | 83.51
T-SFT | 84.49 | 74.61 | 71.29 | 77.89
CIT | 80.78 | 78.87 | 76.18 | 78.48
COT | 85.69 | 77.57 | 72.14 | 78.91
SDRRL | 87.39 | 83.07 | 80.63 | 84.01
en$\rightarrow$xx (COMET)
SFT | 86.78 | 73.22 | 64.05 | 75.41
T-SFT | 85.44 | 66.95 | 59.09 | 72.26
CIT | 85.80 | 74.42 | 69.60 | 77.70
COT | 85.23 | 68.26 | 62.24 | 73.74
SDRRL | 88.70 | 79.03 | 75.97 | 82.34

Table 10: Results of baselines and our SDRRL on FLORES benchmark using SeaLLM. | Tha | Khm | Avg.
---|---|---|---
Performance on Target Language
SFT | 40.68 | 37.04 | 38.86
T-SFT | 48.32 | 38.48 | 43.40
CIT | 48.38 | 39.01 | 43.70
XCOT | 45.07 | 39.00 | 42.04
SDRRL | 49.44 | 39.81 | 44.63
Performance on English Language
SFT | 39.62 | 39.62 | 39.62
T-SFT | 37.92 | 36.94 | 37.43
CIT | 37.64 | 35.69 | 36.67
XCOT | 38.40 | 37.48 | 37.94
SDRRL | 40.66 | 39.97 | 40.32
Table 11: Results of baselines and our SDRRL on MKQA dataset using SeaLLM.

##  Appendix C Experiments with Non-English Language as the Source Language

| NLU Avg. | NLG Avg.
---|---|---
English | +6.29 | +5.31
French | +2.94 | +1.77
Table 12: Distillation gains from SDRRL with English or French as the source
language. Average scores on the natural language understanding task (NLU,
including BELEBELE) and natural language generation tasks (NLG, including
FLORES, XL-SUM ROUGE-L, and MKQA) are reported.

SDRRL aims to transfer the proficiency of LLMs from resource-rich languages to
another target language, indicating that the source language is not limited to
English. In this section, we use French instead of English to generate
responses and replicate experiments conducted in Indonesian, Japanese, and
Korean languages. Table 12 presents the average knowledge distillation gains
from SDRRL over SFT when employing English and French as the source languages,
respectively. Due to the inferior performance of LLaMA-2-7B in French
Bandarkar et al. (2023) compared to English and the decreased effectiveness of
the external translation system NLLB when French is used as the source
language Costa-jussà et al. (2022), the knowledge distillation gains generated
with French as the source language are smaller.

##  Appendix D Case Study

Hungarian |  Question: Mit jelent a DNA?
---|---
|  Before SDRRL: A DNA (deoxyribonucleic acid) is the hereditary material
present in all living organisms. It is a long double-stranded helix made up of
four nitrogenous bases; adenine (A), guanine (G), cytosine (C) and thymine
(T).
|  After SDRRL: A DNA a dezoxiribonukleinsav rövidítése, egy molekula, amely
két szálból áll, amelyek egymáshoz kötődnek, és hidrogénkötések tartják össze
őket. Ez a genetikai anyag, amely az élőlények fejlődését és működését
szabályozza, és a sejtek fehérje- és szénhidrátszintézisét irányító
utasításokat tartalmazza.
Slovenian |  Question: Opišite življenje in vladanje kralja Karla II.
|  Before SDRRL: Kralj Karel II. (1638-1700) je bil francoski krščanski kralj,
ki je vladal od leta 1685 do svoje smrti. Ko je postal kralj, je zavzel
podeželje in se lotil reform, katerek cilj je bil izboljšati upravo in
omogočiti zaščito družbe. Karel II. je bil tudi pomorski kralj, ki si je meril
svet. Under his rule, the French navy became one of the most powerful in the
world. Karel II. je bil tudi velik pokrovitelj umetnosti in književnosti. Že
stoletje po njegovem vrnitemenu je postal Karel II. znan kot eno največjih
francoskih vladarjev zadnjega obdobja.
|  After SDRRL: Kralj Karel II. je bil angleški kralj od leta 1660 do svoje
smrti leta 1685. Bil je sin Karla I., ki je bil odstranjen z oblasti in
usmrčen leta 1649. Po angleški državljanski vojni in obdobju republike je bil
Karel II. ponovno kronan za kralja Anglije leta 1660, kar je označilo začetek
Kraljeve obnove. Bil je znan kot podpornik umetnosti in kulture, ki je
spodbujal delo številnih umetnikov in pisateljev. Njegovo obdobje vladanja je
bilo zaznamovano z obnovitvenimi projekti, vključno z razvojem Londona po
Velikem požaru leta 1666. Karel II. je umrl leta 1685 brez legitimnih moških
potomcev, kar je dolgoročno vplivalo na vprašanje nasledstva. Njegova smrt ni
neposredno vplivala na konec Kraljeve obnove ali angleško državljansko vojno,
ki se je končala pred njegovim vladanjem, niti na začetek angleške suverenosti
v Severni Ameriki.
Bulgarian |  Question: Моля, обяснете историята на ледниковия период.
|  Before SDRRL: ледников период е период в ерата, в която Земята е bila
предимно покрита с лед. това се дължи на увеличение на азъйствата на Северния
полюс, които се оттикват водата в океаните и причинява замразяването на
течности и земя. това води до увеличение на масата лед вglobus и снижаваниеa
нива на температурата на аеросферата. това връвля вanother glacial period or
ice age.
|  After SDRRL: Ледниковият период е период от време в климатичната история на
Земята, през който климатът става много студен и полярните ледени шапки се
разширяват, покривайки голяма част от сушата. Това се случва, когато нивото на
атмосферната температура значително се понижи, което позволява на водата в
океаните да се превърне в лед и води до увеличаване на обема на леда и
връщането на вода към морския лед. Ледниковите периоди обикновено са свързани
с забележително намаляване на глобалните температури, което се случва в
продължение на десетилетия или стотици години. Няколко фактора могат да
доведат до ледников период, включително промени в орбитата на Земята,
намаляване на вулканичните изригвания и промени в океанските течения. Най-
скорошният ледников период, известен като последната ледникова епоха, приключи
преди около 10 000 години, когато климатът започна да се затопля и ледниците
по света започнаха да се топят.
Table 13: Several cases before and after the application of SDRRL. Following
SDRRL, the responses from LLMs become smoother, more comprehensive, and
contain fewer errors (colored in red).

As shown in Table 13, we provide several cases of SDRRL in several languages.
In the case of Hungarian, LLM encounters severe off-target issues, where the
response is in English and is inconsistent with the input language Hungarian,
which is a frequent problem during multilingual generation. After SDRRL, the
severe off-target issue has been effectively mitigated. In the Slovenian case,
LLM produces hallucinations when answering factual questions about history,
leading to factual inaccuracies, noise tokens like emojis, and off-target
English phrases. For example, The time frame mentioned (1685-1700) and the
description does not match any king named Charles. It seems there’s a mix-up
with historical figures. After SDRRL, the hallucination issue has been
alleviated, and the generated content becomes more detailed, refined, and
fluid. In the Bulgarian scenario, the response contains several grammatical
errors, such as ‘‘ледников’’, ‘‘оттикват’’ and ‘‘снижаваниеa’’. In this case,
the SDRRL process enhances the clarity and natural flow of the output text
while also eliminating grammatical errors in Bulgarian. See appendix E for
statistical results regarding off-target issues.

##  Appendix E Off-Target Issue Analysis

Figure 3: The occurrence rate of off-target issues in various languages during
the SDRRL process.

We delve deeper into the effectiveness of SDRRL in alleviating off-target
language issues during LLM responses. We evaluate the responses of LLaMA-2 on
Dolly Conover et al. (2023) and its multilingual extension, Bactrain-X Li et
al. (2023b). As depicted in Figure 3, we showcase the variation in the off-
target occurrence rates across each target language throughout the SDRRL
process. This indicates that SDRRL plays a constructive role in mitigating
off-target issues, ensuring consistency between the input and the response
languages.

##  Appendix F Potential Risks of Our Method

Because our method involves distilling knowledge from other target languages
towards high-resource languages to achieve cross-linguistic alignment, it may
lead to cultural unfairness for mid- and low-resource languages. For instance,
after aligning to English using SDRRL, responses of LLMs in African languages
may also adhere to the cultural practices and social norms of English.


# QTBIPOC PD: Exploring the Intersections of Race, Gender, and Sexual
Orientation in Participatory Design

Naba Rizvi<EMAIL_ADDRESS>0000-0002-2777-700X University of California, San
DiegoSan DiegoUSA ,  Reggie Casanova-Perez, MS<EMAIL_ADDRESS>0000-0002-8062-2947 University of WashingtonSeattleUSA ,  Harshini Ramaswamy
0000-0002-5552-4579 University of California, San DiegoSan DiegoUSA
<EMAIL_ADDRESS>,  Lisa G. Dirks, MS, MLIS  0000-0001-6768-6500 University
of WashingtonSeattleUSA<EMAIL_ADDRESS>,  Emily Bascom  0000-0002-5140-6567
University of WashingtonSeattleUSA<EMAIL_ADDRESS>and  Nadir Weibel, PhD
0000-0002-3457-4227 University of California, San DiegoSan DiegoUSA
<EMAIL_ADDRESS>
(2022)

###### Abstract.

As Human-Computer Interaction (HCI) research aims to be inclusive and
representative of many marginalized identities, there is still a lack of
available literature and research on intersectional considerations of race,
gender, and sexual orientation, especially when it comes to participatory
design. We aim to create a space to generate community recommendations for
effectively and appropriately engaging Queer, Transgender, Black, Indigenous,
People of Color (QTBIPOC) populations in participatory design, and discuss
methods of dissemination for recommendations. Workshop participants will
engage with critical race theory, queer theory, and feminist theory to reflect
on current exclusionary HCI and participatory design methods and practices.

††copyright: acmcopyright††journalyear: 2022††copyright:
rightsretained††conference: CHI Conference on Human Factors in Computing
Systems Extended Abstracts; April 29-May 5, 2022; New Orleans, LA,
USA††booktitle: CHI Conference on Human Factors in Computing Systems Extended
Abstracts (CHI ’22 Extended Abstracts), April 29-May 5, 2022, New Orleans, LA,
USA††doi: 10.1145/3491101.3503733††isbn: 978-1-4503-9156-6/22/04††ccs: Human-
centered computing Participatory design

##  1\. Background

The power inequalities in HCI research involving marginalized communities have
been investigated by researchers utilizing feminism (Bardzell and Bardzell,
2011), critical race theory (Ogbonnaya-Ogburu et al., 2020), decolonial theory
(Wong-Villacres et al., 2020; Alvarado Garcia et al., 2021; Cannanure et al.,
2021), and queer theory (Spiel et al., 2019). While the intersectionality
(Crenshaw, 1990) of race, gender, sexual orientation, and gender identity has
been explored in domains like law (Crenshaw, 2015), healthcare (Douglas et
al., 2019), and education (Schey, 2021), little work has been done on engaging
with Queer, Transgender, Black, Indigenous, People of Color (QTBIPOC)
populations in participatory design research.

Participatory design is a well-known and often utilized method in HCI research
(Schuler and Namioka, 1993), that typically aims to involve end-users and
other stakeholders (family member, employees, customers, partners, etc.) in
the design process. Harrington et al. describe participatory design as a
powerful tool for involving marginalized communities in design (Harrington et
al., 2019). However, to employ inclusive research practices requires that
researchers consider the social and cultural expectations associated with
participatory design methods and how they will impact the communities involved
in the design process, as well as the conventions for representing these
participants when reporting on research, which remain unexplored for the
QTBIPOC community.

In this workshop we focus specifically on how to effectively incorporate
intersectionality in participatory design. The central goal of
intersectionality is to increase the understanding of how an individual’s
social and political identities interact to influence their lived experiences
(Schlesinger et al., 2017). Incorporating intersectionality in participatory
design research will help advance our understanding of a participant’s
overlapping identities which inform their engagement with technology and their
environment. Representing the intersectionality of participants’ identities
and experiences in research and design in participatory design is imperative
to better serve marginalized communities, especially at the intersection of
the Queer-Trans (QT), Black, Indigenous, and People of Color (BIPOC)
communities.

The main structure of our workshop is being developed and facilitated by a
core group of organizers, supported by a steering committee that will help
guide discussions on the intersectionality of race, gender, and sexual
orientation in participatory design research. This committee includes
researchers who have engaged in participatory design with QTBIPOC populations
and community champions working at this intersection.

Throughout the workshop we aim to create opportunities for guided discussions
that will culminate in three distinct dissemination methods: (1) a website
that extends the workshop into a living venue where researchers and community
can consult results of the workshop and further engage in the future, (2) a
public forum (e.g. a Medium blog) that summarizes our discussions on more
inclusive categorizations of race, gender, and sexual orientation in research
studies, and elicits further and broader community discussion beyond the scope
of our workshop, and (3) a research paper outlining recommendations for
engaging in participatory design with QTBIPOC populations that can help the
HCI research community. Our workshop also provides researchers interested in
engaging QTBIPOC people in HCI research an opportunity to network and form a
community.

##  2\. Organizers

The organizers for this workshop have diverse backgrounds. Relevant to this
workshop, we disclose that we have queer and international organizers from
various backgrounds who are from countries in Asia, North America, South
America, and Europe. This group of organizers includes individuals who have
experience engaging in participatory design research with QTBIPOC populations.

Naba Rizvi (she/her), is a queer international student from Pakistan and Saudi
Arabia, a 2nd year Ph.D student at the University of California, San Diego,
and a 2020 NCWIT Collegiate Award recipient. Her research interests include
exploring the intersections of race, gender, disability, and sexual
orientation to guide inclusive participatory design research.

Reggie Casanova-Perez, MS (he/him), is a migrant trans non-binary 4th year
Ph.D student in Biomedical and Health Informatics at University of Washington.
His research interests focuses on making healthcare information systems more
inclusive to gender-diverse patients using feminism and queer theory
approaches.

Harshini Ramaswamy (she/her), is a 4th year South Asian American undergraduate
student studying Cognitive Science with a specialization in Design and
Interaction at the University of California, San Diego. Her research interests
include human-centered design and research in technology and healthcare
settings.

Lisa G. Dirks, MS, MLIS (she/her), is a Ph.D. candidate at the University of
Washington Information School. She is Alaska Native and White and grew up in a
rural Unangan (Aleut) community in the Western Aleutian Islands. A social and
information scientist, her research interests include health informatics,
human-information interaction, user experience, Indigenous health, research
dissemination, and community-engaged and participatory research.

Emily Bascom (she/her), is a 4th year undergraduate student studying
Informatics with a specialization in Human-Computer Interaction at the
University of Washington. Her research interests surround how information is
created, disseminated, and uptaken by the public and how to holistically
represent users in HCI research and informatics systems.

Nadir Weibel, PhD (he/him), is an Associate Professor in the Department of
Computer Science and Engineering at UC San Diego, and a Research Health
Science Specialist at the VA San Diego Health System. He is one of the faculty
of the UCSD Design Lab, Contextual Robotics Institute, Center for Population
and Health Systems, and the Research Center on Optimal Digital Ethics. His
work on Human-centered eXtended Intelligence is situated at the intersection
of computer science, design, and the health sciences, and strive to support a
diverse end-user population through user-centered and participatory design.

##  3\. Website

The workshop’s website (http://qtbipoc-hci.org) will contain all information
about our workshop, including how we solicit submissions. We include a Call
for Participation, information about the organizers, the workshop’s schedule
and outline, the list of accepted submissions and post-workshop documents, and
contact information. The website also serves as a repository for materials for
our reading group and post-workshop plans, including public outreach updates.
Accepted submissions regarding researchers’ experience engaging with the
QTBIPOC community will be available through our website with their authors’
consent.

##  4\. Pre-Workshop Plans

We will promote our workshop through our website and social media channels
(Facebook groups, Twitter, institutional Slack channels, listservs, etc.). To
broaden our pool of participants, we will also advertise the workshop with
relevant organizations that have contact with the QTBIPOC community. Accepted
submissions will be notified by March 20, 2022. Our submission form will
include a section for participants to state their accessibility needs so we
can provide accommodations as needed.

We will have two reading groups between the time when participants are
notified that they have been accepted and the workshop date. These reading
groups will require attendance at a one-hour remote session dedicated to
discussing readings sent to the participants beforehand to prepare
participants for the types of conversations that will be facilitated during
the workshop. Our steering committee will help with selecting appropriate
reading and resources. The goal of these conversations is to create
recommendations for considering the intersectionality of race, gender, and
sexual orientation in participatory design research. Suggested readings and
resources will include peer-reviewed literature and works published through
other mediums such as personal documents, Medium blogs, Ted Talks, and Zines.
The spectrum of source types will provide a deeper, holistic understanding of
both research and personal experiences related to this important subject
matter. Transcripts for the meetings will be provided to all participants for
accessibility and to review following the meeting.

##  5\. In-Person, Hybrid, or Virtual-Only

Our workshop participants will attend two video conferencing calls before the
workshop; each will be a 1-hour pre-workshop reading group discussion. During
our workshop at CHI 2022, participants will have the option to either attend
in-person or conference call in to the meeting. Additionally, attendees will
be able to access all written materials used for the workshops, including
recordings and transcripts from our reading groups, on our website following
each meeting. We will provide accommodations as specified and requested by
participants in the submissions form.

Since our goal is to engage a new community interested in HCI research with
QTBIPOC people, we will open up a hybrid (in-person or remote) workshop
participation to additional participants attending the CHI 2022 conference. We
hope that additional participants will be able to learn and contribute to the
workshop topic, and join the burgeoning community.

##  6\. Asynchronous Engagement

To facilitate asynchronous discussions before the workshop, we will set up
communication channels that are easy to join for our community participants
(e.g. through Slack, Discord, or Discourse). Additionally, all reading
materials, summaries, and notes from our discussions will be available to all
participants through our website. During online engagements, we will offer
live captioning and ensure our slideshow and other materials in the workshop
follow the W3C guidelines for accessibility.

##  7\. Workshop Structure

Table 2 shows our pre-workshop activities, while Table 1 displays the proposed
schedule for our workshop. Time will be allocated at the beginning of the
workshop to make brief introductions, elaborate on community guidelines and
expectations for participation in the workshop, and develop rapport with
participants to facilitate open discussion surrounding the intersectionality
of race, gender, and sexual orientation in HCI and design research.

We will provide a summary of the motivations, goals and desired outcomes of
the workshop. We will invite community champions and advocates working on
intersectionality, race, gender, and sexual orientation to share their
thoughts in short opening keynotes. We will then share the outcomes of our
reading groups, and invite participants to share their perspectives on current
or emerging challenges to be addressed in participatory design research in
relation to intersectionality. We will also include an overview of anti-
racist, queer, and feminist theories. We will then encourage discussion with
participants on whether current standards for inclusive HCI methodologies and
practices incorporate anti-racism, anti-queerphobia, and feminist theory.

Period | Activity
---|---
January-February 2022 | Steering Committee and Organizers select readings and materials
March 20th | Reading Assignments are sent to Participants
4 weeks prior to CHI 2022 Workshop | 1st online reading group meet-up: Race in HCI
2 weeks prior to CHI 2022 Workshop | 2nd online reading group meet-up: Gender and Sexual Orientation in HCI

Table 1. Pre-Workshop Activities

Time (PST) | Activity
---|---
9 am - 9.30 am | Introductions & Review of Workshop Goals
9.30 am - 10.15 am | Invited Community Keynotes
10.15 am - 11.00 am | Summary of Reading Group Discussions:
| -Race in HCI
| -Gender and sexual orientation in HCI
11.00 am - 11.30 am | Break
11.30 am - 12:15 pm | Problems and Opportunities Discussion:
| -Problems with design thinking methodology for the QTBIPOC community
| -How would HCI would benefit from using an intersectional approach
12:15 pm - 12.45 pm | Network and Discover
12:45 pm - 1:45 pm | Lunch Break
1.45 pm - 3:30 PM | Recommendations and Considerations Discussion:
|  -Recommendation generation for community-based participatory design with
the QTBIPOC community.
|  -Brainstorming for instrument development that is respectful of people’s
race, gender and sexual orientation, and their intersections.
3.30 pm - 4 pm | Closing

Table 2. Our proposed workshop schedule. Updated versions will be available on
our website.

111https://www.qtbipoc-pd.org/schedule

##  8\. Post-Workshop Plans

After the workshop, we will compile all material from the reading groups and
the workshop discussions to produce: (1) a website making workshop resources
available to the QTBIPOC community at large, (2) a Medium blog article for the
general public, and (3) a research paper for the academic community.

We expect to release recommendations on how to consider intersectionality in
participatory design research in the planned journal paper, and we will
transform our workshop website into a living community page where we will
disseminate our recommendations to interested community and research partners.

In addition to our dissemination plans, we expect that the workshop will
attract QTBIPOC researchers and offer networking opportunities within this
community. We will nurture these networking opportunities and facilitate them
through the participants’ chosen communication channel such as Slack or
Discord.

##  9\. Call for Participation

We invite participants interested in engaging with the intersection of race,
gender, and sexual orientation in participatory design. Our workshop seeks to
provide a safe space for scholars to share their experiences with
participatory design methodology that excludes the experiences of marginalized
participants. We have two goals: (1) to create recommendations for engaging in
participatory design research with the QTBIPOC community, and (2) determine
methods to disseminate the recommendations and reflections from the workshop.

Participants can join our workshop in two different ways:

  * •

Positionality Submissions \- Emerging QTBIPOC researchers who want to
contribute to the discussion can submit a positionality statement explaining
what being QTBIPOC means to them, challenges they may have encountered as a
QTBIPOC researcher, and how that has influenced their research interests.

  * •

Experience Submissions \- For researchers that have experience working with
QTBIPOC communities who may or may not be QTBIPOC themselves. This submission
will include a detailed explanation on their experience working with LGBTQ+
and/or BIPOC communities. The submissions do not need to focus on the
intersection of the two identities.

Submission deadline is February 24th 2022 (23:59 PT). Positionality
Submissions will not need to follow a specific format, whereas the Experience
Submissions must follow the standard CHI format. All submissions must be sent
as a PDF file using the Google Form available on the website.

Participants will be selected on the basis of the quality of their
submissions. At least one author of each accepted submission must attend the
workshop. The organizing committee can be contacted via email to answer
relevant questions.

## References

  * (1)
  * Alvarado Garcia et al. (2021) Adriana Alvarado Garcia, Juan F Maestre, Manuhuia Barcham, Marilyn Iriarte, Marisol Wong-Villacres, Oscar A Lemus, Palak Dudani, Pedro Reynolds-Cuéllar, Ruotong Wang, and Teresa Cerratto Pargman. 2021\.  Decolonial Pathways: Our Manifesto for a Decolonizing Agenda in HCI Research and Design. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–9.
  * Bardzell and Bardzell (2011) Shaowen Bardzell and Jeffrey Bardzell. 2011.  Towards a feminist HCI methodology: social science, feminism, and HCI. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 675–684.
  * Cannanure et al. (2021) Vikram Kamath Cannanure, Dilrukshi Gamage, Christian Sturm, Heike Winschiers-Theophilus, Juan Fernando Maestre, Naveena Karusala, Pedro Reynolds-Cuéllar, and Neha Kumar. 2021\.  Decolonizing HCI Across Borders. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–5.
  * Crenshaw (1990) Kimberle Crenshaw. 1990\.  Mapping the margins: Intersectionality, identity politics, and violence against women of color.  _Stan. L. Rev._ 43 (1990), 1241.
  * Crenshaw (2015) Kimberlé Crenshaw. 2015\.  Why intersectionality can’t wait.  _The Washington Post_ 24, 09 (2015), 2015.
  * Douglas et al. (2019) Courtland Douglas, Jannis Moody, and Dominique A Broussard. 2019\.  Black beyond the rainbow: Clinical implications for the intersectionality of race, gender, and queer identity.  _Journal of Black Sexuality and Relationships_ 5, 4 (2019), 21–41.
  * Harrington et al. (2019) Christina Harrington, Sheena Erete, and Anne Marie Piper. 2019\.  Deconstructing community-based collaborative design: Towards more equitable participatory design engagements.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–25.
  * Ogbonnaya-Ogburu et al. (2020) Ihudiya Finda Ogbonnaya-Ogburu, Angela DR Smith, Alexandra To, and Kentaro Toyama. 2020.  Critical race theory for HCI. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Schey (2021) Ryan Schey. 2021\.  Race and Queerness in the US Schooling System.  In _Oxford Research Encyclopedia of Education_.
  * Schlesinger et al. (2017) Ari Schlesinger, W Keith Edwards, and Rebecca E Grinter. 2017\.  Intersectional HCI: Engaging identity through gender, race, and class. In _Proceedings of the 2017 CHI conference on human factors in computing systems_. 5412–5427.
  * Schuler and Namioka (1993) Douglas Schuler and Aki Namioka. 1993.  _Participatory design: Principles and practices_.  CRC Press.
  * Spiel et al. (2019) Katta Spiel, Os Keyes, Ashley Marie Walker, Michael A DeVito, Jeremy Birnholtz, Emeline Brulé, Ann Light, Pınar Barlas, Jean Hardy, Alex Ahmed, et al. 2019\.  Queer (ing) HCI: Moving forward in theory and practice. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–4.
  * Wong-Villacres et al. (2020) Marisol Wong-Villacres, Adriana Alvarado Garcia, and Javier Tibau. 2020.  Reflections from the classroom and beyond: Imagining a decolonized hci education. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–14.