# A Hierarchical Interactive Network for Joint Span-based
Aspect-Sentiment Analysis

Wei Chen1,2, Jinglong Du3, Zhao Zhang4, Fuzhen Zhuang2,5∗, Zhongshi He1
1College of Computer Science, Chongqing University, Chongqing, China
2Institute of Artificial Intelligence, Beihang University, Beijing 100191,
China
3College of Medical Informatics, Chongqing Medical University, Chongqing,
China
4Institute of Computing Technology, Chinese Academy of Sciences, Beijing,
China
5SKLSDE, School of Computer Science, Beihang University, Beijing 100191, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding authors.

###### Abstract

Recently, some span-based methods have achieved encouraging performances for
joint aspect-sentiment analysis, which first extract aspects (aspect
extraction) by detecting aspect boundaries and then classify the span-level
sentiments (sentiment classification). However, most existing approaches
either sequentially extract task-specific features, leading to insufficient
feature interactions, or they encode aspect features and sentiment features in
a parallel manner, implying that feature representation in each task is
largely independent of each other except for input sharing. Both of them
ignore the internal correlations between the aspect extraction and sentiment
classification. To solve this problem, we novelly propose a hierarchical
interactive network (HI-ASA) to model two-way interactions between two tasks
appropriately, where the hierarchical interactions involve two steps: shallow-
level interaction and deep-level interaction. First, we utilize cross-stitch
mechanism to combine the different task-specific features selectively as the
input to ensure proper two-way interactions. Second, the mutual information
technique is applied to mutually constrain learning between two tasks in the
output layer, thus the aspect input and the sentiment input are capable of
encoding features of the other task via backpropagation. Extensive experiments
on three real-world datasets demonstrate HI-ASA’s superiority over
baselines111The source codes are released in https://github.com/cwei01/HI-
ASA..

A Hierarchical Interactive Network for Joint Span-based
Aspect-Sentiment Analysis

Wei Chen1,2, Jinglong Du3, Zhao Zhang4, Fuzhen Zhuang2,5∗, Zhongshi
He1††thanks:  Corresponding authors. 1College of Computer Science, Chongqing
University, Chongqing, China 2Institute of Artificial Intelligence, Beihang
University, Beijing 100191, China 3College of Medical Informatics, Chongqing
Medical University, Chongqing, China 4Institute of Computing Technology,
Chinese Academy of Sciences, Beijing, China 5SKLSDE, School of Computer
Science, Beihang University, Beijing 100191, China<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Aspect-sentiment analysis (ASA) Yan et al. (2021); Birjali et al. (2021) aims
at extracting all the aspects and their corresponding sentiments within the
text simultaneously. And it can be divided into two tasks, i.e., aspect
extraction (AE) and sentiment classification (SC). AE is to extract the
aspects Jakob and Gurevych (2010); Poria et al. (2016); Karimi et al. (2021)
in the sentence, and SC aims to predict the sentiments Jiang et al. (2011);
Lin et al. (2019); Karimi et al. (2021) for the extracted aspects.

In recent years, the span-based models Hu et al. (2019); Zhou et al. (2019);
Lin and Yang (2020); Lv et al. (2021) are increasingly becoming an alternative
for ASA because of their inherent advantages (e.g., they can avoid sentiment
inconsistency and huge search space problems in tagging-based models Luo et
al. (2019); Wang et al. (2021)), where the aspects are extracted by directly
predicting the boundary distributions, and the sentiment polarities are
classified based on the aspect-level words. For example, in the sentence “The
screen size is satisfactory but the phone battery capacity is limited.", the
aspect spans are “screen size” and “the phone battery”, and span-level
sentiments are positive and negative, respectively. Formally, most of the
existing methods can be divided into two types: sequential encoding Jebbara
and Cimiano (2016); Zhou et al. (2019) and parallel encoding Hu et al. (2019);
Lv et al. (2021) due to different ways in encoding task-specific features. In
sequential encoding, the task-specific features are extracted sequentially,
i.e., features extracted later have no direct associations with previous ones,
which is a unidirectional interaction strategy. In parallel encoding, task-
specific features are extracted independently except for using shared input,
i.e., the interaction is only present in input sharing. Hence, both encoding
methods above fail to model two-way interactions between AE and SC
appropriately.

In practice, the learning of the AE and SC may mutually influence each other.
On the one hand, sentiment words can be understood better if given the desired
aspects. For example, in the two sentences: “The battery has a large
capacity.” and “The computer case is too large to carry.”, we can find that
the sentiment word “large” expresses positive sentiment when describing the
aspect “hard drive” while negative sentiment towards “computer”. This actually
shows that incorporating the aspect features is conducive to SC. On the other
hand, since sentimental expressions are usually close to aspects, the
potential sentiment features may provide useful signals for AE. For example,
if we observe the word “spicy” appears in a restaurant review, it is likely to
exist an aspect related to food. Thus, incorporating the sentiment feature
also facilitates aspect detection. These two examples illustrate that it is
essential to reasonably establish the two-way associations between two tasks.

Inspired by the above analysis, in this work, we propose a novel Hierarchical
Interactive model for joint span-based Aspect-Sentiment Analysis (HI-ASA).
Specifically, the hierarchical interactions are achieved in two steps:
shallow-level interaction and deep-level interaction. The former learns
semantic-level interactive features to facilitate information sharing between
two tasks in encoding layer. And the latter takes advantage of the technique
of mutual information maximization to ensure two-way interaction between the
two tasks. Extensive experimental comparisons against state-of-the-art
solutions demonstrate the effectiveness of our proposed methods.

Figure 1: The architecture of our HI-ASA.

##  2 Model

In this section, we formally define the problem of joint span-based aspect-
sentiment analysis. Giving a training set
$\\{(\bm{x}_{i},\\{(s_{i}^{j},e_{i}^{j},p_{i}^{j})\\}_{j=1}^{l_{i}})\\}_{i=1}^{N}$,
in each sample
$(\bm{x}_{i},\\{(s_{i}^{j},e_{i}^{j},p_{i}^{j})\\}_{j=1}^{l_{i}})$,
$\bm{x}_{i}=\\{{x}_{i,1},{x}_{i,2},\ldots,{x}_{i,n_{i}}\\}$ is a
$n_{i}$-length sentence, and $(s_{i}^{j},e_{i}^{j},p_{i}^{j})$ identifies an
aspect with $s_{i}^{j}$ and $e_{i}^{j}$ as the start and end boundaries, and
$p_{i}^{j}$ as sentiment polarity. $l_{i}$ is the number of aspects in the
sentence.

The overall architecture is illustrated in Figure 1, which mainly consists of
two parts: AE and SC. We model the hierarchical interactions between two tasks
appropriately to enhance the correlations. For the input section, firstly, we
leverage BERT Devlin et al. (2018) to extract the semantic information for
input sentence. The output through the transformer layers is
$\bm{B}=\\{b_{1},b_{2},\ldots,b_{n}\\}\in\mathbb{R}^{n\times d}$, where $d$ is
the BERT’s embedding size, $n$ is the length of sentence. Then, we stack GRUs
upon $\bm{B}$ for different tasks, here, we define the outputs of AE and SC as
${\bm{H}}_{a}^{0}$ $\in$ $\mathbb{R}^{n\times\hat{d}}$ and ${\bm{H}}_{s}^{0}$
$\in$ $\mathbb{R}^{n\times\hat{d}}$, respectively, where $\hat{d}$ is the
dimension size of GRU.

###  2.1 Shallow-Level Interaction

In previous models Hu et al. (2019); Lin and Yang (2020); Lv et al. (2021),
the feature representations are extracted independently except for using
shared input. In other words, two tasks have no associations with each other,
which is not in line with human ituition. Intuitively, the features of SC are
not only derived from its own features of previous layers, but also come from
the features of AE, and vice versa. Therefore, we design a shallow-level
interaction strategy inspired by the idea of cross-stitch mechanism Misra et
al. (2016). Its core is to selectively combine the different features to
reasonably model two-way interactions. Specifically, the calculation is as
follows:

$\left[\begin{array}[]{c}{\bm{H}}^{1}_{a}\\\
{\bm{H}}^{a}_{s}\end{array}\right]=\left[\begin{array}[]{cc}\bm{\gamma}_{\mathrm{aa}}&\bm{\alpha}_{\mathrm{sa}}\\\
\bm{\alpha}_{as}&\bm{\gamma}_{\mathrm{ss}}\end{array}\right]\left[\begin{array}[]{c}{\bm{H}}^{0}_{a}\\\
{\bm{H}}^{0}_{s}\end{array}\right]$ (1)

where $\bm{\gamma}_{aa}$, $\bm{\gamma}_{ss}$ are the task-specific parameters,
and $\bm{\alpha}_{sa}$, $\bm{\alpha}_{as}$ are task-sharing parameters.
${\bm{H}}_{a}^{1}$ and ${\bm{H}}_{s}^{1}$ are the output features of AE and SC
in the last encoding layer, respectively. We can observe that the cross-stitch
unit is beneficial for the interaction between AE and SC features. To speed up
the training process, we define the following constraints, (1):
$\bm{\gamma}_{aa}=\bm{\gamma}_{ss}$ , $\bm{\alpha}_{sa}=\bm{\alpha}_{as}$ (2):
$\bm{\gamma}_{aa}+\bm{\alpha}_{sa}=1$. Thus, the Eq.1 is simplified to:

$\displaystyle\boldsymbol{H}_{a}^{1}=\alpha\cdot\boldsymbol{H}_{s}^{0}+(1-\alpha)\cdot\boldsymbol{H}_{a}^{0}$
(2)
$\displaystyle\boldsymbol{H}_{s}^{0}=\alpha\cdot\boldsymbol{H}_{a}^{0}+(1-\alpha)\cdot\boldsymbol{H}_{s}^{0}$

where $\alpha$ controls information transferred from the other task.

###  2.2 Deep-Level Interaction

In the last section, we model the two-way interactions from the encoding
layer. Although some semantic-level associations can be captured, we believe
that task-level associations cannot be adequately modeled. To better
understand our proposed method, we give an intuitive explanation in Figure 2.
We observe the span-based aspect extracion models put higher scores on the
entity words (e.g., “Windows” ,“7” ). But with aspect-level sentiment
learning, higher attention weights are not only put on some polarity words
(e.g., “love”), but also on the entity words. Therefore, it would be
interesting if we perform aspect-sentiment mutual learning on both tasks.

Figure 2: The example of our mutual information maximization strategy in
sentence “I love Windows 7.”.

Thus, we propose the deep-level interactive strategy based on mutual
information maximization Kong et al. (2019), where the task-level information
is shared by the two tasks in the output layer. We will explain in detail how
to obtain the aspect scores (i.e., the more a word acts like an aspect word,
the higher its score) and attention scores for these two tasks.

Aspect Extraction ($\mathcal{E}$). We extract the aspects by predicting the
boundaries, i.e., start position and end position. Specifically, we employ a
linear classifier to predict the start scores
$g_{s}=\operatorname{sigmoid}(\bm{V_{s}}\bm{H}_{a}^{1})$, i.e., $\bm{V_{s}}$
is a trainable vector, and end scores $g_{e}$ can be derived in a similar
manner. During training phase, we define the learning object:
$\mathcal{J}_{ae}\\!\\!=\\!\\!-\sum_{i=1}^{n}{\bm{p}}_{s}^{i}\log{(\bm{g^{i}}_{s})}-\sum_{i=1}^{n}{\bm{p}}_{e}^{i}\log{(\bm{g^{i}}_{e})}$,
where $\bm{p}_{s}^{i}$ and $\bm{p}_{e}^{i}$ are the ground truths of the
boundaries. $\bm{g}_{s}^{i}$ and $\bm{g}_{e}^{i}$ are the predicted
distributions.

For the aspect score $\mathcal{E}$, we cannot use the boundary distributions
of aspect (i.e., $g_{s}$ and $g_{e}$) directly due to their different
meanings. In fact, the span of aspect is relatively short, we can use the
average boundary distributions to approximate the aspect scores, so we have:
$\mathcal{E}=\frac{\operatorname{pooling}\left({g_{s}}\right)+\operatorname{pooling}\left({g_{e}}\right)}{2}$,
where $\operatorname{pooling(\cdot)}$ is the mean-pooling function, in our
work, our set window size as $1\times 3$. Therefore, suppose the start score
$g_{s}$ is [0.2,0.3,0.7,0.9,0.1], the first item of
$\operatorname{pooling}\left({g_{s}}\right)$ is $\frac{0+0.2+0.3}{3}=0.17$,
and the $\operatorname{pooling}\left({g_{s}}\right)$ is
[0.17,0.4,0.63,0.57,0.33], such strategy can guarantee that the aspect
distribution approximates the boundary distributions as much as possible.

Sentiment Classification ($\mathcal{A}$). In aspect-level sentiment
classification, the attention weights of words produced during the training
towards the specific aspects could reveal word-polarity information. Thus, we
can leverage the attention distribution as the sentiment features to help
detect the boundaries of aspects. To this end, we choose the over-and-over
attention model Huang et al. (2018) to implement the sentiment classification.
Here, we directly define the attention distribution as $\mathcal{A}$, (i.e.,
$\mathcal{A}$ is attention scores in Figure 2). During training phase, the
optimization goal for SC is:
$\mathcal{J}_{sc}=-\sum_{i=1}^{N}\boldsymbol{y}_{i}^{T}\log\hat{\boldsymbol{y}}_{i}$,
where $\hat{\bm{y}}_{i}$ is the predicted sentiment distribution, and
${\bm{y}}_{i}^{T}$ is the corresponding ground truth, $N$ is the number of
samples.

Mutual Information Maximization. After getting the aspect distribution
$\mathcal{E}$ and sentiment attention distribution $\mathcal{A}$. As discussed
above, it should be interesting if we perform aspect-sentiment mutual learning
on both tasks. In Figure 2, we can find that the aspect scores and attention
weights are both improved after applying the mutual learning technique. We
need to maximize the similarity between two distributions during the training.
An intuitive idea is to use Kullback Leibler (KL) divergence to measure the
distance between distributions, considering the correlations are
bidirectional, we define the following similarity measurement metrics using
Jenson Shannon (JS) divergence:

$JS(\mathcal{E}\|\mathcal{A})=\frac{1}{2}KL\left(\mathcal{E}\|\frac{\mathcal{E}+\mathcal{A}}{2}\right)+\frac{1}{2}KL\left(\mathcal{A}\|\frac{\mathcal{E}+\mathcal{A}}{2}\right)$
(3)

where the JS divergence is symmetric, thus we can leverage it to constrain two
tasks to learn from each other, resulting in balanced interaction between AE
and SC. Whereas in sequential and parallel encoding, sentiment features have
no direct impact on the information of aspect features. Finally, considering
the mutual information maximization between two tasks, HI-ASA is optimaled by
combining three sections:

$\vspace{-0.2cm}\underset{\theta}{\arg\min}\left(\mathcal{J}_{ae}+\mathcal{J}_{sc}+\beta\cdot
JS(\mathcal{E}\|\mathcal{A})\right)\vspace{-0.15cm}$ (4)

where $\theta$ is the set of model’s parameters, $\mathcal{J}_{ae}$ and
$\mathcal{J}_{sc}$ are the optimization goals of AE and SC respectively.
$\beta$ is a balanced parameter. During the test phase, HI-ASA outputs two
parts, one is the scores of boundaries, based on which we leverage the
heuristic algorithm Lin and Yang (2020) to extract all the aspects; the other
one is corresponding sentiment polarities of extracted aspects.

Dataset | #Sentences | #Aspects | #+ | #- | #0
---|---|---|---|---|---
Restaurant | 3900 | 6603 | 4134 | 1538 | 931
Laptop | 1869 | 2936 | 1326 | 900 | 620
Tweets | 2350 | 3243 | 703 | 274 | 2266
Table 1: Statistics of the datasets. “+/-/0” refer to the positive, negative, and neutral sentiment classes. Span-based Models |  | Laptop
---
| Res
---
| Tweets
---
Zhou Zhou et al. (2019) | 59.76 | 71.98 | 51.44
Hu-pipeline Hu et al. (2019) | 68.06 | 74.92 | 57.69
Hu-joint Hu et al. (2019) | 64.59 | 72.47 | 54.55
Hu-collapsedHu et al. (2019) | 48.66 | 57.85 | 48.11
SPRM Lin and Yang (2020) | 68.72 | 79.17 | 59.45
S-AESC Lv et al. (2021) | 65.88 | 74.18 | 54.73
HI-ASA | 70.39 | 79.90 | 60.36
\- w/o SI | 68.05 | 79.10 | 60.04
\- w/o DI | 67.45 | 79.50 | 60.13
Table 2: The performance (F1-score) comparisons with different methods.
Baseline results are retrieved from published papers.

##  3 Experiments

###  3.1 Datasets and Settings

Datasets. In our experiments, we use three public datasets, including Laptop
Pontiki et al. (2014), Restaurant Pontiki et al. (2014, 2015, 2016) and Tweets
Mitchell et al. (2013), which have been widely used in previous works Lv et
al. (2021). The statistic details of experimental dataset are refer to Table
1. For each sentence in datasets, the gold span boundaries and sentiment
polarity labels are available. Specifically, Restaurant is the union set of
the restaurant domain from SemEval2014, SemEval2015 and SemEval2016. Laptop
contains costumer reviews in the electronic product domain, which is collected
from SemEval Challenge. Tweets is composed of twitter posts from different
users.

Settings. In the experiments, the commonly used metric F1-score (F1) is
selected to evaluate for aspect-sentiment analysis, accuracy is applied to
sentiment classification. A correct predicted aspect only when it matches the
gold aspect and the corresponding polarity. We utilize the BERT-Large model as
the backbone network,where the number of transformers is 24 and the hidden
size is 784. In addition, we use Adam optimizer with a learning rate of 3e-5,
the batch size is 32 and the dropout probability of 0.1 is used.

###  3.2 Main Results

The comparisons between HI-ASA and the baselines are presented in Table 2.
Specifically, “- w/o SI” means we remove the shallow-interaction layer, “- w/o
DI” denotes removing the deep-interaction layer. We can observe: (1) Overall,
our proposed HI-ASA consistently achieves the best F1 scores across all the
baselines. More specifically, compared to the state-of-the-art approach SPRM,
HI-ASA improves the performance by about 1.67%, 0.63%, and 0.91% on three
datasets, respectively. These observations indicate the carefully designed HI-
ASA is capable of achieving better performances. The reason can be concluded
two folds: one is that we selectively combine the task-specific features of
each task to reasonably model two-way interactions in the encoding layer; the
second benefit is that we model a balanced task-level interactions. Under this
framework, aspect and sentiment associations are captured appropriately in a
hierarical manner. (2) Besides, we investigate the effectiveness of each
single module, i.e., “- w/o SI” and “- w/o DI”. We can see that when a certain
module is removed, the performance of our model decreases, which indicates the
indispensability of each module.

###  3.3 Ablation Study

Aspect Extraction |  | Laptop
---
| Res
---
| Tweets
---
Hu Hu et al. (2019) | 83.35 | 82.38 | 75.28
SPRM Lin and Yang (2020) | 84.72 | 86.71 | 69.85
S-AESC Lv et al. (2021) | 85.19 | 84.20 | 76.04
HI-ASA | 86.30 | 86.93 | 76.36
\- w/o SI | 84.50 | 87.33 | 75.81
\- w/o DI | 85.10 | 87.09 | 75.99
Sentiment Classifiction | Laptop | Res | Tweets
Hu Hu et al. (2019) | 81.39 | 89.95 | 75.16
SPRM Lin and Yang (2020) | 81.50 | 90.35 | 78.34
HI-ASA | 85.02 | 93.18 | 83.50
\- w/o SI | 84.07 | 92.53 | 82.90
\- w/o DI | 83.75 | 92.62 | 83.49
Table 3: The top part is the performance (F1-score) comparisons with different methods on aspect extraction. And the bottom part is the performance (Accuracy) comparisons with different methods on sentiment classifiction. Dataset$/$$\alpha$ | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
Laptop | 68.05 | 70.39 | 67.62 | 68.63 | 68.58 | 70.36
Restaurant | 79.14 | 78.93 | 79.73 | 79.45 | 79.90 | 78.48
Tweet | 60.04 | 60.36 | 59.70 | 59.89 | 58.79 | 58.19
Table 4: The performance (F1-score) of different sharing ratios ($\alpha$) on
three datasets.

In this section, we go deeper into HI-ASA and analyse the results on both
tasks. The results are shown in Table 3. Generally, we observe that HI-ASA
outperforms baseline competitors on both tasks, which indicates the
effectiveness of the two-way interaction on both aspect extraction and
sentiment classification. For the AE task, HI-ASA can enhance the performance
of most of the baselines. On the SC task, HI-ASA outperforms SPRM by 3.98%,
2.83%, and 5.16% on three datasets, respectively, which implies that it is
more able to boost the performance of SC compared to AE.

###  3.4  Parameter Analysis $\alpha$

In HI-ASA, the parameter $\alpha$ controls information transfer between two
tasks. For a special case, if $\alpha$ = 0, our approach will degenerate to
the parallel encoding. We tune $\alpha$ in the range of
[0.0,0.1,0.2,0.3,0.4,0.5] and the results are presented in Table 4. It is
worth noting that we set the value of $\alpha$ below 0.5. This setting is
inspired by the ituition that the features for each task should come more from
the task itself, rather than the other task. We can notice the best results
are achieved when $\alpha$ is 0.1, 0.4, and 0.1 on three datasets,
respectively, rather than 0.0. This actually shows our interaction is
successful.

Furthermore, we find an interesting phenomenon is that $\alpha$ is small
(i.e., 0.1) on Laptop and Tweets, and large on Restaurant when performing the
best performance. We conjecture the reason lies in that Restaurant has more
samples than the other two datasets, requiring more knowledge interactions for
a better learning process.

##  4 Related Work

Aspect-sentiment Analysis (ASA) Yan et al. (2021); Birjali et al. (2021) is an
essential task in sentiment analysis and can be separated into two tasks,
i.e., aspect extraction (AE) and sentiment classification (SC). Over the past
years, some span-based methods have achieved promising results for ASA, which
first extract aspects by detecting aspect boundaries (AE) and then predict the
span-level sentiments (SC). AE has been widely studied by traditional machine
Jakob and Gurevych (2010) and deep learning algorithms Karimi et al. (2021).
However, the absence of sentiment information may result in redundant and
noisy detection. SC is to predict the sentiment expressed on some specific
aspects in a sentence, which has been studied extensively in NLP community
Tang et al. (2015); Chen et al. (2019, 2020); Karimi et al. (2021). However,
these aspects must be annotated before the AE task. Hu et al. (2019) proposed
an extract-then-classify framework, which extracted aspects with a heuristic
decoding algorithm and then correspondingly classified the span-level
sentiments. Lin and Yang (2020) designed share-private representation for each
task to capture the correlations between two tasks in encoding layer. Besides,
Zhou et al. (2019) introduced a joint model based on span-aware attention
mechanism to predict the sentiment polarity. Although achieved improved
performances, these works fail to model two-way interactions between AE and SC
appropriately.

##  5 Conclusion

In this paper, we proposed a novel aspect-sentiment analysis model named HI-
ASA. The proposed model is equipped with a hierarchical interactive network to
facilitate information sharing between the aspect extraction task and the
sentiment classification task. The experimental results on three benchmark
datasets demonstrated HI-ASA’s effectiveness and generality.

## Acknowledgments

We are grateful to all the anonymous reviewers for their helpful advice on
various aspects of this work. The research work is supported by the National
Key Research and Development Program of China under Grant No. 2021ZD0113604,
the National Natural Science Foundation of China under Grant No. 62176014,
Graduate Research and Innovation Foundation of Chongqing, China under Grant
No.CYC21072.

## References

  * Birjali et al. (2021) Marouane Birjali, Mohammed Kasri, and Abderrahim Beni-Hssane. 2021.  A comprehensive survey on sentiment analysis: Approaches, challenges and trends.  _Knowledge-Based Systems_ , 226:107134.
  * Chen et al. (2020) Wei Chen, Wenxin Yu, Gang He, and Ning Jiang. 2020.  Coarse-to-fine attention network via opinion approximate representation for aspect-level sentiment classification.  In _International Conference on Neural Information Processing_ , pages 704–715. Springer.
  * Chen et al. (2019) Wei Chen, Wenxin Yu, Zhiqiang Zhang, Yunye Zhang, Kepeng Xu, Fengwei Zhang, Yibo Fan, Gang He, and Zhuo Yang. 2019.  Target-based attention model for aspect-level sentiment analysis.  In _International Conference on Neural Information Processing_ , pages 259–269. Springer.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Hu et al. (2019) Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019.  Open-domain targeted sentiment analysis via span-based extraction and classification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 537–546.
  * Huang et al. (2018) Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018.  Aspect level sentiment classification with attention-over-attention neural networks.  In _International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation_ , pages 197–206.
  * Jakob and Gurevych (2010) Niklas Jakob and Iryna Gurevych. 2010.  Extracting opinion targets in a single and cross-domain setting with conditional random fields.  In _Proceedings of the 2010 conference on empirical methods in natural language processing_ , pages 1035–1045.
  * Jebbara and Cimiano (2016) Soufian Jebbara and Philipp Cimiano. 2016.  Aspect-based sentiment analysis using a two-step neural network architecture.  In _Semantic Web Evaluation Challenge_ , pages 153–167. Springer.
  * Jiang et al. (2011) Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011.  Target-dependent twitter sentiment classification.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 151–160.
  * Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021.  Adversarial training for aspect-based sentiment analysis with bert.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 8797–8803. IEEE.
  * Kong et al. (2019) Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2019.  A mutual information maximization perspective of language representation learning.  In _International Conference on Learning Representations_.
  * Lin and Yang (2020) Peiqin Lin and Meng Yang. 2020.  A shared-private representation model with coarse-to-fine extraction for target sentiment analysis.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 4280–4289.
  * Lin et al. (2019) Peiqin Lin, Meng Yang, and Jianhuang Lai. 2019.  Deep mask memory network with semantic dependency and context moment for aspect level sentiment classification.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 5088–5094.
  * Luo et al. (2019) Huaishao Luo, Tianrui Li, Bing Liu, and Junbo Zhang. 2019.  Doer: Dual cross-shared rnn for aspect term-polarity co-extraction.  _arXiv preprint arXiv:1906.01794_.
  * Lv et al. (2021) Yanxia Lv, Fangna Wei, Ying Zheng, Cong Wang, Cong Wan, and Cuirong Wang. 2021.  A span-based model for aspect terms extraction and aspect sentiment classification.  _Neural Computing and Applications_ , 33(8):3769–3779.
  * Misra et al. (2016) Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016.  Cross-stitch networks for multi-task learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3994–4003.
  * Mitchell et al. (2013) Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013\.  Open domain targeted sentiment.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1643–1654.
  * Pontiki et al. (2014) Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  Semeval-2014 task 4: Aspect based sentiment analysis.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 27–35.
  * Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, AL-Smadi Mohammad, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016.  Semeval-2016 task 5: Aspect based sentiment analysis.  In _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_ , pages 19–30.
  * Pontiki et al. (2015) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015.  Semeval-2015 task 12: Aspect based sentiment analysis.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 486–495.
  * Poria et al. (2016) Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016.  Aspect extraction for opinion mining with a deep convolutional neural network.  _Knowledge-Based Systems_ , 108:42–49.
  * Tang et al. (2015) Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015.  Effective lstms for target-dependent sentiment classification.  _arXiv preprint arXiv:1512.01100_.
  * Wang et al. (2021) Xinyi Wang, Guangluan Xu, Zequn Zhang, Li Jin, and Xian Sun. 2021.  End-to-end aspect-based sentiment analysis with hierarchical multi-task learning.  _Neurocomputing_ , 455:178–188.
  * Yan et al. (2021) Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021.  A unified generative framework for aspect-based sentiment analysis.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language_ , pages 2416–2429.
  * Zhou et al. (2019) Yan Zhou, Longtao Huang, Tao Guo, Jizhong Han, and Songlin Hu. 2019.  A span-based joint model for opinion target extraction and target sentiment classification.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 5485–5491.


# Unlocking the Potential of Model Merging for Low-Resource Languages

Mingxu Tao${}^{\text{1,\,2}}$, Chen Zhang${}^{\text{1}}$11footnotemark: 1,
Quzhe Huang${}^{\text{1}}$11footnotemark: 1, Tianyao Ma${}^{\text{1}}$
Songfang Huang${}^{\text{3}}$, Dongyan Zhao${}^{\text{1,\,2}}$, Yansong
Feng${}^{\text{1\,{\char 0\relax}}}$
${}^{\text{1}}$Wangxuan Institute of Computer Technology, Peking University
${}^{\text{2}}$Center for Data Science, Peking University
${}^{\text{3}}$College of Engineering, Peking University
<EMAIL_ADDRESS>
Equal contributions.

###### Abstract

Adapting large language models (LLMs) to new languages typically involves
continual pre-training (CT) followed by supervised fine-tuning (SFT). However,
this CT-then-SFT approach struggles with limited data in the context of low-
resource languages, failing to balance language modeling and task-solving
capabilities. We thus propose model merging as an alternative for low-resource
languages, combining models with distinct capabilities into a single model
without additional training. We use model merging to develop task-solving LLMs
for low-resource languages without SFT data in the target languages. Our
experiments based on Llama-2-7B demonstrate that model merging effectively
endows LLMs for low-resource languages with task-solving abilities,
outperforming CT-then-SFT in scenarios with extremely scarce data. Observing
performance saturation in model merging with more training tokens, we further
analyze the merging process and introduce a slack variable to the model
merging algorithm to mitigate the loss of important parameters, thereby
enhancing performance. We hope that model merging can benefit more human
languages suffering from data scarcity with its higher data efficiency.

Unlocking the Potential of Model Merging for Low-Resource Languages

Mingxu Tao${}^{\text{1,\,2}}$††thanks: Equal contributions., Chen
Zhang${}^{\text{1}}$11footnotemark: 1, Quzhe
Huang${}^{\text{1}}$11footnotemark: 1, Tianyao Ma${}^{\text{1}}$ Songfang
Huang${}^{\text{3}}$, Dongyan Zhao${}^{\text{1,\,2}}$, Yansong
Feng${}^{\text{1\,{\char 0\relax}}}$ ${}^{\text{1}}$Wangxuan Institute of
Computer Technology, Peking University ${}^{\text{2}}$Center for Data Science,
Peking University ${}^{\text{3}}$College of Engineering, Peking University
<EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) demonstrate remarkable capabilities across
various NLP tasks, owing to the vast amounts of high-quality training data
Touvron et al. (2023); Bai et al. (2023). However, developing models with
task-solving abilities for low-resource languages remains challenging due to
limited data availability.

A common practice for constructing task-solving LLMs for a low-resource
language involves continual pertaining (CT) and supervised fine-tuning (SFT)
for the target language Yong et al. (2023); Nguyen et al. (2023), known as CT-
then-SFT. The scarcity of CT data impedes LLMs’ ability to learn effective
language modeling for these target languages. Additionally, it is difficult to
acquire sufficient SFT data in low-resource languages to enhance downstream
task performance. To address this issue, previous works attempt to transfer
capabilities from high-resource languages to low-resource languages by
training on English SFT data Chirkova and Nikoulina (2024); Shaham et al.
(2024). However, this approach can lead to catastrophic forgetting Thrun
(1998); Chen and Liu (2018) of language modeling for the target languages
Mehta et al. (2021); Kotha et al. (2024), resulting in LLMs still failing to
solve tasks due to the loss of language abilities.

To better integrate the language modeling and task-solving capabilities, we
introduce model merging for low-resource languages, which can combine multiple
models with distinct abilities into a single model without additional
training. Previous work Akiba et al. (2024) has shown that an LLM for high-
resource languages can be merged with task-specific models, such as Japanese
language models and math models. In this work, we explore whether model
merging can effectively construct task-solving LLMs for low-resource
languages. Specifically, we investigate the following research questions: RQ1:
What is the viability of constructing task-solving LLMs in low-resource
languages through model merging? RQ2: Is model merging always a better choice
than CT-SFT? RQ3: What factors may affect LLMs in obtaining task-solving
capabilities through model merging?

To answer these questions, we study the adaptation of Llama-2-7B Touvron et
al. (2023), an English-centric LLM, into seven distinct low-resource
languages. We first continually pre-train Llama-2-7B on monolingual texts in
each language. Next, we explore two approaches to inject task-solving
capabilities into this continually pre-trained model: (1) training the LLM
with English SFT data or the data translated to the target low-resource
language; (2) merging the model with an English task-solving LLM. Experiments
show that model merging can effectively equip the CT models with task-solving
capabilities. Notably, when pre-training corpora in the target language are
extremely scarce (¡10B tokens), model merging outperforms CT-then-SFT. As an
LLM is pre-trained with more tokens, the improvements brought by model merging
gradually saturate. Then, model merging can no longer significantly surpass
the SFT method.

To further investigate the factors impeding the continuous improvement of
model merging, we conduct a detailed analysis of the process of merging two
LLMs. We find that when an English SFT model is merged with an LLM continually
pretrained with more tokens in the target language, more parameters from the
SFT model are discarded during merging. The loss of these parameters may lead
to a decline in task-solving capabilities, preventing the merged model from
improving performance on downstream tasks. To mitigate the loss of important
parameters from the SFT model, we propose model merging with slack variables.
This strategy allows for more flexible control over the merging process to
retain the important parameters.

Our contributions are as follows: (1) We are the first to introduce model
merging to construct task-solving LLMs for low-resource languages; (2) We
reveal that model merging is more effective than SFT in the scenarios of
extremely low-resource languages; (3) Through a quantitative study of the
merging process, we explain the performance plateau of model merging with a
larger CT corpus and propose a simple yet effective enhancement to popular
model merging algorithms.

##  2 Related Works

#### Model Merging

Model merging is a promising way to combine the abilities of multiple models.
Pioneering works explore strategies to find the best weights for averaging
Choshen et al. (2022); Wortsman et al. (2022); Matena and Raffel (2022); Jin
et al. (2022). Task Arithmetic Ilharco et al. (2022) employs task vectors,
enabling control through arithmetic operations to steer the merged model’s
behavior. TIES Yadav et al. (2023) further addresses the problem of
information loss by handling parameter conflict more carefully. DARE Yu et al.
(2023) zeros out redundant parameters and amplifies the remaining ones.
Evolutionary Model Merge Akiba et al. (2024) automatically discovers optimal
model combinations through evolutionary algorithms.

There is little discussion of model merging in the context of multilinguality.
Instead, previous works attempt to introduce language-specific and task-
specific modular adapters Pfeiffer et al. (2020); Parovic et al. (2023);
Parović et al. (2024); Zhao et al. (2024), which require additional training.
These works focus on high-resource languages and specific tasks. In contrast,
model merging can utilize existing models without additional training, making
it a versatile approach for building more general task-solving LLMs. Besides,
we are the first to study model merging for low-resource languages.

#### LLMs for Low-Resource Languages

There is a line of works aiming to adapt LLMs to underrepresented human
languages. A common practice is continually pre-training existing LLMs on the
corpus in the target languages Yong et al. (2023); Nguyen et al. (2023); Zhang
et al. (2024b). To improve the efficiency of training, previous works adopt
techniques such as adapters Pfeiffer et al. (2020), script conversion Micallef
et al. (2024), integration of similar languages Senel et al. (2024).

Following pre-training, LLMs typically undergo supervised fine-tuning to
acquire task-solving capabilities Muennighoff et al. (2023); Nguyen et al.
(2023). To address the data scarcity in this step, researchers have employed
various methods to collect SFT data, including crowd-sourcing Singh et al.
(2024), machine translation Muennighoff et al. (2023); Li et al. (2023a), LLM
distillation Li et al. (2024) ,rule-based conversion Cahyawijaya et al.
(2023), et al. However, these methods are not without limitations,
particularly in terms of cost, data quality, and generalizability. In
contrast, the model merging paradigm studied in our work eliminates the need
for expensive and potentially error-prone SFT data collection by leveraging
pre-trained task-solving models from high-resource languages.

Figure 1: Roadmap towards task-solving LLMs for low-resource languages.

##  3 Model Merging for Low-Resource Languages

The conventional CT-then-SFT paradigm struggles to balance language modeling
and task-solving abilities in the context of low-resource languages. We
propose model merging as an alternative, which can construct task-solving LLMs
for low-resource languages without requiring SFT data in the target languages.

###  3.1 Preliminary: Model Merging

Model merging is a technique for combining multiple models possessing
different capabilities into a single versatile model without additional
training. For example, we can merge a model specialized for Japanese and a
model specialized for math to obtain a model that excels at solving
mathematical problems in Japanese Akiba et al. (2024). In this work, we
investigate two commonly-used methods of model merging: weighted averaging
Choshen et al. (2022); Wortsman et al. (2022) and TIES Yadav et al. (2023).
Here we provide a brief overview of these methods.

Weighted averaging is simply averaging the parameters of two models with a
weight tuned on the validation set.

TIES aims to handle the parameter conflicts across multiple models more
meticulously. Suppose we have two models specialized for distinct tasks,
denoted as $\theta_{1}$ and $\theta_{2}$, both trained from the same initial
model $\theta_{\text{init}}$. Task vectors for these models are calculated as
follows: $\tau_{1}=\theta_{1}-\theta_{\text{init}}$ and
$\tau_{2}=\theta_{2}-\theta_{\text{init}}$. The objective is to merge these
task vectors and reintegrate them into the initial model.

The merging process of TIES consists of three steps: (1) Trim: For $\tau_{1}$
and $\tau_{2}$, we trim the redundant parameters by keeping the top-$k_{1}$%
and top-$k_{2}$% values, respectively, creating $\hat{\tau}_{1}$ and
$\hat{\tau}_{2}$. (2) Elect Signs: For each parameter $p$ in $\hat{\tau}_{1}$
and $\hat{\tau}_{2}$, we select the sign (+1 or -1) with the higher magnitude,
denoted as $\gamma^{p}=\mathrm{sgn}(\hat{\tau}_{1}^{p}+\hat{\tau}_{2}^{p})$.
(3) Disjoint Merge: For each parameter $p$, we only keep the parameter values
from $\hat{\tau}_{1}$ and $\hat{\tau}_{2}$ whose signs are the same as the
aggregated elected sign and calculate their mean. Specifically, for each
parameter $p$, its disjoint mean is calculated as
$\tau_{m}^{p}=\mathrm{avg}(S^{p})$, where
$S^{p}=\\{\hat{\tau}_{i}^{p}|\mathrm{sgn}(\hat{\tau}_{i}^{p})=\gamma^{p},i=1,2\\}$.

Given the final merged task vector $\tau_{m}$, we scale it and add it to the
initial model $\theta_{\text{init}}$ to obtain the merged model $\theta_{m}$
as $\theta_{m}=\theta_{\text{init}}+\lambda\cdot\tau_{m}$, where $\lambda$ is
a scaling hyperparameter.

For TIES, we tune three hyperparameters in total on the validation set: two
sparsity rates $k_{1}$, $k_{2}$ and a scaling factor $\lambda$.

Please refer to the original paper of TIES Yadav et al. (2023) for more
details.

###  3.2 Roadmap Towards LLMs for Low-Resource Languages

Given a base model only pre-trained on an English-centric corpus, e.g.,
Llama-2-7B Touvron et al. (2023) in our study, we want to construct a model
capable of solving tasks in a low-resource language. For the target language,
there are very limited pre-training texts, ranging from 1B to 20B tokens, and
almost no data for supervised finetuning (SFT). In this scenario, we
investigate two representative paradigms of constructing such a model: CT-
then-SFT and model merging. We illustrate the roadmap in Figure 1, which
demonstrates the relations between the models.

#### Conventional Practice: CT-then-SFT

The common practice is (1) first continual pre-training (CT) on the
monolingual texts in the target language X to learn the language modeling and
(2) then learning task-solving abilities through SFT Yong et al. (2023);
Nguyen et al. (2023). This approach is referred to as CT-then-SFT. In this
approach, we have the following models:

BASE: We employ the original Llama-2-7B without SFT as the base LLM.

CT-X: We continually pre-train BASE on the corpus in the target language X.
Following previous works d’Autume et al. (2019); Tao et al. (2023), we add 1/4
English corpus for memory replay, to avoid catastrophic forgetting English
language modeling.

CTSFT-X: We train CT-X with SFT data to enhance its task solving ability.
There are two variants using different SFT data:

(1) CTSFT-X-flan: We finetune CT-X with English SFT data, which includes the
original FLAN datasets and the training set of GSM8K111 Since the whole
instruct-tuning datasets contain over 160K instances, we perform necessary
replay with pre-training texts in both English and language X.. This approach
is based on the assumption that task-solving abilities in English can be
transferred to the target language Chirkova and Nikoulina (2024); Shaham et
al. (2024).

(2) CTSFT-X-mt: We translate FLAN and the training set of GSM8K into the
target language X with machine translation (MT) systems222We use NLLB NLLB
Team et al. (2022) for translation. To enhance the model’s ability to follow
English prompts, we randomly sample half of the training instances for
translation. The other half of instances remain in English., which is a common
practice to obtain SFT data for non-English languages Muennighoff et al.
(2023); Li et al. (2023a, b). We then finetune CT-X with the translated data.

#### New Paradigm: Model Merging

By model merging, we can integrate distinct LLMs with various capabilities
into one LLM. To obtain a model capable of solving task in the target language
X, we can merge the following two models:

CT-X: As discussed in the CT-then-SFT procedure, this model learned a certain
amount of language modeling in the language X after CT. However, its task-
solving ability is limited.

SFT-flan: We directly finetune BASE with the SFT data used by CTSFT-X-flan.
The resulting model has sufficiently learned task solving, but the target
language X is still foreign to it.

We merge the two models above to unlock the dual benefits of proficient
language modeling and effective task-solving capabilities. Specifically, we
investigate two methods of model merging: weighted averaging (WAVG, Choshen et
al., 2022; Wortsman et al., 2022) and TIES Yadav et al. (2023). We derive two
variants of merged models, namely WAVG-X-flan and TIES-X-flan.

##  4 Experimental Setup

#### Languages

We use 7 low-resource languages from five distinct language families for
experiments: Tamil, Telugu, Odia, Bengali, Tibetan, Uyghur, and Mongolian (in
the traditional Mongolian script). See their basic information in Table 1.

We select these languages because they are underrepresented in currently
popular LLMs despite their large population (over 475M) worldwide. As shown in
Table 3, the performance of Llama-2-7B in these languages is close to or even
worse than random guessing. Notably, the vocabulary of Llama-2 does not even
contain tokens for Odia and traditional Mongolian, which indicates that the
model has hardly seen these languages during pre-training. Moreover, limited
resources are available for these languages on the internet. Among those
languages, we can only collect fewer than 1B tokens of monolingual texts for
Odia, Tibetan, Uyghur, and traditional Mongolian. The problem of data scarcity
becomes more severe in terms of high-quality data for supervised fine-tuning.

#### Pre-training Corpus

During continual pre-training, we use the largest available corpus for each
language from CulturaX Nguyen et al. (2024), IndicCorp-v2 Doddapaneni et al.
(2023), and MC2 Zhang et al. (2024b). To maximize language coverage with
constraint computational resources, we sample 8B tokens for continual pre-
training of Tamil and Telugu, and 16B tokens for Bengali. The corpus sizes are
shown in Table 2.

Following Llama models Touvron et al. (2023), we employ RedPajama Computer
(2023) with the same sampling proportion for memory replay to reduce
forgetting of English language modeling.

#### SFT Data

We mainly use FLAN Longpre et al. (2023) for SFT, which consists 155K training
instances for 1,411 distinct tasks. Since there are limited math reasoning
tasks in FLAN, we additionally incorporate 7,473 instances from GSM8K Cobbe et
al. (2021) into the supervised training sets.

We translate FLAN into the languages of our study using
NLLB-200-Distilled-1.3B NLLB Team et al. (2022)333NLLB is currently the open-
source MT model with the most extensive language support.. Note that this
model does not support traditional Mongolian and there are no open-source MT
models available for this language currently. We thus adopt a roundabout way:
translating the instructions into Cyrillic Mongolian, which NLLB supports, and
converting them into traditional Mongolian with an open-source transliteration
tool444https://github.com/tugstugi/mongolian-nlp/tree/master/bichig2cyrillic.

#### Evaluation Tasks

Regarding the four languages in India (tam, tel, ory, and ben), we use SIB-200
Adelani et al. (2024) for text classification, Belebele Bandarkar et al.
(2023) for machine reading comprehension, and MGSM Shi et al. (2022) for math
reasoning (only available in Telugu and Bengali). Regarding the three
languages in China (bod, uig, and mvf), we use MLiC-Eval Zhang et al. (2024a),
including the following 4 tasks: text classification (TC), machine reading
comprehension (MRC), response selection (RS), and math reasoning. See
statistics in Appendix A.

#### Implementation Details

We use Megatron-LM for model training Shoeybi et al. (2019) and Arcee’s
MergeKit for model merging Goddard et al. (2024). See more details in Appendix
B.

Name | Family | Script | Population
---|---|---|---
Tamil (tam) | Dravidian | Tamil | 79M
Telugu (tel) | Dravidian | Telugu | 96M
Odia (ory) | Indo-Euro. | Odia | 35M
Bengali (ben) | Indo-Euro. | Bengali | 240M
Tibetan (bod) | Sino-Tibetan | Tibetan | 7M
Uyghur (uig) | Turkic | Arabic | 12M
Mongolian (mvf) | Mongolic | Mongolian | 6M
Table 1: Languages families, writing systems, and populations of the low-resource languages in our study.  Lang. | Corpus | Tokens | Tasks
---|---|---|---
tam | CulturaX | 15.9B | SIB-200, Belebele
tel | CulturaX | 12.4B | SIB-200, Belebele, MGSM
ory | CulturaX | 765M | SIB-200, Belebele
ben | IndicCorp-v2 | 36.4B | SIB-200, Belebele, MGSM
bod | MC2 | 1.00B | MLiC-Eval
uig | MC2 | 412M | MLiC-Eval
mvf | MC2 | 904M | MLiC-Eval
Table 2: The pertaining corpora and evaluation tasks of each language in our study. The number of tokens is obtained with the tokenizer of Llama-2.  | Task | Lang. | Tamil | Telugu | Odia | Bengali | Tibetan | Uyghur | Mongolian | Average
---|---|---|---|---|---|---|---|---|---|---
BASE |  |  | 28.15 | 18.83 | 26.64 | 25.47 | 13.49 | 13.34 | 11.57 | 19.64
SFT-flan | ✓ |  | 29.19 | 17.28 | 25.21 | 24.84 | 23.29 | 22.27 | 19.86 | 23.13
CT-X |  | ✓ | 52.18 | 34.67 | 47.93 | 30.77 | 13.52 | 14.80 | 11.09 | 29.28
CTSFT-X-mt | ✓ | ✓ | 50.57 | 32.90 | 30.14 | 38.40 | 33.85 | 24.85 | 19.57 | 32.90
CTSFT-X-flan | ✓ | ✓ | 53.95 | 37.96 | 44.56 | 42.19 | 42.36 | 49.46 | 24.29 | 42.11
WAVG-X-flan | ✓ | ✓ | 57.56 | 37.58 | 53.59 | 37.19 | 44.30 | 42.64 | 31.09 | 43.42
TIES-X-flan | ✓ | ✓ | 58.46 | 39.50 | 56.49 | 40.31 | 47.86 | 52.43 | 32.56 | 46.80
Table 3: Performance of models built through the roadmap. The best results are
made bold, with the second underlined. The Task and Lang. columns denotes
whether the model have enhanced task-solving abilities and learned the target
languages, respectively.

##  5 Results and Analysis

In this work, we mainly study two categories of common roadmaps, CT-then-SFT
and model merging, to transfer the task-solving ability from a high-resource
language to a low-resource one. We aim to investigate following research
questions. RQ1: What is the viability of constructing task-solving LLMs in
low-resource languages via model merging? RQ2: Is model merging always a
better choice than CT-then-SFT?

We take the settings of BASE, SFT-flan, and CT-X as the baselines. For CT-
then-SFT, we investigate two common methods to build task-solving models,
CTSFT-X-flan and CTSFT-X-mt. For model merging, we study two effective
algorithms to combine the abilities of language modeling and task solving:
weighted averaging and TIES.

Table 3 illustrates the overall performance of different models or setups,
i.e., the average scores over all tasks, for each language. For the
experiments involving continual pre-training, we report the results based on
the last checkpoints of CT-X555Due to the budget of computational resources
and available pre-training data, we use 8B tokens for continual pre-training
on Tamil and Telugu respectively, and 16B tokens for continual pre-training on
Bengali.. See the model performance on individual tasks in Appendix C.1.

###  5.1 Effectiveness of Model Merging

For all the studied languages except Bengali, the models constructed by
merging outperform those built by CT-then-SFT666The exceptional performance in
Bengali may be attributed to its larger CT corpus, which we discuss in Section
5.2.. For example, the merged models based on TIES (TIES-X-flan) achieve an
average score of 46.80% across languages, surpassing CT-then-SFT models
(CTSFT-X-flan) by +4.69%. Although using a naïve merging algorithm, model
merging with WAVG (WAVG-X-flan) still achieves better performance than CT-
then-SFT (CTSFT-X-flan) in four languages.

In the conventional CT-then-SFT approach, LLMs often fail to acquire
sufficient comprehension of the target language from a small-scale CT corpus,
and this ability may be further diminished by supervised fine-tuning. In
contrast, model merging can preserve the language modeling acquired during CT
while incorporating task-solving capabilities by resolving parameter conflicts
carefully.

In conclusion, model merging can be an effective pathway to obtain task-
solving LLMs for low-resource languages.

0.24812163040Tokens of CT in Bengali ($\times$1B)Average Score (%)

0.22463240Tokens of CT in Telugu ($\times$1B)Average Score (%)

Figure 2: Performance of the models based on each checkpoint of CT-ben and CT-
tel. The blue lines illustrate results of TIES-X-flan, with red lines for
CTSFT-X-flan and grey lines for CT-X.

###  5.2 Performance Plateau of Model Merging

Although model merging proves more effective than CT-then-SFT in most
languages of our study, this is not the case for Bengali, which utilizes the
largest corpus in the experiments: CTSFT-ben-flan outperforms TIES-ben-flan.
We guess that the applicability of model merging may be related to the amount
of corpus used in CT. Thus, we examine the performance changes of the merged
models and the CT-then-SFT models under different amounts of CT tokens.

We collect the intermediate checkpoints of CT-ben and CT-tel, which are the
two languages with the largest size of pre-training corpora. Then, we derive
CTSFT-X-flan and TIES-X-flan models based on these checkpoints.

Figure 2 illustrates the performance of CT-only (CT-X), CT-then-SFT (CTSFT-X-
flan), and model merging (TIES-X-flan) based on every checkpoint. For Bengali
and Telugu, TIES-X-flan outperforms CT-X in each checkpoint, indicating model
merging can robustly enhance LLM’s task-solving ability, while CT-then-SFT
demonstrates greater variability. And in both languages, for all checkpoints
where the amount of pre-trained tokens is less than 10.4B tokens, TIES-X-flan
can achieve better results than CTSFT-X-flan.

Model merging can integrate language modeling and task-solving capabilities
more effectively than CT-then-SFT, in scenarios with limited language
resources ($<$10B tokens). Four out of the seven languages in our study
utilize corpora with fewer than 10B tokens. This data scarcity is
characteristic of low-resource languages. For instance, 81% (135 out of 166)
of the languages in the multilingual corpus CulturaX Nguyen et al. (2024) have
fewer than 10B tokens. Consequently, our findings on the effectiveness of
model merging are broadly applicable and have the potential to benefit a wide
range of human languages.

As the amount of CT tokens increases, the CTSFT-X-flan models show more rapid
improvement in task-solving capabilities compared to TIE-X-flan. Specifically,
when pre-trained with more than 14.4B tokens, CTSFT-ben-flan demonstrates
superior performance over TIES-ben-flan. Similarly, in Telugu, the performance
gap between the two models diminishes with additional CT tokens. However, due
to the smaller size of the Telugu corpus compared to Bengali’s, we have not
observed CTSFT-tel-flan overtaking TIES-tel-flan. We note that CT-then-SFT may
be a better method to construct task-solving LLMs in languages with sufficient
resources, for example, Bengali, Vietnamese, Indonesian, etc.

##  6 Understanding the Dynamics of Model Merging

As shown in Figure 2, we find that the performance of TIES-X-flan on
downstream tasks may no longer improve as we use more tokens for CT. In this
section, we want to investigate RQ3: What factors may affect LLMs in obtaining
task-solving capabilities through model merging?

Figure 3: Diagram illustrating the change in the proportion of SFT-flan
parameters discarded during sign election as CT progresses. The colored areas
represent the parameters with sign conflicts. Among them, the cyan parts
represent the parameters which elect the signs of SFT-flan, while the yellow
parts for parameters selecting the signs of CT-ben.

69121521222324Tokens of CT in Bengali ($\times$1B)Proportion (%)

Figure 4: The proportion of parameters removed in SFT-flan during sign
election, relative to the total number of retained parameters of SFT-flan
after trimming.

###  6.1 Quantification of Parameter Conflict

Revisiting the mechanism of TIES, we find that during the merging stage,
parameters from one model may be discarded due to differences in the parameter
signs between the two models. The discarding of model parameters usually leads
to a decline in the corresponding capabilities. Considering that the CT
model’s language modeling ability continuously improves with the increase of
pre-training data, we suspect that it is more likely that the SFT model
discards too much information during the merging process, resulting in the
merged model’s inability to further enhance its task-solving ability in the
target language.

To verify this hypothesis, we take Bengali, the language using the largest
amount of CT corpus in our study, magnitude We explore the changes in the
number of discarded parameters in the SFT model when merging it with CT models
trained with different quantities of data.

As explained in Section 3.1, we first calculate the task vectors of SFT-flan
and each checkpoint of CT-ben. In the trimming stage, we find the optimal
hyperparameters are $k_{\text{sft}}=0.2$ and $k_{\text{ct}}=1.0$ for most
scenarios777If not being optimal, this set of hyperparameters can also result
in comparable performance to the optimal ones.. Thus, to provide a fair
comparison, we freeze $(k_{\text{sft}},\,k_{\text{ct}})$ as $(0.2,\,1.0)$ for
all checkpoints. Then, we examine the signs and magnitudes of each parameter
of the trimmed task vectors of SFT-flan and CT-ben. A parameter that has
contrary signs in two models can be regarded as a parameter with sign
conflict. And if its magnitude in SFT-flan is smaller than that in CT-ben, it
will be removed at the stage of sign election.

Figure 4 illustrates the proportion of parameters that are discarded from SFT-
flan during the merging stage. We can find as the LLM is pre-trained with more
tokens, 4% more parameters are removed in trimmed SFT-flan. We believe that
when using more tokens for CT, TIES discards a larger proportion of parameters
from the SFT model, which may continuously undermine the task-solving
capabilities.

###  6.2 Model Merging with a Slack Variable

To mitigate the information loss in SFT-flan during the merging stage, we
propose TIES-SV, enhancing TIES with a Slack Variable to reduce the number of
discarded parameters in the model with higher information density, i.e., SFT-
flan in this situation.

According to the process of TIES, for each parameter to be discarded from SFT-
flan in the Disjoint Merge step, its magnitude is smaller than the magnitude
of its counterpart parameter in CT-X. To retain the parameters of SFT-flan
while minimizing the information loss of CT-X, we first rank these pairs of
parameters between SFT-flan and CT-X according to their differences in the
magnitude. Next, we select a subset of parameters with the smallest magnitude
differences to reserve.

We evaluate the effectiveness of TIES-SV on the model merging process of SFT-
flan and the last checkpoint of CT-ben. Based on the results in Figure 4, we
reserve 4% parameters in SFT-flan that were to be discarded in the original
algorithm of TIES. Table 4 shows the results of vanilla TIES and our TIES-SV
on three Bengali tasks. In all three tasks, our TIES-SV outperforms vanilla
TIES by 0.52% on average.

| MGSM | SIB-200 | Belebele | Average
---|---|---|---|---
TIES | 7.50 | 79.41 | 34.02 | 40.31
TIES-SV | 8.00 | 79.90 | 34.58 | 40.83
Table 4: Results of TIES and TIES-SV on Bengali tasks.

In conclusion, we propose a simple strategy to improve model merging. It
suggests that during the process of model merging, the importance of different
models’ parameters is not the same. When parameter conflicts occur, we cannot
simply rely on the magnitude of the vectors to decide which model’s parameters
to reserve. Instead, we should use prior information obtained through pilot
studies or other means to reserve the parameters of the more important model.
We hope our TIES-SV can shed light on the study of new model-merging
algorithms in the future.

##  7 Discussions

In this section, we further explore two important questions related to model
merging. First, we investigate the potential of merging more than one low-
resource language into a task-solving model, which could offer a new avenue
for constructing multilingual models. Second, we examine why commonly-used
machine-translated data often fails in the context of low-resource languages.
This failure underscores the advantage of model merging, as it does not
require SFT data in the target language.

###  7.1 Can We Merge Multiple Languages?

Previous work Akiba et al. (2024) shows that model merging algorithms can be
used to combine multiple LLMs with different capabilities, which may enhance
the model to solve complex problems. We wonder whether multiple LLMs adapted
to different low-resource languages can be merged with the same SFT model, to
construct a a task-solving LLM supporting these languages simultaneously.

As a pilot study, we attempt to merge Mongolian and Uyghur LLMs with the
English task-solving model SFT-flan. Due to the limited budget for
computational resources, we cannot conduct a grid search across the
hyperparameter space of the three models. Therefore, we employ the optimal
hyperparameters derived from merging each of the two CT models with SFT-flan.

Model | Ability | mvf | uig
---|---|---|---
SFT-flan | A | 19.86 | 22.27
CT-mvf | B | 11.09 | 13.58
CT-uig | C | 11.65 | 14.80
TIES-mvf-flan | A+B | 32.56 | 27.24
TIES-uig-flan | A+C | 24.45 | 52.43
TIES-mvf&uig-flan | A+B+C | 30.61 | 52.40
Table 5: Results of merging multiple languages into a task-solving model. The
columns mvf and uig refer to the average performance across the Mongolian and
Uyghur tasks, respectively. The best results are made bold, with the second
underlined.

Table 5 illustrates the average scores of tasks in the two languages. The
merged model serving two low-resource languages (TIES-mvf&uig-flan) performs
comparably to the two single-language merged models (TIES-mvf-flan and TIES-
uig-flan) on tasks in the respective languages.

This indicates that model merging has great potential for constructing
multilingual task-solving LLMs. We hope this approach can assist multilingual
speakers, particularly those using underrepresented languages, by combining
multiple existing LLMs in distinct languages without the need for expensive
pre-training.

###  7.2 Why do Machine Translated Data Fail?

Collecting synthetic SFT data through MT is an intuitive method for
constructing task-solving LLM in non-English languages Muennighoff et al.
(2023); Li et al. (2023a). However, the experiment results in Table 3 show
that MT-translated SFT data may not work when it comes to low-resource
languages. The models trained with MT-translated data (CTSFT-X-mt) models
exhibit inferior performance across all languages compared to those trained on
English SFT data (CTSFT-X-flan), with a gap of -3.38%$\sim$-24.61%.

The decline in model performance can be attributed to the low-quality MT
results. Current open-source MT systems have limited abilities of low-resource
languages. For example, we ask Uyghur and Tibetan native speakers to evaluate
sampled translation results by NLLB. They find that there are often irrelevant
contents and undesired code-switching in the translation results, as shown in
Appendix C.2. This kind of noise not only hinders the model’s ability to learn
task-solving strategies but also interferes with language modeling, which
itself is inadequately learned.

In contrast, model merging eliminates the need to collect SFT data in the
target languages. This approach also eliminates the reliance on unreliable
machine translation systems.

##  8 Conclusion

In this paper, we investigate the potential of using model merging to
construct task-solving LLMs for low-resource languages. Our findings
demonstrate that model merging outperforms the conventional CT-then-SFT
paradigm, achieving higher data efficiency. We further analyze the mechanism
behind the performance saturation of model merging with an increased number of
CT tokens, which inspires a simple yet effective improvement to the model
merging algorithm. We hope that model merging can reduce the costs associated
with data collection and model training, benefiting a greater number of
languages suffering from data scarcity.

## Limitations

#### Studied Languages

Due to the high computational cost of continually pretraining LLMs, we cannot
cover a wider range of low-resource languages, only focusing on seven
underrepresented languages in India and China. However, we make efforts to
improve the diversity of selected languages by including different language
families and writing systems.

#### Evaluation Tasks

Most of our evaluated tasks focus on natural language understanding, with less
emphasis on natural language generation. This limitation arises from the
insufficient CT corpus available for the studied low-resource languages, which
is insufficient for the models to learn to perform complex generation in the
target language.

## References

  * Adelani et al. (2024) David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024.  SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.
  * Akiba et al. (2024) Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024.  Evolutionary optimization of model merging recipes.  _arXiv preprint arXiv:2403.13187_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Cahyawijaya et al. (2023) Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023.  InstructAlign: High-and-low resource language alignment via continual crosslingual instruction tuning.  In _Proceedings of the First Workshop in South East Asian Language Processing_ , pages 55–78, Nusa Dua, Bali, Indonesia. Association for Computational Linguistics.
  * Chen and Liu (2018) Zhiyuan Chen and Bing Liu. 2018.  Lifelong supervised learning.  In Ronald J. Brachman and Peter Stone, editors, _Lifelong Machine Learing_ , 2nd edition, pages 35 – 54. Morgan & Claypool Publishers.
  * Chirkova and Nikoulina (2024) Nadezhda Chirkova and Vassilina Nikoulina. 2024.  Zero-shot cross-lingual transfer in instruction tuning of large language model.  _arXiv preprint arXiv:2402.14778_.
  * Choshen et al. (2022) Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022.  Fusing finetuned models for better pretraining.  _arXiv preprint arXiv:2204.03044_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Computer (2023) Together Computer. 2023.  Redpajama: An open source recipe to reproduce llama training dataset.
  * d’Autume et al. (2019) Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019.  Episodic memory in lifelong language learning.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 13132–13141.
  * Doddapaneni et al. (2023) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023.  Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12402–12426, Toronto, Canada. Association for Computational Linguistics.
  * Goddard et al. (2024) Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024.  Arcee’s mergekit: A toolkit for merging large language models.  _arXiv preprint arXiv:2403.13257_.
  * Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022.  Editing models with task arithmetic.  In _The Eleventh International Conference on Learning Representations_.
  * Jin et al. (2022) Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022.  Dataless knowledge fusion by merging weights of language models.  In _The Eleventh International Conference on Learning Representations_.
  * Kotha et al. (2024) Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. 2024.  Understanding catastrophic forgetting in language models via implicit inference.  In _The Twelfth International Conference on Learning Representations_.
  * Li et al. (2023a) Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2023a.  Align after pre-train: Improving multilingual generative models with cross-lingual alignment.  _arXiv preprint arXiv:2311.08089_.
  * Li et al. (2024) Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024.  X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions.  _arXiv preprint arXiv:2405.19744_.
  * Li et al. (2023b) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023b.  Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation.  _arXiv preprint arXiv:2305.15011_.
  * Liashchynskyi and Liashchynskyi (2019) Petro Liashchynskyi and Pavlo Liashchynskyi. 2019.  Grid search, random search, genetic algorithm: A big comparison for nas.  _arXiv preprint arXiv:1912.06059_.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  In _International Conference on Machine Learning_ , pages 22631–22648. PMLR.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Matena and Raffel (2022) Michael S Matena and Colin A Raffel. 2022.  Merging models with fisher-weighted averaging.  _Advances in Neural Information Processing Systems_ , 35:17703–17716.
  * Mehta et al. (2021) Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021.  An empirical investigation of the role of pre-training in lifelong learning.  _arXiv preprint arXiv:2112.09153_.
  * Micallef et al. (2024) Kurt Micallef, Nizar Habash, Claudia Borg, Fadhl Eryani, and Houda Bouamor. 2024.  Cross-lingual transfer from related languages: Treating low-resource Maltese as multilingual code-switching.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1014–1025, St. Julian’s, Malta. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Nguyen et al. (2024) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024.  CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 4226–4237, Torino, Italia. ELRA and ICCL.
  * Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023.  Seallms–large language models for southeast asia.  _arXiv preprint arXiv:2312.00738_.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Parovic et al. (2023) Marinela Parovic, Alan Ansell, Ivan Vulić, and Anna Korhonen. 2023.  Cross-lingual transfer with target language-ready task adapters.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 176–193, Toronto, Canada. Association for Computational Linguistics.
  * Parović et al. (2024) Marinela Parović, Ivan Vulić, and Anna Korhonen. 2024.  Investigating the potential of task arithmetic for cross-lingual transfer.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 124–137, St. Julian’s, Malta. Association for Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Senel et al. (2024) Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, and Goran Glavaš. 2024.  Kardeş-NLU: Transfer to low-resource languages with the help of a high-resource cousin – a benchmark and evaluation for Turkic languages.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1672–1688, St. Julian’s, Malta. Association for Computational Linguistics.
  * Shaham et al. (2024) Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024.  Multilingual instruction tuning with just a pinch of multilinguality.  _arXiv preprint arXiv:2401.01854_.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022.  Language models are multilingual chain-of-thought reasoners.  In _The Eleventh International Conference on Learning Representations_.
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_.
  * Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. 2024.  Aya dataset: An open-access collection for multilingual instruction tuning.  _arXiv preprint arXiv:2402.06619_.
  * Tao et al. (2023) Mingxu Tao, Yansong Feng, and Dongyan Zhao. 2023.  Can BERT refrain from forgetting on sequential tasks? a probing study.  In _The Eleventh International Conference on Learning Representations_.
  * Thrun (1998) Sebastian Thrun. 1998.  Lifelong learning algorithms.  In S. Thrun and L. Pratt, editors, _Learning To Learn_ , pages 181 – 209. Kluwer Academic Publishers.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International conference on machine learning_ , pages 23965–23998. PMLR.
  * Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023.  TIES-merging: Resolving interference when merging models.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Yong et al. (2023) Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. 2023.  BLOOM+1: Adding language support to BLOOM for zero-shot prompting.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11682–11703, Toronto, Canada. Association for Computational Linguistics.
  * Yu et al. (2023) Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023.  Language models are super mario: Absorbing abilities from homologous models as a free lunch.  In _Forty-first International Conference on Machine Learning_.
  * Zhang et al. (2024a) Chen Zhang, Mingxu Tao, and Yansong Feng. 2024a.  MLiC-Eval: An NLP Evaluation Suite for Minority Languages in China.  https://github.com/luciusssss/MLiC-Eval.
  * Zhang et al. (2024b) Chen Zhang, Mingxu Tao, Quzhe Huang, Jiuheng Lin, Zhibin Chen, and Yansong Feng. 2024b.  Mc2: Towards transparent and culturally-aware nlp for minority languages in china.  _arXiv preprint arXiv:2311.08348_.
  * Zhao et al. (2024) Yiran Zhao, Wenxuan Zhang, Huiming Wang, Kenji Kawaguchi, and Lidong Bing. 2024.  Adamergex: Cross-lingual transfer with large language models via adaptive adapter merging.  _arXiv preprint arXiv:2402.18913_.

##  Appendix A Data Statistics

In Table A, we report the statistics of the evaluation datasets used in our
study. We use the development set to tune the hyperparameters in the model
merging algorithms and test the models on the test set.

We follow the license for the data used in our work. Our use of existing
datasets is consistent with their intended use.

##  Appendix B Implementation Details

Since CulturaX and MC${}^{\text{2}}$ are both cleaned and deduplicated corpus,
we do not additionally preprocess these data. In this work, we employ
Megatron-LM Shoeybi et al. (2019) for continual pre-training and supervised
fine-tuning. To obtain the LLMs adapted to each target language, i.e., the
CT-X model mentioned in Section 3.2, we continually pre-train Llama-2-7B with
the texts in corresponding language. Here we use AdamW Loshchilov and Hutter
(2017) as the optimizer, with $\beta_{1}$ and $\beta_{2}$ are set to $0.9$ and
$0.95$ respectively. The maximum learning rate is set to 2e-5, and the batch
size is set to 1M tokens. We also use bfloat16 to train our models.

For model merging, we employ Arcee’s MergeKit Goddard et al. (2024) to merge
the CT model in target language and the English SFT model. Following previous
works Yadav et al. (2023), we use grid search Liashchynskyi and Liashchynskyi
(2019) to select the optimal hyperparameters. For the density of each LLM,
i.e., $k_{1}$ and $k_{2}$ mentioned in Section 3.1, we select the
hyperparameters from $\left\\{0.01,\,0.2,\,0.4,\,0.6,\,0.8,\,1.0\right\\}$.
For the scaling factor $\lambda$, Arcee’s MergeKit can automatically normalize
the magnitudes and self-adaptively selects the optimal scaling factor.

##  Appendix C Additional Experiment Results

###  C.1 Results of Individual Tasks

Here we report the performance of individual tasks for each language: Tamil in
Table 7, Telugu in Table 8, Odia in Table 9, Bengali in Table 10, Tibetan in
Table 11, Uyghur 12 and Mongolian in Table 13.

Dataset | Dev | Test
---|---|---
MGSM | 50 | 200
SIB-200 | 51 | 204
Belebele | 720 | 2,880
TC in MLiC-Eval | 48 | 504
MRC in MLiC-Eval | 20 | 200
RS in MLiC-Eval | 40 | 407
Math in MLiC-Eval | 20 | 200
Table 6: Number of instances for each language in the evaluation datasets. TC is short for text classification. MRC is short for machine reading comprehension. RS is short for response selection. | SIB-200 | Belebele | Average
---|---|---|---
BASE | 30.88 | 25.42 | 28.15
SFT-flan | 30.88 | 27.50 | 29.19
CT-tam | 73.53 | 30.83 | 52.18
CTSFT-tam-mt | 69.61 | 31.53 | 50.57
CTSFT-tam-flan | 71.08 | 36.81 | 53.95
WAVG-tam-flan | 80.39 | 34.72 | 57.56
TIES-tam-flan | 80.39 | 36.53 | 58.46
Table 7: The performance of different models on the Tamil tasks. | MGSM | SIB-200 | Belebele | Average
---|---|---|---|---
BASE | 2.50 | 27.45 | 26.53 | 18.83
SFT-flan | 1.00 | 24.02 | 26.81 | 17.28
CT-tel | 2.00 | 73.53 | 28.47 | 34.67
CTSFT-tel-mt | 3.00 | 61.27 | 34.44 | 32.90
CTSFT-tel-flan | 7.00 | 77.45 | 29.44 | 37.96
WAVG-tel-flan | 3.00 | 76.96 | 32.78 | 37.58
TIES-tel-flan | 8.00 | 77.45 | 33.06 | 39.50
Table 8: The performance of different models on the Telugu tasks. | SIB-200 | Belebele | Average
---|---|---|---
BASE | 26.47 | 26.81 | 26.64
SFT-flan | 25.98 | 24.44 | 25.21
CT-ory | 69.61 | 26.25 | 47.93
CTSFT-ory-mt | 32.25 | 27.92 | 30.14
CTSFT-ory-flan | 61.76 | 27.36 | 44.56
WAVG-ory-flan | 77.45 | 29.72 | 53.59
TIES-ory-flan | 81.86 | 31.11 | 56.49
Table 9: The performance of different models on the Odia tasks. | MGSM | SIB-200 | Belebele | Average
---|---|---|---|---
BASE | 1.00 | 50.00 | 25.42 | 25.47
SFT-ben | 1.50 | 47.06 | 25.97 | 24.84
CT-ben | 1.50 | 63.73 | 27.08 | 30.77
CTSFT-ben-mt | 7.00 | 76.96 | 31.25 | 38.40
CTSFT-ben-flan | 7.50 | 80.88 | 38.19 | 42.19
WAVG-ben-flan | 3.50 | 76.96 | 31.11 | 37.19
TIES-ben-flan | 7.50 | 79.41 | 34.02 | 40.31
Table 10: The performance of different models on the Bengali tasks. | TC | MRC | RS | Math | Average
---|---|---|---|---|---
BASE |  0.60 | 28.50 | 22.36 |  2.50 | 13.49
SFT-bod | 14.48 | 44.00 | 32.68 |  2.00 | 23.29
CT-bod |  0.40 | 28.00 | 18.18 |  7.50 | 13.52
CTSFT-bod-mt | 48.41 | 42.50 | 30.47 | 14.00 | 33.85
CTSFT-bod-flan | 70.24 | 46.50 | 45.70 |  7.00 | 42.36
WAVG-bod-flan | 74.40 | 51.50 | 40.79 | 10.50 | 44.30
TIES-bod-flan | 78.17 | 56.00 | 42.26 | 15.00 | 47.86
Table 11: The performance of different models on the Tibetan tasks. All four tasks are from MLiC-Eval. TC is short for text classification. MRC is short for machine reading comprehension. RS is short for response selection. | TC | MRC | RS | Math | Average
---|---|---|---|---|---
BASE |  0.00 | 26.00 | 23.34 |  4.00 | 13.34
SFT-uig |  7.94 | 43.00 | 34.15 |  4.00 | 22.27
CT-uig |  1.39 | 21.50 | 25.31 | 11.00 | 14.80
CTSFT-uig-mt | 20.63 | 37.00 | 28.26 | 13.50 | 24.85
CTSFT-uig-flan | 90.28 | 53.50 | 38.57 | 15.50 | 49.46
WAVG-uig-flan | 54.56 | 54.00 | 42.01 | 20.00 | 42.64
TIES-uig-flan | 87.50 | 56.00 | 45.21 | 21.00 | 52.43
Table 12: The performance of different models on the Uyghur tasks. All four tasks are from MLiC-Eval. TC is short for text classification. MRC is short for machine reading comprehension. RS is short for response selection. | TC | MRC | RS | Math | Average
---|---|---|---|---|---
BASE |  0.40 | 21.50 | 21.38 |  3.00 | 11.57
SFT-mvf | 10.52 | 34.50 | 33.42 |  1.00 | 19.86
CT-mvf |  0.60 | 12.5 | 27.76 |  3.50 | 11.09
CTSFT-mvf-mt | 20.24 | 31.00 | 27.03 |  0.00 | 19.57
CTSFT-mvf-flan | 40.87 | 30.00 | 25.80 |  0.50 | 24.29
WAVG-mvf-flan | 40.48 | 45.00 | 35.87 |  3.00 | 30.09
TIES-mvf-flan | 45.04 | 50.00 | 32.18 |  3.00 | 32.56
Table 13: The performance of different models on the Mongolian tasks. All four
tasks are from MLiC-Eval. TC is short for text classification. MRC is short
for machine reading comprehension. RS is short for response selection.

###  C.2 Case Study of Machine-Translated Data

Table 14 presents two examples of machine translation (MT) and human
translation applied to SFT data. These examples illustrate the potential
shortcomings of MT compared to human translation.

The MT output, generated by the NLLB model, often introduces irrelevant
content and omits crucial information, as shown in the table’s upper example.
In the lower example, NLLB fails to translate several question options from
English to Tibetan. These issues highlight the significant performance gap
between human and machine translation.

English
---
Input: Give the rationale and then the answer. Is the following statement
true?
“The United States Army Air Forces sent aircraft into the skies.”
Output: The United States Army Air Forces were responsible for aerial
military. The final answer: yes.
Uyghur (Translated by the NLLB Model)
Input:
Output:
Uyghur (Translated by Native Speakers)
Input:
Output:
English
Input: What do people usually tie a ribbon around as a present?
A. floral arrangement B. girl’s hair C. box
D. car E. typewriter
The best answer is
Answer:
Output: A
Tibetan (Translated by the NLLB Model)
Input:
Output: A
Tibetan (Translated by Native Speakers)
Input:
Output: A
Table 14: Translation samples of Uyghur (upper) and Tibetan (lower) SFT data.