# InstructionCP: A fast approach to transfer Large Language Models into target
language

Kuang-Ming Chen1,2 Hung-yi Lee1
1National Taiwan University, Taipei, Taiwan
2ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The rapid development of large language models (LLMs) in recent years has
largely focused on English, resulting in models that respond exclusively in
English. To adapt these models to other languages, continual pre-training (CP)
is often employed, followed by supervised fine-tuning (SFT) to maintain
conversational abilities. However, CP and SFT can reduce a model’s ability to
filter harmful content. We propose Instruction Continual Pre-training (InsCP),
which integrates instruction tags—also known as chat templates—into the CP
process to prevent loss of conversational proficiency while acquiring new
languages. Our experiments demonstrate that InsCP retains conversational and
Reinforcement Learning from Human Feedback (RLHF) abilities. Empirical
evaluations on language alignment, reliability, and knowledge benchmarks
confirm the efficacy of InsCP. Notably, this approach requires only 0.1
billion tokens of high-quality instruction-following data, thereby reducing
resource consumption.

Figure 1: n illustration to demonstrate the difference between the traditional
approach and our method. In the traditional approach, considerable effort is
expended in collecting a plethora of contextual data for continual pre-
training (CP), various types of instruction-following data for instruction
tuning, and significant human resources are allocated to label data for
reinforcement learning from human feedback (RLHF). However, with our method,
Instruction Continual Pre-training (InsCP), these processes are streamlined
into a single step

##  1 Introduction

Large language models (LLMs) have demonstrated remarkable performance across
numerous natural language processing (NLP) tasks(Brown et al., 2020). However,
the majority of LLMs are pre-trained on English corpora, thus restricting
their utility to English language contexts.

While some endeavors opt to train their LLMs from scratch using non-English
data, as exemplified by YI-34B(AI et al., 2024), we recognize the significant
time and computing resources required for such an approach. Drawing
inspiration from Ouyang et al. (2022), many research groups have shifted their
focus towards continual pre-training (CP)(Gupta et al., 2023; Ke et al., 2022)
on target languages to enhance knowledge acquisition and model fluency.
Subsequently, supervised fine-tuning (SFT) is conducted on instruction-
formatted data to ensure that models possess the capability to respond to
questions in a format consistent with English-based pre-trained LLMs, such as
BLOOM(Workshop et al., 2023), LLaMA2(Touvron et al., 2023), and
Mistral-7B(Jiang et al., 2023).

Furthermore, in an effort to align with human preferences, Reinforcement
Learning from Human Feedback (RLHF) has been integrated(Ouyang et al., 2022;
Ziegler et al., 2020). However, the RLHF process is intricate. Direct
Preference Optimization (DPO)(Rafailov et al., 2023) relies on collecting
paired data from human preferences, facilitating more stable and
straightforward model tuning. Nonetheless, gathering both positive and
negative responses from humans still demands substantial effort. In contrast,
Kahneman-Taversky Optimization (KTO)(Ethayarajh et al., 2024) operates with
unpaired data, thus easing the collection process. However, KTO has its
drawbacks. The existence of HALOs (Human-Aware Loss Functions) raises several
questions regarding KTO. Firstly, the KTO loss function is based on the
Kahneman-Tversky value function for monetary gains and losses, which may not
accurately reflect how humans perceive the relative goodness of text.
Nevertheless, LLMs trained using RLHF demonstrate enhanced safety in
completions, a crucial factor for companies and groups intending to open-
source their models(Stiennon et al., 2022). Yet, as highlighted in Qi et al.
(2023), challenges persist in maintaining RLHF capabilities when fine-tuning
GPT-4(OpenAI, 2023) on non-English data. Our experiments validate similar
observations with other LLMs like LLaMA2.

In this work, we propose a novel fine-tuning approach called Instruction
Continual Pre-training (InsCP) for LLMs to adapt to non-English languages.
This process draws inspiration from merging CP and SFT into a unified one-step
training process. Additionally, we investigate whether LLMs, equipped with
their own templates, can recognize tags during CP. Furthermore, we hypothesize
that providing a chat template during CP prevents the model from forgetting
its conversational abilities, as it resembles its original training
conditions. Our approach begins with CP on a specific dataset, where we
augment each piece of data with special instruction tokens, such as
$<|begin\\_of\\_text|>$ in LLaMA3(AI@Meta, 2024). This augmentation enables
the model to respond to target language inputs in the target language and
effectively handle offensive input based on its original RLHF capabilities.

We evaluate the effectiveness of InsCP on LLMs, primarily focusing on the
LLaMA3-instruct model, across three key aspects: language alignment,
reliability, and knowledge benchmarks. Language alignment tests the model’s
proficiency in learning the desired language, while reliability evaluates its
retention of RLHF capabilities. Knowledge benchmarks gauge the pipeline’s
impact on the model’s comprehension ability. Our primary focus for InsCP is
Traditional Chinese as our target language.

The results demonstrate that the model, after undergoing InsCP on
LLaMA3-instruct, effectively performs in Traditional Chinese when prompted
with Traditional Chinese input, surpassing the performance of LLaMA3-instruct.
Moreover, in addition to aligning with Traditional Chinese prompts, the model
retains its ability to respond appropriately to English prompts. Furthermore,
most language benchmarks indicate comparable performance between the model
before and after CP. Additionally, when tested on TruthfulQA(Lin et al.,
2022), a benchmark assessing the model’s reliability, our model exhibits
consistent performance in both English and Traditional Chinese, indicating
that the RLHF ability remains intact without compromising performance, which
typically requires significant investment to develop.

##  2 Related Work

###  2.1 LLMs adapt in other languages

Fine-tuning has been a longstanding technique for enabling models to adapt to
specific domains, particularly in the realm of large language models (LLMs).
Many downstream tasks have been successfully addressed through fine-
tuning(Howard and Ruder, 2018; Devlin et al., 2019; Radford et al., 2018).
While most downstream tasks can be accomplished through instruction fine-
tuning, also known as supervised fine-tuning, adapting an English-based LLM to
other languages, such as in the work of Fujii et al. (2024); Zhao et al.
(2024); Cui et al. (2023); Lin and Chen (2023); YuLan-Team (2023) for fine-
tuning in non-English languages, typically begins with continual pre-training.
This initial step is crucial for ensuring that the models possess the
necessary language proficiency and knowledge. Subsequently, instruction fine-
tuning allows the model to engage in conversational interactions using
specific templates.

###  2.2 Fine-tuning hurts LLMs

Recently, OpenAI introduced the capability to fine-tune GPT-3.5-turbo using
user-provided data. In the (Qi et al., 2023), they collected a limited number
of explicitly harmful examples, identity-shifting data, and the Alpaca dataset
to perform instruction fine-tuning on GPT-3.5-turbo and LLaMA-2-Chat. Their
study evaluated these models against 11 criteria for harmful content, assessed
by GPT-4. They observed that fine-tuning on these models led to an increase in
harmful content generation. Even when employing safety data for fine-tuning,
the resulting impact was still negative, albeit less pronounced than direct
fine-tuning.

###  2.3 Training from human feedback

Ouyang et al. (2022) introduced InstructGPT, a model built upon GPT-3 (Brown
et al., 2020), which they further refined through reinforcement learning from
human feedback (RLHF). In their work, Ouyang et al. (2022) formally outlined
the RLHF algorithm, which comprises three key components: Supervised Fine-
Tuning (SFT), Reward Model training, and reinforcement learning via Proximal
Policy Optimization (PPO) (Schulman et al., 2017). The RLHF algorithm enhances
the model’s ability to adhere to instructions and shows promise in mitigating
the generation of toxic or harmful content.

Recent studies have explored avenues for optimizing human preference without
necessarily relying on learning a reward function. For instance,Rafailov et
al. (2023) proposed Direct Preference Optimization (DPO), refining the policy
through a loss function constructed using the Bradley-Terry reward model.Azar
et al. (2023) introduced Identity Policy Optimization (IPO), advocating for
direct optimization of pairwise human preferences using preference data,
distinct from DPO as IPO does not presuppose a reward model.Ethayarajh et al.
(2024) put forth Kahneman-Tversky Optimization (KTO), suggesting the
utilization of whether a given output is desirable or undesirable for a given
input as the sole criterion to align the model with human preferences.

##  3 Methodology

Continual pre-training (CP) has traditionally served as a method for enhancing
the comprehensive and generative capabilities of LLMs in a target language by
leveraging extensive target language corpora. The underlying principle of the
CP process involves training LLMs to predict the next token based on preceding
content. The loss function guiding this process lists in below.

For our method, Instruction Continual Pre-training, we adopt a similar
approach to CP, but with the addition of the model’s original chat template.
Taking LLaMA3-instruct(AI@Meta, 2024) as an example, to initiate a completion
with LLaMA3-instruct, one must adhere to the following format:

⬇

<|begin_of_text|><|start_header_id|>user<|end_header_id|>
{{inputs}}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{{model_response}}

The inputs in the template represent the prompts provided by the user. In our
context, where the objective is to train LLMs in the target language through
next token prediction tasks while retaining their chat ability, we place the
CP data in the model_response. This arrangement ensures that LLMs generate
tokens based on the target language. The InsCP template is structured as
follows:

⬇

<|begin_of_text|><|start_header_id|>user<|end_header_id|><|eot_id|><|start_header_id|>assistant<|end_header_id|>
{{InsCP_data}<|eot_id|>}

The loss function for CP:

$\begin{split}\mathcal{L}_{pretrain}&=\mathbb{E}_{x\sim\mathcal{D}_{CP}}\Bigg{[}\\\
&-\sum^{S}_{i}\log P(x_{i}\mid
x_{0},...,x_{i-1};\theta_{CP})\Bigg{]}\end{split}$ (1)

The loss function for InsCP:

$\begin{split}\mathcal{L}_{pretrain}&=\mathbb{E}_{x\sim\mathcal{D}_{InsCP}}\Bigg{[}\\\
&-\sum^{S}_{i}\log P(x_{i}\mid
x_{0},...,x_{i-1};\theta_{InsCP})\Bigg{]}\end{split}$ (2)

where $\theta_{CP}$ and $\theta_{InsCP}$ represents the model parameters,
$\mathcal{D_{CP}}$ stands for the data used in continual pre-training,
$\mathcal{D_{InsCP}}$ stands for the data added the chat template and used in
instruct continual pre-training, S represents the length of the input token
sequence, and $x_{i}$ represents the token to be predicted, while
$x_{0},x_{1},...,x_{i-1}$ make up the context.

##  4 Experimental Setup

###  4.1 Training Dataset

We utilize a high-quality dataset comprising paired instruction-following data
for LLaMA3-instruct 8B(AI@Meta, 2024) during the InsCP procedure. The dataset
consists of Traditional Chinese text and has a total size of 0.1 billion
tokens. Throughout the InsCP process, we segregate the questions and answers
into two separate data points. Further details regarding the training process
are provided in the Appendix A.2.

Moreover, to demonstrate the generalizability of our method to other
languages, we extend our approach to Japanese. We utilize a 70M tokens
dataset, structured similarly to the Traditional Chinese dataset, to perform
InsCP on LLaMA3-instruct 8B.

From our experiments, we discovered the critical importance of selecting
appropriate data for InsCP. We aimed to determine the most suitable type of
data for InsCP. Based on our findings, we selected wiki context data with high
perplexity(PPL $\geq$ 30) and two different types of instruction-following
data with low perplexity. We observed that all instruction-following data with
low perplexity(PPL $\leq$ 15)successfully facilitated InsCP. We posit that
this outcome is reasonable because data characterized by instruction-following
and low perplexity are likely to closely resemble the original output of LLMs,
thereby minimizing any adverse effects on the models’ original abilities. The
function of the perplexity is shown below:

$PPL(D|\Theta)=\exp\left(-\frac{1}{M}\sum_{i=1}^{M}\log
p(d_{i}|\Theta)\right)$

Here, $\Theta$ represents the parameters of the language model. Dataset
perplexity can be interpreted as the average perplexity of the model when
predicting the entire dataset. Lower perplexity indicates better predictive
performance of the model on the dataset.

###  4.2 Evaluation Dataset

We introduce three aspects of evaluation datasets to assess our InsCP model:
language alignment, reliability, and knowledge benchmarks. Furthermore, we
employ MT-Bench in our evaluation, we think that MT-Bench can test the LLMs
more comprehensively. Throughout our testing, we maintain uniformity in all
generation strategies, as detailed in the Appendix.

Language alignment We employ the FastText language identification model
(Joulin et al., 2016a, b) to determine the language of 2000 aligned sentences
extracted from the English and Traditional Chinese subset of the NeuLab-
TedTalks language within the tokens generated by our model.

Reliability We employ several common benchmarks to evaluate the reliability of
the model’s output, including TruthfulQA(Lin et al., 2022), ToxiGen(Hartvigsen
et al., 2022), and BOLD(Dhamala et al., 2021) by using lm-evaluation-
harness(Gao et al., 2021). In the TruthfulQA benchmark, we assess the model’s
ability to accurately respond to questions based on factual information.
ToxiGen allows us to evaluate the model’s proficiency in generating non-toxic
responses by utilizing a RoBERTa-based(Liu et al., 2019) approach for
identification, while BOLD assesses the model’s confidence and coherence in
its responses.

Knowledge benchmarks We utilize C-eval-tw, which is a translation of
C-eval(Huang et al., 2023), to evaluate our model. Additionally, we assess our
model using TTQA(Hsu et al., 2023), which focuses on Taiwanese commonsense and
knowledge by 64 expert-selected paragraphs from Wikipedia. For traditional
Chinese multitask benchmarking, we employ TMMLU Plus(Tam et al., 2024). To
ensure that our model’s English-related knowledge does not degrade, we include
ARC(Clark et al., 2018) and Hellaswag(Zellers et al., 2019), which are
benchmarks for English commonsense reasoning. For multitask evaluation,
MMLU(Hendrycks et al., 2020) is a suitable choice.

MT-Bench(Zheng et al., 2023) We utilize MT-Bench to evaluate the comprehensive
abilities of the models, encompassing knowledge, reliability, and language
alignment. Additionally, MT-Bench incorporates multi-conversation scenarios,
allowing us to assess the model’s ability to handle multiple interactions
simultaneously. This enables us to demonstrate that InsCP does not compromise
the RLHF ability of the model.

###  4.3 Evaluation Metrics

Language alignment The FastText language identification model is utilized to
determine the language of the generated text. The model classifies text into
three categories: Chinese and English. The results include the percentage of
sentences identified as Chinese, English, and others from a set of 2000 input
prompts.

Reliability TruthfulQA consists of questions accompanied by multiple
true/false options. Scoring is determined by assigning points based on the
normalized cumulative probability assigned to the correct answers. ToxiGen
utilizes a RoBERTa-based classifier to identify toxic generations and
determine the toxicity score. For BOLD, we employ the Valence Aware Dictionary
and Sentiment Reasoner (VADER(Hutto and Gilbert, 2014)) to calculate the
sentiment score for both the prompt and generated text when combined. We
present the average and standard deviation of the sentiment scores across all
subgroups.

Knowledge benchmarks In ARC and Hellaswag, we utilize length-normalized
accuracy as our metric. For MMLU and TMMLU Plus, we directly calculate
accuracy for each task. In C-eval-tw, we compute metrics by averaging accuracy
across individual tasks. The accuracy computation involves selecting the
option with the highest probabilities. In TTQA, we extract the model’s output
and calculate accuracy based on multiple-choice questions.

MT-Bench In MT-Bench, the GPT-4 score serves as our evaluation metric. GPT-4
now serves as a standard for assessing the generation ability of LLMs,
eliminating the need for expensive human evaluations. For each completed
conversation, we invoke the GPT-4 API, which returns a score ranging from 0 to
10. This score is based on various factors, including instruction following,
harmfulness, and knowledge. Besides, we add the prompt about judging language
alignment in GPT-4 evaluation in order to test model’s language ability.

###  4.4 Baselines

We select LLaMA-3-instruct as our baseline model. To evaluate the performance
of Instruction Continual Pre-training (InsCP), we conduct InsCP using our
baseline model. Furthermore, to compare with the original continual pre-
training process, we also fine-tune a model using this method. However, we
observed that the original method significantly impairs the model’s chat
ability and may cause it to lose its instruction-following capability.
Consequently, it becomes challenging to assess the model’s performance using
certain benchmarks.

model | EN prompt | TW prompt
---|---|---
response | EN% $\uparrow$ | TW% $\downarrow$ | EN% $\downarrow$ | TW% $\uparrow$
LLaMA3-instruct | 1.0 | 0.0 | 0.90 | 0.09
LLaMA3-orgCP | 1.0 | 0.0 | 0.50 | 0.49
LLaMA3-InsCP | 0.99 | 0.01 | 0.01 | 0.99
Table 1: Language alignment benchmark. model | TruthfulQA | ToxiGen | BOLD
---|---|---|---
| mc2 $\uparrow$ | toxicity $\downarrow$ | sentiment $\downarrow$
language | EN | TW | EN | TW | EN | TW
LLaMA3-instruct | 51.6 | 52.7 | 0.10 | 0.14 | 0.54 | 0.55
LLaMA3-orgCP | 50.8 | 50.5 | 0.12 | 0.26 | 0.61 | 0.68
LLaMA3-InsCP | 51.8 | 53.8 | 0.07 | 0.16 | 0.56 | 0.52
Table 2: Reliability benchmark model | ARC | Hellaswag | MMLU | C-eval-tw | TMMLU+ | TTQA
---|---|---|---|---|---|---
| ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$
LLaMA3-instruct | 60.5 | 81.8 | 67.2 | 47.3 | 43.0 | 23.3
LLaMA3-orgCP | 57.5 | 81.3 | 66.1 | 48.5 | 41.3 | 41.3
LLaMA3-InsCP | 61.6 | 81.7 | 65.6 | 48.9 | 41.9 | 48.5
Table 3: Knowledge benchmark model | MT-Bench
---|---
language | EN $\uparrow$ | TW $\uparrow$
LLaMA3-instruct | 7.8 | 4.1
LLaMA3-orgCP | 4.3 | 4.6
LLaMA3-InsCP | 7.6 | 6.7
Table 4: MT-Bench model | MT-Bench-JP
---|---
LLaMA3-instruct | 4.9
LLaMA3-orgCP-JP | 4.8
LLaMA3-InsCP-JP | 6.6
Table 5: MT-Bench-JP

##  5 Experimental Result

In this section, we provide the experimental results of four aspects: language
alignment, reliability, knowledge and MT-Bench. For Traditional Chinese, we
provide comprehensive assessments using MT-Bench. Additionally, we introduce
MT-Bench-JP to evaluate the results specifically for Japanese InsCP.
LLaMA3-InsCP refers to LLaMA3-instruct conducted with instruction CP, while
LLaMA3-orgCP denotes LLaMA3-instruct with original CP.

###  5.1 Language alignment evaluation

We adhere to our evaluation methodology outlined in Section 4.3, presenting
the percentage of responses among 2000 prompts generated by the models. The
experimental findings are summarized in Table 1. Our observations are as
follows: (1)LLaMA3-instruct exhibits poor language alignment: As indicated in
Table 1, when provided with Taiwanese (Traditional Chinese) input prompts,
LLaMA3-instruct frequently generates output in English. This lack of alignment
between the input and output languages can lead to language nonalignment
issues during usage. (2)The same data used with the original CP method fails
to achieve proper alignment: A key distinction between InsCP and the original
CP lies in their respective language learning capabilities. We observed that
with the same data size, InsCP enables LLMs to acquire language proficiency
more rapidly. (3)LLaMA3-InsCP demonstrates remarkable language proficiency:
Regardless of whether provided with English or Traditional Chinese input
prompts, LLaMA3-InsCP consistently responds in the appropriate language.

###  5.2 Reliability evaluation

In Table 2, we present the results of the models’ reliability. Our experiments
were conducted in both English and Chinese to ensure that our model does not
compromise its reinforcement learning from human feedback (RLHF) ability in
either language. Across each benchmark, we observe that the orgCP model
consistently achieves lower scores compared to the other models. We attribute
this outcome to our hypothesis that the model’s RLHF ability diminishes during
continual pre-training (CP) and supervised fine-tuning (SFT). However, both
LLaMA3-instruct and LLaMA3-InsCP retain their RLHF ability, allowing them to
defend against toxic inputs and generate non-harmful context during inference.

###  5.3 Knowledge benchmark

In Table 3, we present the scores from six benchmark tests. We specifically
chose three language-relevant benchmarks to demonstrate that InsCP does not
significantly impact the model’s original English knowledge. Additionally, in
Chinese-related benchmarks, we observed that the model after InsCP exhibited
some improvements compared to both orgCP and the original model. These
findings indicate that InsCP can effectively preserve the LLM’s inherent
abilities while also enhancing its performance in target language domains.

###  5.4 MT-Bench

In Table 4, MT-Bench further highlights the distinctions between orgCP and
InsCP. We note that outputs from orgCP often contain irrelevant text that
deviates from our input prompts. Moreover, the orgCP model appears to forget
how to appropriately conclude conversations. Additionally, due to the
inclusion of language alignment criteria in GPT-4 evaluation, we observe a
significant disparity between the InsCP model and LLaMA3-instruct. While
LLaMA3-instruct predominantly responds in English for most questions, the
InsCP model demonstrates the ability to discern the language input by the
user.

###  5.5 MT-Bench-JP

In Table 5, we observe a distribution similar to that of Traditional Chinese
MT-Bench. Both LLaMA3-instruct and LLaMA3-InsCP-JP successfully generate
responses in the correct format corresponding to the input prompts. However,
LLaMA3-instruct fails to align the responses with the target language.
Conversely, LLaMA3-orgCP-JP notably deviates from the instruction format,
producing text unrelated to the input and sometimes generating repetitive
text.

###  5.6 Limitations of InsCP

As discussed in Section 4.1, the choice of data used in InsCP significantly
influences its outcomes. Our experiments indicate that conducting InsCP
necessitates the utilization of low-perplexity instruction-following data,
which can be challenging to acquire in abundance for certain languages.
Consequently, we opted to perform InsCP using small datasets, which we believe
is a more generalizable approach for languages with limited resources.
Nonetheless, both data size and data quality remain challenges when
implementing InsCP.

##  6 Conclusion

In this work, we introduce a novel pipeline called InsCP designed to
facilitate the transfer of LLMs into non-English domains. Through InsCP, LLMs
can retain their inherent abilities, including reinforcement learning from
human feedback (RLHF) and knowledge in the English domain, while also
acquiring the capability for language alignment in the target language and
gaining knowledge of the target domain. Additionally, we demonstrate that
InsCP does not necessitate extensive data, thereby consuming fewer resources
and less time. Remarkably, even with a small amount of data, InsCP can
transform English-based LLMs into models aligned with the target language, a
stark contrast to the resource-intensive traditional pipeline. InsCP paves the
way for future LLMs, primarily fine-tuned in specific languages, to swiftly
transfer their abilities to other languages.

##  7 Acknowledgements

We extend our appreciation to the ASUS Open Cloud Infrastructure Software
Center for generously providing valuable resources. Special thanks to Steve
Chung-Cheng Chen, Tsung-Ying Yang, Dau-Cheng Lyu, Jen-Hao Cheng, Hsiao-Tsung
Hung, Szu-Hsien Lee for their participation in insightful discussions.

## References

  * AI et al. (2024) 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024.  Yi: Open foundation models by 01.ai.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023.  A general theoretical paradigm to understand learning from human preferences.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  Bold: Dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21. ACM.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024.  Kto: Model alignment as prospect theoretic optimization.
  * Fujii et al. (2024) Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024.  Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.  A framework for few-shot language model evaluation.
  * Gupta et al. (2023) Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023.  Continual pre-training of large language models: How to re-warm your model? In _Workshop on Efficient Systems for Foundation Models @ ICML2023_.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.
  * Hsu et al. (2023) Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da shan Shiu. 2023.  Advancing the evaluation of traditional chinese language models: Towards a comprehensive benchmark suite.
  * Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _arXiv preprint arXiv:2305.08322_.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the international AAAI conference on web and social media_ , volume 8, pages 216–225.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Joulin et al. (2016a) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016a.  Fasttext.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_.
  * Joulin et al. (2016b) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016b.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_.
  * Ke et al. (2022) Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2022.  Continual pre-training of language models.  In _The Eleventh International Conference on Learning Representations_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.
  * Lin and Chen (2023) Yen-Ting Lin and Yun-Nung Chen. 2023.  Language models for taiwanese culture.  Code and models available at https://github.com/MiuLab/Taiwan-LLaMa.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Qi et al. (2023) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023.  Fine-tuning aligned language models compromises safety, even when users do not intend to!
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.
  * Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022.  Learning to summarize from human feedback.
  * Tam et al. (2024) Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, and Hong-Han Shuai. 2024.  An improved traditional chinese evaluation suite for foundation model.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Workshop et al. (2023) BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, and Matthias Gallé et al. 2023\.  Bloom: A 176b-parameter open-access multilingual language model.
  * YuLan-Team (2023) YuLan-Team. 2023.  Yulan-chat: An open-source bilingual chatbot.  https://github.com/RUC-GSAI/YuLan-Chat.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020.  Fine-tuning language models from human preferences.

##  Appendix A Appendix

###  A.1 Training Detail

We utilize LLaMA3-instruct as our base model, and both the original continual
pre-training and instruction continual pre-training are configured with the
following hyperparameters: a learning rate of 3e-5, AdamW optimizer with beta1
of 0.9 and beta2 of 0.95, batch size set to 1 per device (utilizing 64 GPUs),
and training conducted for 10 epochs.

###  A.2 Generation Strategy

We employ vLLM as our generation tool, incorporating LLaMA3’s system prompt in
each generation to harness the full potential of the LLM. For vLLM, we set the
following generation parameters: maximum tokens to 1024, temperature to 0.8,
top-p sampling to 0.9, and seed fixed at 42 to facilitate result
reproducibility. Additionally, we maintain default values for other generation
configurations in vLLM.

###  A.3 MT-Bench evaluation prompt

In the Traditional Chinese MT-Bench, we predominantly adhere to the evaluation
prompts provided by the authors. However, to delve deeper into testing the
LLM’s language alignment ability, we introduce an additional prompt in
Traditional Chinese: ”If the assistant’s answer is in a language other than
Traditional Chinese, please give it a score of 0.” This prompt instructs GPT-4
to assign a score of 0 to responses that are not in the correct language,
thereby enabling a more rigorous assessment of language alignment
capabilities. For Japanese MT-Bench, we also add the prompt in Japanese: ”If
the assistant’s answer is in a language other than Japanese, please give it a
score of 0.”, in order to meet the language alignment requirement we want to
obseve.