# Mixture-of-Instructions: Comprehensive Alignment of a Large Language Model
through the Mixture of Diverse System Prompting Instructions

Bowen Xu Shaoyu Wu 11footnotemark: 1 Kai Liu Lulu Hu
Alibaba Group
<EMAIL_ADDRESS>equal contribution

###### Abstract

With the proliferation of large language models (LLMs), the comprehensive
alignment of such models across multiple tasks has emerged as a critical area
of research. Existing alignment methodologies primarily address single task,
such as multi-turn dialogue, coding, mathematical problem-solving, and tool
usage. However, AI-driven products that leverage language models usually
necessitate a fusion of these abilities to function effectively in real-world
scenarios. Moreover, the considerable computational resources required for
proper alignment of LLMs underscore the need for a more robust, efficient, and
encompassing approach to multi-task alignment, ensuring improved generative
performance. In response to these challenges, we introduce a novel technique
termed Mixture-of-Instructions (MoI), which employs a strategy of instruction
concatenation combined with diverse system prompts to boost the alignment
efficiency of language models. We have also compiled a diverse set of seven
benchmark datasets to rigorously evaluate the alignment efficacy of the MoI-
enhanced language model. Our methodology was applied to the open-source
Qwen-7B-chat model, culminating in the development of Qwen-SFT-MoI. This
enhanced model demonstrates significant advancements in generative
capabilities across coding, mathematics, and tool use tasks.

Figure 1: Performance of Qwen-SFT-MoI and Qwen-7B-chat models, along with
various SFT-aligned models on subdomain datasets, evaluated across seven
datasets encompassing mathematics, programming, tool usage, common sense, and
both single and multi-turn dialogues. Results demonstrate that training with
our Mixture of Instruction method enhances multiple capabilities of language
models, achieving improved alignment

##  1 Introduction

The rise of large language models (LLMs) has underscored the importance of
effective training techniques, encompassing pre-training, Supervised Fine-
Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF), to expand
their knowledge and ensure alignment with human values for greater
reliability. However, despite advancements, LLMs often falter in specialized
tasks such as coding, mathematics, and reasoning. With the growing access to
high-quality, domain-specific datasets(Luo et al., 2023b; Li et al., 2023;
CodeFuse, 2023; Wei et al., 2023; Yue et al., 2023; Yu et al., 2023; Luo et
al., 2023a; Zeng et al., 2023; Yang et al., 2023; Qin et al., 2023), this
study investigates how targeted SFT can boost LLM capabilities in these areas
while preserving existing competencies.

We examined a case where the Qwen-7B-chat (Bai et al., 2023)model failed to
learn Python code for the Boyer-Moore majority vote algorithm. Training with
standard code dataset proved ineffective, with the model producing suboptimal
solutions due to prompt-induced knowledge conflicts. To our surprise, we found
that simply changing to a different system prompt and retraining could easily
overcome this issue, indicating that altering prompts can resolve knowledge
conflicts and improve task performance across tasks.

Leveraging our initial findings, we developed a strategy that assigns unique
system prompts to different tasks and integrates these varying prompts into a
unified instruction set for model alignment. We introduce the Mixture of
Instruction (MoI) methodology, a structured approach to amalgamate open-source
datasets for comprehensive multi-task training, spanning dialogue, code, math,
and agent-based tasks, each with bespoke prompts. This not only recasts
training as a multi-task learning (MTL) endeavor but also tackles MTL
challenges such as dataset bias, where improvement in one task could detriment
another. Through MoI, we adeptly use diverse prompts to ensure balanced task
performance, effectively navigating the complexities of MTL.

Ultimately, we integrated the MoI with multi-turn dialogue, code , mathematics
and agent datasets for comprehensive training, resulting in a significantly
enhanced language model—Qwen-SFT-MoI. This model exhibited substantial
improvement across a range of benchmarks, including MT-Bench(Zheng et al.,
2024a), HumanEval(Chen et al., 2021), MBPP(Austin et al., 2021),
MATH(Hendrycks et al., 2021), GSM8K(Cobbe et al., 2021), MMLU(Hendrycks et
al., 2020), and T-EVAL(Chen et al., 2023), compared to the Qwen-7B-Chat model,
validating our MoI strategy for further alignment on an already SFT model.

This work makes the following contributions:

  * •

We identified and addressed conflicts between new and old knowledge during
Supervised Fine-Tuning (SFT), by introducing modified system prompts that aid
in the integration of new knowledge. This resulted in improved alignment with
and performance on code generation task.

  * •

Our Mixture of Instruction (MoI) method facilitates joint multi-task training,
effectively reducing dataset bias and preserving conversational abilities. The
efficacy of the MoI approach was substantiated through extensive
experimentation.

  * •

By applying MoI alongside high-quality datasets on the Qwen-7B-chat model, we
developed the Qwen-SFT-MoI model. This model demonstrates superior performance
in mathematical reasoning, code generation, tool usage, and chat benchmarks,
thereby highlighting the effectiveness of MoI in enhancing SFT models.

##  2 Related work

###  2.1 LLMs for mathematical reasoning

Solving complex mathematical tasks is challenging for open-source LLMs.
Research focuses on fine-tuning open-source models (e.g., Touvron et al.
(2023a; b); Bai et al. (2023)) with insights from advanced proprietary LLMs
(OpenAI, 2023; 2022). WizardMath (Luo et al., 2023a) employs a reinforced
evol-instruct method, combining supervised fine-tuning with PPO training to
enhance reasoning. Meanwhile, MAmmoTH (Yue et al., 2023) uses Chain of Thought
and Program-of-Thought to enable LLMs to work with external tools, like Python
interpreters, for mathematical problem-solving.

###  2.2 LLMs for code generation

LLMs have been specifically designed for code generation tasks, such as
StarCoder (Li et al., 2023) and DeepSeek-Coder (Guo et al., 2024).
Alternatively, code-generating LLMs can be derived from fine-tuning general-
purpose models, exemplified by CodeLlama (Rozière et al., 2023) and
WizardCoder(Luo et al., 2023b). Our approach aligns with the latter, fine-
tuning general-purpose LLMs to enhance code generation.

###  2.3 LLM for tool usage.

LLMs in real-world scenarios must adeptly select and apply tools from numerous
APIs. Gorilla (Patil et al., 2023) pairs LLMs with a broad API set, while
ToolAlpaca (Tang et al., 2023) documents 3938 tool uses from over 400 real-
world APIs in 50 categories. ToolLLM (Qin et al., 2023) further contributes
with ToolBench, a rich dataset featuring 16464 real-world APIs across 49
categories from RapidAPI Hub, accommodating diverse single and multi-tool use
cases.

###  2.4 Multi-task learning

During multi-task learning, dataset bias (Hadsell et al., 2020) is a
phenomenon where two or more tasks pull the model parameters in different
directions, thus impacting the multi-task learning performance. Mixture-of-
Experts (MoE) framework (Shazeer et al., 2017; Fedus et al., 2022; Lepikhin et
al., 2020)provides a way to model this mechanism. Such architectures are
designed to support conditional computation in which only certain weights of
the network are activated per input as governed by a gating mechanism.In our
work, we propose a method to mitigate dataset bias that focuses on the data
rather than the model.

##  3 Mixture of instructions

###  3.1 System prompt matters

Our study extends an existing aligned model by advancing its proficiency in
programming, mathematical reasoning, and tool utilization. Initially, we
investigated the performance of the Qwen-7B-chat(Bai et al., 2023) model on
the MT-Bench(Zheng et al., 2024a) dataset, a multifaceted benchmark suite that
evaluates models’ efficacy in eight diverse domains through GPT-4 scoring.
Analysis of the Qwen-7B-chat model on MT-Bench’s coding questions revealed
consistent yet suboptimal scores, particularly to specific question ID
127——”Write a function to find the majority element in a given integer array
using the Boyer-Moore Voting Algorithm.”. Although Qwen-7B-chat successfully
identified the majority element, its approach resulted in a higher time
complexity of O(nlog(n)) and a space complexity of O(n), deviating from the
expected O(n) time and O(1) space complexities of the algorithm.Subsequently,
we discovered that Question ID 127 corresponds to an original problem from the
LeetCode repository. Consequently, we downloaded 2612 LeetCode problems along
with their solutions and trained the Qwen-7B-chat model using SFT. However, we
observed that the model, even after SFT, still failed to answer this
particular question.

Figure 2: The figure illustrates attention maps for the responses of three
models to Question ID 127 in the MT-Bench, where greener regions signify
higher attention levels. The maps show attention distribution across the
system prompt, the question, and each model’s answer. Model (a) Qwen-7B-chat
heavily focuses on the prompt and question but erroneously associates ’Boyer’
with frequency tracking, misrepresenting the Boyer-Moore algorithm. Model (b),
post-SFT on coding data, still pays undue attention to the prompt and
question, disregarding ’Boyer’ and fixating on iterative element search. Model
(c), after SFT with an updated prompt, shifts attention towards generating an
answer, correctly hinting at the use of the Boyer-Moore algorithm, which
fundamentally tracks a candidate element.

As depicted in Figure 2, analysis of attention maps for both the original
Qwen-7B-chat and SFT-enhanced models when processing a specific question
revealed that the original Qwen-7B-chat model favored array traversal and
frequency counting, divergent from the Boyer-Moore algorithm’s approach. The
SFT model also disproportionately focused on array operations and unduly on
the system prompt. This suggests that a consistent default prompt (”You are a
helpful assistant”) during SFT may have inadvertently anchored the model’s
attention, preventing it from engaging more relevant computational strategies.
Introducing an atypical system prompt during SFT could recalibrate focus and
facilitate the integration of novel knowledge.

To redirect the attention of a large language model, we replaced its system
prompt with ’You are a programmer’ in the SFT regime. This change, as shown in
Figure 2, led to a realigned attention map that concentrated on the key
’Boyer-Moore Algorithm’ principles, prompting accurate descriptions of the
algorithm’s iteration and candidate tracking. The shift in focus from the
prompt to the algorithmic core problem and solution was significant.
Additionally, we evaluated the effects of varying SFT-driven system prompts on
model performance using the HumanEval, MBPP, and MT-Bench coding benchmark.
Table 3 conveys that the ’You are a programmer’ prompt engendered a notable
uplift in code generation proficiency.

###  3.2 Balanced sampling diverse system prompting instructions

Upon validating that modifying the system prompt can enhance a model’s coding
capabilities, we pondered whether a unique system prompt could stimulate a
model’s distinct abilities. Consequently, we speculated whether combining
datasets from various tasks according to their respective system prompt
directives could enable a model to concurrently acquire competencies from
different fields. To explore this, we gathered open-source datasets for four
tasks: multi-turn dialogue, code, math, and agent, providing a specific system
prompt for each. However, when we aggregated datasets from multiple tasks for
joint training, we observed that the model trained on combined tasks did not
perform as well as models trained on individual tasks separately2. This
revealed a typical phenomenon in multi-task learning, where training on
combined domain data can cause model bias towards a particular domain due to
dataset bias, leading to improved performance in one domain at the expense of
another.

An effective solution to the dataset bias within the multi-task learning field
is the ’mixture of experts’ (MoE) approach. This method utilizes a gating
mechanism and multiple expert feedforward neural networks (FFNs) to learn
domain-specific knowledge, which is a significant research direction in
language models. The principle of MoE is to use a mixture of models to learn a
single instruction. We wondered whether a similar effect could be achieved by
using a single model to learn a mixture of instructions. To address this, we
proposed a balanced sampling scheme, a method that concatenates instructions
in the order of their tasks. The structure of the balanced sampling method is
illustrated in the figure 4 which sequentially collects instructions from
datasets of various tasks, decodes these instructions into tokens, and
combines them.

$\mathcal{L}_{\text{SFT}}(\pi_{\theta})=-\mathbb{E}_{(x,y)\sim\mathcal{D}}[\log\pi_{\theta}(y\mid
x)]$ (1)

The expression of 1 reveals that during the SFT phase. The equation represents
the negative expected value of the log likelihood for the model
$(\pi_{\theta})$ to correctly predict (y) given (x) under a specified data
distribution $(\mathcal{D})$. Our task is to pre-partition datasets
$\mathcal{D}_{\text{chat}}$, $\mathcal{D}_{\text{code}}$,
$\mathcal{D}_{\text{math}}$, $\mathcal{D}_{\text{agent}}$ for each tasks,
enabling the model to optimize all tasks simultaneously during a single back
propagation pass.Figure 5 demonstrates the effectiveness of the balanced
sampling approach in multi-task learning, achieving stable learning
performance across all tasks.

Figure 3: Attention maps from a Qwen-7B-chat model’s first, middle, and last
layers when processing mixture of instructions. The model attends not only to
the current system prompt but also to initial tokens in concatenated
instructions. By prefixing mixed instructions with the default system prompt,
the model implicitly revisits it during learning of diverse prompts, thereby
facilitating cross-domain knowledge transfer to the domain indicated by the
default prompt. Figure 4: (a). ”Sequence of Instructions” involves extracting
instructions from a dataset, tokenizing them, and using padding tokens to
reach a fixed maximum length for SFT in an LLM. (b). ”Concatenation of
Instructions” merges several instructions into a longer single instruction,
minimizing the need for padding and enhancing training efficiency. (c).
”Mixture of Instructions” employs balanced sampling from various datasets and
concatenates instructions to meet a maximum token length, then prioritizes the
instruction with the default system prompt at the start.

###  3.3 Instructions reordering with a default system prompt

While a balanced sampling scheme can mitigate dataset biases and foster
domain-specific learning in models, users often seek a language model adept
across tasks with just a default system prompt. To merge content from diverse
system prompts into a cohesive domain, we suggest a novel method inspired by
attention sinks (Xiao et al., 2023) and our attention mask performance
analyses.

Our findings, depicted in Figure 3, show that language models from
intermediate to final layers heavily focus on initial tokens when encountering
new tokens. Leveraging this, we propose an innovative technique: reordering
mixed instructions to position the instruction with the default system prompt
’You are a helpful assistant.’ at the start. This design is grounded on the
idea of making the model, during SFT, explicitly recognize its domain-specific
prompts while concurrently responding to a universal, default prompt.

This approach seamlessly integrates learning progress from various tasks under
a singular, default domain framework. This enables the model to maximally
utilize its reasoning abilities, even in the absence of domain-specific system
prompts, providing a unified and effective learning trajectory.

###  3.4 Mixture of instructions

Ultimately, we propose a novel SFT scheme, Mixture of Instructions (MoI),
comprising our balanced sampling and instruction reordering strategies.

Beyond the MoI approach, common methods for training datasets from multiple
tasks include sequence instructions and instruction concatenation. Sequence
instructions method entails feeding individual data entries into the language
model for training, padding each entry to meet the prescribed context length
required for the model’s training if its length is insufficient, with padding
tokens excluded from the loss function calculation. On the other hand,
instruction concatenation (also known as instruction packing) addresses the
efficiency loss in sequence instructions caused by excessive padding when
individual instructions are too short. By concatenating multiple instructions
into a single entry while preserving the format of both single and multi-turn
dialogues, instruction concatenation allows the language model to process more
instructions per training instance, thus enhancing training efficiency without
significantly impacting model performance.

Subsequent experiments 2 comparing MoI with concatenation of instructions
validated the effectiveness of our approach. It alleviates dataset bias in
multi-task learning and efficiently transfers knowledge learned from various
system prompts to a default prompt, significantly enhancing model alignment
and performance.

##  4 Experiments

###  4.1 Experiment setup

All of our experiments are performed on 8x80G A100 GPUs, using the
Qwen-7B-chat model as the base for each independent experiment.We utilized the
Huggingface Transformers(Wolf et al., 2019) library to execute our training
and employed DeepSpeed(Rasley et al., 2020) Zero3 optimization to enhance
memory efficiency.

###  4.2 Training Dataset

For training data, in our experiments, we exclusively used open-source
datasets to validate the effectiveness of our method. The data we utilized
comprises solely open-source datasets, which is shown in Table 1

Dataset | Task | System Prompt
---|---|---
ultrachat200k (Ding et al., 2023) | Chat | You are a helpful assistant.
MagicCoder110k (Wei et al., 2023) | Code | You are a programmer.
CodeFeedback157k (Zheng et al., 2024b) | Code | You are a programmer.
EvolInstructCode80k (Luo et al., 2023b) | Code | You are a programmer.
MathInstruct262k (Yue et al., 2023) | Math | You are a mathematics expert.
MetaMath395k (Yu et al., 2023) | Math | You are a mathematics expert.
GPT4Tools75k (Yang et al., 2024) | Agent | You are an Agent with a lot of tools.
ToolbenchG3 (Qin et al., 2023) | Agent | You are an Agent with a lot of tools.
AgentInstruct (Zeng et al., 2023) | Agent | You are an Agent with a lot of tools.
Table 1: All of our training data are sourced from open datasets downloaded
from Huggingface, categorized into four tasks: multi-turn dialogue, code,
mathematics, and tools usage. Custom system prompts were individually crafted
for the dataset of each task to ensure precise alignment with their respective
content.

###  4.3 Evaluation Benchmark

Mathematic reasoning. GSM8K (Cobbe et al., 2021) presents diverse grade school
math word problems, while the MATH dataset (Hendrycks et al., 2021) focuses on
evaluating mathematical reasoning abilities with a collection of problems.

Code generation. HumanEval (Hendrycks et al., 2021) assesses code generation
models’ problem-solving skills through Python programming challenges, and MBPP
(Austin et al., 2021) targets models’ execution-based performance on Python
problems.

Tool usage.T-EVAL (Chen et al., 2023) breaks down the assessment of tool usage
into several sub-domains, shedding light on LLMs’ comprehensive and specific
competencies.

STEM. MMLU (Hendrycks et al., 2020) covers 57 subjects from STEM to humanities
and social sciences, testing models on a wide range of difficulty levels and
world knowledge. Its broad and detailed subject coverage serves as an
excellent tool for spotting models’ blind spots.

Multi-turn dialogue. MT-Bench (Zheng et al., 2024a) features 80 questions
across eight domains, including writing, roleplay, reasoning, and more, with
each domain offering a two-round query-response evaluation using GPT-4.

Model | MMLU | MBPP | HumanEval | GSM8K | MATH | T-Eval | MT-Bench | MT-Bench | Avg
---|---|---|---|---|---|---|---|---|---
| 0-shot | pass@1 | pass@1 | 0-shot | 0-shot | overall | 1st turn | 2nd turn |
Qwen-7B-chat | 57.0 | 31.8 | 37.2 | 50.3 | 6.8 | 60.7 | 5.65 | 4.51 | 49.3
Qwen-SFT-code | 41.5 | 53.0 | 56.1 | 34.0 | 0.0 | 22.3 | 5.70 | 3.13 | 47.9
Qwen-SFT-math | 45.9 | 36.8 | 22.5 | 71.8 | 27.8 | 14.0 | 5.58 | 3.36 | 51.0
Qwen-SFT-agent | 47.7 | 15.0 | 20.7 | 16.1 | 0.1 | 47.1 | 4.41 | 2.87 | 34.6
Qwen-SFT-chat | 57.8 | 12.6 | 22.5 | 54.5 | 0.01 | 60.8 | 5.73 | 4.45 | 44.5
Qwen-SFT-seq | 59.2 | 46.6 | 50.6 | 69.2 | 30.3 | 69.6 | 5.64 | 4.38 | 61.0
Qwen-SFT-concat | 59.7 | 49.0 | 56.1 | 71.3 | 30.3 | 67.1 | 5.65 | 4.56 | 62.4
Qwen-SFT-balanced | 59.4 | 51.6 | 55.5 | 71.9 | 30.0 | 66.9 | 5.66 | 4.31 | 62.2
Qwen-SFT-MoI | 59.3 | 61.0 | 60.4 | 72.4 | 30.0 | 68.2 | 5.68 | 4.62 | 65.0

Table 2: Performance of SFT models on widely-used benchmarks. We report zero-
shot performance of the models.For all benchmark computations, we consistently
used the system prompt ”You are a helpful assistant.” to ensure the
effectiveness of our methodology.When computing the average score, we
multiplied the mt-bench scores by 10 to ensure numerical validity.

###  4.4 SFT experiments

In our experiments, the Qwen-7B-chat model was fine-tuned using domain-
specific datasets over three epochs at a learning rate of 2e-5, except as
noted. Table 3 lists the system prompts utilized for each dataset.

Qwen-SFT-code. Trained on MagicCoder110k, CodeFeedback157k, and
EvolInstructCode80k.

Qwen-SFT-math. Utilized MathInstruct262k and MetaMath395k datasets.

Qwen-SFT-agent. Fine-tuned with GPT4Tools75k, AgentInstruct, and ToolbenchG3.

Qwen-SFT-chat. Focused on ultrachat200k for dialogue interactions.

Qwen-SFT-seq. Combined datasets, following sequence instruction strategy.

Qwen-SFT-concat. Merged datasets with concatenation strategy, learning rate of
4e-5.

Qwen-SFT-balanced. Balanced sampling from all datasets, learning rate of 4e-5.

Qwen-SFT-MoI. Employed the MoI scheme using ”You are a helpful assistant” as
the default system prompt, learning rate of 4e-5,.

###  4.5 Ablation study and analysis

System prompt. As shown in Table 3, by modifying system prompts for SFT, we
can comprehensively enhance the alignment performance of code-related models.
Furthermore, employing the MoI method has also enabled us to acquire new
knowledge under the default system prompts.

Model | Inference system prompt | MBPP | HumanEval | MT-Bench-coding
---|---|---|---|---
|  | pass@1 | pass@1 | 1st turn
Qwen | You are a helpful assistant. | 31.8 | 37.2 | 3.8
Qwen-SFT-code | You are a helpful assistant. | 53.0 | 56.1 | 4.1
Qwen-SFT-code | You are a programmer. | 55.2 | 60.4 | 5.0
Qwen-SFT-MoI | You are a helpful assistant. | 61.0 | 60.4 | 4.7

Table 3: Performance of SFT models on widely-used code generation benchmarks.
We report both zero-shot and few-shot performance of the models. Figure 5: (a)
The model’s performance using two different methods across various iteration
steps on the MBPP dataset. (b) Comparative performance analysis of the model,
utilizing two distinct approaches, over multiple iteration stages on the GSM8K
dataset. (c) An evaluation of the model’s performance, employing two methods,
through successive iteration steps on the HumanEval dataset. (d) The
performance trajectory of the model, under two methodologies, across a range
of iteration steps on the MATH dataset.

Dataset bias. Figure 5 shows our ablation study compared Qwen-SFT-concat and
Qwen-SFT-balanced, demonstrating that balanced sampling rapidly improves GSM8K
accuracy and stabilizes code generation performance during training. In
contrast, concat showed no substantial late-stage gains in GSM8K and a decline
in code capability. With identical datasets and hyperparameters, these
differences highlight balanced sampling superior handling of multi-task
learning in language models.

Code,Math,Agent. Our comparative ablation study in table 2 revealed that while
models trained on math, and code data respectively enhance performance in
their specific tasks, this specialization comes at the cost of diminished
multi-turn conversation ability, as evidenced by a significant drop in MT-
Bench 2nd turn scores for all models. Thus, models trained on these datasets
are recommended for domain-specific tasks rather than as conversational
assistants.Additionally, models trained on the agent dataset did not show
improvement in any metric, suggesting that the T-Eval dataset serves as a
comprehensive test scheme to assess model capabilities.

Model | Params | HumanEval | MBPP | GSM8K | MATH
---|---|---|---|---|---
|  | pass@1 | pass@1 | Accuracy(%) | Accuracy(%)
GPT-3.5(OpenAI, 2022) | - | 73.2 | 81.7 | 80.8 | 37.8
GPT-4(OpenAI, 2023) | - | 86.6 | 83.0 | 97.0 | 69.7
StarCoder Base(Li et al., 2023) | 7B | 24.4 | 33.1 | - | -
Code Llama(Rozière et al., 2023) | 7B | 33.5 | 41.4 | - | -
Code Llama-Instruct(Rozière et al., 2023) | 7B | 34.7 | 44.4 | - | -
Code Llama-Python(Rozière et al., 2023) | 7B | 38.4 | 47.6 | - | -
WizardCoder(Luo et al., 2023b) | 7B | 48.2 | 56.6 | - | -
Magicoder(Wei et al., 2023) | 7B | 60.4 | 64.2 | - | -
WizardMath(Luo et al., 2023a) | 7B | - | - | 54.9 | 10.7
MetaMath(Yu et al., 2023) | 7B | - | - | 66.4 | 19.4
MAmmoTH(Yue et al., 2023) | 7B | - | - | 59.4 | 33.4
Llama2(Touvron et al., 2023b) | 7B | 12.2 | 20.8 | 16.7 | 3.3
Mistral(Jiang et al., 2023) | 7B | 27.4 | 38.6 | 47.5 | 11.3
Qwen(Bai et al., 2023) | 7B | 37.2 | 35.8 | 52.5 | 10.3
Qwen-SFT-MoI(ours) | 7B | 60.4 | 61.0 | 72.4 | 30.0

Table 4: Scores on HumanEval, MBPP, GSM8K, MATH. The pass@1 scores of our
models are computed with greedy decoding. Models are evaluated in 0-shot on
all datasets.

Comparison with open-source models. Table 4 shows that despite the lack of
extensive code pre-training, our model, through high-quality SFT and the MoI
method, matches the performance of contemporary code models with similar
parameters. Comparisons with open-source math reasoning models indicate our
model’s math skills are on par with those fine-tuned solely on math datasets.
Additionally, our model exhibits improved multi-turn dialogue and common sense
reasoning capabilities.

##  5 Conclusion

In conclusion, our study highlights the significant role of system prompts in
enhancing language model performance during SFT. By refining prompts, we
mitigated issues in code generation with the Qwen-7B-chat model, leading to
better handling of complex tasks. Our innovative multi-task learning strategy,
which utilizes curated datasets with tailored prompts, successfully tackled
the dataset bias seen in multi-task training. The Mixture of Instruction (MoI)
method further ensured consistent performance across tasks. Our finalized
Qwen-SFT-MoI model showcased substantial improvements over the original,
underscoring MoI’s potential for improving language model adaptability and
proficiency.

## References

  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chen et al. (2023) Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al.  T-eval: Evaluating the tool utilization capability step by step.  _arXiv preprint arXiv:2312.14033_ , 2023.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * CodeFuse (2023) CodeFuse.  CodeFuse codeexercise-python-27k, 2023.  URL https://huggingface.co/datasets/codefuse-ai/CodeExercise-Python-27k.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  _arXiv preprint arXiv:2305.14233_ , 2023.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _Journal of Machine Learning Research_ , 23(120):1–39, 2022.
  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al.  Deepseek-coder: When the large language model meets programming–the rise of code intelligence.  _arXiv preprint arXiv:2401.14196_ , 2024.
  * Hadsell et al. (2020) Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  _Trends in cognitive sciences_ , 24(12):1028–1040, 2020.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ , 2020.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ , 2021.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_ , 2020.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  StarCoder: May the source be with you!  _CoRR_ , abs/2305.06161, 2023.  doi: 10.48550/arXiv.2305.06161.  URL https://doi.org/10.48550/arXiv.2305.06161.
  * Luo et al. (2023a) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_ , 2023a.
  * Luo et al. (2023b) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  WizardCoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023b.
  * OpenAI (2022) OpenAI.  Introducing ChatGPT, 2022.  URL https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI.  GPT4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 3505–3506, 2020.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code Llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ , 2023.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.  Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.  _arXiv preprint arXiv:2306.05301_ , 2023.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288, 2023b.  doi: 10.48550/arXiv.2307.09288.  URL https://doi.org/10.48550/arXiv.2307.09288.
  * Wei et al. (2023) Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang.  Magicoder: Source code is all you need.  _arXiv preprint arXiv:2312.02120_ , 2023.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xiao et al. (2023) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.  Efficient streaming language models with attention sinks.  _arXiv preprint arXiv:2309.17453_ , 2023.
  * Yang et al. (2023) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  _arXiv preprint arXiv:2305.18752_ , 2023.
  * Yang et al. (2024) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models, 2023.
  * Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  MAmmoTH: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint arXiv:2310.12823_ , 2023.
  * Zheng et al. (2024a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024a.
  * Zheng et al. (2024b) Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement.  _arXiv preprint arXiv:2402.14658_ , 2024b.

##  Appendix A Appendix

###  A.1 Mixture of instructions details

During the training of our language model, we employed the classic ChatML
format:

        <|im_start|>system
        You are a helpful assistant.
        <|im_end|>
        <|im_start|>user
        How are you?
        <|im_end|>
        <|im_start|>assistant
        As I am a large language model, I do not have personal emotions.
        However, I am functioning properly and ready to assist you with any
        information or tasks you need help with. How may I be of service today?
        <|im_end|>

In the example above, the portion labeled with ’system’ is what we have been
discussing as the ’system prompt’ in our paper.

Figure 6 illustrates the effects of the Mixture of Instruction (MoI)
framework. In the initial phase, models subjected to Supervised Fine-Tuning
(SFT) with the default system prompt strive to replicate the canonical
implementation of the Boyer-Moore algorithm. However, as the process advances,
they divert their approach towards devising an algorithm based on iterative
querying. Subsequent to retraining with a modified system prompt, the model
acquires the capability to accurately script the Boyer-Moore algorithm,
demonstrating a clear understanding of its logic and structure. Utilizing the
Mixture of Instructions (MoI) approach for further training enables the
seamless transference of knowledge acquired in the ’You are a programmer’
domain to applications within the ’You are a helpful assistant’ scenario,
showcasing the versatility and adaptability of the learned skills across
different contexts.

Figure 6: The function of the system prompt and the mapping of knowledge
learned from all system prompts onto the default system prompt following the
application of the Mixture of Instructions method.

###  A.2 System prompt for mathematics task

We replicated the experiment of changing the system prompt from our code
generation task in the context of mathematical reasoning tasks, altering the
default system prompt to ’You are a mathematics expert.’ The experimental
results in Table 5 indicate that this role-setting remains effective in
anchoring mathematical knowledge.

Model | Inference system prompt | GSM8K | MATH | MT-Bench-math
---|---|---|---|---
|  | accuracy | accuracy |
Qwen-7B-chat | You are a helpful assistant. | 50.3 | 6.8 | 2.65
Qwen-SFT-math | You are a helpful assistant. | 71.8 | 27.8 | 4.9
Qwen-SFT-math | You are a mathematics expert. | 72.1 | 30.0 | 5.0
Qwen-SFT-MoI | You are a helpful assistant. | 72.4 | 30.0 | 5.0

Table 5: Performance of SFT models on mathematics reasoning benchmark. We
report zero-shot performance of the models.

###  A.3 MT-bench results

Figure 7 shows that employing the MoI technology in language models
concurrently boosts mathematical reasoning and coding skills, paralleling the
proficiency achieved by models trained solely on individual tasks. This
consistency underscores the robustness of our proposed approach.

Figure 7: Detailed scoring charts on MT-Bench for Qwen-7B-chat, Qwen-SFT-code,
Qwen-SFT-math, and Qwen-SFT-MoI models.

###  A.4 T-EVAL results

Table 6 indicates that the agent capabilities within language models exhibit a
direct association with their chat functionalities. Subjecting the models to
SFT using datasets composed exclusively of code, math, or agent-specific
content can significantly impair their agent abilities. The Sequence of
Instructions approach outperforms other methodologies in agent-related tasks,
which we surmise is due to its compliance with the input format used during
the initial SFT, thus maximally preserving the consistency of the language
model. Furthermore, our MoI technique secures the highest scores in techniques
involving instruction concatenation, suggesting that our method also aims to
retain the original competencies of the language model to the greatest
possible extent.

Model | Instruct | Plan | Reason | Retrieve | Understand | Review | overall
---|---|---|---|---|---|---|---
Qwen-7B-chat | 67.6 | 56.3 | 46.9 | 66.1 | 66.5 | 60.6 | 60.7
Qwen-SFT-code | 0.8 | 0.9 | 35.7 | 29.8 | 52.4 | 14.0 | 22.3
Qwen-SFT-math | 0.3 | 9.2 | 21.5 | 11.2 | 30.3 | 11.7 | 14.0
Qwen-SFT-agent | 44.1 | 22.8 | 37.8 | 58.7 | 63.7 | 55.2 | 47.1
Qwen-SFT-chat | 64.0 | 61.1 | 51.3 | 63.4 | 74.4 | 50.3 | 60.8
Qwen-SFT-seq | 83.7 | 65.8 | 54.4 | 77.1 | 78.4 | 58.3 | 69.6
Qwen-SFT-concat | 83.6 | 62.9 | 51.2 | 72.7 | 75.6 | 56.3 | 67.1
Qwen-SFT-balanced | 82.6 | 64.3 | 51.0 | 73.0 | 75.9 | 54.6 | 66.9
Qwen-SFT-MoI | 78.1 | 65.3 | 53.9 | 76.8 | 77.8 | 57.5 | 68.2

Table 6: Detailed scores of SFT models on T-EVAL. We report zero-shot
performance of the models.

###  A.5 Case Study

[Turn #1] Benjamin went to a bookstore and purchased a variety of books. He
bought 5 copies of a sci-fi novel, each priced at $20, 3 copies of a history
book priced at $30 each, and 2 copies of a philosophy book for $45 each. What
was the total cost of his purchases?
---
Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total
cost of the sci-fi novels is 5 * $20 = $100. He also bought 3 copies of a
history book, each priced at $30, so the total cost of the history books is 3
* $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at
$45, so the total cost of the philosophy books is 2 * $45 = $90. The total
cost of his purchases is $100 + $90 + $90 = $280.
280
The answer is 280
[Turn #2] Suppose Benjamin decides to sell each of these books at a 25% markup
from the price he purchased them. What would be his total revenue if he sold
all the books he bought?
Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total
cost of the sci-fi novels is 5 * $20 = $100. He also bought 3 copies of a
history book, each priced at $30, so the total cost of the history books is 3
* $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at
$45, so the total cost of the philosophy books is 2 * $45 = $90. The total
cost of his purchases is $100 + $90 + $90 = $280. If Benjamin decides to sell
each of these books at a 25% markup from the price he purchased them, the
markup would be 25% of $280, which is 0.25 * $280 = $70. So, the total revenue
Benjamin would get if he sold all the books he bought is $280 + $70 = $350.
350
The answer is 350
[Turn #3]After achieving his profit goal by selling books at a markup,
Benjamin decides to donate a portion of his total revenue to a local library.
If he chooses to donate 15% of his total revenue, how much money will the
local library receive?
Benjamin’s total revenue is $350. If he chooses to donate 15% of his total
revenue, the amount he will donate is 15/100 * $350 = $52.50. So, the local
library will receive $52.50.
52.5
The answer is 52.5
[Turn #4]If Benjamin aims to achieve a profit of $100 over the initial cost of
his purchases by selling all the books, what percentage markup should he apply
to each book’s selling price?
Benjamin aims to achieve a profit of $100 over the initial cost of his
purchases, which is $280. To calculate the markup percentage, we need to find
the difference between the selling price and the cost price, and then divide
that by the cost price. The selling price is the cost price plus the markup,
so the selling price is $280 + $100 = $380. The markup is the difference
between the selling price and the cost price, which is $380 - $280 = $100. To
find the markup percentage, we divide the markup by the cost price and
multiply by 100. The markup percentage is ($100 / $280) * 100 = 35.71%.
35.71
The answer is 35.71
Table 7: Our MoI model demonstrated performance on the MT-Bench math, and we
proceeded to inquire about the third and fourth rounds.

[Turn #1]Write a simple website in HTML. When a user clicks the button, it
shows a random joke from a list of 4 jokes.
---
Sure, here is a simple HTML code that does what you asked for:
<!DOCTYPE html>
<html>
<head>
<title>Joke Generator</title>
<style>
button {
margin: 10px;
padding: 10px;
font-size: 18px;
}
</style>
</head>
<body>
<h1>Joke Generator>/h1>
<button onclick=”displayJoke()”>Generate Joke</button>
<p id=”joke”></p>
<script >
const jokes = [
”Why don’t scientists trust atoms? Because they make up everything!”,
”Why did the scarecrow win an award? Because he was outstanding in his
field!”,
”Why don’t eggs tell jokes? Because they’d crack each other up!”,
”Why don’t programmers like nature? Because they prefer to create their own
world!”
];
function displayJoke() {
const randomJoke = Math.floor(Math.random() * jokes.length);
document.getElementById(”joke”).innerText = jokes[randomJoke];
}
</script >
</body >
</html >
This code creates a simple webpage with a button that, when clicked, generates
a random joke from the list of jokes and displays it in a paragraph element.
[Turn #2]How to use CSS to change the color of jokes to red?
You can use CSS to change the color of the jokes. Here is how you can do it:
<!DOCTYPE html>
<html>
<head>
<title>Random Joke Generator</title>
<style>
body {
font-family: Arial, sans-serif;
}
button {
margin-top: 20px;
}
#joke {
color: read;
}
</style>
</head>
<body>
<h1>Random Joke Generator</h1>

<button onclick=”displayJoke()”>Generate Joke</button>
---
<p id=”joke”></p>
<script >
const jokes = [
”Why don’t scientists trust atoms? Because they make up everything!”,
”Why did the scarecrow win an award? Because he was outstanding in his
field!”,
”Why don’t eggs tell jokes? Because they’d crack each other up!”,
”Why don’t programmers like nature? Because they prefer to create their own
world!”
];
function displayJoke() {
var randomJoke = jokes[Math.floor(Math.random() * jokes.length)];
document.getElementById(”joke”).innerHTML = randomJoke;
}
</script >
</body >
</html >
In this code, the CSS rule p { color: red; } is used to change the color of
all paragraph elements to red.
[TURN #3]How to dynamically retrieve a list of jokes from an external file or
API
You can dynamically retrieve a list of jokes from an external file or API
using JavaScript’s fetch function. Here is an example:
<!DOCTYPE html>
<html>
<head>
<title>Random Joke Generator</title>
<style>
body {
font-family: Arial, sans-serif;
}
button {
margin-top: 20px;
}
#joke {
color: read;
}
</style>
</head>
<body>
<h1>Random Joke Generator</h1>
<button onclick=”displayJoke()”>Generate Joke</button>
<p id=”joke”></p>
<script >
function displayJoke() {
fetch(’https://your-api-url.com/jokes’)
.then(response =>response.json())
.then(jokes =>{
var randomJoke = jokes[Math.floor(Math.random() * jokes.length)];
document.getElementById(”joke”).innerHTML = randomJoke;
})
.catch(error =>console.error(’Error:’, error));
}
</script >
</body >
</html >
Table 8: Our MoI model demonstrated performance on the MT-Bench code, and we
proceeded to inquire about the third round. Questions:
---
You have access to the following API: [
[
’AirbnbSearch.get_property_details’, {
’name’: ’AirbnbSearch.get_property_details’,
’description’: ’This function retrieves the details of a property given its
ID.’,
’required_parameters’: [
{’name’: ’propertyId’,’type’: ’NUMBER’, ’description’: ’The ID of the
property.’}],
’optional_parameters’: [],
’return_data’: [
{ ’name’: ’title’, ’description’: ’The title of the property.’},
{’name’: ’propertyType’, ’description’: ’The type of the property.’},
{’name’: ’personCapacity’, ’description’: ’The capacity of the property.’},
{’name’: ’rating’, ’description’: ’The rating of the property.’},
{’name’: ’latitude’, ’description’: ’The latitude of the property.’},
{’name’: ’longitude’, ’description’: ’The longitude of the property.’}]}
],
[
’AirbnbSearch.get_property_reviews’, {
’name’: ’AirbnbSearch.get_property_reviews’,
’description’: ’This function retrieves the reviews of a property given its
ID.’,
’required_parameters’: [
{’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the
property.’}],
’optional_parameters’: [],
’return_data’: [
{’name’: ’reviews’, ’description’: ’The reviews of the property, containing
comment, rating, and date. At most 3 reviews are returned.’}]}
],
[
’AirbnbSearch.check_availability’,{
’name’: ’AirbnbSearch.check_availability’,
’description’: ’This function checks the availability of a property given its
ID.’,
’required_parameters’: [
{’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the
property.’}],
’optional_parameters’: [
{’name’: ’date’, ’type’: ’STRING’, ’description’: ’The date to check, it must
in the format of YYYY-MM-DD.’}],
’return_data’: [
{’name’: ’availability’, ’description’: ’The availability of the property for
the given date or the next 10 days.’}]}
],
[
’AirbnbSearch.search_property_by_place’, {
’name’: ’AirbnbSearch.search_property_by_place’,
’description’: ’This function takes various parameters to search properties on
Airbnb.’,
’required_parameters’: [
{’name’: ’place’, ’type’: ’STRING’, ’description’: ’The name of the
destination.’}],
’optional_parameters’: [],
’return_data’: [
{’name’: ’property’, ’description’: ’a list of at most 3 properties,
containing id, name, and address.’}]}
]
]
Please generate a plan for answer user’s questions, which should be a list of
actions with the following format:
“‘
---
[
{
// id of the action
”id”: number;
// the name of the action
”name”: string;
// input params required by this action
”args”: ”str(Record<string, any>)”;
},
…
]
“‘
You can imagine args when you plan the action, and these instructions will be
executed sequentially.For example, if you want to call ‘api1‘ with ‘arg1‘ and
‘arg2‘, you can write the following plan:
“‘
[
{
”id”: 0,
”name”: ”api1”,
”args”: ”{’arg1’: ’value1’, ’arg2’: ’value2’, …}”,
},
…
]
“‘
The args should be a dictionary in string format. PLEASE use ” in args
dictionary and use ”” in other places, DO NOT print args with value None or
null.You should only generate a list in json format. The list should be the
full planning list without ‘…‘. DO NOT generate any text to explain the json.
Here is the instruction:
I’m helping a client find the first available rental property in Los Angeles.
Please find a property in Los Angeles. Then, check the availability of the
property for the coming week. Next, get the details of that property. Finally,
retrieve the reviews of the property.
Answer:
[
{
”id”: 0,
”name”: ”AirbnbSearch.search_property_by_place”,
”args”: ”{’place’: ’Los Angeles’}”
},
{
”id”: 1,
”name”: ”AirbnbSearch.check_availability”,
”args”: ”{’propertyId’:<property_id>, ’date’: ’2022-01-01’}”
},
{
”id”: 2,
”name”: ”AirbnbSearch.get_property_details”,
”args”: ”{’propertyId’: <property_id>}”
},
{
”id”: 3,
”name”: ”AirbnbSearch.get_property_reviews”,
”args”: ”{’propertyId’: <property_id>}”
}
]
Table 9: Our MoI model demonstrated performance on the T-EVAL plan.

###  A.6 Solve math problem with code interpreter

In this section, we demonstrate that models trained using the Mixture of
Instructions (MoI) approach can perform functions akin to the code interpreter
capabilities of GPT-4. This case necessitates a comprehensive evaluation of
the model’s mathematical reasoning, code writing, and tool utilisation
proficiencies.

We feed prompts into the Qwen-SFT-MoI system in the following format, adhering
to the ReACT standard during tool use—prompting the model to comply with a
Thought, Action, Action Input sequence to generate the tool’s output:

        You are a helpful assistant.
        Answer the following questions as best you can. You have access to the
        following APIs:

        code_interpreter: available for executing Python code. The drive at
        ’workspace/’ can be used to save and persist user files. Internet access
        for this session is disabled. Do not make external web requests or API
        calls as they will fail.

        Parameters:
        {
          "type": "object",
          "properties": {
              "code": {"type": "string", "description": "executable code"}
           }
        }

        Use the following format:
        Question: the input question you must answer
        Thought: you should always think about what to do
        Action: the action to take, should be one of [code_interpreter]
        Action Input: the input to the action
        Observation: the result of the action
        ... (this Thought/Action/Action Input/Observation can be
        repeated zero or more times)
        Thought: I now know the final answer
        Final Answer: the final answer to the original input question

        Begin!

This interaction informs the model to employ a code interpreter tool to
address mathematical queries. A model proficient in following instructions
should, upon encountering the user’s question, compose code that meets the
formatting requirements. For this functionality, we have developed an entire
pipeline. It filters and extracts the relevant substance from the model’s code
output, then passes it into a Python code interpreter to obtain execution
results, which are subsequently compared with the standard outcomes.

        Questions:
        Henry and 3 of his friends order 7 pizzas for lunch. Each pizza is
        cut into 8 slices. If Henry and his friends want to share the pizzas
        equally, how many slices can each of them have?

        Answer:
        Thought:Let’s solve this problem using Python code.
        Action:code_interpreter
        Action Input: {"code":"number_of_pizzas = 7
        slices_per_pizza = 8
        number_of_friends = 3
        number_of_slices_per_person = number_of_pizzas * slices_per_pizza /
        (number_of_friends + 1)
        number_of_slices_per_person
        "}
        Observation: 14.0
        Thought: I now know the final answer.
        Final Answer: Thus each of them can have \boxed{14} slices.

Our model’s efficacy is validated against the GSM8K benchmark, where our Qwen-
SFT-MoI model achieves an accuracy of 69.14% under a zero-shot setting.
Contrastingly, when the model infers separately on the problems, the accuracy
reaches 72%. This case illustrates that our model not only possesses
commendable mathematical reasoning, code writing, and tool utilisation
capacities but can also synergise these abilities to resolve problems
encountered in practical processes.