# ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language
Models

Jing Liu1,2, Ruihao Gong2,3, Mingyang Zhang4, Yefei He4, Jianfei Cai1, Bohan
Zhuang1
1ZIP Lab, Monash University 2SenseTime Research
3Beihang University 4Zhejiang University
Work done during an internship at SenseTime Research.Corresponding author.
Email: $\tt<EMAIL_ADDRESS>
###### Abstract

The typical process for LLM’s development involves pre-training a general
foundation model on massive data, followed by fine-tuning on task-specific
data to obtain a series of specialized experts. Serving these experts can pose
significant challenges, as loading all experts onto devices is impractical,
and frequent switching between experts in response to user requests can incur
substantial I/O costs, leading to a notable increase in latency and serving
expenses. Previous approaches decompose the expert weights into those of the
pre-trained model and residual delta weights, followed by quantizing the delta
weights using output channel-wise step sizes to reduce the model size.
However, these methods overlook the fact that certain input channels of delta
weights can cause significant quantization errors at extremely low bitwidths.
Additionally, existing methods assume that the appropriate model for a user
request is known in advance, which is not the case in practice. To this end,
we introduce ME-Switch, a memory-efficient expert switching framework tailored
for LLM serving. To condense the number of bits required for describing the
delta weights, ME-Switch employs mixed-precision quantization that selectively
quantizes non-salient input channels of delta weights to extremely low bits
while keeping the salient ones intact, which significantly reduces storage
demand while maintaining performance. Moreover, we develop a routing method
that efficiently directs user queries to the most suitable expert by
transforming the model selection problem into a domain classification problem.
Extensive experiments show the promising memory efficiency and routing
performance of ME-Switch. For example, when serving three models from
Mistral-7B family, ME-Switch reduces the model size by $1.74\times$ and
maintains nearly lossless performance on instruction, mathematical reasoning
and code generation tasks. Furthermore, our method can efficiently serve 16
models from the Mistral-7B family on a NVIDIA A100 GPU.

##  1 Introduction

Large language models (LLMs) such as GPT-4 [6] and Gemini [43], have achieved
significant advancements in natural language processing (NLP). Trained on
large-scale text datasets, these models develop a broad foundation of language
understanding, enabling them to excel at tasks requiring common-sense
knowledge. To acquire task-specific knowledge, LLMs can be further fine-tuned
on specialized tasks, thereby adapting them for diverse applications such as
interactive agents [45, 22], code generation [34, 31], and mathematical
problem solving [33, 31], showing the remarkable versatility of LLMs.
Nevertheless, given that no single model can master all tasks simultaneously,
serving multiple LLMs, each precisely tailored for specific tasks, becomes
essential for good performance.

Figure 1: An illustration of the input channel-wise maximum and minimum values
for the delta weights of Speechless-Code-Mistral-7B. The variability across
input channels highlights that certain salient channels, irrespective of their
magnitude, can cause significant quantization errors when quantized with ultra
low-bitwidth, which underscores their critical role in preserving performance.

However, serving multiple models poses several major challenges. First, even
with a relatively small number of models, the storage demands are significant
due to the extensive number of parameters each model contains. For example,
three LLaMA-2-70B models would collectively require over 384GB of storage,
calculated as 128GB per model times three. Second, the substantial memory
requirements of these models may make it impractical to load all of them into
GPU memory simultaneously. While dynamically swapping model weights in and out
of GPU memory as needed is feasible, the large size of the models makes this
process slow and inefficient, significantly delaying response times and
adversely affecting user experience.

To address the above challenges, existing methods [28, 50] decompose the
weights of fine-tuned models into the weights of the pre-trained model and the
delta weights introduced during fine-tuning. By applying per-tensor
quantization [35] to these delta weights, it is possible to significantly
reduce storage requirements, thereby enabling efficient sharing of the base
model’s storage across multiple fine-tuned models. Nevertheless, using per-
tensor quantization ignores the distinction in delta weight values across
different input and output channels, resulting in substantial information
loss. To mitigate this issue, finer granularity quantization, such as output
channel-wise quantization [48, 47, 27] where each output channel is allocated
its own learnable step size, still neglects the variations in input channels,
as shown in Figure 1. To further mitigate information loss, one may rescale
the input channels of delta weights before quantization [26]. However, with
extremely low-bitwidth quantization, rescaling provides only limited
alleviation. Moreover, existing methods assume that the appropriate model to
utilize in response to a user request is known in advance. In reality,
determining which model to use dynamically is crucial for handling diverse
user queries efficiently. This gap highlights the need for an intelligent
model selection mechanism that can adaptively choose the best-suited model
based on the specific requirements of each user request.

In this paper, we propose ME-Switch, a memory-efficient expert switching
framework tailored for LLMs. To reduce the quantization error while
simultaneously reducing storage needs, we develop a mixed-precision
quantization method that quantizes non-salient input channels of delta weights
to extremely low bits while preserving those salient ones, which could
substantially increase quantization errors when quantized at very low
bitwidths, in full precision. Since the number of salient input channels is
relatively small, incorporating a limited amount of high-precision delta
weights incurs negligible memory overhead and inference cost. To identify the
important input channels of delta weights, one may select based on their
magnitudes [13], as shown in Figure 2(a), which however fails to capture those
leading to high quantization error. In contrast, our approach identifies
salient input channels of delta weights based on their impact on
reconstruction errors in the output activations, as illustrated in Figure
2(b). Additionally, we develop a routing method that selects the most suitable
LLM in response to a user request, enabling efficient and accurate model
switching. To simplify the routing problem, we assume that each LLM
specializes in a distinct domain, such as code generation or mathematics
reasoning. This allows us to treat model selection as a multiple-choice
question-answering task where each option represents a target domain,
effectively transforming the problem into a domain classification task, as
shown in Figure 3. Notably, dialogue LLMs exhibit strong instruction-following
capabilities. Leveraging this, we propose using a small pre-trained dialogue
LLM as a model-level router to solve the multiple-choice question-answering
task. We then construct a multiple-choice question-answering dataset and fine-
tune the router to improve its performance. Given its smaller size compared to
the specialized LLMs, the router operates efficiently without adding
significant overhead.

Our contributions can be summarized as follows. 1) We propose a mixed-
precision quantization method that significantly reduces the storage demands
of serving multiple LLMs while maintaining performance. This is achieved by
selectively quantizing the non-salient input channels of delta weights,
leaving the salient ones unchanged. 2) We introduce a routing method that
dynamically selects the most appropriate LLM for a given query, ensuring
efficient and accurate model switching. This is accomplished by transforming
the model selection problem into a domain classification task and then solving
it with a small LLM. 3) We conduct extensive experiments demonstrating the
promising memory efficiency and routing performance of ME-Switch. Remarkably,
when serving three models from the Mistral-7B family, ME-Switch not only
delivers near-lossless performance on instruction, mathematical reasoning, and
code generation tasks but also reduces the model size by $1.74\times$. Even
more impressively, our method is able to simultaneously serve up to $16$
models for Mistral-7B family on a single NVIDIA A100 GPU without encountering
the out-of-memory issue.

##  2 Related Work

Efficient LLM serving. LLMs can be efficiently deployed on GPUs for high-
throughput serving using several inference frameworks, such as vLLM [24] and
Orca [51]. Given that a single LLM cannot excel across all domains, it is
crucial to serve multiple LLMs simultaneously to handle diverse user queries
effectively. To determine which model to use during inference, Zooter [32]
utilizes a task-level router to load different LLMs based on the incoming task
requirements. This approach, however, introduces significant memory
challenges, as hosting all LLMs on a GPU simultaneously can excessively strain
the available VRAM. One solution is to serve multiple Low-Rank Adaptation
(LoRA) modules within a multi-tenant serving system, such as Punica [9] and
S-LoRA [38]. Another viable strategy to address the above issue involves
dynamically loading different LLMs from CPU memory to GPU memory as needed,
thereby reducing peak GPU memory utilization. Nevertheless, this dynamic
loading competes for GPU memory bandwidth with model-swapping operations,
presenting a significant bottleneck. To mitigate these challenges, techniques
like Deltazip [50] and BitDelta [28] have been developed to compress the delta
parameters. These methods allow task-specific delta parameters to be loaded
into GPU memory on-demand, ensuring that only a single pre-trained model’s
parameters reside permanently in GPU memory. This approach aims to achieve
low-latency inference while minimizing the costs associated with maintaining
numerous fine-tuned models in GPU memory. Our approach stands out by serving
multiple LLMs with a minimal memory footprint with nearly lossless
performance, which is achieved through an advanced delta weights quantization
strategy and an adaptive model-level router.

Delta compression. In recent years, numerous approaches have focused on
reducing the storage overhead for maintaining different task-specific models
through delta parameter compression. Model merging strategies often
incorporate multiple tasks’ delta parameters [14] into the pretrained
parameters to minimize the number of parameters needed for multi-task
operations. To address the parameter conflicts that often arise during model
merging, Yu et al. [53] proposed a method involving massive unstructured
random pruning (achieving 90% sparsity) of delta parameters and subsequent
rescaling. This approach ensures that the accuracy of downstream tasks is not
compromised. In a similar vein, Ties-merging [49] developed a pruning strategy
based on the magnitude and sign of delta parameters to further reduce
conflicts and storage needs. Additionally, the LoRA-based PEFT methods [19,
46] introduces a novel approach by learning one or several low-rank matrices
to represent the delta parameters. ZipLoRA [37] explores the sparsity within
these low-rank matrices, allowing the fusion of low-rank matrices from
different tasks to further reduce the spatial overhead of multi-task models.
However, these merged models often suffer from notable performance
degradation. Conversely, some studies focus solely on compressing delta
parameters for each task without merging, thereby mitigating performance
degradation across multiple tasks. For example, Ryu et al. [36] combine
quantization and low-rank estimation techniques to reduce the storage size of
delta parameters. Liu et al. [28] push this further by quantizing each task’s
delta parameters to 1 bit, achieving more than a tenfold compression. However,
these quantization methods often lack robust handling of outliers, resulting
in performance declines compared to uncompressed delta parameters. Moreover,
models that do not integrate multi-task delta parameters require manual
activation of specific delta parameters for each task, which reduces the
model’s applicability in multi-task environments.

##  3 Preliminaries

Figure 2: An illustration comparison between the magnitude-based selection of
salient delta weights and our reconstruction-error-based selection method.
Given a delta weight matrix $\Delta\in\mathbb{R}^{m\times n}$, its quantized
version $\hat{\Delta}$, and input ${\bf x}\in\mathbb{R}^{m}$, where $m$ and
$n$ denote the number of input and output channels, respectively, our method
measures the importance of each input delta channel by $\sum_{j=1}^{n}\|{\bf
x}_{i}\Delta_{ij}-{\bf x}_{i}\hat{\Delta}_{ij}\|_{2}^{2}$.

For simplicity, we employ uniform quantization [21] to compress the models.
Given a matrix ${\bf X}$ with floating-point values (_e.g._ , FP16 or FP32),
the quantization process can be expressed as:

$\begin{array}[]{ll}\hat{{\bf X}}=\mathrm{quant}({\bf
X})=\mathrm{clamp}\left(\lfloor\frac{{\bf
X}}{s}\rceil,-Q_{N},Q_{P}\right)\times s,\end{array}$ (1)

where the function $\mathrm{clamp}({\bf V},{\bf V}_{\mathrm{min}},{\bf
V}_{\mathrm{max}})$ clamps all elements in ${\bf V}$ within the range $[{\bf
V}_{\mathrm{min}},{\bf V}_{\mathrm{max}}]$, the operator $\lfloor\cdot\rceil$
rounds a given value to the nearest integer, and $s$ is a learnable
quantization step size initialized by $\max(|{\bf X}|)/(2^{b-1}-1)$. Here,
$Q_{N}$ and $Q_{P}$ denote the number of negative and positive quantization
levels, respectively. For $b$-bit quantized weights, $Q_{N}$ and $Q_{P}$ are
set to $2^{b-1}$ and $2^{b-1}-1$, respectively. Since the rounding function is
not differentiable, we use the straight-through estimator (STE) [15] for
gradient approximation following [15]. For binary quantization, we use the
quantization method following [35].

Recent studies [28, 50] have shown that the weights of a fine-tuned model can
be decomposed into the weights of the pre-trained model and the delta weights
introduced during fine-tuning. Let ${\bf W}\in\mathbb{R}^{m\times n}$ and
${\bf W}_{\mathrm{FT}}\in\mathbb{R}^{m\times n}$ be the weight matrices of the
pre-trained model and the fine-tuned model, respectively, where $m$ represents
the number of input channels and $n$ denotes the output number of channels.
The delta weight is defined as $\Delta={\bf W}_{\mathrm{FT}}-{\bf W}$. To
reduce storage requirements, one can perform quantization using Eq. (1) to
compress $\Delta$. However, this method often results in significant
performance degradation because it assumes all delta weight channels are
equally sensitive to quantization noise. Additionally, existing methods assume
the appropriate model for a user request is predetermined. In practice,
dynamically choosing the most suitable model based on the specific nature of
each user’s query is more effective.

##  4 Proposed Method

In this section, we propose ME-Switch, a memory-efficient expert switching
framework designed to optimize the deployment of LLMs. ME-Switch addresses the
challenges of reducing storage demands while maintaining performance and
introduces model-level routing to efficiently manage task-specific queries. We
will introduce the details of ME-Switch in the following.

###  4.1 Salient-Aware Delta Compression

In this section, we introduce our salient-aware delta compression approach,
which specifically targets the preservation of salient input channels during
quantization. While quantization methods with learnable step sizes for each
output channel can handle variations in output channels effectively [48, 27],
variations in input channels pose a significant challenge to maintaining model
performance after quantization, as shown in Figure 1. Some salient input
channels of delta weights, regardless of their magnitude, can cause
substantial information loss and degrade model performance when quantized to a
very low bitwidth. To address this, rescaling the input channels of delta
weights before quantization [26] offers a solution to mitigate some of the
quantization error, which however provides only limited alleviation under the
context of extremely low-bit quantization.

Salient-aware delta selection. To protect salient input channels of delta
weights, we develop a mixed-precision quantization method, where the majority
of input channels of delta weights are quantized to low-bit precision, while
only a small number of critical input channels are represented in 16-bit. This
approach achieves significant reductions in storage requirements with minimal
performance loss. The remaining challenge lies in identifying these salient
input channels. A naive approach might involve selecting the input channels
with large magnitudes as the salient delta weights [13], as shown in Figure
2(a). However, this method neglects the influence of input activations on the
outputs, failing to identify channels that lead to high quantization error. To
address this, inspired by the pruning metric [41], we introduce a new salient
delta weights selection metric based on reconstruction errors in the outputs,
considering both weights and input activations, as shown in Figure 2(b).
Specifically, given an input ${\bf x}\in\mathbb{R}^{m}$, the reconstruction
error for the input channel $i$ can be calculated by

$\begin{array}[]{ll}\sum_{j=1}^{n}\|{\bf x}_{i}\Delta_{ij}-{\bf
x}_{i}\hat{\Delta}_{ij}\|_{2}^{2},\end{array}$ (2)

where $\hat{\Delta}$ is the quantized delta weights and $n$ denotes the output
channel number. With the reconstruction error defined, we choose those input
channels with the top-$k$ largest reconstruction errors as the salient ones
and retain them in 16-bit representation to maintain performance.

Efficient distillation. The quantization step size defined in Eq. (1) plays an
important role in the final performance. To learn the quantization step size,
we employ knowledge distillation to guide the alignment of the output logits
of the quantized model with those of the full-precision fine-tuned model
following [28]. To reduce training overhead, we freeze the model weights and
focus solely on optimizing the quantization step size ${\bf s}$ using a small
calibration dataset $\mathcal{X}$ by solving problem

$\begin{array}[]{ll}\arg\min_{{\bf
s}}\|f({\mathcal{X}})-\hat{f}(\mathcal{X};{\bf s})\|_{2}^{2},\end{array}$ (3)

where $f(\cdot)$ and $\hat{f}(\cdot)$ denote the output logits of the fine-
tuned and quantized models, respectively. Thanks to the reduced number of
trainable parameters, this training process is highly efficient.

On-demand swapping. The extremely low-bit compression for delta weights
significantly reduces the model size, thereby alleviating storage demands.
This approach enables us to maintain a single pre-trained model while storing
multiple sets of compressed delta weights, facilitating efficient on-demand
swapping. In this scenario, the pre-trained model remains in GPU memory, and
the corresponding compressed delta weights are loaded dynamically based on the
user query. Compared with directly serving multiple models, our method is more
GPU memory-efficient. Since LLM decoding is memory-bound [29, 39] due to the
auto-regressive nature, reducing model size effectively decreases the
parameter loading time, thereby improving decoding latency. To achieve fast
inference, we decouple the matrix multiplication during inference into two
components:

$\begin{array}[]{ll}{\bf y}={\bf x}{\bf W}_{\mathrm{FT}}={\bf x}({\bf
W}+\Delta)\approx{\bf x}{\bf W}+{\bf x}\tilde{\Delta},\end{array}$ (4)

where $\tilde{\Delta}$ represents the compressed delta weights, including both
the quantized unsalient and the full-precision salient delta weights. For
${\bf x}{\bf W}$, the computation is performed using a FP16 batched GEMM
kernel. For ${\bf x}\tilde{\Delta}$, we implement an efficient Triton kernel
[44] that fuses dequantization and matrix multiplication for efficient
computation.

Figure 3: An illustration of the model-level routing. We first prompt the
model-level router with the user query using a template (See Section A for
more details) that presents a list of potential domains. The router then
assesses these options and selects the most relevant domain by answering a
multiple-choice question, effectively classifying the query into the
corresponding category.

###  4.2 Model-level Routing

In this section, we present a method to determine the appropriate model for a
given user query. Consider a set of LLMs represented as
$\mathcal{F}=\\{f_{1},f_{2},\cdots,f_{M}\\}$, where $M$ denotes the number of
models. Given a user query $q$, we aim to find the most suitable LLM by
solving the following problem $\arg\max_{f\in\mathcal{F}}P(q,f(q))$, where $P$
is a function that measures the quality or performance of the LLM response. To
simplify the routing process, we assume that each LLM in $\mathcal{F}$
specializes in distinct domains such as code generation or mathematical
problem solving, This setup allows us to treat the routing challenge as a
multiple-choice question-answering task, where each option corresponds to a
specific domain, thereby transforming the problem into a domain classification
problem. Note that dialogue LLMs like Qwen1.5-1.8B-Chat [8] exhibit
capabilities in following instructions, which inspires us to utilize a small
pre-trained LLM as a model-level router. As illustrated in Figure 3, we first
prompt the router with the user’s question using a template designed to elicit
domain classification. For the prompt template, please refer to Section A of
the supplementary material. Based on the router’s response, we then
dynamically load the corresponding compressed delta weights for the selected
domain-specific model, such as a mathematical model, to generate outputs.

Since the router is not explicitly trained for query domain classification,
its initial routing performance may be suboptimal. To improve the routing
performance, we construct a multiple-choice question-answering dataset
tailored for our routing problem. We collect instruction-following data from
various domains and insert the query into the prompt template as shown in
Figure 3. The responses are constructed by considering the correct domain-
specific model choice. We then fine-tune the router with our constructed
dataset using supervised fine-tuning to further improve the routing accuracy.

Discussion with Mixture of Experts (MoE). Besides model-level routing, another
approach to handle diverse user queries efficiently is to construct a mixture-
of-experts using a set of pre-trained models. This can be achieved by
integrating the feedforward layers from all pre-trained LLMs into a single MoE
module at each attention-FFN block, and merging other layers, such as self-
attention layers, by simply averaging their weights [40]. An additional gate
network is introduced for each MoE module to perform token-level routing.
However, this approach requires extensive fine-tuning of the entire network
parameters and the gating network, as there is a significant gap between MoE
experts and pre-trained LLMs. Notably, each expert in an MoE tends to become a
generalist across all domains due to the load balancing loss, which encourages
an even distribution of the workload among experts [16, 23]. This contrasts
with pre-trained LLMs, which are typically specialists within their specific
domains.

Model size reduction analysis. Let $\Psi$ be the model size of an FP16 pre-
trained model. When storing $M$ FP16 models, the total model size is $M\Psi$.
In contrast, using our method, we store a single base model, a model-level
router and $M$ compressed delta models. The total model size is
$\Psi+M\tilde{\Psi}+\Phi$, where $\tilde{\Psi}$ represents the size of the
compressed delta model and $\Phi$ denotes the storage requirement for the
router. Therefore, the compression ratio can be computed by
$M\Psi/(\Psi+M\tilde{\Psi}+\Phi)$.

##  5 Experiments

Candidate LLMs. We apply our ME-Switch to two model families, Mistral-7B [22]
and LLaMA-2-13B [45]. For the Mistral family, we include
Dolphin-2.2.1-Mistral-7B [3] as the instruction expert, Speechless-Code-
Mistral-7B [5] as the code expert, and MetaMath-Mistral-7B [52] as the math
expert. For the LLaMA-2-13B family, LLaMA-2-13B-Chat [45] serves as the
instruction expert, MetaMath-13B [52] as the math expert, and
LLaMA2-Chinese-13B-Chat [4] as the Chinese expert. The above models are fine-
tuned based on pre-trained backbones. We use Qwen1.5-1.8B-Chat [8] as the
model-level router.

Training and testing datasets. We collect a diverse set of instruction samples
from various open-source datasets, including Alpaca [42] for the instruction
domain, MetaMathQA [52] for the mathematics domain, Code-74k-ShareGPT [2] for
the code domain, and Chinese Alpaca [1] for the Chinese domain. To evaluate
the performance of the resulting LLMs, we report accuracy on several
benchmarks across different domains: MMLU [17] for the instruction, GSM8K [11]
and MATH [18] for the mathematics, HumanEval [10] and MBPP [7] for the code,
and C-Eval [20] and C-MMLU [25] for the Chinese. We use the WizardCoder
toolbox to evaluate on HumanEval and MBPP, and the OpenCompass toolbox [12] to
evaluate the other datasets.

Implementation details. For salient-aware delta compression, we construct a
calibration set from each domain-specific dataset and use these sets to
compress the corresponding delta weights for each domain. Each calibration set
consists of 1600 randomly sampled sequences, each with a length of 128 tokens.
The bitwidth $b$ and the number of FP16 input channels $k$ are set to 2 and 8,
respectively. We employ the AdamW optimizer [30] with a learning rate of
$10^{-5}$ for 1 epoch, with a mini-batch size of 4. Delta weights compression
experiments for the Mistral family are conducted on two NVIDIA A100 80G GPUs,
while for the LLaMA-2-13B model, we use four NVIDIA A100 80G GPUs.

For the model-level router training, we construct the training data using
samples from various domains, as mentioned in Section 4.2. To balance the
dataset, we extract an equal number of samples from each domain-specific
dataset. This constructed dataset is used to fine-tune model-level router
through supervised fine-tuning for 4 epochs on a machine with 8 $\times$ A100
GPUs. We use the AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.95$,
setting the learning rate to $3\times 10^{-4}$ and applying a linear learning
rate warmup. The weight decay is set to 0.01. We set the per-device mini-batch
size to 8 and use gradient accumulation steps of 2.

Table 1: Main results for Mistral-7B and LLaMA-13B families.

Model | MMLU (%) $\uparrow$ | Mathematical Reasoning (%) $\uparrow$ | Code Generation (%) $\uparrow$
---|---|---|---
STEM | Hums. | Social | Other | Avg. | GSM8K | Math | Avg. | HumanEval | MBPP | Avg.
Dolphin-2.2.1-Mistral-7B | 52.05 | 68.83 | 73.42 | 65.43 | 63.43 | 63.68 | 12.80 | 38.24 | 42.70 | 54.90 | 48.80
MetaMath-Mistral-7B | 50.45 | 66.82 | 71.63 | 64.60 | 61.87 | 73.92 | 20.62 | 47.27 | 0.00 | 21.60 | 10.80
Speechless-Code-Mistral-7B | 51.82 | 68.35 | 73.74 | 65.69 | 63.36 | 61.18 | 13.52 | 37.35 | 51.20 | 60.40 | 55.80
ME-Switch | 51.49 | 68.37 | 73.60 | 66.08 | 63.32 | 73.39 | 20.30 | 46.85 | 51.80 | 60.70 | 56.25
Model | MMLU (%) $\uparrow$ | Mathematical Reasoning (%) $\uparrow$ | Chinese (%) $\uparrow$
STEM | Hums. | Social | Other | Avg. | GSM8K | Math | Avg. | C-Eval | C-MMLU | Avg.
LLaMA-2-13B-Chat | 44.26 | 59.79 | 63.20 | 56.57 | 54.60 | 43.75 | 5.20 | 24.48 | 36.13 | 38.71 | 37.42
MetaMath-13B | 37.81 | 52.77 | 56.00 | 50.05 | 47.84 | 69.14 | 8.48 | 38.81 | 33.62 | 32.70 | 33.16
LLaMA2-Chinese-13B-Chat | 45.24 | 60.01 | 62.47 | 55.92 | 54.67 | 38.89 | 4.54 | 21.72 | 40.28 | 39.16 | 39.72
ME-Switch | 44.25 | 59.65 | 63.18 | 56.64 | 54.57 | 69.98 | 13.14 | 41.56 | 40.46 | 39.75 | 40.11

Table 2: Effect of salient-aware delta compression for Speechless-Code-
Mistral-7B and MetaMath-Mistral-7B. $\tilde{\Delta}$ denotes the compressed
delta weights.

Model | Bit | # FP Channels | Model Size (GB) | Code Generation (%) $\uparrow$
---|---|---|---|---
HumanEval | MBPP | Avg.
Speechless-Code-Mistral-7B | 16 | 0 | 13.48 | 51.20 | 60.40 | 55.80
1-9 Speechless-Code-Mistral-7B $\tilde{\Delta}$ | 1 | 0 | 1.30 | 47.00 | 58.40 | 52.70
Speechless-Code-Mistral-7B $\tilde{\Delta}$ | 2 | 0 | 2.11 | 47.00 | 59.10 | 53.05
Speechless-Code-Mistral-7B $\tilde{\Delta}$ | 2 | 8 | 2.13 | 51.80 | 60.70 | 56.25
Model | Bit | # FP Channels | Model Size (GB) | Mathematical Reasoning (%) $\uparrow$
GSM8K | Math | Avg.
MetaMath-Mistral-7B | 16 | 0 | 13.48 | 73.92 | 20.62 | 47.27
1-9 MetaMath-Mistral-7B $\tilde{\Delta}$ | 1 | 0 | 1.30 | 53.45 | 1.50 | 27.48
MetaMath-Mistral-7B $\tilde{\Delta}$ | 2 | 0 | 2.11 | 73.31 | 20.44 | 46.88
MetaMath-Mistral-7B $\tilde{\Delta}$ | 2 | 8 | 2.13 | 73.62 | 20.48 | 47.05

###  5.1 Main Results

To evaluate the efficacy of our proposed model, we apply ME-Switch to the
Mistral-7B and LLaMA-13B model families. The experimental results, detailed in
Table 1, demonstrate that ME-Switch, even with extremely compressed delta
weights, achieves performance comparable to that of the respective unquantized
expert models across various downstream tasks. For the Mistral-7B family, on
MMLU, ME-Switch trails the instruction expert by only 0.1% and lags behind the
math expert by just 0.4% in mathematical reasoning tasks. Notably, ME-Switch
consistently outperforms the code expert in code generation tasks. The good
performance of ME-Switch can be attributed to our accurate selection of
salient delta weight channels, efficient distillation for step size learning,
and precise routing for user queries. Furthermore, larger models typically
contain a higher proportion of redundancy. Consequently, our ME-Switch
exhibits even better performance on the LLaMA-13B family. For example, ME-
Switch achieves an average accuracy improvement of 0.39% over the Chinese
expert in the Chinese domain.

###  5.2 Ablation Studies

Effect of salient-aware delta compression. To validate the effect of our
proposed salient-aware delta compression, we compress the delta weights of
Speechless-Code-Mistral-7B and MetaMath-Mistral-7B with different bitwidths
and evaluate their performance on code generation and mathematical reasoning
tasks, respectively. The results presented in Table 2, indicate that
quantizing delta weights to only 1-bit leads to a significant performance
drop. Increasing the bitwidth to 2 improves performance, but it still lags
behind that of the full-precision model. Notably, despite maintaining a
minimal number of FP16 channels, the model size of the mixed-precision model
(2.13 GB) is nearly unchanged compared to the fully quantized model (2.11 GB).
However, introducing a small number of FP channels significantly improves
performance. For example, our compressed Speechless-Code-Mistral-7B, with a
model size reduced by 6.33$\times$, outperforms the full-precision counterpart
by 0.45% in average accuracy on code generation tasks. This underscores the
capability of salient-aware delta compression to minimize model size while
preserving comprehensive model performance.

Table 3: Performance comparisons with other weight-only quantization methods.

Method | FP | 1-bit | AWQ | Random | Wanda | Magnitude | Ours
---|---|---|---|---|---|---|---
HumanEval | 51.20 | 47.00 | 47.00 | 46.30 | 48.20 | 48.20 | 47.60
MBPP | 60.40 | 58.40 | 58.10 | 58.60 | 58.60 | 58.90 | 60.70
Avg. | 55.80 | 52.70 | 52.55 | 52.45 | 53.40 | 53.55 | 54.15

Performance comparisons with other weight-only quantization methods. To
demonstrate the promising performance of our salient-aware delta compression,
we also include the following weight-only quantization methods: AWQ: we use
AWQ [26] to rescale input channels of delta weights before quantization to
mitigate quantization errors. Random: using our salient-aware delta
compression, we randomly select some input channels from the delta weights as
important channels. Wanda: leveraging our salient-aware delta compression, we
select important input channels of delta weights using the pruning metric from
Wanda [41]. Magnitude: within our salient-aware delta compression framework,
we select sensitive input channels of delta weights based on their weight
magnitude, following the method proposed by [13]. For fair comparisons, we
apply all methods to compress the delta weights of Speechless-Code-Mistral-7B
to 1-bit. All mixed-precision methods retain 32 FP16 channels. We also include
a 1-bit baseline for comparison. The results are shown in Table 3. From the
results, we observe that AWQ achieves comparable performance to the 1-bit
baseline, highlighting the limitations of rescaling in extremely low-bitwidth
quantization. In contrast, keeping the salient delta input channels performs
favourably against the rescaling input channel counterpart. Moreover, our
salient channel selection demonstrates superior performance than Random,
Wanda, and Magnitude metrics. For example, our salient-aware delta compression
outperforms Wanda by 0.75% on the average accuracy, showcasing its
effectiveness in selecting the salient delta weights.

Figure 4: Effect of supervised fine-tuning (SFT) in model-level routing. We
assess the performance of routing by measuring the accuracy on a 4-domain
classification task (instruction, mathematics, code, and Chinese).

Figure 5: Model size reduction results in terms of Mistral-7B family. The
model sizes for the a single 16-bit floating-point model, a compressed model,
and the router are 13.48 GB, 2.13 GB, and 3.42 GB, respectively.

Effect of supervised fine-tuning in model-level routing. To investigate the
effect of supervised fine-tuning (SFT) on model-level routing, we evaluate the
domain classification performance of the router (_i.e._ , Qwen1.5-1.8B-Chat)
across four domains: instruction, mathematics, code, and Chinese. As shown in
Figure 5, the pre-trained router performs poorly in domain classification
without fine-tuning, achieving a Top-1 accuracy of only $5.80\%$ on C-Eval.
However, with SFT, the router’s performance improves significantly, reaching
nearly 100% accuracy across all domains. This demonstrates that supervised
fine-tuning greatly enhances the instruction-following capabilities of the
router, thereby improving its routing performance.

Figure 6: Model size reduction results in terms of LLaMA-13B family. The model
sizes for a single FP16 model, a compressed model, and the router are 24.23
GB, 3.60 GB, and 3.42 GB, respectively.

Figure 7: Decoding latency for Mistral-7B. “Naive” denotes the naive inference
with $M$ fine-tuned models. “Ours” represents batch inference with our method.
Out-of-memory scenarios are indicated as “OOM”.

Model size reduction analysis. To investigate the model size reduction as
discussed in Section 4.2, we compare the total storage requirements of full-
precision models with those of our compressed models for the Mistral-7B and
LLaMA-13B families across varying model counts, as shown in Figures 5 and 7.
As the number of models increases, the compression ratios improve
substantially. For instance, with nine models, our method achieves a
3.36$\times$ reduction compared to full-precision models. These savings become
even more pronounced with larger model sizes, reaching up to a 3.63$\times$
reduction for LLaMA-13B family.

Latency analysis. To assess the latency improvements from delta weights
compression, we measured the end-to-end decoding latency of the Mistral-7B
model with an input sequence length of 128 on a single NVIDIA A100. Decoding
latency is critical, as it typically dominates processing time in LLM
operations [26, 29]. Our efficient Triton kernel, which enables batched matrix
multiplication between multiple compressed weight matrices and high-precision
input activations, is compared against the conventional approach of
individually processing multiple models. Results depicted in Figure 7
illustrate that while our method may perform slightly slower than the naive
approach for a small number of models, it provides lower latency as the number
of models is greater than 4. More importantly, unlike the naive approach, our
method is able to simultaneously serve 16 models on GPUs without running into
out-of-memory (OOM) issues, demonstrating better scalability and efficiency in
high-load scenarios.

##  6 Conclusion and Future Work

In this paper, we have introduced ME-Switch, a memory-efficient expert
switching framework designed for LLMs. Our method has addressed the critical
challenge of balancing model performance with storage efficiency. The core of
our ME-Switch lies in a novel mixed-precision quantization method that
selectively compresses non-salient delta weights to extremely low-bit
precision while preserving salient delta weights. Additionally, we have
developed a model-level router that dynamically selects the most suitable LLM
for a given query by transforming the model selection problem into a domain
classification task. Extensive experiments on Mistral-7B and LLaMA-13B
families have demonstrated that ME-Switch achieves performance comparable to
unquantized expert models across various tasks while significantly reducing
model size. In terms of limitations, our ME-Switch relies on accurate domain
classification by the model-level router, which can sometimes misclassify and
impact performance. Future work could explore more advanced or hybrid routing
strategies, such as Mixture of Experts (MoE), to enhance accuracy.
Additionally, quantizing the base model itself could further reduce the
overall model size. This approach would require careful consideration of the
combined effects of quantizing both the base model and the delta weights to
ensure performance is maintained. Furthermore, reducing the bitwidth of the KV
Cache could accelerate the decoding speed, offering additional efficiency
improvements.

Broader Impacts. The substantial storage savings and competitive accuracy
highlight ME-Switch’s potential for scalable and cost-effective AI
deployments. The improved efficiency and scalability of ME-Switch could lead
to the proliferation of LLMs, making it easier to deploy these models in
various applications, including those that might be used maliciously. The
widespread availability of powerful LLMs increases the risk of misuse in
generating harmful content, spreading misinformation, and exacerbating issues
related to bias and discrimination inherent in the training data.

## References

  * [1] alpaca-zh-51k.  https://huggingface.co/datasets/hfl/alpaca_zh_51k.
  * [2] Code-74k-ShareGPT.  https://huggingface.co/datasets/ajibawa-2023/Code-74k-ShareGPT.
  * [3] Dolphin-2.2.1-Mistral-7B.  https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b.
  * [4] Llama2-Chinese-13b-Chat.  https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat.
  * [5] Speechless-Code-Mistral-7b-V1.0.  https://huggingface.co/uukuguy/speechless-code-mistral-7b-v1.0.
  * [6] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [7] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al.  Program synthesis with large language models.  arXiv preprint arXiv:2108.07732, 2021.
  * [8] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [9] L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy.  Punica: Multi-tenant lora serving.  arXiv preprint arXiv:2310.18547, 2023.
  * [10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [11] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [12] O. Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023.
  * [13] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer.  Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.  NeurIPS, 35:30318–30332, 2022.
  * [14] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  Nature Machine Intelligence, 5(3):220–235, 2023.
  * [15] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha.  Learned step size quantization.  In ICLR, 2020.
  * [16] W. Fedus, B. Zoph, and N. Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  JMLR, 23(120):1–39, 2022.
  * [17] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.  Measuring massive multitask language understanding.  In ICLR, 2021.
  * [18] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.  Measuring mathematical problem solving with the MATH dataset.  In NeurIPS, 2021.
  * [19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [20] Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, Y. Fu, et al.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  NeurIPS, 36, 2024.
  * [21] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In CVPR, pages 2704–2713, 2018.
  * [22] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [23] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [24] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica.  Efficient memory management for large language model serving with pagedattention.  In SOSP, 2023.
  * [25] H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin.  Cmmlu: Measuring massive multitask language understanding in chinese.  arXiv preprint arXiv:2306.09212, 2023.
  * [26] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han.  Awq: Activation-aware weight quantization for llm compression and acceleration.  In MLSys, 2024.
  * [27] J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang.  QLLM: Accurate and efficient low-bitwidth quantization for large language models.  In ICLR, 2024.
  * [28] J. Liu, G. Xiao, K. Li, J. D. Lee, S. Han, T. Dao, and T. Cai.  Bitdelta: Your fine-tune may only be worth one bit.  arXiv preprint arXiv:2402.10193, 2024.
  * [29] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re, et al.  Deja vu: Contextual sparsity for efficient llms at inference time.  In ICML, pages 22137–22176, 2023.
  * [30] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [31] A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al.  Starcoder 2 and the stack v2: The next generation.  arXiv preprint arXiv:2402.19173, 2024.
  * [32] K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou.  Routing to the expert: Efficient reward-guided ensemble of large language models.  arXiv preprint arXiv:2311.08692, 2023.
  * [33] H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  arXiv preprint arXiv:2308.09583, 2023.
  * [34] Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  arXiv preprint arXiv:2306.08568, 2023.
  * [35] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In ECCV, pages 525–542, 2016.
  * [36] S. Ryu, S. Seo, and J. Yoo.  Efficient storage of fine-tuned models via low-rank approximation of weight residuals.  arXiv preprint arXiv:2305.18425, 2023.
  * [37] V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani.  Ziplora: Any subject in any style by effectively merging loras.  arXiv preprint arXiv:2311.13600, 2023.
  * [38] Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, et al.  S-lora: Serving thousands of concurrent lora adapters.  arXiv preprint arXiv:2311.03285, 2023.
  * [39] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang.  Flexgen: High-throughput generative inference of large language models with a single gpu.  In ICML, pages 31094–31116, 2023.
  * [40] S. Sukhbaatar, O. Golovneva, V. Sharma, H. Xu, X. V. Lin, B. Rozière, J. Kahn, D. Li, W.-t. Yih, J. Weston, et al.  Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.  arXiv preprint arXiv:2403.07816, 2024.
  * [41] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter.  A simple and effective pruning approach for large language models.  In ICLR, 2024.
  * [42] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [43] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [44] P. Tillet, H.-T. Kung, and D. Cox.  Triton: an intermediate language and compiler for tiled neural network computations.  In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019.
  * [45] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [46] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi.  Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation.  arXiv preprint arXiv:2210.07558, 2022.
  * [47] X. Wei, Y. Zhang, Y. Li, X. Zhang, R. Gong, J. Guo, and X. Liu.  Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling.  In EMNLP, pages 1648–1665, 2023.
  * [48] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In ICML, pages 38087–38099, 2023.
  * [49] P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal.  Ties-merging: Resolving interference when merging models.  NeurIPS, 36, 2024.
  * [50] X. Yao and A. Klimovic.  Deltazip: Multi-tenant language model serving via delta compression.  arXiv preprint arXiv:2312.05215, 2023.
  * [51] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun.  Orca: A distributed serving system for $\\{$Transformer-Based$\\}$ generative models.  In OSDI, pages 521–538, 2022.
  * [52] L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  arXiv preprint arXiv:2309.12284, 2023.
  * [53] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li.  Language models are super mario: Absorbing abilities from homologous models as a free lunch.  arXiv preprint arXiv:2311.03099, 2023.

Supplemental Material

##  Appendix A More Details about Prompt Template for Model-level Routing

In this section, we presents the prompt template for our model-level routing,
as illustrated in Table A. When a user query is received, it is embedded into
the prompt template to form a structured question. This structured question is
then processed by the router to perform domain classification.

Table A:  Prompt template for model-level routing.  Prompt template for model-
level routing
---
Classify the query based on the required expertise. Route the query to the
appropriate model for a precise response. Only output the letter corresponding
to the best category (A, B, C, …, F). Query: {Insert the user’s query here. }
Options: A) Instruct - For general guidance, explanations, or broad advice. B)
Code - For programming-related queries, like debugging or coding. C) Math -
For mathematical inquiries, such as problems or theories. D) Chinese Language
Expert - For inquiries related to the Chinese language, including translation,
grammar, and usage. … F) {Specify additional categories and their descriptions
here.} Response should be only ’A’, ’B’, ‘C’, … or ‘F”, with no additional
text.