# QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language
Models

Jing Liu1,2, Ruihao Gong2,3, Xiuying Wei2, Zhiwei Dong2,4, Jianfei Cai1, Bohan
Zhuang1
1ZIP Lab, Monash University 2SenseTime Research 3Beihang University
4University of Science and Technology Beijing Work done during an internship
at SenseTime Research.Corresponding author. Email: $\tt
<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have demonstrated unparalleled efficacy in
natural language processing. However, their high computational demands and
memory overheads hinder their broad deployment. To address this, two
quantization strategies emerge, including Quantization-Aware Training (QAT)
and Post-Training Quantization (PTQ). For LLMs, the billions of parameters
make the QAT impractical due to the prohibitive training cost and thus PTQ
becomes more prevalent. In existing studies, activation outliers in particular
channels are identified as the biggest challenge to PTQ accuracy. They propose
to transform the magnitudes from activations to weights, which however offers
limited alleviation or suffers from unstable gradients, resulting in a severe
performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate
and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an
adaptive channel reassembly technique that reallocates the magnitude of
outliers to other channels, thereby mitigating their impact on the
quantization range. This is achieved by channel disassembly and channel
assembly, which first breaks down the outlier channels into several sub-
channels to ensure a more balanced distribution of activation magnitudes. Then
similar channels are merged to maintain the original channel number for
efficiency. Additionally, an adaptive strategy is designed to autonomously
determine the optimal number of sub-channels for channel disassembly. To
further compensate for the performance loss caused by quantization, we propose
an efficient tuning method that only learns a small number of low-rank weights
while freezing the pre-trained quantized model. After training, these low-rank
parameters can be fused into the frozen weights without affecting inference.
Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM is able to obtain
accurate quantized models efficiently. For example, QLLM quantizes the 4-bit
LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the
previous state-of-the-art method by 7.89% on the average accuracy across five
zero-shot tasks.

##  1 Introduction

Recently, Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023) and LLaMA
(Touvron et al., 2023a; b) have achieved unprecedented advancements in natural
language processing (NLP). These models excel in a range of tasks, from
advanced reasoning in code and mathematics to classification and question
answering. However, their extraordinary performance is accompanied by
substantial computational demands and vast model sizes. For example, GPT-3
(Brown et al., 2020), the precursor to GPT-4, already contains a stunning 175
billion parameters, requiring a minimum of 325 GB of memory for storage in
half-precision (FP16) format. This necessitates the use of at least
5$\times$80GB NVIDIA A100 or 8$\times$48GB NVIDIA A40 GPUs during the
inference phase. As a result, deploying these models to real-world
applications poses significant challenges.

Figure 1: An illustration of the channel-wise maximum and minimum values for
the input activations of a linear layer in LLaMA-65B for (a) original pre-
trained model (b) after SmoothQuant (Xiao et al., 2023) and (c) after our
channel reassembly.

In light of the aforementioned challenges, network quantization (Zhou et al.,
2016) emerges as a compelling solution, which maps weights and/or activations
to lower-bit representations, resulting in a much lower memory footprint and
faster inference. Existing quantization methods for LLMs can be classified
into two types: quantization-aware training (QAT) (Liu et al., 2023) and post-
training quantization (PTQ) (Wei et al., 2022b; 2023; Xiao et al., 2023).
Although with promising performance, QAT suffers from unbearable training
costs as it needs to fine-tune the whole quantized model with quantization
parameters using a large amount of data, rendering it impractical for the
efficient deployment of LLMs. This practical limitation has shifted the
spotlight towards PTQ which only uses a little data to tune the quantized
weights. However, when it comes to extremely low-bitwidth quantization for
LLMs, _e.g._ , 4-bit weight and/or activation quantization, existing PTQ
methods (Xiao et al., 2023; Dettmers et al., 2022) suffer from significant
performance degradation.

Recent studies (Dettmers et al., 2022; Xiao et al., 2023; Wei et al., 2023)
have revealed a unique pattern in LLMs’ activations that is they contain
specific outlier channels with significantly large magnitudes. This renders
existing quantization methods less effective, as the outliers amplify the
quantization range of layer activations, causing the vast majority of normal
activation values to be quantized imprecisely and consequently leading to
notable performance degradation. This issue will worsen with the prevalent use
of layer-wise or token-wise activation quantization, a common practice for
maximizing hardware efficiency. To tackle this challenge, recent studies (Xiao
et al., 2023; Wei et al., 2022b; 2023; Shao et al., 2023) have focused on
smoothing activation outliers by transitioning the magnitudes from activations
to weights through a mathematically equivalent transformation. Such a
transformation can be learned using either gradient-free methods (Xiao et al.,
2023; Wei et al., 2022b; 2023) or gradient-based methods (Shao et al., 2023).
However, as shown in Figure 1, for exceedingly pronounced activation outliers
(those 50 $\times$ larger than others), the former offers only limited
alleviation while the latter suffers from unstable gradients. As a result,
both methods leads to significant performance degradation in low-bitwidth
quantization. To compensate for the performance drop of quantization, a widely
adopted PTQ strategy (Wei et al., 2023; Shao et al., 2023; Yao et al., 2022)
further proposes to tune the quantized LLM directly by minimizing the block-
wise reconstruction error. In LLMs, the tuned block refers to the Attention-
FFN module. However, considering the huge number of parameters in an LLM, this
approach still requires substantial training overheads and demands a
significant amount of GPU memory.

In this paper, we propose QLLM, an accurate and efficient low-bitwidth post-
training quantization method tailored for LLMs. To handle the outlier issue,
we introduce a gradient-free channel reassembly technique that redistributes
the large activation magnitude of the outlier channels across the channels.
Specifically, we first disassemble the outlier channels into several sub-
channels. By spreading the magnitude of outliers, it ensures a more uniform
activation range across channels, facilitating a balanced and precise
quantization and thus improving the performance of quantized LLMs. We then
introduce channel assembly, which fuses similar channels together to maintain
the original channel count. Moreover, given the varying outlier patterns
across different layers and the existence of extreme outliers, we propose an
adaptive strategy to determine the optimal number of disassembled channels for
each layer, which is based on minimizing the reassembly error between the
original output activations and the counterpart with the reassembled input
activations.

To further improve the performance of the quantized LLMs, motivated by low-
rank parameter-efficient fine-tuning paradigm LoRA (Hu et al., 2022; Dettmers
et al., 2023a), we further propose an efficient gradient-based error
correction strategy that freezes the pre-trained model and introduces a small
set of learnable low-rank weights into each layer of the LLM. Then, QLLM
learns the low-rank weights by minimizing block-wise quantization error
sequentially. Owing to the reduced number of trainable parameters, both the
training time and GPU memory requirements are significantly reduced. Such
efficiency gain enables us to perform a multi-block reconstruction that
simultaneously reconstructs a collection of consecutive Attention-FFN blocks,
further mitigating the quantization error accumulation during propagation in
low-bit LLMs. Notably, after training, these learnable low-rank weights can be
seamlessly merged with the frozen weights followed by quantization, thereby
ensuring no additional computational burden during inference.

Our contributions can be summarized as follows: 1) We introduce a simple yet
effective channel reassembly method to suppress activation outliers in LLMs,
which is accomplished by initially disassembling the outlier channels to make
activations more quantization-friendly and subsequently merging similar
channels so as to preserve the original channel count for efficiency. We also
propose to determine the optimal number of disassembled channels for each
layer, considering the diverse outlier patterns across layers and the presence
of extreme outliers. The overall process is gradient-free and enjoys high
efficiency. 2) An efficient error correction mechanism is proposed to further
enhance the gradient-free channel reassembly. It leverages the learning of
low-rank parameters to counteract quantization error in a structured way,
leading to a substantial reduction in training time and GPU memory
requirements without incurring any additional inference overhead. 3) Extensive
experiments show the promising performance and training efficiency of QLLM.
For example, QLLM quantizes 4-bit LLaMA-2-70B within 10 hours, and outperforms
previous SOTA methods by 7.89% on the average accuracy across five zero-shot
tasks.

##  2 Related Work

Network quantization. Network quantization (Zhou et al., 2016) which
represents the weights, activations, and even gradients with low precision, is
an effective method to reduce the model size and computational burden.
Existing techniques fall into two primary categories: quantization-aware
training (QAT) (Esser et al., 2020; Kim et al., 2021; Li et al., 2022) and
post-training quantization (PTQ) (Nagel et al., 2020; Li et al., 2021; Wei et
al., 2022a). QAT incorporates the quantization process directly into the
training phase and jointly learning the quantizer as well as model parameters
(Zhang et al., 2018; Jung et al., 2019; Choi et al., 2019; Bhalgat et al.,
2020; Esser et al., 2020; Liu et al., 2022) with the help of straight-through
estimator (STE) (Bengio et al., 2013), which greatly mitigates the accuracy
degradation caused by compression. However, the training cost of QAT can be
prohibitively high, primarily because it requires fine-tuning the quantized
model on the original training dataset of the pre-trained model. PTQ offers a
less resource-intensive alternative, allowing models to be quantized after
being fully trained with only a small amount of data. To reduce the
performance drop, several methods have been proposed to perform layer-wise
(Nagel et al., 2019; 2020; Wu et al., 2020; Hubara et al., 2020; Li et al.,
2021) or even block-wise calibration (Li et al., 2021). Further innovations
delve into outlier mitigation, adopting strategies like clipping (Banner et
al., 2019; McKinstry et al., 2019; Choukroun et al., 2019) or value splitting
(Zhao et al., 2019) for weights and activations to improve the precision by
allocating more bits to the intermediate values. However, for LLMs, a recent
study (Liu et al., 2023) has found that MinMax quantization, which maintains
the full value range, performs better than clipping-based methods, as outliers
are critical to the performance. Different from these methods, our QLLM
targets quantization for LLMs.

Quantization on LLMs. Given constraints such as limited training data and
intensive computational demands, prevailing quantization techniques for LLMs
are primarily based on PTQ. Existing LLM quantization approaches can be
classified into two categories: weight-only quantization (Frantar et al.,
2022; Park et al., 2023; Lin et al., 2023; Dettmers et al., 2023b; Chai et
al., 2023; Cheng et al., 2023; Dettmers et al., 2023a; Kim et al., 2023; Chee
et al., 2023; Lee et al., 2023) and weight-activation quantization (Dettmers
et al., 2022; Xiao et al., 2023; Wei et al., 2022b; 2023; Yao et al., 2022;
2023; Yuan et al., 2023; Liu et al., 2023; Wu et al., 2023). The former
focuses on compressing the vast number of weights in LLMs to reduce the memory
footprint, while the latter compresses both weights and activations into low-
bit values, aiming to accelerate computation-intensive matrix multiplication.
To handle the different value ranges of weight matrices, recent studies have
delved into more fine-grained quantization, such as channel-wise quantization
(Frantar et al., 2022) or group-wise quantization (Frantar et al., 2022; Lin
et al., 2023). To further compensate for the performance drop for extremely
low-bitwidth quantization, QLoRA (Dettmers et al., 2023a), and INT2.1 (Chai et
al., 2023) introduce additional full-precision weights (Yao et al., 2023).
While our method also presents a small set of low-rank weights, it stands
apart from QLoRA and INT2.1 as our learnable parameters can be reparameterized
into pretrained weights followed by quantization. Recent research (Dettmers et
al., 2022) has shown that activation outliers exist in some feature dimensions
across different tokens. Several works (Wei et al., 2022b; 2023; Xiao et al.,
2023; Shao et al., 2023) have been proposed to migrate the quantization
difficulty from activations to weights within the same channel, based on
gradient-free methods (Wei et al., 2022b; Xiao et al., 2023; Wei et al., 2023)
or gradient-based methods (Shao et al., 2023). However, when dealing with very
pronounced activation outliers, the existing methods often show limited
improvement or incur unstable gradients. In a notable difference, our proposed
QLLMs method efficiently redistributes the large activation magnitudes of
outlier channels among all channels, offering a distinctive approach compared
to these existing methods.

##  3 Preliminaries

Basic notations. In this paper, matrix is marked as ${\bf X}$ and vector is
denoted by ${\bf x}$. The LLMs usually have two core parts: multi-head self-
attention (MSA) layers and feed-forward network (FFN) layers, which are mainly
composed of linear layers. Here, we give the formulation of linear layers at
the output channel $k$:

$\begin{array}[]{ll}{\bf y}_{k}=\sum_{i=1}^{{M}}{\bf x}_{i}{\bf
W}_{ik},\end{array}$ (1)

where ${\bf x}\in\mathbb{R}^{{M}}$ refers to input, ${\bf
W}\in\mathbb{R}^{{M}\times{N}}$ denotes the weight, and ${\bf
y}\in\mathbb{R}^{{N}}$ stands for the output. In this way, the numbers of
input and output channels are ${M}$ and ${N}$, respectively.

Quantization. We adopt uniform quantization for both weights and activations
because of its hardware-friendly nature (Jacob et al., 2018). For matrix ${\bf
X}$ with floating-point values such as FP16 or FP32, the $b$-bit quantization
quantizes it in the following way:

$\begin{array}[]{ll}{\bf X}_{q}=\mathrm{quant}({\bf
X})=\mathrm{clamp}\left(\lfloor\frac{{\bf
X}}{\alpha}\rceil+\beta,0,2^{b}-1\right),\mathrm{where}~{}\alpha=\frac{\mathrm{max}({\bf
X})-\mathrm{min}({\bf
X})}{2^{b}-1},\beta=-\left\lfloor\frac{\mathrm{min}(\mathbf{X})}{\alpha}\right\rceil,\end{array}$
(2)

where the function $\mathrm{clamp}(v,v_{\mathrm{min}},v_{\mathrm{max}})$ clips
any value $v$ into the range of $[v_{\mathrm{min}},v_{\mathrm{max}}]$ and
$\lfloor\cdot\rceil$ is a rounding operator that returns the nearest integer
of a given value. Here, $\alpha$ denotes the scaling factor and $\beta$
represents the zero-point value.

Recent studies (Dettmers et al., 2022; Xiao et al., 2023; Wei et al., 2022b)
point out that there are extremely large outliers in certain channels of
activations in LLMs, which makes the quantization challenging to balance the
accurate representation for large values and small numbers. To tackle this
problem, some approaches (Bondarenko et al., 2021; Yuan et al., 2023) adopt
fine-grained quantization scheme, which assigns different quantization
parameters for different channels. However, such a way needs delicate kernel
design and clearly increases computation overhead for inference. Also, some
works Wei et al. (2022b); Xiao et al. (2023) propose to use channel-wise
scaling between activation and weights, which still remains outliers under
extreme cases, as shown in Figure 1.

##  4 Proposed Method

In this section, we propose the adaptive channel reassembly framework to
redistribute input activation outliers across multiple channels. The framework
consists of three components: channel disassembly for decomposing the outlier
channel, channel assembly for balancing the efficiency, and an adaptive
strategy to find the suitable reassembly ratio for each layer. The channel
reassembly technique is gradient-free and efficient to implement. What’s more,
it can be equipped with a gradient-based and well-designed error correction
module for further enhancement.

###  4.1 Adaptive Channel Reassembly

####  4.1.1 Channel Disassembly

In this part, we introduce our channel disassembly to decompose the input
outlier channels into several sub-channels, which can reduce the outlier
magnitude and make the activations more quantization-friendly without altering
the layer output.

Considering that outliers tend to be concentrated in specific channels across
various inputs and the desire to preserve their information during
quantization, we propose to break down these outlier channels into several
sub-channels to redistribute their large values. Without loss of generality,
by assuming the $M$-th channel as the outlier channel, we can disassemble it
into $\frac{{\bf x}_{M}}{T}$ and replicate this channel $T$ times, reducing
the outlier magnitude by a factor of $T$. Simultaneously, it is also natural
to duplicate the corresponding weight channel $T$ times, enabling us to
maintain the equivalent output:

$\begin{array}[]{ll}{\bf y}_{k}=\sum_{i=1}^{{M}-1}{\bf x}_{i}{\bf
W}_{ik}+\underbrace{\frac{{\bf x}_{{M}}}{T}{\bf W}_{Mk}+\cdots+\frac{{\bf
x}_{{M}}}{T}{\bf W}_{{M}k}}_{T~{}\text{times}}.\end{array}$ (3)

The equation above produces the same output with the original linear layer
equation in Eq. (1) and introduces an additional $T-1$ channels for both the
input and the weight.

Taking into account that the quantization range impacts accuracy, we introduce
an outlier threshold, denoted as $\theta$, to identify the outlier channels
and determine the number of sub-channels together, with
$T=\lceil\mathrm{max}({\bf x}_{M})/\theta\rceil$. This approach ensures that
channels with values smaller than $\theta$ remain unchanged with $T=1$, while
the magnitude of outliers are divided by $T$.

Our channel disassembly method allows us to retain outlier information with an
equivalent output and ease the quantization difficulty with a much smaller
value range. Its only drawback is the increase in the number of channels,
which may lead to additional computational costs and will be addressed in the
next subsection.

####  4.1.2 Channel Assembly

Note that the input channel count increases to $M+T-1$ after channel
disassembly. Given the substantial quantity of channels in LLMs, it is
possible to omit some unimportant channels or merge similar input channels to
keep the original channel count $M$ for efficiency while maintaining outputs.
To achieve this, a straightforward method is to use channel pruning (Ma et
al., 2023; Sun et al., 2023) that removes the unimportant channels directly.
However, such method may result in substantial information loss, especially
when $T$ is large. Motivated by recent studies (Bolya et al., 2023; Bolya &
Hoffman, 2023) that combine similar tokens, we propose a channel assembly
method that delves into merging $T-1$ similar input channels. Given channels
$i$ and $j$, in alignment with token merging techniques (Bolya et al., 2023;
Bolya & Hoffman, 2023), our goal is to aggregate them by calculating the
average of their input features, denoted as $\frac{{\bf x}_{i}+{\bf
x}_{j}}{2}$ and utilizing the aggregated feature in subsequent computations,
which is defined as:

$\begin{array}[]{ll}{\bf x}_{i}{\bf W}_{ik}+{\bf x}_{j}{\bf
W}_{jk}\approx\frac{{\bf x}_{i}+{\bf x}_{j}}{2}\left({\bf W}_{ik}+{\bf
W}_{jk}\right),\end{array}$ (4)

where ${\bf W}_{ik}+{\bf W}_{jk}$ represents the merged weight. With the aim
of minimizing the information loss of channel assembly in Eq. (4), we can
define a distance metric $D(i,j)$ between channels $i$ and $j$ as

$\begin{array}[]{ll}D(i,j)=\left\|\frac{{\bf x}_{i}\left({\bf W}_{ik}-{\bf
W}_{jk}\right)}{2}+\frac{{\bf x}_{j}\left({\bf W}_{jk}-{\bf
W}_{ik}\right)}{2}\right\|_{2}^{2},\end{array}$ (5)

where $\left\|\cdot\right\|_{2}$ represents the $\ell_{2}$ norm. The above
distance metric takes into account the difference in both input activations
and weights between the two channels.

With the channel distance defined, the next step is to determine which
channels to aggregate efficiently, with the goal of reducing the total channel
count by $T-1$. To address this, we propose using bipartite soft matching
(Bolya et al., 2023; Bolya & Hoffman, 2023) that first partitions the channels
into two sets, each containing roughly equal sizes, and subsequently finds the
$T-1$ most similar pairs between these two sets (see Appendix A for details).
Note that we do not assemble the channels that are disassembled from the
outlier channels since they play a critical role in the performance of LLMs.
After the channel reassembly, including both disassembly and assembly, we
acquire the reassembled input activations that are more amenable to
quantization, along with the corresponding reassembled weights for layer $l$.

####  4.1.3 Adaptive Reassembly

In this section, we present a method to adaptively determine the appropriate
reassembly ratio for each layer. For channel disassembly, selecting a high
value for $T$ with a small $\theta$ substantially reduces outlier magnitudes
and benefits quantization, while resulting in a larger increase in channel
merging error due to a higher merging ratio. Conversely, choosing a small $T$
with a large $\theta$ will not increase the channel count much, making it
easier for the assembly stage to keep the information while likely still
retaining outliers, causing significant quantization errors. Therefore, it is
crucial to carefully determine the outlier threshold $\theta$ or the
reassembly channel number $T$.

However, it is hard to choose $\theta$ in practice as distinct layers have
different patterns of outliers, as shown in Figure B. Motivated by (Wei et
al., 2023), we propose an adaptive strategy to find the optimal $\theta$ by
minimizing the reassembly error between the original output activations and
their counterparts generated with the reassembled input activations for each
layer.

Note that our channel reassembly technique can yield the reassembled
activation $\hat{{\bf X}}\in\mathbb{R}^{L\times M}$ with a sequence length of
$L$, which can then be fed into a MSA layer or a FFN layer. For example, let
us consider a case where $\hat{{\bf X}}$ is fed into a MSA layer. A standard
MSA layer calculates queries, keys and values with three learnable projection
matrices ${\bf W}_{Q},{\bf W}_{K},{\bf W}_{V}\in\mathbb{R}^{M\times N}$ as
${\bf Q}={\bf X}{\bf W}_{Q},{\bf K}={\bf X}{\bf W}_{K},{\bf V}={\bf X}{\bf
W}_{V}$, where ${\bf X}\in\mathbb{R}^{L\times M}$ represents the original
input activation. Let $\hat{{\bf W}}_{Q}$, $\hat{{\bf W}}_{K}$, $\hat{{\bf
W}}_{V}$ be the reassembled projection weights. In this way, the reconstructed
queries, keys, and values can be formulated as:

$\begin{array}[]{ll}\tilde{{\bf Q}}=\mathrm{quant}(\hat{{\bf
X}})\mathrm{quant}(\hat{{\bf W}_{Q}}),\tilde{{\bf K}}=\mathrm{quant}(\hat{{\bf
X}})\mathrm{quant}(\hat{{\bf W}_{K}}),\tilde{{\bf V}}=\mathrm{quant}(\hat{{\bf
X}})\mathrm{quant}(\hat{{\bf W}_{V}}).\end{array}$ (6)

We then find $\theta$ by solving the following problem:

$\begin{array}[]{ll}\arg\min_{\theta}\left\|\mathrm{Softmax}({\bf Q}{\bf
K}^{\top}){\bf V}-\mathrm{Softmax}(\tilde{{\bf Q}}\tilde{{\bf
K}}^{\top})\hat{{\bf V}}\right\|_{F}^{2},\end{array}$ (7)

where $\left\|\cdot\right\|_{F}$ denotes the Frobenius norm. To solve problem
(7) efficiently, we use grid search following (Choukroun et al., 2019; Wei et
al., 2023) (see Algorithm 1 in Appendix for details).

###  4.2 Efficient Gradient-based Error Correction

Based on the above gradient-free adaptive channel reassembly, an efficient
gradient-based error correction technique is further proposed for improving
the performance of the quantized LLMs using a small set of calibration data.

Inspired by recent developments in parameter-efficient fine-tuning methods (Hu
et al., 2022; Dettmers et al., 2023a), the efficient error correction
introduces two low-rank parameters ${\bf A}\in\mathbb{R}^{M\times r}$ and
${\bf B}\in\mathbb{R}^{r\times N}$ with a rank of $r$ into each projection
layer of our QLLM. Then, we can obtain the output ${\bf Y}$ of a quantized
linear layer by ${\bf Y}=\mathrm{quant}({\bf X})\mathrm{quant}({\bf
W})+\mathrm{quant}({\bf X}){\bf A}{\bf B}.$ Instead of directly tuning the
quantized weights, we learn the introduced low-rank parameters by minimizing
the reconstruction error between the original and the quantized outputs of the
Attention-FFN block. Thanks to the reduced number of trainable parameters,
both the optimization cost and GPU memory usage can be significantly reduced.
Such efficiency gain allows us to further suppress the accumulation of
quantization error during forward propagation via a structured reconstruction,
_i.e._ , performing multi-block reconstruction for QLLM, which simultaneously
adjusts a collection of consecutive Attention-FFN blocks by focusing on
reconstructing the final block output.

After the reconstruction, we only need to store the quantized weight
$\mathrm{quant}({\bf W}+{\bf A}{\bf B})$, which does not introduce extra
inference costs. Note that it is inevitable that the absorption process will
introduce additional quantization errors. To counteract this, following (He et
al., 2017; Nagel et al., 2020; Hubara et al., 2020), we perform reconstruction
sequentially rather than in parallel, which enables us to account for the
quantization error stemming from the previous layers.

###  4.3 Efficiency Discussion

Reassembly efficiency. Our adaptive channel reassembly stands out for its
efficiency, mainly attributed to its gradient-free nature, which excludes the
need for backward propagation. The main source of computational expense of our
method comes from the channel assembly, which requires the calculation of
pairwise distances. Fortunately, the utilization of efficient bipartite soft
matching eliminates the need to compute distances for every pair of channels,
enhancing the efficiency. For the gradient-based error correction, the reduced
number of parameters significantly lowers its optimization cost, rendering it
more efficient than directly adjusting the quantized weights.

Inference efficiency.  Both channel disassembly and assembly can be
implemented efficiently if the previous layer $l-1$ is a linear layer. Please
refer to Appendix B for more details. In cases where the preceding layer $l-1$
is a non-linear layer, such as a layer normalization (Ba et al., 2016), we
introduce additional disassembly and assembly layers that are designed to
decompose and aggregate channels during runtime, with the channel indexes for
decomposition and aggregation calculated offline using calibration data, which
only introduces a minor extra overhead.

##  5 Experiments

Table 1: Performance comparisons of different methods for weights and
activations quantization on LLaMA-1 model family. PPL denotes the perplexity.

Model | #Bits | Method | PPL $\downarrow$ | Accuracy (%) $\uparrow$
---|---|---|---|---
WikiText2 | C4 | Avg. | PIQA | ARC-e | ARC-c | HellaSwag | Winogrande | Avg.
LLaMA-1-7B | W16A16 | - | 5.68 | 7.08 | 6.38 | 77.37 | 52.48 | 41.38 | 72.99 | 66.93 | 62.23
W6A6 | SQ | 6.15 | 7.61 | 6.88 | 76.65 | 53.11 | 40.10 | 71.52 | 61.88 | 60.65
W6A6 | OS+ | - | - | - | 76.82 | 51.35 | 41.13 | 71.42 | 65.98 | 61.34
W6A6 | OmniQuant | 5.96 | 7.43 | 6.70 | 77.09 | 51.89 | 40.87 | 71.61 | 65.03 | 61.30
W6A6 | QLLM | 5.89 | 7.34 | 6.62 | 77.26 | 52.02 | 41.04 | 71.40 | 65.19 | 61.38
2-14 | W4A4 | SQ | 52.85 | 104.35 | 78.60 | 49.80 | 30.40 | 25.80 | 27.40 | 48.00 | 36.28
| W4A4 | LLM-QAT | - | - | - | 51.50 | 27.90 | 23.90 | 31.10 | 51.90 | 37.26
| W4A4 | LLM-QAT+SQ | - | - | - | 55.90 | 35.50 | 26.40 | 47.80 | 50.60 | 43.24
| W4A4 | OS+ | - | - | - | 62.73 | 39.98 | 30.29 | 44.39 | 52.96 | 46.07
| W4A4 | OmniQuant | 11.26 | 14.51 | 12.89 | 66.15 | 45.20 | 31.14 | 56.44 | 53.43 | 50.47
| W4A4 | QLLM | 9.65 | 12.29 | 10.97 | 68.77 | 45.20 | 31.14 | 57.43 | 56.67 | 51.84
LLaMA-1-13B | W16A16 | - | 5.09 | 6.61 | 5.85 | 79.05 | 59.84 | 44.62 | 76.22 | 70.09 | 65.96
W6A6 | SQ | 5.50 | 7.03 | 6.27 | 77.80 | 56.36 | 42.58 | 75.11 | 68.11 | 63.99
W6A6 | OS+ | - | - | - | 78.29 | 56.90 | 43.09 | 75.09 | 69.22 | 64.52
W6A6 | OmniQuant | 5.28 | 6.84 | 6.06 | 78.40 | 57.28 | 42.91 | 75.82 | 68.27 | 64.54
W6A6 | QLLM | 5.28 | 6.82 | 6.05 | 77.91 | 57.70 | 42.92 | 75.02 | 69.14 | 64.54
2-14 | W4A4 | SQ | 79.35 | 120.24 | 99.80 | 55.55 | 34.51 | 26.71 | 41.56 | 48.70 | 41.41
| W4A4 | OS+ | - | - | - | 63.00 | 40.32 | 30.38 | 53.61 | 51.54 | 47.77
| W4A4 | OmniQuant | 10.87 | 13.78 | 12.33 | 69.69 | 47.39 | 33.10 | 58.96 | 55.80 | 52.99
| W4A4 | QLLM | 8.41 | 10.58 | 9.50 | 71.38 | 47.60 | 34.30 | 63.70 | 59.43 | 55.28
LLaMA-1-30B | W16A16 | - | 4.10 | 5.98 | 5.04 | 80.09 | 58.92 | 45.39 | 79.21 | 72.77 | 67.28
W6A6 | SQ | - | - | - | 77.14 | 57.61 | 42.91 | 78.07 | 69.92 | 65.13
W6A6 | OS+ | - | - | - | 80.14 | 58.92 | 45.05 | 77.96 | 71.98 | 66.81
W6A6 | OmniQuant | 4.38 | 6.22 | 5.30 | 79.81 | 58.79 | 45.22 | 78.95 | 72.21 | 67.00
W6A6 | QLLM | 4.30 | 6.17 | 5.24 | 79.65 | 58.08 | 44.11 | 78.38 | 73.24 | 66.69
2-14 | W4A4 | SQ | 399.65 | 245.87 | 322.76 | 50.16 | 28.11 | 26.71 | 31.97 | 51.14 | 37.62
| W4A4 | OS+ | - | - | - | 67.63 | 46.17 | 34.30 | 54.32 | 52.64 | 51.01
| W4A4 | OmniQuant | 10.33 | 12.49 | 11.41 | 71.21 | 49.45 | 34.47 | 64.65 | 59.19 | 55.79
| W4A4 | QLLM | 8.37 | 11.51 | 9.94 | 73.83 | 50.67 | 38.40 | 67.91 | 58.56 | 57.87
LLaMA-1-65B | W16A16 | - | 3.56 | 5.62 | 4.59 | 80.85 | 58.75 | 46.25 | 80.73 | 77.11 | 68.74
W6A6 | SQ | 4.00 | 6.08 | 5.04 | 77.97 | 54.67 | 44.62 | 77.51 | 72.61 | 65.48
W6A6 | OS+ | - | - | - | 79.67 | 55.68 | 45.22 | 78.03 | 73.95 | 66.51
W6A6 | OmniQuant | 3.75 | 5.82 | 4.79 | 81.01 | 58.12 | 46.33 | 79.91 | 75.69 | 68.21
W6A6 | QLLM | 3.73 | 5.80 | 4.77 | 80.14 | 57.79 | 45.05 | 79.74 | 74.59 | 67.46
2-14 | W4A4 | SQ | 112.02 | 118.96 | 115.49 | 61.81 | 40.15 | 32.08 | 46.19 | 50.83 | 46.21
| W4A4 | OS+ | - | - | - | 68.06 | 43.98 | 35.32 | 50.73 | 54.30 | 50.48
| W4A4 | OmniQuant | 9.17 | 11.28 | 10.23 | 71.81 | 48.02 | 35.92 | 66.81 | 59.51 | 56.41
| W4A4 | QLLM | 6.87 | 8.98 | 7.93 | 73.56 | 52.06 | 39.68 | 70.94 | 62.9 | 59.83

Models and datasets. We apply QLLM to quantize the LLaMA-1 (Touvron et al.,
2023a) and LLaMA-2 (Touvron et al., 2023b) families. To evaluate the
performance of the quantized LLM, we report the zero-shot accuracy on various
benchmarks, including PIQA (Bisk et al., 2020), ARC (Clark et al., 2018),
HellaSwag (Zellers et al., 2019), and WinoGrande (Sakaguchi et al., 2021).
Additionally, we evaluate the perplexity, a key indicator of a model’s
generative performance that correlates significantly with zero-shot outcomes,
on WikiText2 (Merity et al., 2017), PTB (Marcus et al., 1993) and C4 (Raffel
et al., 2020).

Quantization settings. In alignment with prior research (Dettmers et al.,
2022; Shao et al., 2023), we use per-channel weight quantization and per-token
activation quantization. Following (Shao et al., 2023; Liu et al., 2023), we
quantize all weights and intermediate activations, with the exception of the
Softmax output probability, which is maintained at full precision. Following
OmniQuant (Shao et al., 2023), we focus on 4- and 6-bit weights and
activations quantization. We exclude 8-bit quantization as SmoothQuant (Xiao
et al., 2023) is able to achieve lossless performance.

Compared methods. We compare our QLLM with several state-of-the-art (SOTA) PTQ
quantization methods, such as OmniQuant (Shao et al., 2023), SmoothQuant (SQ)
(Xiao et al., 2023), Outlier Suppression+ (OS+) (Wei et al., 2023) and recent
QAT method LLM-QAT (Liu et al., 2023). For fair comparisons, we reproduce
SmoothQuant and Outlier Suppression+ with per-channel weight quantization and
per-token activation quantization.

Implementation details. Following OmniQuant (Shao et al., 2023), we randomly
sample 128 samples, each with a sequence length of 2048, from WikiText2 to
construct the calibration set. Our QLLM begins by applying channel reassembly
prior to all linear projection layers, excluding the attention output
projection layer, followed by performing error correction on the resulting
model. The rank $r$ of the introduced low-rank parameters is set to 4, and
these parameters are trained for 10 epochs with a mini-batch size of 1. We
carry out the reconstruction using four Attention-FFN blocks. AdamW
(Loshchilov & Hutter, 2019) with a linear learning rate decay scheduler is
used following (Yao et al., 2022). The learning rate is set to $5\times
10^{4}$. All training experiments are conducted on a single NVIDIA A100 80G
GPU. We use the Language Model Evaluation Harness toolbox (Gao et al., 2021)
for evaluation. The torch-int toolbox111https://github.com/Guangxuan-
Xiao/torch-int is employed to assess inference time.

###  5.1 Main Results

We report the results on LLaMA-1 and LLaMA-2 families in Table 1, and Table A
in Appendix. For 6-bit quantization, QLLM delivers comparable performance with
the SOTA PTQ quantization methods. When focusing on extremely low-bitwidth
quantization (_i.e._ , 4-bit), it is pertinent to note that activation
outliers serve as the performance bottleneck, thereby highlighting the
importance of suppressing the outliers. In this case, our QLLM achieves
significantly higher zero-shot accuracy and much lower perplexity than the
contenders. For example, QLLM quantized 4-bit LLaMA-1-65B outperforms
OmniQuant counterpart by an average of 3.42% in accuracy across five zero-shot
tasks. Remarkably, for LLaMA-7B, our QLLM even surpasses the QAT method, LLM-
QAT + SQ, by 8.6% on the average accuracy, which strongly demonstrates the
efficacy of our QLLM.

###  5.2 Ablation Studies

Effect of different components in channel reassembly. To show the
effectiveness of the diverse components involved in channel reassembly, we
apply different methods with our efficient error correction to yield 4-bit
LLaMA-13B and show the results in Table 4. For channel disassembly, we
determine $\theta$ by exploring different channel expansion ratios $\gamma$.
We observe that our method with channel disassembly significantly surpasses
the counterpart that does not utilize it. With the increasing expansion ratio
$\gamma$, the performance of the quantized model can be further improved.
These results strongly show that channel disassembly is able to make
activations more quantization-friendly by decomposing the outlier channels.

Table 2: Perplexity results of different components in channel reassembly.
“CD” stands for channel disassembly. “CA” represents channel assembly. “CP”
indicates channel pruning. “Adaptive” refers to the adaptive strategy.
“$\gamma$” is the channel expansion ratio.

CD | CA | CP | Adaptive | $\gamma$ | LLaMA-1-13B
---|---|---|---|---|---
WikiText2 | PTB | C4 | Avg.
|  |  |  | - | 189.35 | 539.59 | 303.45 | 344.13
✓ |  |  |  | 0.01 | 8.31 | 14.44 | 10.74 | 11.16
✓ |  |  |  | 0.03 | 8.01 | 13.52 | 10.27 | 10.60
✓ |  |  |  | 0.05 | 7.85 | 13.38 | 10.13 | 10.45
✓ |  |  |  | 0.07 | 7.81 | 13.35 | 10.11 | 10.42
✓ | ✓ |  |  | 0.01 | 8.68 | 15.16 | 11.12 | 11.65
✓ | ✓ |  |  | 0.03 | 8.72 | 14.99 | 11.03 | 11.58
✓ | ✓ |  |  | 0.05 | 8.95 | 15.34 | 11.29 | 11.86
✓ | ✓ |  |  | 0.07 | 9.39 | 15.98 | 11.84 | 12.40
✓ |  | ✓ |  | 0.01 | 8.98 | 16.34 | 11.37 | 12.23
✓ |  | ✓ |  | 0.03 | 9.51 | 18.29 | 12.7 | 13.50
✓ |  | ✓ |  | 0.05 | 9.60 | 18.11 | 13.4 | 13.70
✓ |  | ✓ |  | 0.07 | 11.23 | 21.61 | 19.79 | 17.54
✓ | ✓ | - | ✓ | - | 8.41 | 14.38 | 10.58 | 11.12

Table 3: The training time (GPU Hours) comparisons of our QLLM with OmniQuant.

Method | OmniQuant | QLLM
---|---|---
LLaMA-2-7B | 1.98 | 1.05
LLaMA-2-13B | 3.46 | 1.79
LLaMA-2-70B | 14.52 | 9.05

Table 4: Inference time (ms) of our QLLM for 8-bit LLaMA-1-7B with a mini-
batch size of 1.

$L$ | 256 | 512 | 1024
---|---|---|---
FP | 85 | 151 | 291
INT8 | 46 | 71 | 154
QLLM | 53 | 90 | 184

Furthermore, by incorporating channel assembly, our method manages to preserve
the original channel count with little performance drop. In comparison to
channel pruning, our channel assembly leads to lower information loss, thereby
achieving much better performance, especially at higher $\gamma$. Rather than
determining $\theta$ using a predefined expansion ratio, our method, equipped
with an adaptive strategy, is capable of autonomously finding optimal
$\theta$, resulting in near-lossless performance compared to the approach
utilizing only channel disassembly. The resulting expansion ratios for
different layers are shown in Figure A of the Appendix.

Table 5: Performance comparisons of our channel reassembly (CR) with previous
outlier handling methods across five zero-shot tasks.

Model | #Bits | Method | PIQA | ARC-e | ARC-c | HellaSwag | Winogrande | Avg.
---|---|---|---|---|---|---|---|---
LLaMA-1-7B | W6A6 | SQ | 76.65 | 53.11 | 40.10 | 71.52 | 61.88 | 60.65
W6A6 | OS+ | 76.82 | 51.35 | 41.13 | 71.42 | 65.98 | 61.34
W6A6 | CR | 76.88 | 52.31 | 40.87 | 71.37 | 64.33 | 61.15
2-9 | W4A4 | SQ | 49.80 | 30.40 | 25.80 | 27.40 | 48.00 | 36.28
| W4A4 | OS+ | 62.73 | 39.98 | 30.29 | 44.39 | 52.96 | 46.07
| W4A4 | CR | 66.92 | 42.55 | 32.34 | 54.31 | 50.04 | 49.23
LLaMA-1-13B | W6A6 | SQ | 77.80 | 56.36 | 42.58 | 75.11 | 68.11 | 63.99
W6A6 | OS+ | 78.29 | 56.90 | 43.09 | 75.09 | 69.22 | 64.52
W6A6 | CR | 78.02 | 56.69 | 42.41 | 74.70 | 70.01 | 64.37
2-9 | W4A4 | SQ | 55.55 | 34.51 | 26.71 | 41.56 | 48.70 | 41.41
| W4A4 | OS+ | 63.00 | 40.32 | 30.38 | 53.61 | 51.54 | 47.77
| W4A4 | CR | 67.57 | 43.77 | 31.48 | 60.78 | 56.04 | 51.93

Channel reassembly vs. other outlier handling methods. To further show the
effectiveness of channel reassembly, we compare our method with previous
outlier handling methods which employ gradient-free methods to learn
mathematically equivalent transformations. For fair comparisons, we do not
apply efficient error correction. From Table 5, all methods exhibit comparable
performance at 6-bit quantization. However, for 4-bit quantization, channel
reassembly significantly surpasses other methods by a large margin,
particularly for larger models. For instance, the 4-bit LLaMA-1-13B model with
channel reassembly surpasses OS+ by 4.16% in average accuracy.

Table 6: Perplexity comparisons between efficient error correction (EEC) and
tuning quantized weights directly (TQW) for 4-bit LLaMA-1-7B. “OOM” indicates
out of memory.

#Attn-FFN Block | Method | WikiText2 | PTB | C4 | Avg. | Training Time (GPU Hours) | GPU Memory (GB)
---|---|---|---|---|---|---|---
- | CR | 14.12 | 25.30 | 16.58 | 18.67 | - | -
1-9 1 | TQW | 10.10 | 16.19 | 12.95 | 13.08 | 1.49 | 10.9
EEC | 11.21 | 19.29 | 14.06 | 14.85 | 1.06 | 7.95
1-9 2 | TQW | 9.74 | 14.79 | 11.68 | 12.07 | 1.49 | 17.62
EEC | 10.61 | 18.56 | 13.64 | 14.27 | 1.05 | 11.62
1-9 4 | TQW | 8.90 | 14.75 | 11.63 | 11.76 | 1.48 | 30.95
EEC | 9.65 | 16.56 | 12.29 | 12.83 | 1.05 | 18.91
1-9 8 | TQW | - | - | - | - | - | OOM
EEC | 9.18 | 14.98 | 11.63 | 11.93 | 1.05 | 33.50
1-9 16 | TQW | - | - | - | - | - | OOM
EEC | 9.13 | 14.95 | 11.60 | 11.89 | 1.05 | 62.70

Effect of efficient gradient-based error correction. After channel reassembly,
we implement our QLLM to produce 4-bit LLaMA-7B models with our efficient
gradient-based error correction (EEC) and tuning quantized weights directly
(TQW) outlined in Section 4.2 to further improve the performance of quantized
LLMs and show the results in Table 6. The results show that employing EEC not
only maintains comparable performance but also markedly improves training
speed and significantly reduces GPU memory usage over TQW. It is worth noting
that there is a trade-off between GPU memory (_i.e._ , #Attn-FFN blocks) and
performance. Leveraging EEC even allows us to perform reconstruction for 16
Attention-FFN blocks simultaneously, thereby significantly improving
performance while preserving a similar training speed and a reasonable
increase in GPU memory.

Training efficiency. We assess the training efficiency of our method in
comparison to OmniQuant, a recent SOTA PTQ method. The GPU training hours for
both methods are presented in Table 4. The results reveal that the training
cost of our QLLM can be up to 1.93$\times$ faster than OmniQuant, showing the
exceptional training efficiency of our QLLM.

Inference efficiency. To assess the inference efficiency of our channel
reassembly technique, we measure the inference speed of QLLM on a single
NVIDIA 3090 GPU. Our focus is on 8-bit quantization with different sequence
lengths $L$, as 4-bit and 6-bit quantization are not well-supported by
existing hardware. The results are presented in Table 4. Compared to INT8,
QLLM incurs only a minor additional cost but operates significantly faster
than its floating-point (FP) counterpart. Notably, our channel reassembly
strategy substantially mitigates losses attributed to quantizing outliers (see
Table 5), with only a slight extra computational overhead.

##  6 Conclusion and Future Work

In this paper, we have proposed an accurate and efficient post-training
quantization approach for low-bit LLMs, dubbed QLLM. The core of our QLLM lies
in a novel adaptive channel reassembly paradigm that effectively addresses
activation outliers, a pivotal factor contributing to the performance
bottleneck in quantizing LLMs. The key idea involves reallocating outlier
magnitudes to other channels, accomplished through a process of channel
disassembly followed by assembly. We have further proposed a quantization-
aware, parameter-efficient fine-tuning strategy that leverages calibration
data to compensate for the information loss resulting from quantization.
Extensive experiments on LLaMA model series have demonstrated the promising
performance and training efficiency of our QLLM. In terms of limitations, our
proposed channel reassembly involves introducing additional operations to
decompose and aggregate channels during runtime, thereby incurring additional
inference costs. A potential solution to improve inference efficiency is to
explore kernel fusing (Wang et al., 2010), aiming to fuse disassembly,
assembly and layer normalization into a single operator. Another way is to
aggregate more similar or unimportant channels (Sun et al., 2023) than those
disassembled to achieve higher speedup.

## References

  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.  Gqa: Training generalized multi-query transformer models from multi-head checkpoints.  _arXiv preprint arXiv:2305.13245_ , 2023.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Banner et al. (2019) Ron Banner, Yury Nahshan, and Daniel Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  _NeurIPS_ , 32, 2019.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Bhalgat et al. (2020) Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In _CVPR_ , pp. 696–697, 2020.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  Piqa: Reasoning about physical commonsense in natural language.  In _AAAI_ , volume 34, pp. 7432–7439, 2020.
  * Bolya & Hoffman (2023) Daniel Bolya and Judy Hoffman.  Token merging for fast stable diffusion.  In _CVPR_ , pp. 4598–4602, 2023.
  * Bolya et al. (2023) Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  In _ICLR_ , 2023.
  * Bondarenko et al. (2021) Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort.  Understanding and overcoming the challenges of efficient transformer quantization.  In _EMNLP_ , pp. 7947–7969, 2021.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , 33:1877–1901, 2020.
  * Chai et al. (2023) Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu-Yeon Wei.  Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation.  _arXiv preprint arXiv:2306.08162_ , 2023.
  * Chee et al. (2023) Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa.  Quip: 2-bit quantization of large language models with guarantees.  _arXiv preprint arXiv:2307.13304_ , 2023.
  * Cheng et al. (2023) Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, and Kaokao Lv.  Optimize weight rounding via signed gradient descent for the quantization of llms.  _arXiv preprint arXiv:2309.05516_ , 2023.
  * Choi et al. (2019) Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang.  Accurate and efficient 2-bit quantized neural networks.  _PMLR_ , 1:348–359, 2019.
  * Choukroun et al. (2019) Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev.  Low-bit quantization of neural networks for efficient inference.  In _ICCVW_ , pp. 3009–3018. IEEE, 2019.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.  _NeurIPS_ , 35:30318–30332, 2022.
  * Dettmers et al. (2023a) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_ , 2023a.
  * Dettmers et al. (2023b) Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.  Spqr: A sparse-quantized representation for near-lossless llm weight compression.  _arXiv preprint arXiv:2306.03078_ , 2023b.
  * Esser et al. (2020) Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha.  Learned step size quantization.  In _ICLR_ , 2020.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Optq: Accurate quantization for generative pre-trained transformers.  In _ICLR_ , 2022.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.  URL https://doi.org/10.5281/zenodo.5371628.
  * He et al. (2017) Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In _ICCV_ , pp. 1389–1397, 2017.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _ICLR_ , 2022.
  * Hubara et al. (2020) Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.  Improving post training neural quantization: Layer-wise calibration and integer programming.  _arXiv preprint arXiv:2006.10518_ , 2020.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _CVPR_ , pp. 2704–2713, 2018.
  * Jung et al. (2019) Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In _CVPR_ , pp. 4350–4359, 2019.
  * Kim et al. (2023) Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee.  Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization.  _arXiv preprint arXiv:2305.14152_ , 2023.
  * Kim et al. (2021) Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.  I-bert: Integer-only bert quantization.  In _ICML_ , pp. 5506–5518. PMLR, 2021.
  * Lee et al. (2023) Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park.  Owq: Lessons learned from activation outliers for weight quantization in large language models.  _arXiv preprint arXiv:2306.02272_ , 2023.
  * Li et al. (2022) Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo.  Q-vit: Accurate and fully quantized low-bit vision transformer.  _NeurIPS_ , 35:34451–34463, 2022.
  * Li et al. (2021) Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  Brecq: Pushing the limit of post-training quantization by block reconstruction.  In _ICLR_ , 2021.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_ , 2023.
  * Liu et al. (2022) Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen.  Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation.  In _CVPR_ , pp. 4942–4952, 2022.
  * Liu et al. (2023) Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.  Llm-qat: Data-free quantization aware training for large language models.  _arXiv preprint arXiv:2305.17888_ , 2023.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang.  Llm-pruner: On the structural pruning of large language models.  _arXiv preprint arXiv:2305.11627_ , 2023.
  * Marcus et al. (1993) Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of english: The penn treebank.  _Computational Linguistics_ , 19(2):313–330, 1993.
  * McKinstry et al. (2019) Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra S Modha.  Discovering low-precision networks close to full-precision networks for efficient inference.  In _2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)_ , pp. 6–9. IEEE, 2019.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _ICLR_ , 2017.
  * Nagel et al. (2019) Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling.  Data-free quantization through weight equalization and bias correction.  In _ICCV_ , pp. 1325–1334, 2019.
  * Nagel et al. (2020) Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In _ICML_ , pp. 7197–7206. PMLR, 2020.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.  URL https://api.semanticscholar.org/CorpusID:257532815.
  * Park et al. (2023) Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee.  Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models.  _arXiv preprint arXiv:2206.09557_ , 2023.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 21(1):5485–5551, 2020.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106, 2021.
  * Shao et al. (2023) Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo.  Omniquant: Omnidirectionally calibrated quantization for large language models.  _arXiv preprint arXiv:2308.13137_ , 2023.
  * Sun et al. (2023) Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter.  A simple and effective pruning approach for large language models.  _arXiv preprint arXiv:2306.11695_ , 2023.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971, 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. (2010) Guibin Wang, YiSong Lin, and Wei Yi.  Kernel fusion: An effective method for better power efficiency on multithreaded gpu.  In _2010 IEEE/ACM Int’l Conference on Green Computing and Communications & Int’l Conference on Cyber, Physical and Social Computing_, pp. 344–350, 2010.
  * Wei et al. (2022a) Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu.  QDrop: Randomly dropping quantization for extremely low-bit post-training quantization.  In _ICLR_ , 2022a.
  * Wei et al. (2022b) Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu.  Outlier suppression: Pushing the limit of low-bit transformer language models.  _NeurIPS_ , 35:17402–17414, 2022b.
  * Wei et al. (2023) Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu.  Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.  _arXiv preprint arXiv:2304.09145_ , 2023.
  * Wu et al. (2020) Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang.  Easyquant: Post-training quantization via scale optimization.  _arXiv preprint arXiv:2006.16669_ , 2020.
  * Wu et al. (2023) Xiaoxia Wu, Zhewei Yao, and Yuxiong He.  Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats.  _arXiv preprint arXiv:2307.09782_ , 2023.
  * Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _ICML_ , pp. 38087–38099. PMLR, 2023.
  * Yao et al. (2022) Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.  _NeurIPS_ , 35:27168–27183, 2022.
  * Yao et al. (2023) Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He.  Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation.  _arXiv preprint arXiv:2303.08302_ , 2023.
  * Yuan et al. (2023) Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu.  Rptq: Reorder-based post-training quantization for large language models.  _arXiv preprint arXiv:2304.01089_ , 2023.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In _ACL_ , pp. 4791–4800, 2019.
  * Zhang et al. (2018) Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In _ECCV_ , pp. 365–382, 2018.
  * Zhao et al. (2019) Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang.  Improving neural network quantization without retraining using outlier channel splitting.  In _ICML_ , pp. 7543–7552, 2019.
  * Zhou et al. (2016) Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  _arXiv preprint arXiv:1606.06160_ , 2016.

Appendix

##  Appendix A More details about bipartite soft matching

As mentioned in Section 4.1.2, we use bipartite soft matching (Bolya et al.,
2023) to determine which channels to aggregate efficiently. Support that we
want to aggregate $T-1$ channels. The step-by-step bipartite soft matching
algorithm is shown as follows:

  1. 1.

Divide the channels into two sets $\mathbb{A}$ and $\mathbb{B}$, each of
approximately equal size.

  2. 2.

For each channel in $\mathbb{A}$, construct an edge to its most similar
counterpart in $\mathbb{B}$.

  3. 3.

Select the $T-1$ most similar edges.

  4. 4.

Aggregate the channels that remain connected, according to Eq. (4).

  5. 5.

Concatenate the two sets to form the assembled channel set.

##  Appendix B More details about the efficient implementation for channel
reassembly

As mentioned in Section 4.3, the channel disassembly and assembly can be
implemented efficiently if the previous layer $l-1$ is a linear layer.
Specifically, let ${\bf W}^{l-1}\in\mathbb{R}^{C\times M}$ be the weights of
preceding linear layer, where $C$ and $M$ denotes the input and output channel
number for layer $l-1$, respectively. For channel disassembly, we enlarge the
output channels of the preceding linear layer weights by:

${\bf W}_{:i}^{l-1}=\begin{cases}{\bf W}_{:i}^{l-1}\quad&\text{if}~{}i\leq
M-1\\\ \frac{{\bf W}_{:M}^{l-1}}{T},\quad&\text{otherwise},\\\ \end{cases}$
(A)

and adjust the input channels of the current layer’s weight by:

${\bf W}_{i:}^{l}=\begin{cases}{\bf W}_{i:}^{l}\quad&\text{if}~{}i\leq M-1\\\
{\bf W}_{M:}^{l},\quad&\text{otherwise}.\\\ \end{cases}$ (B)

Similarly, for channel assembly, suppose that we are aggregating channel $j$
to channel $i$. Then, channel assembly can be implemented by reducing the
output weight channels of the preceding linear layer $l-1$ by:

${\bf W}_{:i}^{l-1}=\frac{{\bf W}_{:i}^{l-1}+{\bf W}_{:j}^{l-1}}{2},$ (C)

and adjusting the input channels of the current layer’s weight $l$ by:

${\bf W}_{i:}^{l}={\bf W}_{i:}^{l}+{\bf W}_{j:}^{l}.$ (D)

##  Appendix C Algorithm of Adaptive Channel Reassembly

We summarize our proposed adaptive channel reassembly in Algorithm 1.

Input: Input activation ${\bf x}\in\mathbb{R}^{M}$, linear layer weight ${\bf
W}\in\mathbb{R}^{M\times N}$, grid search iteration $P$.

Set ${\mathcal{L}}^{*}$ to $\infty$.

Function  _ReAssembly(_$\theta$_)_:

// Channel disassembly

Calculate the total sub-channels number by
$n=\sum_{i=1}^{M}\lceil\mathrm{max}({\bf x}_{{i}})/\theta\rceil$.

Perform channel disassembly using Eq. (3).

// Channel assembly

Find the $n$ most similar channel pairs using bipartite soft matching with the
distance metric in Eq. (5).

Perform channel assembly using Eq. (4).

return ReAssembly Error ${\mathcal{L}}$ using Eq. (7);

Calculate the max value of each channel ${\bf m}\in\mathbb{R}^{M}$.

for  _$p\in\\{1,2,\dots,P\\}$_ do

Calculate the threshold by $\theta=\mathrm{min}({\bf
m})+\frac{p}{P}\cdot(\mathrm{max}({\bf m})-\mathrm{min}({\bf m}))$.

${\mathcal{L}}$=ReAssembly(_$\theta$_);

if  _${\mathcal{L}} <{\mathcal{L}}^{*}$_ then

${\mathcal{L}}^{*}\leftarrow{\mathcal{L}}$, $\theta^{*}\leftarrow\theta$.

Adopt the final channel reassembly using the found $\theta^{*}$:
ReAssembly(_$\theta^{*}$_).

return One reassembled layer of LLM.

Algorithm 1 Algorithm of Adaptive Channel Reassembly for one layer in LLM.
Table A: Performance comparisons of different methods for weights and
activations quantization on LLaMA-2 model family.

Model | #Bits | Method | PPL $\downarrow$ | Accuracy (%) $\uparrow$
---|---|---|---|---
WikiText2 | C4 | Avg. | PIQA | ARC-e | ARC-c | HellaSwag | Winogrande | Avg.
LLaMA-2-7B | W16A16 | - | 5.47 | 6.97 | 6.22 | 76.82 | 53.62 | 40.53 | 72.87 | 67.25 | 62.22
W6A6 | SQ | 6.37 | 7.84 | 7.11 | 75.57 | 53.62 | 39.93 | 71.76 | 66.14 | 61.40
W6A6 | OS+ | - | - | - | 76.22 | 52.74 | 40.70 | 71.89 | 65.19 | 61.35
W6A6 | OmniQuant | 5.87 | 7.48 | 6.68 | 76.77 | 52.90 | 40.61 | 71.86 | 64.09 | 61.25
W6A6 | QLLM | 5.72 | 7.31 | 6.52 | 77.48 | 52.99 | 39.33 | 71.38 | 65.98 | 61.43
2-14 | W4A4 | SQ | 101.77 | 93.21 | 97.49 | 60.17 | 35.23 | 27.13 | 37.08 | 49.57 | 41.84
| W4A4 | OS+ | - | - | - | 63.11 | 39.10 | 28.84 | 47.31 | 51.3 | 45.93
| W4A4 | OmniQuant | 14.61 | 18.39 | 16.50 | 65.94 | 43.94 | 30.80 | 53.53 | 55.09 | 49.86
| W4A4 | QLLM | 11.75 | 13.26 | 12.51 | 67.68 | 44.40 | 30.89 | 58.45 | 56.59 | 51.60
LLaMA-2-13B | W16A16 | - | 4.88 | 6.47 | 5.68 | 78.84 | 57.91 | 44.28 | 76.63 | 69.85 | 65.50
W6A6 | SQ | 5.19 | 6.77 | 5.98 | 78.29 | 57.41 | 43.86 | 75.02 | 66.93 | 64.30
W6A6 | OS+ | - | - | - | 78.29 | 59.13 | 43.34 | 75.37 | 67.56 | 64.74
W6A6 | OmniQuant | 5.14 | 6.74 | 5.94 | 78.56 | 57.11 | 43.60 | 75.36 | 68.35 | 64.60
W6A6 | QLLM | 5.08 | 6.71 | 5.90 | 78.78 | 58.29 | 43.77 | 75.10 | 68.43 | 64.87
2-14 | W4A4 | SQ | 29.82 | 44.08 | 36.95 | 62.30 | 40.28 | 30.72 | 42.24 | 49.96 | 45.10
| W4A4 | OS+ | - | - | - | 64.47 | 41.46 | 32.17 | 59.30 | 51.38 | 49.76
| W4A4 | OmniQuant | 12.28 | 14.64 | 13.46 | 69.80 | 47.22 | 33.79 | 59.34 | 55.49 | 53.13
| W4A4 | QLLM | 9.09 | 11.13 | 10.11 | 70.46 | 48.48 | 34.39 | 62.80 | 55.41 | 54.31
LLaMA-2-70B | W16A16 | - | 3.32 | 5.52 | 4.42 | 81.01 | 59.68 | 47.95 | 80.87 | 76.95 | 69.29
W6A6 | SQ | 3.69 | 5.88 | 4.79 | 79.87 | 57.32 | 45.65 | 79.01 | 74.03 | 67.18
W6A6 | OS+ | - | - | - | 79.33 | 59.09 | 47.18 | 79.46 | 75.06 | 68.02
W6A6 | OmniQuant∗ | 3.71 | 5.91 | 4.81 | 80.20 | 60.27 | 46.84 | 80.55 | 76.01 | 68.77
W6A6 | QLLM | 3.55 | 5.76 | 4.66 | 80.63 | 59.01 | 45.99 | 79.64 | 75.37 | 68.13
2-14 | W4A4 | SQ | 26.01 | 34.61 | 30.31 | 64.09 | 41.84 | 32.00 | 54.21 | 51.07 | 48.64
| W4A4 | OS+ | - | - | - | 66.16 | 42.72 | 34.90 | 56.93 | 52.96 | 50.73
| W4A4 | OmniQuant∗ | 41.10 | 54.33 | 47.72 | 52.99 | 31.14 | 23.89 | 33.88 | 52.01 | 38.78
| W4A4 | QLLM | 7.00 | 8.89 | 7.95 | 74.27 | 50.59 | 37.2 | 71.62 | 59.43 | 58.62

  * •

∗ indicates no learnable equivalent transformation (Shao et al., 2023) on
queries, keys, values, or attention output due to incompatibility with
grouped-query attention (Ainslie et al., 2023) in LLaMA-2-70B model.

##  Appendix D More results on LLaMA-2 family

We provide additional results for the LLaMA-2 family in Table A. The
observations from these results are consistent with the phenomena identified
in the LLaMA-1 family. Notably, our QLLM significantly outperforms the state-
of-the-art post-training quantization (PTQ) methods, demonstrating a
substantial margin of improvement in 4-bit quantization. For example, QLLM
quantized 4-bit LLaMA-2-65B outperforms SmoothQuant counterpart by an average
of 7.89% on the accuracy, which shows the promising results of our method.

##  Appendix E Effect of different calibration sets

We apply QLLM to yield 4-bit LLaMA-7B using different calibration sets and
report the results in Table B. From the results, we observe that the choices
of calibration set have a minor effect, as the performance remains relatively
consistent across different sets. For fair comparisons, following OmniQuant
(Shao et al., 2023), we use WikiText2 as a calibration set by default.

Table B: Effect of different calibration sets. We report the perplexity
$\downarrow$ of W4A4 LLaMA-7B on WikiText2 (Merity et al., 2017), PTB (Marcus
et al., 1993) and C4 (Raffel et al., 2020).

|  | Evaluation Set |
---|---|---|---
|  | WikiText2 | PTB | C4 | Average
Calibration Set | WikiText2 | 9.65 | 16.56 | 12.29 | 12.83
PTB | 10.73 | 13.89 | 12.44 | 12.35
C4 | 10.56 | 17.44 | 12.08 | 13.36

##  Appendix F Effect of different numbers of calibration samples

We apply QLLM to yield 4-bit LLaMA-7B using different numbers of calibration
samples from WikiText2 and show the results in Table C. The results reveal a
positive correlation between the performance of the quantized model and the
number of calibration samples, indicating that utilizing more samples
generally leads to better performance. This trend underscores the importance
of the calibration phase, where leveraging a larger sample pool can provide a
more comprehensive representation of the data distribution, enabling more
accurate quantization. However, it is also imperative to consider the
computational and memory overhead associated with an increasing number of
calibration samples. There is an inherent trade-off between achieving higher
model performance and maintaining computational efficiency.

Table C: Effect of different # calibration samples. We report the perplexity
$\downarrow$ of W4A4 LLaMA-7B on WikiText2 (Merity et al., 2017), PTB (Marcus
et al., 1993) and C4 (Raffel et al., 2020).

#Samples | WikiText2 | PTB | C4 | Average
---|---|---|---|---
16 | 10.72 | 18.43 | 13.66 | 14.27
32 | 10.12 | 17.35 | 12.84 | 13.44
64 | 10.15 | 16.23 | 12.26 | 12.88
128 | 9.65 | 16.56 | 12.29 | 12.83
256 | 9.60 | 15.75 | 11.79 | 12.38

##  Appendix G More results about the expansion ratios of the quantized LLM

In this section, we illustrate the detailed expansion ratios for the input
activations of different layers in the 4-bit LLaMA-1-13B obtained by our
adaptive strategy in Figure A. From the results, our adaptive strategy
allocates higher expansion ratios to the shallower MSA layers and to the
deeper down projection layer in the FFN, which indicates that these layers
possess a greater number of outliers. To substantiate this observation, we
further plot the channel-wise maximum and minimum values for the input
activations across different layers in Figure B. These visual representations
further underscore the effectiveness of our adaptive strategy in identifying
and addressing the presence of outliers in different layers.

Figure A: An illustration of the searched expansion ratios using our adaptive
strategy for 4-bit LLaMA-1-13B. Figure B: An illustration of the channel-wise
maximum and minimum input activation values for the MSA, up projection and
down projection layers in FFN of different blocks in LLaMA-1-13B.