#  CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained
Models using Greedy Coordinate Descent

Pranav Ajit Nair
Google DeepMind
<EMAIL_ADDRESS>&Arun Sai Suggala
Google DeepMind
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) ††Work done when the authors were at Google
Research. have recently demonstrated remarkable performance across diverse
language tasks. But their deployment is often constrained by their substantial
computational and storage requirements. Quantization has emerged as a key
technique for addressing this challenge, enabling the compression of large
models with minimal impact on performance. The recent GPTQ algorithm, a post-
training quantization (PTQ) method, has proven highly effective for
compressing LLMs, sparking a wave of research that leverages GPTQ as a core
component. Recognizing the pivotal role of GPTQ in the PTQ landscape, we
introduce CDQuant, a simple and scalable alternative to GPTQ with improved
performance. CDQuant uses coordinate descent to minimize the layer-wise
reconstruction loss to achieve high-quality quantized weights. Our algorithm
is easy to implement and scales efficiently to models with hundreds of
billions of parameters. Through extensive evaluation on the PaLM2 model
family, we demonstrate that CDQuant consistently outperforms GPTQ across
diverse model sizes and quantization levels. In particular, for INT2
quantization of PaLM2-Otter, CDQuant achieves a $10\%$ reduction in perplexity
compared to GPTQ.

##  1 Introduction

Large language models (LLMs) have shown remarkable ability to handle various
language tasks (Touvron et al., 2023; OpenAI, 2023; Google, 2023), but their
widespread adoption is hampered by their substantial computational and memory
demands. To tackle this challenge, researchers have explored techniques like
quantization, pruning, and distillation, with quantization being a
particularly promising avenue for reducing model size and inference time
without significantly sacrificing performance (Miao et al., 2023).
Quantization techniques broadly fall into two categories: post-training
quantization (PTQ) and quantization-aware training (QAT). QAT, while
potentially yielding better results, poses a significant challenge for LLMs
due to the immense resources required to train these large models. As a
result, a growing body of research has focused on PTQ, which is generally less
computationally intensive and can be applied to large, pre-trained models.

A seminal work in PTQ for LLMs is GPTQ (Frantar et al., 2022), which
introduced a one-shot weight quantization approach that minimizes the
following layer-wise output reconstruction loss: $\|X(W-\hat{W})\|_{F}^{2}$,
where $X=[\mathbf{x}_{1},\mathbf{x}_{2}\dots\mathbf{x}_{n}]$ represents the
layer inputs and $W,\widehat{W}\in\mathbb{R}^{d_{\text{in}}\times
d_{\text{out}}}$ are the original and quantized weight matrices, respectively.
This method has sparked a wave of research in PTQ. Subsequent works have built
upon GPTQ, addressing its limitations and improving it’s performance. For
instance, SpQR (Dettmers et al., 2023) and OWQ (Lee et al., 2023) improve GPTQ
by explicitly addressing outlier weights, maintaining them in full precision,
while quantizing the remaining weights using GPTQ. This hybrid approach
improves quantization accuracy, particularly at lower bit-widths. Another line
of research focuses on transforming the weight space before applying GPTQ. For
instance, QuIP (Chee et al., 2024), FrameQuant (Adepu et al., 2024) transform
the weights into a more quantization-friendly space and apply GPTQ in the
transformed space. AWQ (Lin et al., 2023), SmoothQuant (Xiao et al., 2023)
reduce the effect of activation outliers by performing feature scaling before
quantizing the weights using standard techniques.

Given the central role GPTQ plays in the landscape of PTQ methods for LLMs,
improving its quality directly translates to better quantization across
numerous techniques that build upon it. Consequently, in this work, we revisit
the core optimization problem addressed by GPTQ and introduce a novel
coordinate descent based algorithm, CDQuant, to minimize the objective.
CDQuant is an iterative optimization technique that greedily selects
coordinates to descend along in each iteration. This approach contrasts with
GPTQ, which cycles through the coordinates only _once_ , in a predetermined
order, often leading to suboptimal solutions for the layer-wise objective (see
Section 2 for more details). CDQuant is simple to implement and scales
effectively to models with billions of parameters. We further extend CDQuant
to group/sub-channel quantization, a technique gaining popularity of late
(Dettmers et al., 2021; Dettmers and Zettlemoyer, 2023). Extensive experiments
on PaLM 2 (Anil et al., 2023) demonstrate that CDQuant consistently
outperforms GPTQ across various model sizes and quantization precision levels.
For example, for INT2 quantization of PaLM2-Otter, CDQuant achieves a $10\%$
reduction in perplexity compared to GPTQ.

##  2 Related Work

The literature on quantization is huge. In this section, we only review the
works that are related to quantization of large pre-trained models, and those
that are related to our work.

#### Weight-only Quantization.

Optimal Brain Quantization (OBQ) (Frantar and Alistarh, 2022), a post-training
quantization (PTQ) technique, draws inspiration from the Optimal Brain Surgeon
(OBS) framework proposed by LeCun et al. (1989). Frantar and Alistarh (2022)
introduced exact yet efficient algorithms for implementing the OBS greedy
strategy. However, OBQ remains computationally expensive and struggles to
scale effectively to large language models (LLMs). Frantar et al. (2022)
subsequently introduced GPTQ, which employs heuristics to accelerate the OBQ
algorithm. Specifically, GPTQ updates coordinates once in a cyclic manner
rather than the greedy approach used in OBQ. This modification significantly
speeds up the algorithm but comes at the cost of reduced performance. In our
work, we address this performance drop by proposing greedy coordinate descent
algorithms that are both straightforward and easy to implement.

Recent works have sought to improve upon GPTQ. One popular strategy here is to
identify and isolate problematic weights before applying GPTQ. For example,
SpQR (Dettmers et al., 2023) isolates outlier weights and keeps them at full
precision, quantizing only the remaining values using GPTQ. Similarly, OWQ
(Lee et al., 2023) identifies weights that are highly sensitive to small
perturbations and excludes them from the GPTQ quantization process. Another
promising direction involves transforming the weights into a different basis
before quantization. Both QuIP (Chee et al., 2024), and FrameQuant (Adepu et
al., 2024) take this route and perform quantization in the transformed space
using GPTQ. In all these approaches, we could potentially substitute GPTQ with
CDQuant and expect to see further improvements in performance. Other promising
methods such as AWQ (Lin et al., 2023), AffineQuant (Ma et al., 2024),
suppress the effect of outliers in input activations by adjusting their scale
and transferring it to weights. They then perform the standard MinMax
quantization on the rescaled weights. One could replace MinMax with GPTQ,
CDQuant to improve the performance of these techniques (see Table 3 for a
comparison of CDQuant with AWQ).

While working on our paper, we came across QuantEase (Behdin et al., 2023), a
parallel research effort sharing similar goals as ours for improving GPTQ.
While both methods leverage the concept of coordinate descent, QuantEase
adopts a cyclic approach for weight updates, whereas our work employs a greedy
strategy. Although our experiments (Appendix F) indicate comparable
performance between the two methods, our work extends beyond QuantEase by
introducing specialized algorithms for group/sub-channel quantization, not
just full-channel quantization. Additionally, we develop novel block
coordinate descent algorithms that further improve the performance. That being
said, both these works (QuantEase and ours) collectively highlight the
potential of coordinate descent algorithms in outperforming GPTQ.

#### Weight+Activation Quantization.

SmoothQuant (Xiao et al., 2023), OS+(Wei et al., 2023, 2022) have a similar
flavour as AWQ, but quantize both weights and activations after scaling them
appropriately. OmniQuant (Shao et al., 2023) performs quantization of the
entire transformer block in a single shot. This encompasses both activation
and weight quantization. Furthermore, it subsumes SmoothQuant, OS+ by using
both feature scaling and outlier suppression. QLLM (Liu et al., 2023) tackles
the issue of outliers in the activations by splitting the outlier features
into multiple sub-channels and then recombining them, effectively reducing
their influence. QLLM also incorporates a fine-tuning step at the end,
introducing low-rank weights into each layer of the LLM. LLM.int8() (Dettmers
et al., 2022) quantizes both acativations and weights to $8$-bits and also
identifies outliers and stores them in full precision.

#### Efficient End-to-End Quantization.

To recover the drop in performance from quantization Chai et al. (2023);
Dettmers et al. (2024); Hu et al. (2021) perform low-rank parameter-efficient
fine-tuning.

##  3 CDQuant

#### Notation.

Throughout the paper, we denote vectors by bold faced letters ($\mathbf{a}$),
and matrices by capital letters ($A$).
$\|\mathbf{a}\|_{2}=\sqrt{\sum_{i}\mathbf{a}_{i}^{2}}$ is the Euclidean norm
and $\|A\|_{F}=\sqrt{\sum_{i,j}A_{ij}^{2}}$ is the Frobenius norm of a matrix.
$\mbox{diag}(\mathbf{a})$ represents a diagonal matrix with $\mathbf{a}$ as
its diagonal entries. $d_{\text{in}},d_{\text{out}}$ denote the input, output
dimensions of a layer. $W\in\mathbb{R}^{d_{\text{in}}\times d_{\text{out}}}$
is the weight matrix of the layer, and
$X=[\mathbf{x}_{1},\mathbf{x}_{2}\dots\mathbf{x}_{n}]\in\mathbb{R}^{n\times
d_{\text{in}}}$ is the matrix containing $n$ datapoints that are sent as input
to the layer. $H=X^{T}X$ is the Hessian matrix for our objective in Equation
(1). $c$ denotes the number of bits of precision used in quantization. In
quantization, we aim to represent $W$ as
$Q\times\mbox{diag}(\mathbf{a})+\mathbf{1}\mathbf{b}^{T}$, where
$\mathbf{a},\mathbf{b}\in\mathbb{R}^{d_{\text{out}}}$ represent the scale and
bias parameters and $Q\in\\{0,1,\dots 2^{c}-1\\}^{d_{\text{in}}\times
d_{\text{out}}}$ is the quantized matrix.

Many existing PTQ techniques, including GPTQ, aim to solve the following
layer-wise optimization objective:
$\min_{\mathbf{a},\mathbf{b},Q}\|X(W-Q\times\mbox{diag}(\mathbf{a})-\mathbf{1}\mathbf{b}^{T})\|_{F}^{2}$.
Observe that this problem breaks down into $d_{\text{out}}$ independent
problems across the output dimension. So, in the sequel, we focus on the
following problem of quantizing a $d_{\text{in}}$-dimensional vector

$\displaystyle\min_{a,b,\mathbf{q}}\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}.$
(1)

For a fixed $(a,b)$, this problem is called _Integer Linear Regression_
problem. It turns out, finding optimal solutions to this problem is NP-hard
(Chrétien and Corset, 2009; Park and Boyd, 2018). So, several works have
designed heuristics to solve this problem (Nagel et al., 2020; Li et al.,
2021; Frantar and Alistarh, 2022; Frantar et al., 2022; Hubara et al., 2021).
Within the context of LLMs, GPTQ is perhaps the most popular among these
techniques, as it scales efficiently to models with billions of parameters
(Frantar et al., 2022). In this work, we aim to improve upon GPTQ by designing
better heuristics, while maintaining its scalability aspect. To this end, we
rely on performing coordinate descent on objective (1), which we describe
below.

###  3.1 Greedy Coordinate Descent

Algorithm 1 Greedy Coordinate Descent (CD)

1:Input: $T$ \- coordinate descent steps, $X$ \- input data matrix,
$\mathbf{w}$ \- vector to be quantized, $a$ \- scale, $b$ \- bias,
$\mathbf{q}_{0}$ \- initial estimate

2:Compute Hessian $H$ as: $H\leftarrow X^{T}X$

3:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow
2H(\mathbf{q}_{0}-a^{-1}(\mathbf{w}-b))$

4:for $t\in[1:T]$ do

5: Find the coordinate that leads to the largest reduction in loss
$i^{*},r^{*}=\operatorname*{arg\,min}_{i\in\\{0,1,\dots
d_{\text{in}}-1\\},r\in\\{0,1,\dots
2^{c}-1\\}}(r-\mathbf{q}_{t-1,i})^{2}H_{i,i}+(r-\mathbf{q}_{t-1,i})\mathbf{g}_{i}$

6: Update gradient $\mathbf{g}$ as
$\mathbf{g}\leftarrow\mathbf{g}+2(r^{*}-\mathbf{q}_{t-1,i^{*}})H_{i^{*},\cdot},$
where $H_{i^{*},\cdot}$ is the $i^{*}$ column of $H$

7: Update $\mathbf{q}_{t-1}$ as
$\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1}+(r^{*}-\mathbf{q}_{t-1,i^{*}})\mathbf{e}_{i^{*}},$
where $\mathbf{e}_{i^{*}}$ is the standard basis vector with $1$ in $i^{*}$
position and $0$ everywhere else

8:end for

In this section, we assume we have suitable values for scale ($a$) and bias
($b$) parameters already at hand, and focus on optimizing $\mathbf{q}$. For a
more in-depth discussion of how we determine these values, please refer to the
final part of the section. As the name suggests, in greedy coordinate descent,
at each round, we find the coordinate that leads to the biggest reduction in
the objective and descend along that coordinate. Letting
${\mathcal{L}}(\mathbf{q})\coloneqq\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}$ be
the objective in Equation (1), we try to find a coordinate $i$ and value $r$,
such that updating the $i^{th}$ coordinate to $r$ gives the biggest reduction
in loss

$\min_{i,r}{\mathcal{L}}(\mathbf{q}+(r-\mathbf{q}_{i})\mathbf{e}_{i})-{\mathcal{L}}(\mathbf{q}),$

where $\mathbf{e}_{i}$ is the standard basis vector with $1$ in $i$ position
and $0$ everywhere else. Luckily for us, this can be implemented extremely
efficiently as we have analytical expressions for the objective. In
particular, one can easily show that

${\mathcal{L}}(\mathbf{q}+(r-\mathbf{q}_{i})\mathbf{e}_{i})-{\mathcal{L}}(\mathbf{q})=(r-\mathbf{q}_{i})^{2}H_{i,i}+(r-\mathbf{q}_{i})\mathbf{g}_{i},$

where $H,\mathbf{g}$ are the Hessian and gradient of ${\mathcal{L}}$ evaluated
at $\mathbf{q}$. Algorithm 1 describes this procedure. Observe that _line 5_
is the most expensive step of the algorithm. It requires finding the minimum
value among $d_{\text{in}}\times 2^{c}$ possibilities, a task that is well-
suited for parallelization on GPUs.

#### Extension to Block Coordinate Descent.

A natural extension to Algorithm 1 is block coordinate descent (BCD), where
multiple coordinates are updated simultaneously in each iteration. While a
greedy approach to BCD could, in principle, optimize the objective much
better, the computational cost becomes prohibitive. Specifically, updating $k$
coordinates at a time necessitates evaluating $(d_{\text{in}}\times
2^{c})^{k}$ possible combinations of coordinates and their corresponding
values. To overcome this, we propose a randomized BCD strategy (Algorithm 2),
where we randomly partition the coordinates into $d_{\text{in}}/k$ blocks, and
only search over these blocks and their corresponding values. This
significantly reduces the search space to a more manageable
$d_{\text{in}}/k\times 2^{kc}$ possibilities, making the algorithm practical
for larger models. In our experiments, we primarily set $k=2$ and work with
$c\in\\{2,3,4\\}$.

#### Initializing $a,b,\mathbf{q}$.

To initialize $a,b,\mathbf{q}$ in Algorithms 1, 2, we introduce a technique
called Optimal Weight Clipping (OWC), which draws inspiration from the
Learnable Weight Clipping (LWC) mechanism used in OmniQuant (Shao et al.,
2023). In OWC, we quantize weight $\mathbf{w}$ as follows

$\displaystyle\mathbf{q}=\text{clamp}\left(\left\lfloor\frac{\mathbf{w}-b}{a}\right\rceil,0,2^{c}-1\right),\quad
a=\frac{\gamma(\max(\mathbf{w})-\min(\mathbf{w}))}{2^{c}-1},\quad
b=\min(\mathbf{w}).$ (2)

Here, $\gamma\in[0,1]$ represents the clipping strength. We determine the
optimal $\gamma$ by minimizing the following layer-wise loss:

$\min_{\gamma\in[0,1]}\|X(\mathbf{w}-a\mathbf{q}-b)\|_{2}^{2}.$

Note that while not explicitly stated, both $\mathbf{w},a$ in the above
objective are implicitly dependent on $\gamma$. This optimization can be
efficiently solved using a simple grid search. In contrast, LWC (Shao et al.,
2023) optimizes a different objective function, focusing on end-to-end
quantization of entire transformer block using gradient-based techniques. It
is worth noting that setting $\gamma=1$ in OWC recovers the widely used MinMax
quantization scheme, which is used in many existing quantization methods,
including GPTQ (Frantar et al., 2022), SmoothQuant‘(Xiao et al., 2023). In our
experiments, we observed that OWC provides a much better initialization
compared to MinMax quantization, for both GPTQ and CDQuant.

Algorithm 2 Block Coordinate Descent with Random Blocks (BCD)

1:Input: $T$ \- coordinate descent steps, $k$ \- block size, $X$ \- input data
matrix, $\mathbf{w}$ \- vector to be quantized, $a$ \- scale, $b$ \- bias,
$\mathbf{q}_{0}$ \- initial estimate

2:Compute Hessian $H$ as: $H\leftarrow X^{T}X$

3:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow
2H(\mathbf{q}_{0}-a^{-1}(\mathbf{w}-b))$

4:for $t\in[1:T]$ do

5: Randomly partition the set $\\{0,1,\dots d_{\text{in}}-1\\}$ into
$d_{\text{in}}/k$ blocks, each of size $k$

6: Find the block that leads to the largest reduction in loss
$i^{*},r^{*}=\operatorname*{arg\,min}_{\begin{subarray}{c}i\in\\{0,1,\dots
d_{\text{in}}/k-1\\},\\\ r\in\\{0,1,\dots
2^{c}-1\\}^{k}\end{subarray}}(r-\mathbf{q}_{t-1,i})^{T}H_{i,i}(r-\mathbf{q}_{t-1,i})+(r-\mathbf{q}_{t-1,i})^{T}\mathbf{g}_{i},$
where $\mathbf{q}_{t-1,i},H_{i,i}$ are the sub-vector, sub-matrix of
$\mathbf{q}_{t-1},H$ corresponding to block $i$.

7: Update gradient $\mathbf{g}$ as
$\mathbf{g}\leftarrow\mathbf{g}+2H_{i^{*},\cdot}(r^{*}-\mathbf{q}_{t-1,i^{*}}),$

8: Update $\mathbf{q}_{t-1}$ as  $\mathbf{q}_{t-1,i^{*}}\leftarrow
r^{*},\quad\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1},$

9:end for

###  3.2 Extension to Sub-channel Quantization

In this section, we consider sub-channel (or group) quantization, which is a
more fine-grained quantization that divides the weight vector $\mathbf{w}$
into multiple groups and assigns a quantization scale to each group (Dettmers
et al., 2021; Dettmers and Zettlemoyer, 2023). Letting $g$ be the group size,
we divide weight $\mathbf{w}$ into $d_{\text{in}}/g$ groups
$\\{\mathbf{w}^{(0)},\mathbf{w}^{(1)},\dots\mathbf{w}^{(d_{\text{in}}/g-1)}\\}$
each of size $g$, and quantize $\mathbf{w}^{(i)}$ as
$a^{(i)}\mathbf{q}^{(i)}+b^{(i)}$. To learn the optimal parameters for this
sub-channel quantization, we solve the following optimization problem:

$\displaystyle\min_{\\{a^{(i)},b^{(i)},\mathbf{q}^{(i)}\\}_{i=0}^{d_{\text{in}}/g-1}}\big{|}\big{|}\sum_{i=0}X^{(i)}(\mathbf{w}^{(i)}-a^{(i)}\mathbf{q}^{(i)}-b^{(i)})\big{|}\big{|}_{2}^{2}.$
(3)

Here, $X^{(i)}$ represents the columns of $X$ corresponding to the indices
within group $i$. To solve this optimization problem, we employ a coordinate
descent approach, similar to Algorithms 1 and 2 described earlier. That is,
given initial values for the scaling and bias parameters, we iteratively
optimize the quantized representation $\mathbf{q}$ using coordinate descent.
Due to space constraints, we present the resulting algorithms in Appendix C
(see Algorithms 4, 5).

#### Initialization.

Next, we tackle the initialization of parameters
$\\{a^{(i)},b^{(i)},\mathbf{q}^{(i)}\\}_{i=0}^{d_{\text{in}}/g-1}$ for our
coordinate descent procedure. Our approach draws inspiration from the OWC
algorithm described above, adapting its core idea to this problem. In essence,
we reframe the initialization problem as one of selecting optimal clipping
strengths $\\{\gamma^{(i)}\\}_{i=0}^{d_{\text{in}}/g-1}$ for groups
$\\{0,\dots d_{\text{in}}/g-1\\}$. This leads us to the following problem

$\displaystyle\min_{\gamma^{(0)},\dots\gamma^{(d_{\text{in}}/g-1)}}\big{|}\big{|}\sum_{i=0}X^{(i)}(\mathbf{w}^{(i)}-a^{(i)}\mathbf{q}^{(i)}-b^{(i)})\big{|}\big{|}_{2}^{2},$
(4)

where

$\mathbf{q}^{(i)}=\text{clamp}\left(\left\lfloor\frac{\mathbf{w}^{(i)}-b^{(i)}}{a^{(i)}}\right\rceil,0,2^{c}-1\right),\
a^{(i)}=\frac{\gamma^{(i)}(\max(\mathbf{w}^{(i)})-\min(\mathbf{w}^{(i)}))}{2^{c}-1},\
b^{(i)}=\min(\mathbf{w}^{(i)}).$

We use greedy coordinate descent to optimize Equation (4). In each iteration,
we update the $\gamma^{(i)}$ that leads to biggest drop in loss. This
procedure, which we call OWC-CD, is described in Algorithm 3.

Algorithm 3 Coordinate Descent for Optimal Weight Clipping (OWC-CD)

1:Input: $T$ \- coordinate descent steps, $g$ \- group size, $X$ \- input data
matrix, $\mathbf{w}$ \- weight vector, $\Gamma$\- grid of possible values for
clipping strength

2:for $\beta\in\Gamma$ do

3: for $i\in[0:d_{\text{in}}/g-1]$ do

4: Compute quantization residual $\Delta(i,\beta)$ for group $i$ with clipping
strength $\beta$ as:
$\Delta(i,\beta)\leftarrow\mathbf{w}^{(i)}-a^{(i)}(\beta)\mathbf{q}^{(i)}(\beta)-b^{(i)}(\beta)$
. where $a^{(i)}(\beta),b^{(i)}(\beta),\mathbf{q}^{(i)}(\beta)$ are as defined
in Equation (2).

5: end for

6:end for

7:Initialize clipping strengths for each group
$\gamma^{(0)},\dots\gamma^{(d_{\text{in}}/g-1)}$

8:Compute Hessian $H$ as: $H\leftarrow X^{T}X$

9:$\mathbf{v}_{i}\leftarrow-2H_{i}(a\mathbf{q}+b-w)$ where $H_{i}$ is the sub-
matrix of $H$ corresponding to the columns of group $i$.

10:for $t\in[1:T]$ do

11: Find the group that leads to the largest reduction in loss  $\displaystyle
i^{*},\beta^{*}=\operatorname*{arg\,min}_{\begin{subarray}{c}i\in\\{0,1,\dots
d_{\text{in}}/g-1\\},\\\ \beta\in\Gamma\end{subarray}}$
$\displaystyle(\Delta(i,\gamma^{(i)})-\Delta(i,\beta))^{T}H_{i,i}(\Delta(i,\gamma^{(i)})-\Delta(i,\beta))$
$\displaystyle+\mathbf{v}_{i}^{T}(\Delta(i,\gamma^{(i)})-\Delta(i,\beta))$
where $H_{i,i}$ is the block diagonal element of $H$ corresponding to group
$i$.

12: Update
$\mathbf{v}\leftarrow\mathbf{v}+2(\Delta(i^{*},\gamma^{(i^{*})})-\Delta(i^{*},\beta^{*}))^{T}H_{i^{*}}$

13: Update $\gamma^{(i^{*})}\leftarrow\beta^{*}$

14:end for

##  4 Experiments

#### Quantization.

In our experiments, we focus on weight-only quantization. We present results
for two scenarios: (1) quantizing the FFN layers and (2) quantizing both the
FFN and attention weights. In the former setting, all attention layers are
quantized to INT8 using MinMax quantization, and low-bit quantization schemes
are only applied to the feed-forward layers. It’s worth pointing out that this
setting is very relevant in practice because FFN layers account for majority
of the latency, and quantizing attention weights yields negligible latency
reduction at the expense of model quality (Samaga et al., 2024). We perform
both per-channel and sub-channel quantization (with a group size of $128$). We
mainly focus on INT2, INT3 and INT4 quantization.

#### Models.

We use PaLM2 models (Anil et al., 2023) in all our experiments. In particular,
we use PaLM2-Gecko, PaLM2-Otter, PaLM2-Bison models. These models vary in
size, with PaLM2-Gecko being the smallest, followed by PaLM2-Otter, and
finally PaLM2-Bison.

#### Baselines.

Since our primary goal is to demonstrate that CDQuant gets improved
performance over GPTQ, we have chosen it as the primary baseline in most of
our experiments. To further demonstrate the our technique can be used as a
plug-and-play replacement for GPTQ, we also include comparisons with AWQ (Lin
et al., 2023). For all our experiments, we initialize GPTQ with OWC, and run
GPTQ for $T=d_{\text{in}}$ steps. To ensure stability and generalization, GPTQ
regularizes its Hessian matrix by adding a scaled identity matrix ($\lambda
I$). Tuning this $\lambda$ for every (model, layer) pair is infeasible. So, we
determine a single optimal value using the PaLM2-Gecko model, and apply it
universally.

####  CDQuant.

We evaluate both the coordinate descent variants described in Section 3.1: CD
(Algorithm 1), BCD (Algorithm 2). For per-channel quantization, we initialize
CD with Optimal Weight Clipping (OWC), and for sub-channel quantization, we
additionally include initialization with OWC-CD. BCD is always initialized
with CD in our experiments. Unless otherwise stated, both CD and BCD are run
for $T=d_{\text{in}}$ iterations, OWC-CD is run for $d_{\text{in}}/g$
iterations, where $g$ is the group size. Similar to GPTQ, we regularize the
Hessian matrix used in CD, BCD by adding $\lambda I$. We determine a
reasonable value for $\lambda$ using the PaLM2-Gecko model, and use it in all
our experiments.

#### Evaluation.

Following recent works (Frantar et al., 2022; Ma et al., 2024), we evaluate
all algorithms using two key metrics: perplexity and downstream task
performance. To ensure a robust evaluation, we calculate perplexity using a
$100$ million token subset derived from the PaLM2 training mixture. We chose
to do this since the evaluation sets for PTB (Marcus et al., 1994) and
WikiText2 (Merity et al., 2017) datasets, that are commonly used in the field,
are orders of magnitude smaller (83K and 246K tokens in their respective test
sets, and 512K tokens subsampled from the C4 validation set (Frantar et al.,
2022)), and thus can have high variance, and may not accurately reflect the
model’s true performance. We use TriviaQA (Joshi et al., 2017), SQuAD
(Rajpurkar et al., 2018), NaturalQuestions (Kwiatkowski et al., 2019) and
WebQuestions (Berant et al., 2013) to evaluate generation capabilities of the
quantized models, and to evaluate their reasoning capabilities, we test on
ARC-c, ARC-e (Clark et al., 2018), HellaSwag (Zellers et al., 2019), BoolQ
(Clark et al., 2019), PIQA (Bisk et al., 2020) and WinoGrande (Sakaguchi et
al., 2020). For downstream evaluations, we consider both zero-shot, and one-
shot settings. Finally, to determine which optimization technique is most
effective at solving the layer-wise objective in Equation (1), we evaluate the
final solution’s objective value using the same $100$ million tokens that we
used to compute perplexity.

#### Training.

All techniques are calibrated using $1280$ data points, where each data point
has $2048$ tokens. For OWC, we use a grid size of $50$ to find the most
optimal $\gamma$. We used 8 Nvidia H$100$ GPUs for quantizing the models.

Table 1: Table presents _pretraining perplexity_ numbers for GPTQ, CD, BCD for
INT3, INT4 quantization of FFN weights. $wx,ay,gz$ in the config column
corresponds to $x$-bit weights, $y$-bit activations and group/sub-channel size
of $z$.

Config | Method | PaLM2-Gecko | PaLM2-Otter | PaLM2-Bison
---|---|---|---|---
w16a16 | - | 7.948 | 5.984 | 5.298
w3a16 | OWC | $12.570$ | $17.928$ | $6.169$
GPTQ | $11.347$ | $7.176$ | $5.774$
| CD | $10.920$ | $7.002$ | $5.739$
| BCD(k=2) | $\mathbf{10.898}$ | $\mathbf{6.979}$ | $\mathbf{5.733}$
w3a16g128 | OWC | $11.597$ | $8.342$ | $5.847$
GPTQ | $10.414$ | $6.635$ | $5.677$
| CD | $10.273$ | $6.655$ | $5.656$
| BCD(k=2) | $10.259$ | $6.545$ | $5.654$
| OWC-CD | $10.706$ | $6.635$ | $5.686$
|  OWC-CD \+ CD | $10.143$ | $6.528$ | $5.650$
|  OWC-CD \+ BCD(k=2) | $\mathbf{10.138}$ | $\mathbf{6.527}$ | $\mathbf{5.647}$
w4a16 | OWC | $8.946$ | $6.693$ | $5.475$
GPTQ | $8.764$ | $6.249$ | $5.417$
| CD | $8.694$ | $6.195$ | $5.407$
| BCD(k=2) | $\mathbf{8.691}$ | $\mathbf{6.192}$ | $\mathbf{5.405}$
w4a16g128 | OWC | $8.613$ | $6.264$ | $5.401$
GPTQ | $8.498$ | $6.112$ | $5.377$
| CD | $8.456$ | $6.097$ | $5.373$
| BCD(k=2) | $8.454$ | $6.097$ | $5.372$
| OWC-CD | $8.519$ | $6.106$ | $5.377$
|  OWC-CD \+ CD | $8.436$ | $6.092$ | $5.371$
|  OWC-CD \+ BCD(k=2) | $\mathbf{8.434}$ | $\mathbf{6.091}$ | $\mathbf{5.371}$

Table 2: Table presents _downstream evaluation_ (one-shot) numbers for GPTQ,
CD, BCD for INT3, INT4 quantization of FFN weights. _Gen_ , _Rank_ columns
correspond to generation and ranking tasks.

Config | Method | PaLM2-Gecko | PaLM2-Otter | PaLM2-Bison
---|---|---|---|---
|  | Gen. | Rank | Avg. | Gen. | Rank | Avg. | Gen. | Rank | Avg.
w16a16 | - | $25.85$ | $61.44$ | $47.2$ | $44.31$ | $75.43$ | $62.98$ | $51.08$ | $78.28$ | $68.00$
w3a16 | OWC | $18.95$ | $55.74$ | $41.02$ | $21.02$ | $65.78$ | $47.87$ | $47.52$ | $77.76$ | $65.66$
GPTQ | $19.42$ | $57.71$ | $42.39$ | $37.26$ | $73.95$ | $59.27$ | $48.55$ | $78.58$ | $66.57$
| CD | $20.69$ | $58.05$ | $43.11$ | $38.12$ | $73.37$ | $59.27$ | $48.47$ | $78.56$ | $66.53$
| BCD(k=2) | $20.52$ | $58.44$ | $43.27$ | $38.53$ | $73.46$ | $59.49$ | $48.08$ | $78.66$ | $66.43$
w3a16g128 | OWC | $22.83$ | $57.02$ | $43.35$ | $33.18$ | $70.81$ | $55.76$ | $48.08$ | $78.34$ | $66.24$
GPTQ | $22.24$ | $58.66$ | $44.09$ | $40.30$ | $74.12$ | $60.59$ | $49.15$ | $78.58$ | $66.81$
| CD | $22.79$ | $58.81$ | $44.40$ | $41.33$ | $74.27$ | $61.10$ | $49.17$ | $78.49$ | $66.76$
| BCD(k=2) | $22.64$ | $58.85$ | $44.37$ | $41.19$ | $73.94$ | $60.84$ | $49.02$ | $78.51$ | $66.71$
| OWC-CD | $22.21$ | $58.69$ | $44.10$ | $40.84$ | $74.33$ | $60.93$ | $48.54$ | $78.29$ | $66.39$
|  OWC-CD \+ CD | $22.44$ | $58.84$ | $44.28$ | $40.87$ | $74.41$ | $61.00$ | $49.22$ | $78.86$ | $67.00$
|  OWC-CD \+ BCD(k=2) | $22.15$ | $58.77$ | $44.13$ | $41.19$ | $74.65$ | $61.27$ | $49.08$ | $78.3$ | $66.61$
w4a16 | OWC | $23.27$ | $60.22$ | $45.44$ | $41.70$ | $74.21$ | $61.21$ | $50.05$ | $78.72$ | $67.25$
GPTQ | $24.51$ | $60.72$ | $46.24$ | $43.00$ | $74.86$ | $62.12$ | $50.28$ | $79.15$ | $67.6$
| CD | $24.60$ | $60.62$ | $46.22$ | $43.21$ | $75.17$ | $62.39$ | $50.09$ | $79.21$ | $67.56$
| BCD(k=2) | $24.52$ | $60.62$ | $46.18$ | $43.16$ | $75.50$ | $62.56$ | $50.25$ | $79.14$ | $67.59$
w4a16g128 | OWC | $24.63$ | $61.27$ | $46.61$ | $42.31$ | $74.76$ | $61.78$ | $50.49$ | $79.38$ | $67.82$
GPTQ | $25.40$ | $61.13$ | $46.84$ | $43.54$ | $75.02$ | $62.43$ | $50.63$ | $79.21$ | $67.78$
| CD | $25.18$ | $61.42$ | $46.92$ | $43.78$ | $74.93$ | $62.47$ | $50.73$ | $79.30$ | $67.87$
| BCD(k=2) | $24.95$ | $61.49$ | $46.87$ | $43.77$ | $75.04$ | $62.53$ | $50.84$ | $79.56$ | $68.07$
| OWC-CD | $24.59$ | $61.40$ | $46.67$ | $43.67$ | $75.31$ | $62.65$ | $50.65$ | $79.04$ | $67.69$
|  OWC-CD \+ CD | $25.12$ | $61.44$ | $46.91$ | $44.00$ | $75.27$ | $62.76$ | $50.62$ | $79.21$ | $67.78$
|  OWC-CD \+ BCD(k=2) | $25.00$ | $61.50$ | $46.90$ | $43.75$ | $75.20$ | $62.62$ | $50.64$ | $79.17$ | $67.76$

#### Results.

Table 1 presents the perplexity numbers for different quantization techniques
applied to FFN layers. It can be seen that both our CD and BCD methods have a
clear advantage over GPTQ, leading to lower perplexity scores for all models
and quantization levels. The difference is more pronounced for lower bit
quantization. This is further highlighted in Table 5, which focuses
specifically on INT2 quantization. Here, we see almost $10\%$ improvement in
perplexity over GPTQ. We also present (1-shot) downstream evals in Tables 2, 5
(for 0-shot results, and a detailed breakdown of results for each dataset,
please refer to Appendix D). In Table 5 we see $2$-$2.5\%$ relative
improvement in downstream evals over GPTQ. Furthermore, Table 2 demonstrates
that our methods consistently match or exceed GPTQ’s performance across all
the settings, with the most substantial improvements observed in low-bit
quantization, and smaller models. Finally, we also observe that coordinate
descent techniques are better at optimizing the layer-wise objective in
Equation (1), than GPTQ. For instance, the average objective value (relative
to all $0$’s solution) for the GPTQ solution for the $1^{\text{st}}$ feed-
forward layer is $0.164$, whereas for CD it is $0.158$, and for BCD(k=$2$) it
is $0.157$.

Next, observe that Table 5 also investigates the effect of multiple epochs of
CD and BCD, where each epoch corresponds to $d_{\text{in}}$ iterations. For
BCD with $k=2$, additional epochs enhance performance. However, for BCD with
$k=4$, multiple epochs appear to lead to overfitting, indicating a need for
stronger regularization.

FFN+Attention Quantization. In addition to FFN quantization, we also perform
FFN + attention weight quantization. However, the inputs to the attention
layer are often aligned in a handful of directions. Consequently, performing
quantization using such a data leads to a huge drop in performance, as the
algorithms would primarily focus on a few directions and ignore the rest. To
mitigate this, we clip the largest eigenvalues of the Hessian to ensure a more
balanced Hessian. This technique, reminiscent of the weight clipping in
OmniQuant (Shao et al., 2023), improves the performance of both GPTQ and
CDQuant111One could also rely on existing techniques such as AWQ and
SmoothQuant to reduce the effect of outliers. But in our experiments we
noticed that both AWQ and SmoothQuant performed poorly compared to the simple
eigenvalue clipping technique.. For instance, for PaLM2-Gecko, the perplexity
for w3a16-GPTQ improves from $34.872$ to $24.054$ with clipping, and for
w4a16-GPTQ, it improves from $10.966$ to $10.005$. Table 4 presents the
results from this experiment, where we run all the algorithms on the clipped
Hessian. We once again notice that in almost all the settings, our coordinate
descent algorithms outperform GPTQ.

Table 3: Perplexity numbers for GPTQ, CD, BCD for quantization of FFN layers
of PaLM2-Otter, with and without (w/o) AWQ.

Method | PaLM2-Otter
---|---
| w3a16 | w3a16g128
| with AWQ | w/o AWQ | with AWQ | w/o AWQ
OWC | $22.661$ | $17.928$ | $6.813$ | $8.342$
GPTQ | $7.381$ | $7.176$ | $6.519$ | $6.635$
CD | $7.244$ | $7.002$ | $6.475$ | $6.655$
BCD(k=2) | $\mathbf{7.211}$ | $\mathbf{6.979}$ | $\mathbf{6.474}$ | $\mathbf{6.545}$

AWQ. A major strength of our algorithm is its versatility. It seamlessly
replaces GPTQ in any quantization technique that relies on it. To illustrate
this, we focus on the AWQ algorithm. We demonstrate that our algorithm, when
layered on top of AWQ, surpasses the performance of GPTQ layered on top of
AWQ. Table 3 presents the result from this experiment. It can be seen that BCD
always outperforms GPTQ, for both per-channel and group quantization 222The
performance of AWQ on PaLM2-Bison is much worse than not performing AWQ.
Hence, in Table 3, we only present results for PaLM2-Otter..

#### Runtime.

The runtimes of various techniques for quantizing PaLM2-Otter and PaLM2-Bison
are presented in Table 6. To be precise, the table presents time taken to
quantize all the FFN, attention layers in the model, for per-channel
quantization. For group quantization runtimes, please refer to Appendix E. It
can be seen that CD and GPTQ have comparable runtimes. While BCD takes longer
than CD, the majority of its time is spent quantizing the $2^{nd}$ FFN layer,
where the quantization dimension is equivalent to the model’s hidden
dimension. So, in practice, to speed up the entire process, one could rely on
CD for quantizing FFN2 and use BCD for quantizing the rest of the weight
matrices.

Table 4: Table presents _pretraining perplexity_ numbers for GPTQ, CD, BCD for
INT3, INT4 quantization of both FFN and attention weights.

Config | Method | PaLM2-Gecko | PaLM2-Otter | PaLM2-Bison
---|---|---|---|---
w16a16 | - | 7.948 | 5.984 | 5.298
w3a16 | OWC | $41.751$ | $54.499$ | $6.446$
GPTQ | $24.054$ | $\mathbf{11.384}$ | $5.801$
| CD | $19.692$ | $11.542$ | $5.770$
| BCD(k=2) | $\mathbf{19.312}$ | $11.392$ | $\mathbf{5.762}$
w3a16g128 | OWC | $20.301$ | $9.785$ | $5.973$
GPTQ | $15.042$ | $7.501$ | $5.698$
| CD | $14.757$ | $7.454$ | $5.679$
| BCD(k=2) | $14.691$ | $\mathbf{7.438}$ | $5.673$
| OWC-CD | $16.268$ | $7.617$ | $5.746$
|  OWC-CD \+ CD | $14.285$ | $7.466$ | $5.675$
|  OWC-CD \+ BCD(k=2) | $\mathbf{14.242}$ | $7.458$ | $\mathbf{5.670}$
w4a16 | OWC | $10.683$ | $7.194$ | $5.508$
GPTQ | $10.005$ | $6.645$ | $5.425$
| CD | $9.991$ | $6.627$ | $5.415$
| BCD(k=2) | $\mathbf{9.965}$ | $\mathbf{6.621}$ | $\mathbf{5.413}$
w4a16g128 | OWC | $9.402$ | $6.415$ | $5.420$
GPTQ | $9.160$ | $6.241$ | $5.381$
| CD | $9.124$ | $6.226$ | $5.377$
| BCD(k=2) | $9.120$ | $\mathbf{6.225}$ | $5.376$
| OWC-CD | $9.241$ | $6.261$ | $5.387$
|  OWC-CD \+ CD | $9.094$ | $6.244$ | $5.375$
|  OWC-CD \+ BCD(k=2) | $\mathbf{9.093}$ | $6.242$ | $\mathbf{5.374}$

Table 5: Table presents the perplexity, and _downstream evaluations_ (1-shot)
for GPTQ, CD, BCD for INT2 quantization of FFN weights. _Gen_ , _Rank_ columns
correspond to generation and ranking tasks.

Method | Epochs | PaLM2-Otter | PaLM2-Bison
---|---|---|---
w2a16g128 |  | Perplexity | Generation | Rank | Avg. | Perplexity | Generation | Rank | Avg.
GPTQ | - | $10.816$ | $26.98$ | $67.14$ | $51.07$ | $7.230$ | $39.32$ | $74.19$ | $60.24$
CD | $1$ | $9.917$ | $29.87$ | $66.76$ | $52.00$ | $7.123$ | $38.73$ | $73.56$ | $59.63$
BCD(k=2) | $1$ | $9.822$ | $30.03$ | $66.82$ | $52.11$ | $7.094$ | $38.85$ | $73.99$ | $59.93$
$2$ | $9.760$ | $30.09$ | $\mathbf{67.45}$ | $\mathbf{52.51}$ | $\mathbf{7.088}$ | $\mathbf{39.35}$ | $74.10$ | $60.20$
| $3$ | $9.719$ | $\mathbf{30.19}$ | $67.09$ | $52.33$ | $7.094$ | $39.30$ | $\mathbf{74.24}$ | $\mathbf{60.26}$
BCD(k=4) | $1$ | $\mathbf{9.708}$ | $30.11$ | $67.09$ | $52.30$ | $7.099$ | $38.98$ | $74.20$ | $60.11$
$2$ | $9.752$ | $30.00$ | $66.70$ | $52.02$ | $7.099$ | $39.09$ | $74.08$ | $60.08$
| $3$ | $9.749$ | $30.07$ | $66.93$ | $52.19$ | $7.099$ | $39.05$ | $74.10$ | $60.08$

Table 6: Table presents the per-channel quantization runtime (in minutes) of
CD, BCD, GPTQ (each run for $d_{\text{in}}$ iterations) on PaLM2-Otter and
PaLM2-Bison.

Config | Method | PaLM2-Otter | PaLM2-Bison
---|---|---|---
|  |  | FFN runtime
---
(in minutes)
| Attn. runtime
---
(in minutes)
| FFN runtime
---
(in minutes)
| Attn. runtime
---
(in minutes)
w3a16 | GPTQ |  $0.90$m |  $0.72$m |  $6.42$m |  $0.63$m
CD |  $2.94$m |  $0.49$m |  $16.13$m |  $0.43$m
BCD(k=2) |  $14.03$m |  $2.08$m |  $82.13$m |  $1.81$m
w4a16 | GPTQ |  $0.90$m |  $0.64$m |  $6.60$m |  $0.56$m
CD |  $3.59$m |  $0.59$m |  $21.19$m |  $0.52$m
BCD(k=2) |  $34.85$m |  $4.72$m |  $235.14$m |  $4.13$m

##  5 Conclusion and Future Work

In this work, we developed a coordinate descent framework (CDQuant) for
quantization of LLMs. CDQuant is a simple and effective alternative to GPTQ,
that consistently outperformed it on PaLM2 models. The simplicity of our
algorithm makes it a seamless substitute for GPTQ in various algorithmic
contexts where GPTQ currently functions as a sub-routine. Our future work aims
to further improve the performance of CDQuant. In particular, we aim to speed
up our BCD algorithm, and make it as fast as CD. Furthermore, we will focus on
developing layer-wise loss functions that are more closely aligned with end-
to-end loss, thereby reducing the performance gap between full-precision and
quantized models. Finally, we will also explore integrating CDQuant with QuIP
and FrameQuant to evaluate potential performance gains for extreme low-bit
quantization.

## Acknowledgements

We are grateful to Kyuyeun Kim, Wonpyo Park, and Jaehong Kim for assisting in
setting up the calibration pipelines, and Prateek Jain for helpful
discussions, support, and feedback.

## References

  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  Technical report, 2023.  URL https://cdn.openai.com/papers/gpt-4.pdf.
  * Google [2023] Gemini Team Google.  Gemini: A family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Miao et al. [2023] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia.  Towards efficient generative large language model serving: A survey from algorithms to systems.  _arXiv preprint arXiv:2312.15234_ , 2023.
  * Frantar et al. [2022] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_ , 2022.
  * Dettmers et al. [2023] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.  Spqr: A sparse-quantized representation for near-lossless llm weight compression.  _arXiv preprint arXiv:2306.03078_ , 2023.
  * Lee et al. [2023] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park.  Owq: Lessons learned from activation outliers for weight quantization in large language models.  _arXiv preprint arXiv:2306.02272_ , 2023.
  * Chee et al. [2024] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa.  Quip: 2-bit quantization of large language models with guarantees.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Adepu et al. [2024] Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, and Vikas Singh.  Framequant: Flexible low-bit quantization for transformers.  _arXiv preprint arXiv:2403.06082_ , 2024.
  * Lin et al. [2023] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_ , 2023.
  * Xiao et al. [2023] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pages 38087–38099. PMLR, 2023.
  * Dettmers et al. [2021] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.  8-bit optimizers via block-wise quantization.  _arXiv preprint arXiv:2110.02861_ , 2021.
  * Dettmers and Zettlemoyer [2023] Tim Dettmers and Luke Zettlemoyer.  The case for 4-bit precision: k-bit inference scaling laws.  In _International Conference on Machine Learning_ , pages 7750–7774. PMLR, 2023.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Frantar and Alistarh [2022] Elias Frantar and Dan Alistarh.  Optimal brain compression: A framework for accurate post-training quantization and pruning.  _Advances in Neural Information Processing Systems_ , 35:4475–4488, 2022.
  * LeCun et al. [1989] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  _Advances in neural information processing systems_ , 2, 1989.
  * Ma et al. [2024] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji.  Affinequant: Affine transformation quantization for large language models.  _arXiv preprint arXiv:2403.12544_ , 2024.
  * Behdin et al. [2023] Kayhan Behdin, Ayan Acharya, Aman Gupta, Qingquan Song, Siyu Zhu, Sathiya Keerthi, and Rahul Mazumder.  Quantease: Optimization-based quantization for language models.  _arXiv preprint arXiv:2309.01885_ , 2023.
  * Wei et al. [2023] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu.  Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.  _arXiv preprint arXiv:2304.09145_ , 2023.
  * Wei et al. [2022] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu.  Outlier suppression: Pushing the limit of low-bit transformer language models.  _Advances in Neural Information Processing Systems_ , 35:17402–17414, 2022.
  * Shao et al. [2023] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo.  Omniquant: Omnidirectionally calibrated quantization for large language models.  _arXiv preprint arXiv:2308.13137_ , 2023.
  * Liu et al. [2023] Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang.  Qllm: Accurate and efficient low-bitwidth quantization for large language models.  _arXiv preprint arXiv:2310.08041_ , 2023.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.  _Advances in Neural Information Processing Systems_ , 35:30318–30332, 2022.
  * Chai et al. [2023] Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu-Yeon Wei.  Int2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation.  _arXiv preprint arXiv:2306.08162_ , 2023.
  * Dettmers et al. [2024] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Chrétien and Corset [2009] Stéphane Chrétien and Franck Corset.  Using the eigenvalue relaxation for binary least-squares estimation problems.  _Signal processing_ , 89(11):2079–2091, 2009.
  * Park and Boyd [2018] Jaehyun Park and Stephen Boyd.  A semidefinite programming method for integer convex quadratic minimization.  _Optimization Letters_ , 12:499–518, 2018.
  * Nagel et al. [2020] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In _International Conference on Machine Learning_ , pages 7197–7206. PMLR, 2020.
  * Li et al. [2021] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  Brecq: Pushing the limit of post-training quantization by block reconstruction.  _arXiv preprint arXiv:2102.05426_ , 2021.
  * Hubara et al. [2021] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.  Accurate post training quantization with small calibration sets.  In _International Conference on Machine Learning_ , pages 4466–4475. PMLR, 2021.
  * Samaga et al. [2024] Yashas Samaga, Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, and Praneeth Netrapalli.  Hire: High recall approximate top-k estimation for efficient llm inference.  _ArXiv_ , abs/2402.09360, 2024.  URL https://api.semanticscholar.org/CorpusID:267657774.
  * Marcus et al. [1994] Mitchell P. Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger.  The penn treebank: Annotating predicate argument structure.  In _Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8-11, 1994_. Morgan Kaufmann, 1994.  URL https://aclanthology.org/H94-1020/.
  * Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Byj72udxe.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In Regina Barzilay and Min-Yen Kan, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 1601–1611. Association for Computational Linguistics, 2017.  doi: 10.18653/V1/P17-1147.  URL https://doi.org/10.18653/v1/P17-1147.
  * Rajpurkar et al. [2018] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 784–789. Association for Computational Linguistics, 2018.  doi: 10.18653/V1/P18-2124.  URL https://aclanthology.org/P18-2124/.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: a benchmark for question answering research.  _Trans. Assoc. Comput. Linguistics_ , 7:452–466, 2019.  doi: 10.1162/TACL\\_A\\_00276.  URL https://doi.org/10.1162/tacl_a_00276.
  * Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1533–1544. ACL, 2013.  URL https://aclanthology.org/D13-1160/.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457, 2018.  URL http://arxiv.org/abs/1803.05457.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4791–4800. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/P19-1472.  URL https://doi.org/10.18653/v1/p19-1472.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2924–2936. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/N19-1300.  URL https://doi.org/10.18653/v1/n19-1300.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  PIQA: reasoning about physical commonsense in natural language.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7432–7439. AAAI Press, 2020.  doi: 10.1609/AAAI.V34I05.6239.  URL https://doi.org/10.1609/aaai.v34i05.6239.
  * Sakaguchi et al. [2020] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8732–8740. AAAI Press, 2020.  doi: 10.1609/AAAI.V34I05.6399.  URL https://doi.org/10.1609/aaai.v34i05.6399.

##  Appendix A Broader Impact

We introduce a Coordinate Descent based approach, CDQuant, for compressing
Large Language Models. Our method uses only a small amount of data for
calibration. We do not foresee any ethical implications arising from the
technical aspects of our approach. However, compressing LLMs may give rise to
bias effects, a study of which seems essential given the extensive use of
LLMs. Our work may be of assistance to such studies. Also, since quantization
allows for easier deployment of LLMs, it could have potential societal
implications which seem difficult to predict.

##  Appendix B Limitations

  * •

While both CD and BCD outperformed GPTQ in our experiments, BCD achieves
slightly better performance than CD. However, BCD is not as fast as CD and can
be expensive for large models. In future, we aim to develop techniques to
speed up BCD.

  * •

Our algorithms still don’t bridge the gap between QAT and PTQ, especially on
smaller models. To bridge this gap, we believe one should move away from the
$\ell_{2}^{2}$ surrogate loss that is being considered by most of the existing
work. Instead, we should design surrogate losses that are more closely aligned
with end-to-end loss.

  * •

Due to computational reasons, we couldn’t run experiments where we replace
GPTQ component with CDQuant in algorithms such as QuIP, FrameQuant.

##  Appendix C CDQuant for sub-channel quantization

To simplify the explanation in this section, we introduce a slightly modified
notation. Given a weight vector $\mathbf{w}$, we represent it’s sub-channel
quantization as $\mathbf{a}\odot\mathbf{q}+\mathbf{b}$, where $\odot$ is the
elementwise multiplication, and
$\mathbf{a},\mathbf{b}\in\mathbb{R}^{d_{\text{in}}}$ are the scale, and bias
parameters that satisfy the following constraints:
$\mathbf{a}_{k}=\mathbf{a}_{l},\mathbf{b}_{k}=\mathbf{b}_{l}$, for any two
indices $k,l$ that fall in the same group. With this notation, for any given
$\mathbf{a},\mathbf{b}$, the optimization problem in Equation (3) can be
rewritten as

$\displaystyle\min_{\mathbf{q}}\|X(\mathbf{w}-\mathbf{a}\odot\mathbf{q}-\mathbf{b})\|_{2}^{2}.$
(5)

Letting
$D_{\mathbf{a}}=\mbox{diag}(\mathbf{a}),\tilde{X}=XD_{\mathbf{a}},\tilde{\mathbf{w}}=D_{\mathbf{a}}^{-1}\mathbf{w},\tilde{\mathbf{b}}=D_{\mathbf{a}}^{-1}\mathbf{b}$,
the above problem can be further rewritten as

$\displaystyle\min_{\mathbf{q}}\|\tilde{X}(\tilde{\mathbf{w}}-\mathbf{q}-\tilde{\mathbf{b}})\|_{2}^{2}.$
(6)

Observe that this problem is the same as the per-channel quantization problem
described in Equation (1), but with modified parameters
$\tilde{X},\tilde{\mathbf{w}},\tilde{\mathbf{b}}$. So extending CDQuant to
sub-channel quantization simply involves running Algorithms 1, 2 with these
modified parameters. Algorithms 4, 5 present the resulting algorithms.

Algorithm 4 Greedy Coordinate Descent (CD)

1:Input: $T$ \- coordinate descent steps, $X$ \- input data matrix,
$\mathbf{w}$ \- vector to be quantized, $a$ \- scale, $b$ \- bias,
$\mathbf{q}_{0}$ \- initial estimate

2:$\tilde{X}\leftarrow
X\mbox{diag}(\mathbf{a}),\tilde{\mathbf{w}}=\mbox{diag}(\mathbf{a})^{-1}\mathbf{w},\tilde{\mathbf{b}}=\mbox{diag}(\mathbf{a})\mathbf{b}$

3:Compute Hessian $H$ as: $H\leftarrow\tilde{X}^{T}\tilde{X}$

4:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow
2H(\mathbf{q}_{0}-(\tilde{\mathbf{w}}-\tilde{\mathbf{b}}))$

5:for $t\in[1:T]$ do

6: Find the coordinate that leads to the largest reduction in loss
$i^{*},r^{*}=\operatorname*{arg\,min}_{i\in\\{0,1,\dots
d_{\text{in}}-1\\},r\in\\{0,1,\dots
2^{c}-1\\}}(r-\mathbf{q}_{t-1,i})^{2}H_{i,i}+(r-\mathbf{q}_{t-1,i})\mathbf{g}_{i}$

7: Update gradient $\mathbf{g}$ as
$\mathbf{g}\leftarrow\mathbf{g}+2(r^{*}-\mathbf{q}_{t-1,i^{*}})H_{i^{*},\cdot},$
where $H_{i^{*},\cdot}$ is the $i^{*}$ column of $H$

8: Update $\mathbf{q}_{t-1}$ as
$\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1}+(r^{*}-\mathbf{q}_{t-1,i^{*}})\mathbf{e}_{i^{*}},$
where $\mathbf{e}_{i^{*}}$ is the standard basis vector with $1$ in $i^{*}$
position and $0$ everywhere else

9:end for

Algorithm 5 Block Coordinate Descent with Random Blocks (BCD)

1:Input: $T$ \- coordinate descent steps, $k$ \- block size, $X$ \- input data
matrix, $\mathbf{w}$ \- vector to be quantized, $a$ \- scale, $b$ \- bias,
$\mathbf{q}_{0}$ \- initial estimate

2:$\tilde{X}\leftarrow
X\mbox{diag}(\mathbf{a}),\tilde{\mathbf{w}}=\mbox{diag}(\mathbf{a})^{-1}\mathbf{w},\tilde{\mathbf{b}}=\mbox{diag}(\mathbf{a})\mathbf{b}$

3:Compute Hessian $H$ as: $H\leftarrow\tilde{X}^{T}\tilde{X}$

4:Compute gradient $\mathbf{g}$ as: $\mathbf{g}\leftarrow
2H(\mathbf{q}_{0}-(\tilde{\mathbf{w}}-\tilde{\mathbf{b}}))$

5:for $t\in[1:T]$ do

6: Randomly partition the set $\\{0,1,\dots d_{\text{in}}-1\\}$ into
$d_{\text{in}}/k$ blocks, each of size $k$

7: Find the block that leads to the largest reduction in loss
$i^{*},r^{*}=\operatorname*{arg\,min}_{\begin{subarray}{c}i\in\\{0,1,\dots
d_{\text{in}}/k-1\\},\\\ r\in\\{0,1,\dots
2^{c}-1\\}^{k}\end{subarray}}(r-\mathbf{q}_{t-1,i})^{T}H_{i,i}(r-\mathbf{q}_{t-1,i})+(r-\mathbf{q}_{t-1,i})^{T}\mathbf{g}_{i},$
where $\mathbf{q}_{t-1,i},H_{i,i}$ are the sub-vector, sub-matrix of
$\mathbf{q}_{t-1},H$ corresponding to block $i$.

8: Update gradient $\mathbf{g}$ as
$\mathbf{g}\leftarrow\mathbf{g}+2H_{i^{*},\cdot}(r^{*}-\mathbf{q}_{t-1,i^{*}}),$

9: Update $\mathbf{q}_{t-1}$ as  $\mathbf{q}_{t-1,i^{*}}\leftarrow
r^{*},\quad\mathbf{q}_{t}\leftarrow\mathbf{q}_{t-1},$

10:end for

##  Appendix D Detailed Downstream Evaluation Results

Table 7: Table presents _downstream evaluation_ (one-shot) numbers of
PaLM2-Bison - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Bison
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $27.84$ | $75.19$ | $77.42$ | $23.87$ | $59.04$ | $84.64$ | $88.87$ | $82.50$ | $82.43$ | $78.22$
w3a16 | OWC | $22.16$ | $75.84$ | $70.31$ | $21.75$ | $57.34$ | $83.50$ | $86.94$ | $79.60$ | $81.12$ | $78.06$
GPTQ | $24.29$ | $74.65$ | $72.29$ | $22.98$ | $58.70$ | $85.02$ | $87.58$ | $80.81$ | $81.72$ | $77.66$
| CD | $23.60$ | $74.92$ | $72.25$ | $23.13$ | $58.53$ | $84.68$ | $88.20$ | $80.60$ | $81.61$ | $77.74$
| BCD(k=2) | $23.24$ | $74.78$ | $72.06$ | $22.24$ | $58.62$ | $84.55$ | $88.04$ | $80.69$ | $81.83$ | $78.22$
w3a16g128 | OWC | $23.88$ | $73.59$ | $73.31$ | $21.56$ | $57.85$ | $83.84$ | $88.23$ | $80.67$ | $80.79$ | $78.69$
GPTQ | $25.32$ | $73.96$ | $74.10$ | $23.23$ | $58.02$ | $84.39$ | $88.50$ | $80.67$ | $81.83$ | $78.06$
| CD | $25.24$ | $74.16$ | $74.11$ | $23.18$ | $57.51$ | $83.75$ | $88.38$ | $81.15$ | $82.15$ | $77.98$
| BCD(k=2) | $24.93$ | $74.05$ | $73.90$ | $23.18$ | $56.74$ | $84.05$ | $88.26$ | $81.06$ | $82.43$ | $78.53$
| OWC-CD | $24.60$ | $74.03$ | $73.80$ | $21.75$ | $57.17$ | $84.18$ | $88.2$ | $80.97$ | $81.94$ | $77.27$
|  OWC-CD \+ CD | $25.12$ | $74.37$ | $73.99$ | $23.38$ | $58.36$ | $85.44$ | $88.65$ | $80.93$ | $81.77$ | $77.98$
|  OWC-CD \+ BCD(k=2) | $25.32$ | $73.54$ | $74.30$ | $23.18$ | $57.34$ | $83.96$ | $88.59$ | $81.01$ | $81.39$ | $77.51$
w4a16 | OWC | $26.57$ | $74.66$ | $75.84$ | $23.13$ | $58.28$ | $83.42$ | $88.75$ | $81.39$ | $81.66$ | $78.85$
GPTQ | $27.06$ | $74.87$ | $76.45$ | $22.74$ | $58.87$ | $84.60$ | $88.53$ | $81.72$ | $82.48$ | $78.69$
| CD | $26.79$ | $74.40$ | $76.35$ | $22.83$ | $59.22$ | $84.68$ | $88.99$ | $81.60$ | $82.10$ | $78.69$
| BCD(k=2) | $27.06$ | $74.54$ | $76.28$ | $23.13$ | $59.81$ | $84.76$ | $88.56$ | $81.70$ | $82.26$ | $77.74$
w4a16g128 | OWC | $27.04$ | $75.37$ | $76.42$ | $23.13$ | $59.73$ | $84.97$ | $88.87$ | $81.94$ | $82.32$ | $78.45$
GPTQ | $27.12$ | $75.63$ | $76.42$ | $23.38$ | $58.45$ | $84.76$ | $89.05$ | $82.01$ | $82.15$ | $78.85$
| CD | $27.31$ | $75.52$ | $76.45$ | $23.62$ | $59.04$ | $85.02$ | $88.50$ | $82.04$ | $82.54$ | $78.69$
| BCD(k=2) | $27.37$ | $75.44$ | $76.92$ | $23.62$ | $58.87$ | $85.35$ | $88.72$ | $81.98$ | $82.97$ | $79.48$
| OWC-CD | $27.31$ | $75.70$ | $76.64$ | $22.93$ | $58.36$ | $84.64$ | $88.56$ | $81.98$ | $82.59$ | $78.14$
|  OWC-CD \+ CD | $27.01$ | $75.52$ | $76.69$ | $23.28$ | $58.79$ | $85.10$ | $88.56$ | $81.98$ | $82.15$ | $78.69$
|  OWC-CD \+ BCD(k=2) | $26.98$ | $75.42$ | $76.84$ | $23.33$ | $58.79$ | $85.35$ | $88.47$ | $81.93$ | $82.32$ | $78.14$

Table 8: Table presents _downstream evaluation_ (one-shot) numbers of
PaLM2-Otter - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Otter
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $19.78$ | $70.3$ | $69.1$ | $18.06$ | $54.1$ | $80.13$ | $85.47$ | $79.51$ | $79.43$ | $73.95$
w3a16 | OWC | $5.96$ | $50.17$ | $24.8$ | $3.15$ | $45.99$ | $66.84$ | $74.83$ | $66.98$ | $72.47$ | $67.56$
GPTQ | $13.77$ | $66.12$ | $56.15$ | $12.99$ | $52.56$ | $78.58$ | $85.17$ | $76.75$ | $78.35$ | $72.3$
| CD | $14.49$ | $65.71$ | $58.83$ | $13.44$ | $52.47$ | $79.92$ | $83.03$ | $75.81$ | $77.09$ | $71.9$
| BCD(k=2) | $14.63$ | $65.96$ | $59.46$ | $14.07$ | $52.73$ | $79.84$ | $83.09$ | $75.93$ | $77.42$ | $71.74$
w3a16g128 | OWC | $10.89$ | $67.98$ | $45.03$ | $8.81$ | $48.29$ | $74.83$ | $79.14$ | $74.64$ | $75.52$ | $72.45$
GPTQ | $16.12$ | $69.8$ | $60.82$ | $14.47$ | $53.16$ | $79.34$ | $84.07$ | $77.12$ | $77.86$ | $73.16$
| CD | $15.46$ | $71.62$ | $62.07$ | $16.19$ | $52.3$ | $78.79$ | $84.34$ | $77.02$ | $78.35$ | $74.82$
| BCD(k=2) | $16.04$ | $71.04$ | $62.02$ | $15.65$ | $51.71$ | $78.32$ | $84.83$ | $77.04$ | $77.69$ | $74.03$
| OWC-CD | $15.62$ | $71.78$ | $61.19$ | $14.76$ | $52.99$ | $79.34$ | $84.1$ | $76.92$ | $77.42$ | $75.22$
|  OWC-CD \+ CD | $16.2$ | $70.77$ | $61.8$ | $14.71$ | $53.41$ | $79.42$ | $84.5$ | $77.22$ | $78.29$ | $73.64$
|  OWC-CD \+ BCD(k=2) | $16.7$ | $71.3$ | $61.87$ | $14.91$ | $53.84$ | $79.8$ | $84.37$ | $77.25$ | $78.45$ | $74.19$
w4a16 | OWC | $16.81$ | $71.7$ | $61.75$ | $16.54$ | $53.33$ | $78.24$ | $85.17$ | $77.32$ | $79.33$ | $71.9$
GPTQ | $19$ | $70.66$ | $66.01$ | $16.34$ | $53.67$ | $79.46$ | $85.57$ | $78.5$ | $79.05$ | $72.93$
| CD | $18.7$ | $70.97$ | $66.11$ | $17.08$ | $54.27$ | $80.89$ | $85.87$ | $78.55$ | $79.05$ | $72.38$
| BCD(k=2) | $19.14$ | $70.9$ | $65.72$ | $16.88$ | $54.78$ | $80.6$ | $85.6$ | $78.64$ | $79.65$ | $73.72$
w4a16g128 | OWC | $18.03$ | $69.97$ | $65.28$ | $15.94$ | $53.41$ | $79.42$ | $84.89$ | $78.1$ | $79.16$ | $73.56$
GPTQ | $19.17$ | $70.34$ | $67.46$ | $17.18$ | $53.58$ | $79.88$ | $85.02$ | $79$ | $79.11$ | $73.56$
| CD | $19.25$ | $71.12$ | $67.61$ | $17.13$ | $53.33$ | $80.18$ | $84.92$ | $79.09$ | $79.11$ | $72.93$
| BCD(k=2) | $19.45$ | $70.81$ | $67.75$ | $17.08$ | $54.27$ | $80.3$ | $84.74$ | $79.05$ | $78.94$ | $72.93$
| OWC-CD | $19.2$ | $70.58$ | $67.66$ | $17.22$ | $53.24$ | $80.51$ | $85.05$ | $78.84$ | $79.49$ | $74.74$
|  OWC-CD \+ CD | $19.64$ | $70.8$ | $68.02$ | $17.52$ | $53.75$ | $80.77$ | $84.98$ | $79.09$ | $78.78$ | $74.27$
|  OWC-CD \+ BCD(k=2) | $19.31$ | $70.41$ | $67.91$ | $17.37$ | $53.67$ | $80.3$ | $84.95$ | $79.07$ | $78.78$ | $74.43$

Table 9: Table presents _downstream evaluation_ (one-shot) numbers of
PaLM2-Gecko - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Gecko
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $7.67$ | $49.04$ | $37.72$ | $8.96$ | $37.12$ | $68.1$ | $66.15$ | $61.55$ | $73.99$ | $61.72$
w3a16 | OWC | $3.71$ | $48.17$ | $18.84$ | $5.07$ | $30.03$ | $60.1$ | $59.69$ | $53.45$ | $70.24$ | $60.93$
GPTQ | $4.38$ | $46.91$ | $20.43$ | $5.95$ | $31.23$ | $62.84$ | $63.55$ | $56.51$ | $71.65$ | $60.46$
| CD | $4.63$ | $48.1$ | $22.79$ | $7.23$ | $32.59$ | $63.51$ | $62.14$ | $57.6$ | $72.09$ | $60.38$
| BCD(k=2) | $4.52$ | $48.84$ | $22.09$ | $6.64$ | $32.51$ | $64.14$ | $63.24$ | $57.53$ | $72.2$ | $61.01$
w3a16g128 | OWC | $4.43$ | $57.48$ | $20.99$ | $8.42$ | $32.59$ | $63.09$ | $62.05$ | $53.45$ | $71.38$ | $59.59$
GPTQ | $4.85$ | $53.14$ | $23.25$ | $7.73$ | $33.11$ | $64.73$ | $64.43$ | $56.19$ | $72.42$ | $61.09$
| CD | $4.49$ | $54.43$ | $24.12$ | $8.12$ | $33.28$ | $65.15$ | $64.22$ | $56.56$ | $71.98$ | $61.64$
| BCD(k=2) | $4.29$ | $53.58$ | $24.41$ | $8.27$ | $33.7$ | $65.32$ | $64.25$ | $56.25$ | $71.71$ | $61.88$
| OWC-CD | $4.6$ | $53.86$ | $23.23$ | $7.14$ | $32.85$ | $64.81$ | $64.65$ | $56.31$ | $71.82$ | $61.72$
|  OWC-CD \+ CD | $4.52$ | $52.4$ | $25.07$ | $7.78$ | $32.76$ | $64.44$ | $65.11$ | $56.85$ | $71.93$ | $61.96$
|  OWC-CD \+ BCD(k=2) | $4.65$ | $51.15$ | $25.13$ | $7.68$ | $33.19$ | $64.9$ | $64.68$ | $56.88$ | $71.82$ | $61.17$
w4a16 | OWC | $6.09$ | $48.4$ | $31.06$ | $7.53$ | $35.24$ | $65.07$ | $64.59$ | $59.59$ | $72.96$ | $63.85$
GPTQ | $6.81$ | $50.67$ | $32.5$ | $8.07$ | $36.43$ | $65.53$ | $65.05$ | $60.16$ | $73.29$ | $63.85$
| CD | $6.81$ | $51.07$ | $31.87$ | $8.66$ | $35.58$ | $66.92$ | $64.65$ | $60.22$ | $73.23$ | $63.14$
| BCD(k=2) | $6.65$ | $50.86$ | $31.67$ | $8.91$ | $35.15$ | $67.21$ | $64.4$ | $60.31$ | $73.45$ | $63.22$
w4a16g128 | OWC | $7.12$ | $47.85$ | $33.58$ | $9.99$ | $35.58$ | $67.09$ | $68.38$ | $60.15$ | $73.5$ | $62.9$
GPTQ | $7.2$ | $51.6$ | $33.94$ | $8.86$ | $36.18$ | $67.21$ | $66.51$ | $59.99$ | $73.5$ | $63.38$
| CD | $7.59$ | $48.87$ | $35.22$ | $9.06$ | $35.92$ | $68.01$ | $67.09$ | $60.49$ | $73.45$ | $63.54$
| BCD(k=2) | $7.29$ | $48.72$ | $34.67$ | $9.1$ | $36.69$ | $68.01$ | $67.28$ | $60.7$ | $73.34$ | $62.9$
| OWC-CD | $7.31$ | $48.52$ | $33.2$ | $9.3$ | $35.75$ | $67.59$ | $66.97$ | $60.8$ | $74.21$ | $63.06$
|  OWC-CD \+ CD | $7.15$ | $49.15$ | $34.77$ | $9.4$ | $36.52$ | $67.8$ | $66.33$ | $60.74$ | $73.61$ | $63.61$
|  OWC-CD \+ BCD(k=2) | $7.29$ | $48.88$ | $34.68$ | $9.15$ | $36.43$ | $68.01$ | $65.75$ | $60.77$ | $74.05$ | $64.01$

Table 10: Table presents _downstream evaluation_ (one-shot) numbers of
PaLM2-Otter - for GPTQ, CD, BCD for INT2 quantization of FFN weights.

Method | Epochs | PaLM2-Otter
---|---|---
w2a16g128 |  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
GPTQ | - | $6.93$ | $64.89$ | $28.36$ | $7.73$ | $44.28$ | $71.04$ | $78.81$ | $68.29$ | $72.69$ | $67.72$
CD | $1$ | $8.23$ | $68.12$ | $32.98$ | $10.14$ | $43.69$ | $72.43$ | $76.91$ | $68.11$ | $72.96$ | $66.46$
BCD(k=2) | $1$ | $8.09$ | $69.2$ | $32.85$ | $9.99$ | $43.43$ | $73.02$ | $77$ | $68.03$ | $73.07$ | $66.38$
$2$ | $8.2$ | $69.43$ | $32.98$ | $9.74$ | $43.52$ | $73.7$ | $77.03$ | $68.56$ | $74.1$ | $67.8$
| $3$ | $8.31$ | $69.79$ | $33.03$ | $9.65$ | $43.52$ | $73.11$ | $77.09$ | $68.76$ | $73.07$ | $67.01$
BCD(k=4) | $1$ | $7.87$ | $69.52$ | $33.24$ | $9.79$ | $43.17$ | $72.6$ | $77.83$ | $68.61$ | $73.88$ | $66.46$
$2$ | $8.31$ | $69.06$ | $33.03$ | $9.6$ | $43$ | $72.6$ | $76.64$ | $68.51$ | $73.18$ | $66.3$
| $3$ | $8.31$ | $69.8$ | $32.69$ | $9.5$ | $42.15$ | $72.43$ | $77.06$ | $68.34$ | $73.72$ | $67.88$

Table 11: Table presents _downstream evaluation_ (one-shot) numbers of
PaLM2-Bison - for GPTQ, CD, BCD for INT2 quantization of FFN weights.

Method | Epochs | PaLM2-Bison
---|---|---
w2a16g128 |  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
GPTQ | - | $15.26$ | $69.34$ | $55.6$ | $17.08$ | $51.02$ | $80.13$ | $86.02$ | $74.44$ | $78.78$ | $74.74$
CD | $1$ | $15.6$ | $67.43$ | $55.06$ | $16.83$ | $50.43$ | $79.97$ | $83.76$ | $74.35$ | $78.35$ | $74.51$
BCD(k=2) | $1$ | $15.62$ | $67.32$ | $55.82$ | $16.63$ | $51.45$ | $79.97$ | $83.61$ | $74.46$ | $79$ | $75.45$
$2$ | $15.62$ | $68.45$ | $56.36$ | $16.98$ | $51.02$ | $80.09$ | $84.56$ | $74.99$ | $78.89$ | $75.06$
| $3$ | $16.01$ | $67.99$ | $56.31$ | $16.88$ | $51.19$ | $79.88$ | $84.65$ | $74.95$ | $79.33$ | $75.45$
BCD(k=4) | $1$ | $15.9$ | $67.46$ | $56.12$ | $16.44$ | $51.28$ | $80.35$ | $84.01$ | $74.89$ | $78.84$ | $75.85$
$2$ | $16.04$ | $67.61$ | $56.34$ | $16.39$ | $51.54$ | $80.09$ | $83.91$ | $74.83$ | $78.73$ | $75.37$
| $3$ | $16.12$ | $67.54$ | $56.3$ | $16.24$ | $51.02$ | $80.22$ | $83.98$ | $74.93$ | $78.51$ | $75.93$

Table 12: Table presents _downstream evaluation_ (zero-shot) numbers for GPTQ,
CD, BCD for INT3, INT4 quantization of FFN weights. _Gen_ , _Rank_ columns
correspond to generation and ranking tasks.

Config | Method | PaLM2-Gecko | PaLM2-Otter | PaLM2-Bison
---|---|---|---|---
|  | Gen. | Rank | Avg. | Gen. | Rank | Avg. | Gen. | Rank | Avg.
w16a16 | - | $25.85$ | $61.44$ | $47.2$ | $44.31$ | $75.43$ | $62.98$ | $51.08$ | $78.28$ | $68.00$
w3a16 | OWC | $15.45$ | $55.73$ | $39.62$ | $15.06$ | $59.29$ | $41.60$ | $39.58$ | $75.45$ | $61.10$
GPTQ | $13.08$ | $56.92$ | $39.39$ | $31.80$ | $71.94$ | $55.88$ | $41.42$ | $76.70$ | $62.58$
| CD | $15.82$ | $58.19$ | $41.24$ | $32.34$ | $72.03$ | $56.16$ | $41.66$ | $76.71$ | $62.69$
| BCD(k=2) | $16.16$ | $57.85$ | $41.17$ | $32.04$ | $72.22$ | $56.15$ | $41.37$ | $76.78$ | $62.62$
w3a16g128 | OWC | $18.54$ | $56.95$ | $41.59$ | $26.62$ | $68.09$ | $51.50$ | $41.54$ | $76.51$ | $62.52$
GPTQ | $17.96$ | $58.30$ | $42.16$ | $33.43$ | $72.98$ | $57.16$ | $43.47$ | $76.65$ | $63.38$
| CD | $16.81$ | $58.10$ | $41.58$ | $32.73$ | $72.62$ | $56.66$ | $42.95$ | $77.03$ | $63.4$
| BCD(k=2) | $16.33$ | $57.81$ | $41.22$ | $32.13$ | $72.67$ | $56.46$ | $42.67$ | $76.85$ | $63.18$
| OWC-CD | $16.73$ | $57.74$ | $41.34$ | $32.90$ | $72.40$ | $56.60$ | $42.23$ | $77.02$ | $63.11$
|  OWC-CD \+ CD | $16.37$ | $58.22$ | $41.48$ | $33.53$ | $72.64$ | $56.99$ | $43.09$ | $76.91$ | $63.38$
|  OWC-CD \+ BCD(k=2) | $16.47$ | $58.45$ | $41.66$ | $33.71$ | $72.73$ | $57.12$ | $43.07$ | $76.70$ | $63.25$
w4a16 | OWC | $16.07$ | $59.42$ | $42.08$ | $33.76$ | $72.22$ | $56.83$ | $43.59$ | $77.13$ | $63.71$
GPTQ | $18.77$ | $59.5$ | $43.21$ | $34.03$ | $73.53$ | $57.73$ | $44.16$ | $77.54$ | $64.19$
| CD | $18.20$ | $59.95$ | $43.25$ | $35.01$ | $73.42$ | $58.06$ | $43.97$ | $77.55$ | $64.12$
| BCD(k=2) | $17.89$ | $59.64$ | $42.94$ | $35.10$ | $73.31$ | $58.02$ | $43.95$ | $77.77$ | $64.24$
w4a16g128 | OWC | $19.00$ | $60.42$ | $43.85$ | $34.52$ | $72.83$ | $57.5$ | $44.45$ | $77.71$ | $64.41$
GPTQ | $19.07$ | $60.40$ | $43.87$ | $35.34$ | $73.60$ | $58.29$ | $44.25$ | $77.65$ | $64.29$
| CD | $20.35$ | $60.94$ | $44.70$ | $36.20$ | $73.67$ | $58.68$ | $44.21$ | $77.72$ | $64.31$
| BCD(k=2) | $20.44$ | $60.69$ | $44.59$ | $36.11$ | $73.54$ | $58.57$ | $44.17$ | $77.56$ | $64.2$
| OWC-CD | $19.80$ | $60.39$ | $44.15$ | $36.33$ | $73.62$ | $58.71$ | $44.75$ | $77.70$ | $64.52$
|  OWC-CD \+ CD | $20.47$ | $60.85$ | $44.70$ | $35.98$ | $73.59$ | $58.54$ | $44.77$ | $77.52$ | $64.42$
|  OWC-CD \+ BCD(k=2) | $20.15$ | $60.49$ | $44.36$ | $36.00$ | $73.62$ | $58.57$ | $44.70$ | $77.70$ | $64.50$

Table 13: Table presents _downstream evaluation_ (zero-shot) numbers of
PaLM2-Bison - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Bison
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$
w3a16 | OWC | $15.87$ | $73.01$ | $58.86$ | $10.58$ | $52.47$ | $76.43$ | $86.91$ | $79.76$ | $81.01$ | $76.09$
GPTQ | $17.31$ | $71.25$ | $65.53$ | $11.56$ | $54.44$ | $79.25$ | $87.22$ | $80.78$ | $81.61$ | $76.87$
| CD | $17.84$ | $71.24$ | $64.76$ | $12.8$ | $54.44$ | $79.17$ | $87.58$ | $80.79$ | $82.1$ | $76.16$
| BCD(k=2) | $17.81$ | $71.3$ | $64.29$ | $12.06$ | $53.5$ | $79.08$ | $87.65$ | $80.91$ | $82.21$ | $77.35$
w3a16g128 | OWC | $15.98$ | $71.28$ | $68.02$ | $10.88$ | $53.58$ | $79.55$ | $86.73$ | $80.9$ | $81.61$ | $76.72$
GPTQ | $18.95$ | $71.49$ | $69.67$ | $13.78$ | $53.24$ | $79.84$ | $86.79$ | $81.05$ | $82.05$ | $76.95$
| CD | $18.31$ | $71.81$ | $68.56$ | $13.14$ | $53.41$ | $79.8$ | $87.55$ | $81.31$ | $82.43$ | $77.66$
| BCD(k=2) | $18.53$ | $71.55$ | $67.62$ | $12.99$ | $53.58$ | $79.59$ | $87.49$ | $81.14$ | $82.81$ | $76.48$
| OWC-CD | $17.26$ | $71.79$ | $67.53$ | $12.35$ | $54.69$ | $80.85$ | $85.72$ | $81.02$ | $81.88$ | $77.98$
|  OWC-CD \+ CD | $18.75$ | $72.05$ | $68.35$ | $13.19$ | $54.1$ | $80.18$ | $86.36$ | $80.86$ | $82.48$ | $77.51$
|  OWC-CD \+ BCD(k=2) | $19.36$ | $71.47$ | $68.45$ | $12.99$ | $53.58$ | $80.47$ | $86.02$ | $81.15$ | $82.26$ | $76.72$
w4a16 | OWC | $20$ | $72.51$ | $69.34$ | $12.5$ | $54.61$ | $80.39$ | $87.06$ | $81.5$ | $82.05$ | $77.19$
GPTQ | $21.25$ | $72.39$ | $70.6$ | $12.4$ | $55.2$ | $80.64$ | $87.58$ | $81.98$ | $82.26$ | $77.58$
| CD | $21.5$ | $71.57$ | $70.04$ | $12.8$ | $55.29$ | $80.56$ | $88.07$ | $81.78$ | $82.59$ | $77.03$
| BCD(k=2) | $21.16$ | $71.92$ | $70.01$ | $12.7$ | $55.63$ | $81.19$ | $87.77$ | $81.82$ | $82.81$ | $77.43$
w4a16g128 | OWC | $21.22$ | $73.06$ | $70.07$ | $13.44$ | $55.8$ | $80.47$ | $87.77$ | $82.08$ | $82.64$ | $77.51$
GPTQ | $20.72$ | $73.17$ | $70.12$ | $12.99$ | $55.38$ | $80.89$ | $87.55$ | $82.14$ | $82.59$ | $77.35$
| CD | $21.8$ | $73.37$ | $69.06$ | $12.6$ | $54.78$ | $80.72$ | $87.25$ | $82.16$ | $83.19$ | $78.22$
| BCD(k=2) | $21.5$ | $73.06$ | $69.34$ | $12.8$ | $55.29$ | $80.98$ | $87.31$ | $82.14$ | $82.54$ | $77.11$
| OWC-CD | $21.36$ | $73.85$ | $70.07$ | $13.73$ | $55.12$ | $81.23$ | $87.71$ | $82.27$ | $82.59$ | $77.27$
|  OWC-CD \+ CD | $21.41$ | $73.32$ | $71.46$ | $12.89$ | $54.18$ | $81.4$ | $87.28$ | $82.17$ | $82.37$ | $77.74$
|  OWC-CD \+ BCD(k=2) | $21.19$ | $73.24$ | $71.64$ | $12.75$ | $55.03$ | $81.52$ | $87.28$ | $82.06$ | $82.64$ | $77.66$

Table 14: Table presents _downstream evaluation_ (zero-shot) numbers of
PaLM2-Otter - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Otter
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$
w3a16 | OWC | $2.27$ | $49.45$ | $8.02$ | $0.49$ | $41.98$ | $60.14$ | $48.78$ | $66.91$ | $71.87$ | $66.06$
GPTQ | $9.09$ | $64.53$ | $46.38$ | $7.19$ | $49.83$ | $73.99$ | $81.96$ | $76.87$ | $78.89$ | $70.09$
| CD | $10.72$ | $64.01$ | $48.03$ | $6.59$ | $50.77$ | $76.6$ | $81.01$ | $75.77$ | $77.97$ | $70.09$
| BCD(k=2) | $10.55$ | $64.31$ | $47.25$ | $6.05$ | $50.85$ | $76.43$ | $81.41$ | $75.91$ | $78.45$ | $70.24$
w3a16g128 | OWC | $6.18$ | $66.37$ | $30.48$ | $3.44$ | $47.44$ | $70.16$ | $69.63$ | $74.61$ | $76.88$ | $69.85$
GPTQ | $10.17$ | $66.53$ | $49.29$ | $7.73$ | $51.37$ | $75.93$ | $82.35$ | $77$ | $79.11$ | $72.14$
| CD | $7.17$ | $68.73$ | $49.42$ | $5.61$ | $50.26$ | $76.18$ | $82.39$ | $76.79$ | $78.13$ | $71.98$
| BCD(k=2) | $6.68$ | $68.18$ | $48.39$ | $5.27$ | $50.26$ | $75.93$ | $81.41$ | $76.84$ | $79$ | $72.61$
| OWC-CD | $8.98$ | $68.37$ | $48.33$ | $5.91$ | $50.51$ | $75.46$ | $81.35$ | $76.83$ | $78.18$ | $72.06$
|  OWC-CD \+ CD | $9.72$ | $67.87$ | $49.87$ | $6.64$ | $51.37$ | $76.68$ | $81.59$ | $76.78$ | $77.91$ | $71.51$
|  OWC-CD \+ BCD(k=2) | $9.94$ | $68.26$ | $49.91$ | $6.74$ | $50.85$ | $76.22$ | $81.68$ | $76.99$ | $78.89$ | $71.74$
w4a16 | OWC | $11.19$ | $66.67$ | $49.56$ | $7.63$ | $50$ | $73.91$ | $80$ | $77.36$ | $79.11$ | $72.93$
GPTQ | $12.27$ | $66.65$ | $48.72$ | $8.46$ | $51.71$ | $75.84$ | $82.02$ | $78.2$ | $80.09$ | $73.32$
| CD | $12.52$ | $66.85$ | $53.16$ | $7.53$ | $52.3$ | $76.43$ | $81.28$ | $78.3$ | $79.6$ | $72.61$
| BCD(k=2) | $12.33$ | $67$ | $53.45$ | $7.63$ | $51.96$ | $75.97$ | $80.92$ | $78.47$ | $80.47$ | $72.06$
w4a16g128 | OWC | $11.33$ | $65.98$ | $53.57$ | $7.19$ | $49.66$ | $74.92$ | $82.17$ | $78.22$ | $79.54$ | $72.45$
GPTQ | $11.5$ | $65.81$ | $56.36$ | $7.68$ | $51.71$ | $76.52$ | $82.08$ | $78.86$ | $79.27$ | $73.16$
| CD | $11.72$ | $66.85$ | $57.63$ | $8.61$ | $51.19$ | $76.6$ | $82.51$ | $78.89$ | $79.49$ | $73.32$
| BCD(k=2) | $11.66$ | $66.73$ | $57.5$ | $8.56$ | $51.19$ | $76.3$ | $82.29$ | $78.88$ | $79.27$ | $73.32$
| OWC-CD | $12.27$ | $66.28$ | $58.21$ | $8.56$ | $51.62$ | $76.85$ | $82.32$ | $78.85$ | $79.49$ | $72.61$
|  OWC-CD \+ CD | $11.63$ | $66.28$ | $58.06$ | $7.92$ | $51.62$ | $76.94$ | $82.45$ | $78.72$ | $79.11$ | $72.69$
|  OWC-CD \+ BCD(k=2) | $11.72$ | $65.96$ | $58.29$ | $8.02$ | $51.79$ | $76.77$ | $82.17$ | $78.8$ | $79.49$ | $72.69$

Table 15: Table presents _downstream evaluation_ (zero-shot) numbers of
PaLM2-Gecko - for GPTQ, CD, BCD for INT3, INT4 quantization of FFN weights.

Config | Method | PaLM2-Gecko
---|---|---
|  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
w16a16 | - | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$ | $aa$
w3a16 | OWC | $2.55$ | $47.57$ | $9.81$ | $1.87$ | $29.69$ | $56.73$ | $64.8$ | $53.28$ | $69.59$ | $60.3$
GPTQ | $3.35$ | $44.13$ | $3.97$ | $0.89$ | $30.2$ | $62.12$ | $63.79$ | $56.33$ | $71.22$ | $57.85$
| CD | $3.55$ | $45.71$ | $11.32$ | $2.71$ | $30.63$ | $60.56$ | $66.45$ | $57.2$ | $71.87$ | $62.43$
| BCD(k=2) | $3.63$ | $47.04$ | $11.22$ | $2.76$ | $30.29$ | $60.52$ | $65.32$ | $57.33$ | $71.82$ | $61.8$
w3a16g128 | OWC | $3.68$ | $55.38$ | $12.5$ | $2.61$ | $31.66$ | $60.14$ | $66.12$ | $53.19$ | $71.71$ | $58.88$
GPTQ | $4.02$ | $50.04$ | $14.64$ | $3.15$ | $32.42$ | $61.91$ | $66.64$ | $56.05$ | $73.01$ | $59.75$
| CD | $3.38$ | $52.01$ | $9.62$ | $2.21$ | $31.23$ | $63.13$ | $65.9$ | $55.98$ | $72.47$ | $59.91$
| BCD(k=2) | $3.1$ | $50.95$ | $9.36$ | $1.92$ | $31.14$ | $62.25$ | $65.41$ | $56.16$ | $72.91$ | $58.96$
| OWC-CD | $3.6$ | $51.69$ | $9.33$ | $2.31$ | $31.83$ | $61.74$ | $65.02$ | $56.23$ | $72.2$ | $59.43$
|  OWC-CD \+ CD | $3.82$ | $50.39$ | $9.5$ | $1.77$ | $32.51$ | $62.25$ | $66.33$ | $56.91$ | $71.27$ | $60.06$
|  OWC-CD \+ BCD(k=2) | $3.63$ | $49.87$ | $10.26$ | $2.12$ | $32.76$ | $62.88$ | $66.24$ | $56.79$ | $71.82$ | $60.22$
w4a16 | OWC | $4.02$ | $46.96$ | $10.63$ | $2.66$ | $33.19$ | $61.99$ | $66.7$ | $59.26$ | $73.67$ | $61.72$
GPTQ | $4.46$ | $48.69$ | $18.84$ | $3.1$ | $34.47$ | $62.96$ | $64.77$ | $60.09$ | $73.61$ | $61.09$
| CD | $4.13$ | $48.88$ | $16.64$ | $3.15$ | $33.53$ | $63.89$ | $66.79$ | $60.31$ | $73.78$ | $61.4$
| BCD(k=2) | $4.24$ | $48.31$ | $15.73$ | $3.3$ | $33.28$ | $63.34$ | $66.39$ | $60.26$ | $73.56$ | $61.01$
w4a16g128 | OWC | $4.99$ | $46.1$ | $20.62$ | $4.28$ | $34.73$ | $64.48$ | $67.83$ | $59.91$ | $74.32$ | $61.25$
GPTQ | $5.18$ | $49.79$ | $18.17$ | $3.15$ | $35.15$ | $64.6$ | $66.24$ | $60.21$ | $74.32$ | $61.88$
| CD | $5.73$ | $46.96$ | $24.17$ | $4.53$ | $36.18$ | $66.04$ | $67.8$ | $60.7$ | $73.67$ | $61.25$
| BCD(k=2) | $5.79$ | $47.28$ | $24.11$ | $4.58$ | $36.01$ | $65.7$ | $67.06$ | $60.73$ | $73.61$ | $61.01$
| OWC-CD | $5.54$ | $47.24$ | $21.71$ | $4.72$ | $34.3$ | $64.73$ | $68.01$ | $60.72$ | $73.56$ | $61.01$
|  OWC-CD \+ CD | $5.62$ | $47.6$ | $24$ | $4.68$ | $35.49$ | $66.16$ | $67.06$ | $60.78$ | $74.1$ | $61.48$
|  OWC-CD \+ BCD(k=2) | $5.57$ | $47.17$ | $23.05$ | $4.82$ | $35.24$ | $65.82$ | $66.02$ | $60.9$ | $73.23$ | $61.72$

Table 16: Table presents _downstream evaluation_ (zero-shot) numbers for GPTQ,
CD, BCD for INT2 quantization of FFN weights. _Gen_ , _Rank_ columns
correspond to generation and ranking tasks.

Method | Epochs | PaLM2-Otter | PaLM2-Bison
---|---|---|---
w2a16g128 |  | Generation | Rank | Avg. | Generation | Rank | Avg.
GPTQ | - | $23.06$ | $66.62$ | $49.2$ | $33.62$ | $72.74$ | $57.09$
CD | 1 | $25.12$ | $65.73$ | $49.48$ | $33.63$ | $71.80$ | $56.53$
BCD(k=2) | $1$ | $25.42$ | $65.57$ | $49.51$ | $33.97$ | $72.16$ | $56.88$
$2$ | $24.73$ | $65.78$ | $49.36$ | $34.01$ | $72.39$ | $57.04$
| $3$ | $25.28$ | $66.01$ | $49.72$ | $34.27$ | $72.67$ | $57.31$
BCD(k=4) | $1$ | $25.15$ | $66.04$ | $49.68$ | $33.94$ | $72.10$ | $56.84$
$2$ | $24.86$ | $65.71$ | $49.37$ | $33.87$ | $71.97$ | $56.73$
| $3$ | $25.70$ | $65.30$ | $49.46$ | $33.85$ | $72.06$ | $56.78$

Table 17: Table presents _downstream evaluation_ (zero-shot) numbers of
PaLM2-Otter - for GPTQ, CD, BCD for INT2 quantization of FFN weights.

Method | Epochs | PaLM2-Otter
---|---|---
w2a16g128 |  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
GPTQ | - | $4.32$ | $63.43$ | $22.08$ | $2.41$ | $42.32$ | $66.96$ | $80.09$ | $68.54$ | $73.78$ | $68.03$
CD | $1$ | $4.99$ | $66.51$ | $25.53$ | $3.44$ | $40.87$ | $70.33$ | $74.89$ | $68.13$ | $72.96$ | $67.17$
BCD(k=2) | $1$ | $4.93$ | $67.51$ | $25.56$ | $3.69$ | $41.47$ | $69.7$ | $74.98$ | $67.97$ | $73.72$ | $65.59$
$2$ | $4.6$ | $67.74$ | $23.6$ | $3$ | $40.78$ | $69.49$ | $75.47$ | $68.27$ | $73.39$ | $67.25$
| $3$ | $5.1$ | $67.91$ | $24.53$ | $3.59$ | $42.41$ | $69.87$ | $75.02$ | $68.52$ | $73.88$ | $66.38$
BCD(k=4) | $1$ | $4.9$ | $67.5$ | $25.01$ | $3.2$ | $42.15$ | $68.69$ | $76.18$ | $68.51$ | $73.61$ | $67.09$
$2$ | $4.79$ | $67.9$ | $23.55$ | $3.2$ | $41.81$ | $69.82$ | $73.39$ | $68.5$ | $73.5$ | $67.25$
| $3$ | $5.46$ | $68.29$ | $26$ | $3.05$ | $41.21$ | $69.02$ | $73.98$ | $68.06$ | $72.52$ | $67.01$

Table 18: Table presents _downstream evaluation_ (zero-shot) numbers of
PaLM2-Bison - for GPTQ, CD, BCD for INT2 quantization of FFN weights.

Method | Epochs | PaLM2-Bison
---|---|---
w2a16g128 |  | NatualQ. | SQuAD | TriviaQA | WebQ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
GPTQ | - | $11.55$ | $66.72$ | $48.79$ | $7.43$ | $47.27$ | $76.89$ | $84.56$ | $74.36$ | $79.54$ | $73.8$
CD | $1$ | $11.25$ | $64.49$ | $50.84$ | $7.92$ | $48.12$ | $75.97$ | $79.39$ | $74.28$ | $79.11$ | $73.95$
BCD(k=2) | $1$ | $11.77$ | $64.17$ | $51.36$ | $8.56$ | $48.38$ | $75.88$ | $81.01$ | $74.6$ | $78.84$ | $74.27$
$2$ | $11.61$ | $64.69$ | $51.23$ | $8.51$ | $47.87$ | $76.09$ | $81.74$ | $74.72$ | $79$ | $74.9$
| $3$ | $11.72$ | $64.5$ | $52.15$ | $8.71$ | $48.46$ | $76.18$ | $82.54$ | $74.82$ | $79.6$ | $74.43$
BCD(k=4) | $1$ | $11.69$ | $64.48$ | $51.07$ | $8.51$ | $47.7$ | $75.63$ | $80.89$ | $74.55$ | $79.6$ | $74.27$
$2$ | $11.75$ | $64.39$ | $51.36$ | $7.97$ | $48.46$ | $75.04$ | $80.58$ | $74.48$ | $79.38$ | $73.88$
| $3$ | $11.8$ | $64.41$ | $51.16$ | $8.02$ | $48.12$ | $75.55$ | $80.86$ | $74.55$ | $78.73$ | $74.59$

##  Appendix E Quantization Runtime

Table 19: Table presents the time taken (in minutes) to quantize PaLM2-Otter
and PaLM2-Bison. All models were quantized with 8 80GB H100s, and run for 1
epoch (i.e., $T=d_{\text{in}}$).

Config | Method | PaLM2-Otter | PaLM2-Bison
---|---|---|---
|  |  | FFN runtime
---
(in minutes)
| Attn. runtime
---
(in minutes)
| FFN runtime
---
(in minutes)
| Attn. runtime
---
(in minutes)
w3a16 | GPTQ |  $0.90$m |  $0.72$m |  $6.42$m |  $0.63$m
CD |  $2.94$m |  $0.49$m |  $16.13$m |  $0.43$m
BCD(k=2) |  $14.03$m |  $2.08$m |  $82.13$m |  $1.81$m
w3a16g128 | GPTQ |  $1.02$m |  $0.75$m |  $6.70$m |  $0.66$m
CD |  $3.32$m |  $0.59$m |  $18.44$m |  $0.52$m
BCD(k=2) |  $14.73$m |  $2.12$m |  $88.10$m |  $1.86$m
w4a16 | GPTQ |  $0.90$m |  $0.64$m |  $6.60$m |  $0.56$m
CD |  $3.59$m |  $0.59$m |  $21.19$m |  $0.52$m
BCD(k=2) |  $34.85$m |  $4.72$m |  $235.14$m |  $4.13$m
w4a16g128 | GPTQ |  $0.98$m |  $0.69$m |  $6.962$m |  $0.60$m
CD |  $3.99$m |  $0.69$m |  $23.48$m |  $0.60$m
BCD(k=2) |  $42.37$m |  $5.41$m |  $192.59$m |  $4.74$m

##  Appendix F Comparison to QuantEase

We also compare our per-channel quantization results with those of QuantEase,
a parallel study to ours. It’s important to note that QuantEase does not have
a publicly available implementation. So, we implemented the cyclic coordinate
descent strategy used by QuantEase, and used the same initialization and
regularization strength as our algorithms (although the QuantEase paper
doesn’t provide these details). We then ran QuantEase for the recommended
number of iterations specified in the paper ($20$ epochs or
$T=20d_{\text{in}}$ iterations). The findings are presented in Table 20. On
PaLM2-Gecko, PaLM2-Otter, and PaLM2-Bison, both these approaches have similar
performance. These results collectively highlight the potential of coordinate
descent algorithms in outperforming GPTQ.

Config | Method | PaLM2-Gecko | PaLM2-Otter | PaLM2-Bison
---|---|---|---|---
w3a16 | GPTQ | $11.347$ | $7.176$ | $5.774$
QuantEase (epochs=20) | $10.731$ | $6.996$ | $5.741$
CD | $10.920$ | $7.002$ | $5.739$
| BCD(k=2) | $10.898$ | $6.979$ | $5.733$
w4a16 | GPTQ | $8.764$ | $6.249$ | $5.417$
QuantEase (epochs=20) | $8.670$ | $6.197$ | $5.408$
CD | $8.694$ | $6.195$ | $5.407$
| BCD(k=2) | $8.691$ | $6.192$ | $5.405$

Table 20: Comparison of QuantEase with CD and BCD for FFN quantization.