# IM-Unpack: Training and Inference with Arbitrarily Low Precision Integers

Zhanpeng Zeng Karthikeyan Sankaralingam Vikas Singh
University of Wisconsin–Madison
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

GEneral Matrix Multiply (GEMM) is a central operation in deep learning and
corresponds to the largest chunk of the compute footprint. Therefore,
improving its efficiency is an active topic of ongoing research. A popular
strategy is the use of low bit-width integers to approximate the original
entries in a matrix. This allows efficiency gains, but often requires
sophisticated techniques to control the rounding error incurred. In this work,
we first verify/check that when the low bit-width restriction is removed, for
a variety of Transformer-based models, whether integers are sufficient for all
GEMMs need – for both training and inference stages, and can achieve parity
with floating point counterparts. No sophisticated techniques are needed. We
find that while a large majority of entries in matrices (encountered in such
models) can be easily represented by low bit-width integers, the existence of
a few heavy hitter entries make it difficult to achieve efficiency gains via
the exclusive use of low bit-width GEMMs alone. To address this issue, we
develop a simple algorithm, Integer Matrix Unpacking (IM-Unpack), to unpack a
matrix with large integer entries into a larger matrix whose entries all lie
within the representable range of arbitrarily low bit-width integers. This
allows equivalence with the original GEMM, i.e., the exact result can be
obtained using purely low bit-width integer GEMMs. This comes at the cost of
additional operations – we show that for many popular models, this overhead is
quite small.

##  1 Introduction

Calculating the product of two matrices using GEneral Matrix Multiply (GEMM)
is one of the most widely used operations in modern machine learning. Given
matrices $\mathbf{A}$ and $\mathbf{B}$ of size $n\times d$ and $h\times d$
respectively, the output of a GEMM is calculated as

$\mathbf{C}=\mathbf{A}\mathbf{B}^{\top}$ (1)

Choosing the appropriate numerical precision or data type (FP32, FP16, or
BF16) for GEMM is often important, and hinges on several factors including the
specific application, characteristics of the data, model architecture, as well
as numerical behavior such as convergence. This choice affects compute and
memory efficiency most directly, since a disproportionately large chunk of the
compute footprint of a model involves the GEMM operator. A good example is the
large improvement in latency and memory achieved via low bit-width GEMM, and
made possible due to extensive ongoing work on quantization (to low bit-width
data types) and low-precision training [2, 21, 14, 6, 15, 28, 6, 20, 18, 17,
15, 29, 8, 16, 25, 27, 30, 26]. Integer quantization is being actively pursued
for inference efficiency, and the use of low bit-width integers is universal
to deliver the efficiency gains. However, this strategy often incurs large
rounding errors when representing all matrix entries as low bit-width
integers, and explains the drop in performance and thereby, a need for error
correction techniques [11, 28, 4]. So how much of the performance degradation
is due to (a) rounding to integers versus (b) restricting to low bit-width
integers? To answer this question, it appears worthwhile to check whether
integer GEMMs will achieve parity without sophisticated techniques (for the
inference stage, and more aspirationally, for training) for popular models if
we do not restrict to low bit-width integers.

Figure 1: Overall Illustration. We verify the Efficacy of Integers
(Contribution 1) in §2, but note that the integer matrices contain heavy
hitters (§3). Then, we describe our proposed algorithm, IM-Unpack
(Contribution 2), to resolve these heavy hitters in §4.

Overview. The starting point of our work is to first experimentally verify
that the aforementioned hypothesis – that integer GEMM may work – is true (see
§2). But by itself, this finding offers no value proposition for efficiency.
Still, this experiment is useful for the following reason. For a particular
class of models (e.g., Transformers), we can easily contrast the corresponding
input matrices $\mathbf{A}$ and $\mathbf{B}$ between (a) integer GEMM and (b)
low bit-width integer GEMM and probe if any meaningful structure can be
exploited. While there is a clear difference in the outputs of (a) integer
GEMM versus (b) low bit-width integer GEMM, we find that a large majority of
entries of $\mathbf{A}$ and $\mathbf{B}$ can be well-represented using low
bit-width integers – and the difference in the outputs can be entirely
attributed to a few heavy hitter entries in $\mathbf{A}$ and $\mathbf{B}$,
that cannot be represented using low bit-width integers. Other works have also
run into this issue of “outliers” and suggest using high precision [6] or a
separate quantization for these entries [29, 28]. Driven by the simple
observation that we can represent a large integer by a series of smaller
integers, our algorithm, Integer Matrix Unpack (IM-Unpack), enables unpacking
any integer into a series of low-bit integers. The key outcome is that the
calculation can be carried out entirely using low bit-width integer arithmetic
and thus unifies the computation needed for heavy hitters and the remaining
entries (which were already amenable to low-bit integer arithmetic).
Specifically, IM-Unpack unpacks an integer matrix such that all values of the
unpacked matrices always stay within the representable range of low bit-width
integers (bit-width can be chosen arbitrarily). We obtain the exact result of
the original integer GEMM using purely low bit-width integer GEMMs. Since the
bit-width of integer arithmetic is independent of the actual range of the
original matrices, the construction will greatly simplify the
hardware/compiler support by only needing support for one specific bit-width.
The overall structure/contributions of this paper is shown in Fig. 1.

Notations. To simplify the presentation, we will narrow the scope of our
discussion exclusively to Transformer-based models. We first define notations
for all relevant GEMMs. For the linear layer, let the input activation and
parameter matrix be $\mathbf{X}$ and $\mathbf{W}$. Let the query, key, value
matrices involved in self-attention computation be
$\mathbf{Q},\mathbf{K},\mathbf{V}$. Below, we itemize all GEMMs used in a
Transformer model:

$\begin{split}\mathbf{Y}=\mathbf{X}\mathbf{W}^{\top}\quad\mathbf{P}=\mathbf{Q}\mathbf{K}^{\top}\quad\mathbf{O}=\mathbf{M}\mathbf{V}\end{split}$
(2)

where $\mathbf{M}$ is the attention score between $\mathbf{Q}$ and
$\mathbf{K}$ defined as $\mathbf{M}=\text{softmax}(\mathbf{P})$ (omitting
scaling factors). Now, given the gradient for
$\mathbf{Y},\mathbf{P},\mathbf{O}$ denoted as
$\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}$, the other
gradients are calculated via GEMMs as well:

$\begin{split}\nabla_{\mathbf{X}}=\nabla_{\mathbf{Y}}\mathbf{W}\quad\nabla_{\mathbf{Q}}&=\nabla_{\mathbf{P}}\mathbf{K}\quad\nabla_{\mathbf{M}}=\nabla_{\mathbf{O}}\mathbf{V}^{\top}\\\
\nabla_{\mathbf{W}}=\nabla_{\mathbf{Y}}^{\top}\mathbf{X}\
\quad\nabla_{\mathbf{K}}&=\nabla_{\mathbf{P}}^{\top}\mathbf{Q}\quad\
\nabla_{\mathbf{V}}=\mathbf{M}^{\top}\nabla_{\mathbf{O}}\\\ \end{split}$ (3)

These notations will help refer to each type of GEMM later.

##  2 Round to Nearest: What do we lose?

Table 1: Inference: Comparison on LLaMA-7B zero-shot performance and ViT
ImageNet classification when using quantized computations in all linear
layers. HS: HellaSwag, WG: WinoGrande. The super-script ‡ indicates that
LLM.int8() uses mixed-precision (INT8+FP16) to process outliers using FP16.

LLaMA-7B | Method | $\beta$ | Type | ARC-c | ARC-e | BoolQ | HS | PIQA | WG
---|---|---|---|---|---|---|---|---|---
Full-Precision | - | BF16 | 43.1 | 76.3 | 77.8 | 57.2 | 78.0 | 68.8
LLM.int8() | - |  INT8‡ | 43.8 | 75.5 | 77.8 | 57.4 | 77.6 | 68.7
SmoothQuant | - | INT8 | 37.4 | 74.4 | 74.0 | 55.0 | 77.5 | 69.6
LLM-QAT | - | INT4 | 30.2 | 50.3 | 63.5 | 55.6 | 64.3 | 52.9
LLM-FP4 | - | FP4 | 33.6 | 65.9 | 64.2 | 47.8 | 73.5 | 63.7
RTN | 5 | INT | 39.3 | 72.8 | 69.9 | 53.4 | 74.9 | 66.4
| 7 | INT | 42.6 | 73.9 | 72.3 | 55.9 | 77.0 | 67.4
| 11 | INT | 43.9 | 76.1 | 77.3 | 56.3 | 77.3 | 69.3
| 15 | INT | 43.0 | 75.7 | 77.5 | 57.0 | 78.0 | 69.2
| 31 | INT | 42.7 | 76.1 | 76.1 | 57.3 | 77.3 | 69.3

ViT | Method | $\beta$ | Type | Tiny | Small | Base | Large | Huge
---|---|---|---|---|---|---|---|---
Full-Precision | - | FP32 | 75.5 | 81.4 | 85.1 | 85.8 | 87.6
RTN | 5 | INT | 3.9 | 36.9 | 78.7 | 83.6 | 85.3
| 7 | INT | 41.0 | 70.9 | 82.8 | 84.9 | 86.7
| 15 | INT | 71.4 | 79.8 | 84.6 | 85.6 | 87.5

Table 2: Inference: Comparison on LLaMA-7B and ViT when quantize computation
in all GEMMs. ∗: PTQ4ViT uses a twin uniform quantization so GEMMs cannot be
performed on INT6 directly and requires some modifications.

LLaMA-7B | Method | $\beta$ | Type | ARC-c | ARC-e | BoolQ | HS | PIQA | WG
---|---|---|---|---|---|---|---|---|---
Full-Precision | - | BF16 | 43.1 | 76.3 | 77.8 | 57.2 | 78.0 | 68.8
RTN | 5 | INT | 23.5 | 34.3 | 54.8 | 32.5 | 57.6 | 49.7
| 7 | INT | 34.2 | 64.0 | 64.6 | 50.1 | 70.3 | 61.2
| 11 | INT | 41.6 | 72.4 | 68.7 | 55.1 | 75.4 | 65.1
| 15 | INT | 44.0 | 75.0 | 74.6 | 56.4 | 77.0 | 66.3
| 31 | INT | 43.4 | 75.8 | 76.8 | 57.5 | 77.4 | 68.4

ViT | Method | $\beta$ | Type | Tiny | Small | Base | Large | Huge
---|---|---|---|---|---|---|---|---
Full-Precision | - | FP32 | 75.5 | 81.4 | 85.1 | 85.8 | 87.6
FQ-ViT | - | INT8 | - | - | 83.3 | 85.0 | -
I-ViT | - | INT8 | - | 81.3 | 84.8 | - | -
PTQ4ViT | - |  INT6∗ | 66.7 | 78.3 | 82.9 | 84.9 | 86.6
APQ-ViT | - | INT4 | 17.6 | 48.0 | 41.4 | - | -
RepQ-ViT | - | INT4 | - | 65.1 | 68.5 | - | -
RTN | 5 | INT | 3.5 | 28.5 | 76.9 | 83.2 | 84.9
| 7 | INT | 39.0 | 69.9 | 82.1 | 84.7 | 86.5
| 15 | INT | 71.1 | 79.8 | 84.5 | 85.6 | 87.5

Let us start by using the simplest Rounding To Nearest (RTN) to map FP to
integers, and check the extent to which integer GEMMs work satisfactorily for
both training and inference, if we do not restrict to low bit-width integers.
Specifically, for matrix $\mathbf{A}$, all entries of $\mathbf{A}$ are
quantized via

$\mathbf{A}_{q}=\text{round}(0.5\beta/\alpha_{p}(\mathbf{A})\mathbf{A})$ (4)

where $\alpha_{p}(\mathbf{A})$ gives the $p$-th percentile (see §7.1) based on
the magnitude of entries in $\mathbf{A}$, i.e., $p\%$ of entries in
$\mathbf{A}$ fall in the interval
$[-\alpha_{p}(\mathbf{A}),\alpha_{p}(\mathbf{A})]$. We only need
$\alpha_{p}(\mathbf{A})$ as a meaningful estimate of the approximate range of
values, and so we set $p=95\%$ for all experiments except a few cases
discussed explicitly. The hyperparameter $\beta$ is the number of distinct
integers that we want to use to encode values that are within
$[-\alpha_{p}(\mathbf{A}),\alpha_{p}(\mathbf{A})]$. Then, after quantization,
the GEMM for the original matrices can be approximated (because we incur a
rounding error) in the quantized domain using integer GEMMs. The approximated
GEMM is computed using the quantized $\mathbf{A}$ and $\mathbf{B}$:

$\mathbf{C}\approx\frac{\alpha_{p}(\mathbf{A})\alpha_{p}(\mathbf{B})}{(0.5\beta)^{2}}\mathbf{A}_{q}\mathbf{B}_{q}^{\top}$
(5)

The scaling factor in (5) is used to undo the scaling in (4). Here,
$\mathbf{A}_{q}\mathbf{B}_{q}^{\top}$ is an integer GEMM, as desired. For
notational simplicity, if clear from context, we will drop the $q$ subscript
from $\mathbf{A}$ and $\mathbf{B}$.

###  2.1 Efficacy of Integers: Inference

A majority of the literature on quantized low precision calculations focuses
on inference efficiency [11, 4, 18, 29, 11, 4, 18, 29, 17, 15, 29, 8, 16].
Here, given a trained model, quantization seeks to reduce the precision of
parameters and input activations to low precision. This allows faster low
precision arithmetic for compute efficiency while maintaining model
performance. So, we first evaluate how well RTN preserves model performance
compared to baselines in this inference regime. See §7.2 for results for
quantizing parameters for memory saving and see §7.5 for other experiments.
Most quantization schemes for LLMs focus on quantizing GEMMs in Linear layers,
while quantization methods for Vision Transformers are more ambitious and
quantize all GEMMs in a Transformer. We follow this convention for baselines,
but present all variants for RTN.

Figure 2: Training: Comparison of RoBERTa loss curves.

Quantize GEMMs in Linear layers. It is common [28, 18] to try and quantize the
weight and input activation of linear layers to low precision for compute
efficiency. We summarize our comparisons in Tab. 1. Here, we compare RTN to
[28, 6, 20, 18]. As shown in Tab. 1, a simple RTN works remarkably well
compared to other baselines. We use INT as a data type for RTN here; in §4, we
show that we can compute integer GEMMs of any bit-widths using arbitrarily low
bit-width GEMMs.

Quantize all GEMMs. A more ambitious goal is to quantize every GEMM in a
Transformer model for higher efficiency. The comparison results with [17, 15,
29, 8, 16] are summarized in Tab. 2. We can draw a similar conclusion that a
simple RTN offers strong performance.

###  2.2 Efficacy of Integers: Training

Table 3: Training: Validation log perplexity of RoBERTa.  Size | FP32 | BF16 | $\beta=255$ | $\beta=31$ | $\beta=15$
---|---|---|---|---|---
Small | 1.869 | 1.868 | 1.823 | 1.840 | 1.891
Base | 1.611 | - | - | 1.601 | -

The transition from FP32 to FP16 and BF16 for GEMMs has doubled the compute
efficiency of modern deep learning models. However, far fewer efforts have
focused on low precision training (relative to inference) and this usually
requires more sophisticated modifications [25, 27, 30, 26]. In this
subsection, we evaluate how well quantizing all GEMMs using RTN works for
training Transformer models. To ensure that the updates can be properly
accumulated for the parameters, we use FP32 for storing the parameters and use
the quantized version for GEMMs. To limit the amount of compute but still
gather strong evidence, we evaluate RTN on RoBERTa [19] pretraining using
masked language modeling [7] on the English Wikipedia corpus [10] and ImageNet
classification [5] using ViT [9] (and see T5-Large [23] finetuning in §7.6).
All hyperparameters (including random seed) are the same for full-precision
and RTN quantized training. See §7.3 for more details of training
configurations.

Table 4: Training: Validation top-1 accuracy of ViT-Small. FP32 | FP16 | $\beta=63^{\dagger}$ | $\beta=31^{\dagger}$ | $\beta=31^{*}$
---|---|---|---|---
78.91 | 79.16 | 78.94 | 79.33 | 79.17

RoBERTa. As shown in Fig. 2, when $p=95\%$, for both Small and Base models,
the RTN quantized training gives an almost identical log perplexity (loss)
curves as FP32 training for $\beta\in\\{15,31,255\\}$. For larger $\beta$, the
curve is even closer to the FP32 training curve. We see that $\beta=31$
already gives a remarkably good result. Surprisingly, despite a marginally
higher training log perplexity when using RTN, the validation log perplexity
of RTN ($\beta=31$ and $\beta=255$) is marginally lower than FP32 and BF16,
see Tab. 3.

ViT. For ViT, compared to RoBERTa pretraining, we found that it may be
necessary to allow the gradients
$\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}$ of the model to
have higher bit-width. As shown in Fig. 3, when $\beta$ is the same
($\beta=31$ and $\beta=127$ for the set
$\\{\mathbf{X},\mathbf{W},\mathbf{Q},\mathbf{K},\mathbf{M},\mathbf{V}\\}$ and
$\\{\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}\\}$, we see
divergence in the middle of training. Alternatively, when using larger $\beta$
for only the set
$\\{\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}\\}$, the loss
curve of RTN quantized training is almost identical to FP32 training.
Surprisingly, we observed similar results as RoBERTa training: despite
marginally higher training loss when using RTN, the validation top-1 accuracy
of RTN is higher than FP32 as shown in Fig. 3 and Tab. 4.

##  3 What happens with Low Bit-Width?

Figure 3: Training: Comparison of ViT-Small. † and ∗: we set $\beta=16383$ and
$\beta=1023$, respectively, for the set
$\\{\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}\\}$.

Converting floating point to integers alone will not provide efficiency
benefits. Rather, we want to use a representation that can be efficiently
computed (and why low bit-width integers are common in integer quantization).
Notice that as a direct consequence of RTN, by (4), 95% of values can be
represented using $\beta$ distinct numbers, which requires only
$\log_{2}(\beta+1)$ bits. For example, if $\beta=15$, then we can represent
these 95% of values with 4-bit signed integers, which is already low bit-
width. So, is there still a problem?

It turns out that the difficulty involves dealing with the remaining 5% of
entries. To get a sense of how large these values are, we calculate the ratio
$\alpha_{100}(\cdot)/\alpha_{95}(\cdot)$ between the maximum and 95th-
percentile of the magnitude of each matrix in GEMMs when performing (a)
inference (forward pass) of LLaMA-7B and ViT-Large and (b) training (forward
pass and backward pass) of RoBERTa-Small at different training phases. We can
check the ratios in Tab. 5 and Tab. 6, respectively. We see extremely large
values across both training and inference and across the entire duration of
training, so simply increasing the representation bit width of low precision
integers by a few more bits will not be sufficient to represent these heavy
hitters.

We performed experiments studying different ways of handling these heavy
hitters when quantizing all GEMMs (linear layers and self-attention
computation) in Transformer models. Unless $\beta$ is inordinately large
(based on Tab. 5 and Tab. 6, more than $10^{5}$ times larger than our choice
of $\beta$ for $p=95\%$), simply ensuring that the heavy hitters lie within
the representable range of $\beta$ for $\beta=255$ or $\beta=127$ results in a
huge performance drop as shown in Tab. 7. On the other hand, clipping the
extreme heavy hitters (at the $99.5$-percentile) also fails as shown in Tab.
7. Our observations for training are similar – we can see the loss curves for
$p=100\%,\beta=255$ and $p=95\%,\beta=31$ in Fig. 2.

Table 5: Maximal ratios between the maximum and 95-percentile of magnitudes of each matrix involved in GEMMs. Model | $\mathbf{X}$ | $\mathbf{W}$ | $\mathbf{Q}$ | $\mathbf{K}$ | $\mathbf{M}$ | $\mathbf{V}$
---|---|---|---|---|---|---
LLaMA-7B | 141312.0 | 47.8 | 8.4 | 8.1 | 4448.0 | 36.2
ViT-Large | 284402.4 | 34.8 | 4.3 | 4.3 | 120.0 | 8.9
Table 6: Maximal ratios between the maximum and 95-percentile of magnitudes of each matrix involved in GEMMs during the training of RoBERTa-Small. Progress | $\mathbf{X}$ | $\mathbf{W}$ | $\nabla_{\mathbf{Y}}$ | $\mathbf{Q}$ | $\mathbf{K}$ | $\nabla_{\mathbf{P}}$ | $\mathbf{M}$ | $\mathbf{V}$ | $\nabla_{\mathbf{O}}$
---|---|---|---|---|---|---|---|---|---
1/3 | 28.7 | 7.1 | 292.5 | 3.7 | 3.0 | 309365.2 | 3924.6 | 3.1 | 25.8
2/3 | 25.7 | 13.8 | 235.4 | 4.2 | 2.7 | 283742.8 | 2283.3 | 3.3 | 32.4
3/3 | 22.0 | 16.0 | 290.3 | 4.0 | 3.0 | 218376.0 | 2018.6 | 3.4 | 28.9
Table 7: Catastrophic performance degradation when restricting outliers to a
representable range of quantized domain or clipping the outliers on zero-shot
inference of LLaMA-7B and ImageNet classification of quantized ViT models.
$p=100$ means we keep outliers within representable range of $\beta$ distinct
integers. $\beta=\infty$ means that we do not quantize the values. Clip means
we clip the values that is larger than $p$-percentile.

LLaMA-7B | $p$ | $\beta$ | Clip | ARC-c | ARC-e | BoolQ | HS | PIQA | WG
---|---|---|---|---|---|---|---|---|---
Full-Precision | 43.1 | 76.3 | 77.8 | 57.2 | 78.0 | 68.8
100 | 255 | No | 35.8 | 66.2 | 57.8 | 47.4 | 71.3 | 63.9
99.5 | $\infty$ | Yes | 21.4 | 25.5 | 60.2 | 25.8 | 53.5 | 49.9
95 | 31 | No | 43.4 | 75.8 | 76.8 | 57.5 | 77.4 | 68.4

ViT | $p$ | $\beta$ | Clip | Tiny | Small | Base | Large | Huge
---|---|---|---|---|---|---|---|---
Full-Precision | 75.5 | 81.4 | 85.1 | 85.8 | 87.6
100 | 127 | No | 53.9 | 69.1 | 72.0 | 81.6 | 83.6
99.5 | $\infty$ | Yes | 11.3 | 24.1 | 9.0 | 15.8 | 0.6
95 | 15 | No | 71.1 | 79.8 | 84.5 | 85.6 | 87.5

As briefly mentioned earlier, some ideas have been proposed to process these
so-called outliers. The approach in [6] exploits the location structure of
where these outliers occur and moves the columns or rows of each matrix (with
these outliers) into a different matrix, then GEMM is performed using FP16.
The authors in [28] propose to smooth the outliers in activation and mitigate
the quantization difficulty via a transformation. This strategy requires
specialized GEMM hardware support for different precisions and may even lower
the performance as shown in our baseline comparisons in §2.1 and §2.2.

Goals. We desire an approach that does not alter the results of integer GEMMs;
in other words, all results in §2.1 and §2.2 must remain exactly the same, yet
we should not need calculations using different precisions. This may appear
unrealistic but our simple procedure, IM-Unpack, allows representing heavy
hitters using low bit-width integers. Calculations are carried out using low
bit-width integer arithmetic. Specifically, IM-Unpack unpacks a matrix
containing heavy hitters into a larger unpacked matrix (we study how large the
expansion will be in §4.2) whose values are all representable by low bit-width
integers. IM-Unpack obtains the exact output of the original GEMM using purely
low bit-width integer GEMMs on these unpacked matrices.

##  4 IM-Unpack: Integer Matrix Unpacking

Our approach starts with a simple observation that, for example, a 32-bit
integer $v$ can be represented as

$v=v_{0}+128v_{1}+128^{2}v_{2}+128^{3}v_{3}+128^{4}v_{4}$ (6)

where $v_{i}$ are 8-bit integers. Multiplication/addition of two 32-bit
integers can be performed on these decomposed 8-bit integers followed by some
post-processing steps (scaling via bit shifting and accumulation). This
unpacking does enable performing high bit-width arithmetic using lower bit-
widths, but it achieves this at the cost of requiring more operations. For
example, one 32-bit addition now becomes five 8-bit additions with some follow
up processing, and one 32-bit multiplication becomes twenty five 8-bit
multiplications (distributive law).

######  Remark 4.1.

The reason why this unpacking is still useful is because the additional work
depends on the number and spatial distribution of the heavy hitters/outliers.
We harvest gains because outliers account for a very small portion of the
matrices that appear in practice in training/inference stages of Transformer
models.

Figure 4: Illustration of unpacking row vectors. The solid, dashed, and dotted
arrows correspond to lines 5, 4, and 6 in Algo. 1 Algorithm 1
UnpackRow($\mathbf{A},b$)

1: Let $\mathbf{\Pi}\leftarrow\mathbf{I}$ and $s\leftarrow 2^{b-1}$ and
$i\leftarrow 0$

2: while $\mathbf{A}[i,:]$ exists do

3: if $\mathbf{A}[i,:]$ contains OB entries then

4: Append $\text{floor}(\mathbf{A}[i,:]/s)$ as a new row to $\mathbf{A}$

5: $\mathbf{A}[i,:]\leftarrow\mathbf{A}[i,:]\text{ mod }s$

6: Append $s\mathbf{\Pi}[:,i]$ as a new column to $\mathbf{\Pi}$

7: end if

8: $i\leftarrow i+1$

9: end while

10: return $\mathbf{A},\mathbf{\Pi}$

Let $b$ be the target bit-width of low bit-width integers and $s=2^{b-1}$ be
the representable range of bit-width $b$: $b$-bit integers can represent a set
$\\{-s+1,\cdots,0,\cdots,s-1\\}$. We refer to any integers inside of this set
as In-Bound (IB) values and any integers outside of this set as Out-of-Bound
(OB) values, which will be used in later discussion to refer to the values
that need to be unpacked. We will first show how to unpack a vector to
multiple low bit-width vectors. Then, we will discuss how to unpack a matrix
using different strategies to achieve better results in different cases.
Lastly, we will evaluate how well does IM-Unpack work.

Unpacking an integer vector. Let $\mathbf{v}$ be an integer vector and define
a function:

$\begin{split}m(\mathbf{v},s,i)=\text{floor}(\mathbf{v}/s^{i})\text{ mod
}s\end{split}$ (7)

such that for all $i$, all entries of $m(\mathbf{v},s,i)$ are bounded (IB),
i.e., lie in the interval $[-s+1,s-1]$. When $s$ is clear from the context, we
shorten the LHS of (7) to just $m(\mathbf{v},i)$. Then,

$\begin{split}\mathbf{v}&=\sum_{i=0}^{\infty}s^{i}m(\mathbf{v},i)\end{split}$
(8)

Note that $\mathbf{v}/s^{i}$ decreases to $0$ exponentially fast, so we are
able to unpack a vector with just a few low bit-width vectors.

###  4.1 Variants of Matrix Unpacking

In this subsection, we discuss different strategies of matrix unpacking for
different structure-types of matrices. First, we discuss the case where
$\mathbf{A}$ is the matrix containing OB values to be unpacked and
$\mathbf{B}$ is a matrix whose values are all IB. Next, we discuss how
unpacking works when both $\mathbf{A}$ and $\mathbf{B}$ contains OB values.

Unpacking row vectors. We start with the simplest way of unpacking a matrix:
unpacking the row vectors. Given a matrix $\mathbf{A}$, if one row of
$\mathbf{A}$ contains OB values, we can unpack the row to multiple rows whose
entries are all bounded. The exact procedure is described in Alg. 1 and
illustrated in Fig. 4. In Fig. 4, when the second row in $\mathbf{A}$ contains
OB values, we can unpack it to two row vectors (the second and fifth row) and
the post-processing step takes the form of applying $\mathbf{\Pi}_{A}$ to the
unpacked matrix $\mathbf{A}_{u}$.

Figure 5: Illustration of unpacking column vectors. The blue solid, dashed,
and dotted arrows correspond to lines 5, 4, and 7 in Algo. 1, and the gray
dashed arrow corresponds to line 6 in Algo. 1.

Reconstructing $\mathbf{A}$. $\mathbf{A}$ can be reconstructed using the
unpacked matrix $\mathbf{A}_{u}$ whose entries are IB and a sparse matrix
$\mathbf{\Pi}$ whose column contains only one non-zero:

$\begin{split}\mathbf{A}_{u},\mathbf{\Pi}_{A}&=\text{UnpackRow}(\mathbf{A},b)\\\
\mathbf{A}&=\mathbf{\Pi}_{A}\mathbf{A}_{u}\end{split}$ (9)

Here, applying $\mathbf{\Pi}_{A}$ to $\mathbf{A}_{u}$ can be efficiently
computed easily (for example, via torch.index_add).

Figure 6: Left: Failure case for unpacking rows. Right: Failure case for
unpacking rows or columns alone.

Are we done? If we do not care about maximizing efficiency, then the above
scheme already provides a way to perform high bit-width GEMM using low bit-
width GEMM. However, this might not be the optimal unpacking strategy for some
matrices. For example, consider the left matrix shown in Fig. 6. Since every
row of this matrix contains OB values, every row need to be unpacked,
resulting in a much larger matrix. In this case, it might be better to try and
unpack the column vectors. Let us apply a similar idea of unpacking row
vectors to unpack column vectors of $\mathbf{A}$:

$\begin{split}\mathbf{A}&=\mathbf{A}_{u}^{\prime}\mathbf{\Pi}^{\prime}_{A}\\\
\mathbf{A}\mathbf{B}^{\top}&=\mathbf{A}_{u}^{\prime}\mathbf{\Pi}^{\prime}_{A}\mathbf{B}^{\top}\end{split}$
(10)

While unpacking column vectors is reasonable, the sparse matrix
$\mathbf{\Pi}^{\prime}_{A}$ creates an problem. When performing a GEMM of two
lower bit-width matrices: $\mathbf{\Pi}^{\prime}_{A}$ has to be applied to
$\mathbf{A}_{u}^{\prime}$ or $\mathbf{B}^{\top}$ before GEMM, but the
result/output may contain OB entries after the application, disabling low bit-
width integer GEMM. This problem is similar to per-channel quantization. It is
not simple to handle and become more involved when $\mathbf{B}$ also need to
be unpacked.

Unpacking column vectors. Alternatively, let us look at how
$\mathbf{A}\mathbf{B}^{\top}$ is computed via outer product of column vectors:

$\mathbf{C}=\mathbf{A}\mathbf{B}^{\top}=\sum_{i=1}^{d}\mathbf{A}[:,i]\mathbf{B}[:,i]^{\top}$
(11)

Let us look at the $i$-th outer product. Let us try unpacking
$\mathbf{A}[:,i]$ using (8), then we have

$\begin{split}\mathbf{A}[:,i]\mathbf{B}[:,i]^{\top}&=\sum_{j=0}^{\infty}s^{j}m(\mathbf{A}[:,i],j)\mathbf{B}[:,i]^{\top}\end{split}$
(12)

Suppose that $m(\mathbf{A}[:,i],j)=0$ for $j\geq k$, then we can unpack one
outer product to $k$ outer products. This is equivalent to appending
$m(\mathbf{A}[:,i],j)$ for $0\leq j<k$ to the columns of $\mathbf{A}$,
appending $k$ identical $\mathbf{B}[:,i]$ to the columns of $\mathbf{B}$, and
maintaining a diagonal matrix to keep track of the scaling factor $s^{j}$. The
exact procedure is described in Alg. 2, and Fig. 5 shows a visualization of
unpacking columns. Using column unpacking, we have

$\begin{split}\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u}&=\text{UnpackColumn}(\mathbf{A},\mathbf{B},\mathbf{I},b)\\\
\mathbf{A}\mathbf{B}^{\top}&=\mathbf{A}_{u}\mathbf{S}_{u}\mathbf{B}_{e}^{\top}\end{split}$
(13)

Naively, this still suffers from the same problem as discussed in (10) in that
there is a diagonal scaling matrix between two low bit-width matrices making
low bit-width GEMMs difficult. However, since $\mathbf{S}_{u}$ is a diagonal
matrix whose diagonal entries consist of a few distinct factors in
$\\{1,s,s^{2},...\\}$, we can easily compute one GEMM for each distinct
diagonal entry as shown in Alg. 3.

$\begin{split}\mathbf{A}\mathbf{B}^{\top}&=\text{ScaledMatMul}(\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u})\end{split}$
(14)

Further, since $s$ is a power of $2$, the scaling can be efficiently
implemented via bit shifting.

Algorithm 2 UnpackColumn$(\mathbf{A},\mathbf{B},\mathbf{S},b)$

1: Let $s\leftarrow 2^{b-1}$ and $i\leftarrow 0$

2: while $\mathbf{A}[:,i]$ exists do

3: if $\mathbf{A}[:,i]$ contains OB entries then

4: Append $\text{floor}(\mathbf{A}[:,i]/s)$ as new column to $\mathbf{A}$

5: $\mathbf{A}[:,i]\leftarrow\mathbf{A}[:,i]\text{ mod }s$

6: Append $\mathbf{B}[:,i]$ as new column to $\mathbf{B}$

7: Append $s\mathbf{S}[i,i]$ as new diagonal entry to $\mathbf{S}$

8: end if

9: $i\leftarrow i+1$

10: end while

11: return $\mathbf{A},\mathbf{B},\mathbf{S}$

Algorithm 3 ScaledMatMul($\mathbf{A},\mathbf{B},\mathbf{S}$)

1: Let $\mathbf{C}\leftarrow 0$

2: for all distinct diagonal entry $s^{i}$ in $\mathbf{S}$ do

3: Let $\mathcal{I}$ be the index set where $\mathbf{S}[j,j]=s^{i}$ for
$j\in\mathcal{I}$

4:
$\mathbf{C}\leftarrow\mathbf{C}+s^{i}\mathbf{A}[:,\mathcal{I}]\mathbf{B}[:,\mathcal{I}]^{\top}$

5: end for

6: return $\mathbf{C}$

Figure 7: Illustration of unpacking both rows and columns based on the OOB
counts. The red solid, dashed, and dotted arrows correspond to lines 8, 7, and
9 in Algo. 4. The blue solid, dashed, and dotted arrows correspond to lines
12, 11, and 14 in Algo. 4, and the gray dashed arrow corresponds to line 13 in
Algo. 4.

Are we done yet? Unpacking columns is efficient for the left matrix shown in
Fig. 6. However, neither unpacking rows nor unpacking columns will be
efficient for unpacking the right matrix shown in Fig. 6. All rows and columns
contains OB values. Unpacking rows or columns alone will not be ideal. For the
right matrix in Fig. 6, a better strategy is to unpack the second row and the
second column simultaneously.

Algorithm 4 UnpackBoth$(\mathbf{A},\mathbf{B},\mathbf{S},b)$

1: Let $s\leftarrow 2^{b-1}$ and

2: while True do

3: Let $(c_{0},i),(c_{1},j)$ be the tuples of top OB count in row/column
vectors and corresponding index

4: if $c_{0}=0$ and $c_{1}=0$ then

5: break

6: else if $c_{0}\geq c_{1}$ then

7: Append $\text{floor}(\mathbf{A}[i,:]/s)$ as new row to $\mathbf{A}$

8: $\mathbf{A}[i,:]\leftarrow\mathbf{A}[i,:]\text{ mod }s$

9: Append $s\mathbf{\Pi}[:,i]$ as new column to $\mathbf{\Pi}$

10: else

11: Append $\text{floor}(\mathbf{A}[:,j]/s)$ as new column to $\mathbf{A}$

12: $\mathbf{A}[:,j]\leftarrow\mathbf{A}[:,j]\text{ mod }s$

13: Append $\mathbf{B}[:,j]$ as new column to $\mathbf{B}$

14: Append $s\mathbf{S}[j,j]$ as new diagonal entry to $\mathbf{S}$

15: end if

16: end while

17: return $\mathbf{A},\mathbf{B},\mathbf{S},\mathbf{\Pi}$

Unpacking both rows and columns simultaneously. Our final strategy combines
row and column unpacking together and selectively performs row unpack or
column unpack based on the number of OB values that can be eliminated. The
procedure is described in Alg. 4, and we provide an illustration of unpacking
both dimensions in Fig. 7. With this procedure, we can obtain the output of
high bit-width GEMM using low bit-width as:

$\begin{split}\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u},\mathbf{\Pi}_{A}&=\text{UnpackBoth}(\mathbf{A},\mathbf{B},\mathbf{I},b)\\\
\mathbf{A}\mathbf{B}^{\top}&=\mathbf{\Pi}_{A}\mathbf{A}_{u}\mathbf{S}_{u}\mathbf{B}_{e}^{\top}\\\
\end{split}$ (15)

Here, $\mathbf{A}_{u}\mathbf{S}_{u}\mathbf{B}^{\top}$ can be calculated via
Alg. 3, and applying $\mathbf{\Pi}_{A}$ can be performed efficiently as
discussed.

Combining everything. Since we have different strategies for unpacking, let us
first define a unified interface in Alg. 5. One can verify that for any
strategies $s_{A}$:

$\begin{split}\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u},\mathbf{\Pi}_{A}&=\text{Unpack}(\mathbf{A},\mathbf{B},\mathbf{I},b,s_{A})\\\
\mathbf{A}\mathbf{B}^{\top}&=\mathbf{\Pi}_{A}\mathbf{A}_{u}\mathbf{S}_{u}\mathbf{B}_{e}^{\top}\end{split}$
(16)

In the previous discussion, $\mathbf{B}$ was assumed to have all IB values.
When $\mathbf{B}$ contains OB values, we note that $\mathbf{B}$ can be
unpacked in a similar manner, and the choice of unpacking strategies for
$\mathbf{B}$ is independent of the unpacking strategy for $\mathbf{A}$. For
example, $\mathbf{A}$ can be unpacked row-wise, while $\mathbf{B}$ is unpacked
column-wise. By taking the unpacked
$\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u},\mathbf{\Pi}_{A}$ from (16), we
can further unpack $\mathbf{B}$ using strategy $s_{B}$:

$\begin{split}\mathbf{B}_{eu},\mathbf{A}_{ue},\mathbf{S}_{uu},\mathbf{\Pi}_{B}&=\text{Unpack}(\mathbf{B}_{e},\mathbf{A}_{u},\mathbf{S}_{u},b,s_{B})\\\
\mathbf{A}\mathbf{B}^{\top}&=\mathbf{\Pi}_{A}\mathbf{A}_{ue}\mathbf{S}_{uu}\mathbf{B}_{eu}^{\top}\mathbf{\Pi}_{B}^{\top}\end{split}$
(17)

Here, values in both $\mathbf{A}_{ue}$ and $\mathbf{B}_{eu}$ are IB, and the
result can be obtained similar to discussion in Eq. (15).

Summary. We introduced three strategies to unpack a matrix to low bit-width
integer matrices for different structures of OB values in a matrix. While
these strategies work for arbitrary matrices, we can clearly see that these
unpacking strategies are most efficient when the OB values concentrate in a
few columns and rows. Luckily, the matrices of interest in Transformer models
indeed have this property, which is studied and exploited in several works [6,
28].

Algorithm 5 Unpack$(\mathbf{A},\mathbf{B},\mathbf{S},b,\text{strategy})$

1: if strategy is UnpackRow then

2: $\mathbf{A}_{u},\mathbf{\Pi}_{A}\leftarrow\text{UnpackRow}(\mathbf{A},b)$

3: $\mathbf{S}_{u},\mathbf{B}_{e}\leftarrow\mathbf{S},\mathbf{B}$

4: else if strategy is UnpackColumn then

5:
$\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u}\leftarrow\text{UnpackColumn}(\mathbf{A},\mathbf{B},\mathbf{S},b)$

6: $\mathbf{\Pi}_{A}\leftarrow\mathbf{I}$

7: else

8:
$\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u},\mathbf{\Pi}_{A}\leftarrow\text{UnpackBoth}(\mathbf{A},\mathbf{B},\mathbf{S},b)$

9: end if

10: return $\mathbf{A}_{u},\mathbf{B}_{e},\mathbf{S}_{u},\mathbf{\Pi}_{A}$

###  4.2 Evaluating Unpacking Overhead

Table 8: Averaged unpack ratios of each type of GEMMs in LLaMA-7B: linear layers (computing $\mathbf{Y}$), attention score (computing $\mathbf{P}$), and attention output (computing $\mathbf{O}$) when using different unpack strategies and integer bit-width $b$ under quantization $\beta$ settings. AS: Attention Score, AO: Attention Output.  $\beta$ | 5 | 15 | 31
---|---|---|---
Integer Bits $b$ | 3 | 4 | 5 | 4 | 5 | 6 | 5 | 6 | 7
Linear ($\mathbf{Y}$) |  $\mathbf{X}$ | Row |  $\mathbf{W}$ | Row | 2.67 | 1.93 | 1.57 | 2.47 | 2.02 | 1.73 | 2.12 | 2.00 | 1.74
Row | Col | 10.76 | 2.35 | 1.61 | 9.91 | 5.36 | 1.84 | 8.44 | 5.62 | 1.86
Row | Both | 5.46 | 1.95 | 1.57 | 5.15 | 2.20 | 1.73 | 4.71 | 2.24 | 1.75
Col | Row | 3.80 | 1.32 | 1.06 | 3.98 | 1.64 | 1.16 | 3.93 | 1.68 | 1.17
Col | Col | 15.40 | 1.62 | 1.09 | 16.00 | 4.01 | 1.25 | 15.69 | 4.33 | 1.27
Col | Both | 5.21 | 1.34 | 1.06 | 6.04 | 1.76 | 1.16 | 5.98 | 1.82 | 1.17
Mix | 2.6 | 1.27 | 1.06 | 2.44 | 1.4 | 1.15 | 2.1 | 1.42 | 1.16
AS ($\mathbf{P}$) |  $\mathbf{Q}$ | Row |  $\mathbf{K}$ | Row | 1.97 | 1.60 | 1.0 | 2.00 | 1.87 | 1.15 | 2.00 | 1.87 | 1.18
Row | Col | 3.22 | 1.64 | 1.0 | 5.35 | 2.07 | 1.17 | 5.36 | 2.09 | 1.20
Col | Row | 1.81 | 1.04 | 1.0 | 2.91 | 1.14 | 1.01 | 2.91 | 1.15 | 1.01
Col | Col | 3.36 | 1.08 | 1.0 | 8.66 | 1.32 | 1.03 | 8.67 | 1.35 | 1.03
Mix | 1.72 | 1.03 | 1.0 | 1.95 | 1.13 | 1.01 | 1.95 | 1.14 | 1.01
AO ($\mathbf{O}$) |  $\mathbf{M}$ | Row |  $\mathbf{V}$ | Row | 6.02 | 4.18 | 3.27 | 4.72 | 3.65 | 3.02 | 3.93 | 3.24 | 2.81
Row | Col | 15.10 | 4.53 | 3.35 | 18.21 | 4.64 | 3.16 | 15.07 | 4.21 | 2.95
Col | Row | 16.29 | 8.14 | 5.12 | 11.28 | 6.98 | 4.91 | 8.41 | 5.84 | 4.42
Col | Col | 42.21 | 8.76 | 5.21 | 43.57 | 9.11 | 5.09 | 32.31 | 7.74 | 4.61
Mix | 5.98 | 4.11 | 3.16 | 4.7 | 3.62 | 2.97 | 3.92 | 3.22 | 2.77
Table 9: Averaged unpack ratios of each type of quantized GEMMs in both forward and backward of a RoBERTa-Small when using different integer bit length $b$ at different training phrases of the $\beta=31$ experiment in Fig. 2. The optimal strategies (Mix as in Tab. 8) for each GEMM is used.  Progress | 1/3 | 2/3 | 3/3
---|---|---|---
Integer Bits $b$ | 5 | 6 | 7 | 5 | 6 | 7 | 5 | 6 | 7
Linear | $\mathbf{Y}$ | 2.00 | 1.31 | 1.08 | 2.00 | 1.32 | 1.07 | 2.00 | 1.32 | 1.05
$\nabla_{\mathbf{X}}$ | 1.50 | 1.31 | 1.15 | 1.50 | 1.30 | 1.16 | 1.50 | 1.30 | 1.15
$\nabla_{\mathbf{W}}$ | 1.98 | 1.25 | 1.04 | 1.98 | 1.25 | 1.03 | 1.98 | 1.25 | 1.03
AS | $\mathbf{P}$ | 1.66 | 1.04 | 1.00 | 1.42 | 1.05 | 1.0 | 1.40 | 1.04 | 1.00
$\nabla_{\mathbf{Q}}$ | 2.22 | 1.90 | 1.71 | 2.22 | 1.91 | 1.7 | 2.24 | 1.92 | 1.71
$\nabla_{\mathbf{K}}$ | 1.79 | 1.06 | 1.00 | 1.49 | 1.07 | 1.0 | 1.45 | 1.07 | 1.00
AO | $\mathbf{O}$ | 3.11 | 2.71 | 2.30 | 3.10 | 2.68 | 2.24 | 3.10 | 2.62 | 2.22
$\nabla_{\mathbf{M}}$ | 1.21 | 1.10 | 1.04 | 1.21 | 1.10 | 1.04 | 1.21 | 1.10 | 1.04
$\nabla_{\mathbf{V}}$ | 2.88 | 2.52 | 2.12 | 2.87 | 2.48 | 2.10 | 2.86 | 2.41 | 2.09
Table 10: Averaged ratios of quantized GEMMs ($\beta=15$) in linear layers on ViT-Large when using different strategies and a range of integer bit-widths $b$ to the lowest bit-width possible.  Integer Bits $b$ | 2 | 3 | 4 | 5 | 6 | 7
---|---|---|---|---|---|---
$\mathbf{X}$ | Row |  $\mathbf{W}$ | Row | 7.24 | 3.80 | 2.63 | 2.22 | 1.54 | 1.43
Row | Col | 194.89 | 27.52 | 10.46 | 4.31 | 1.62 | 1.43
Row | Both | 85.92 | 13.80 | 6.22 | 2.76 | 1.56 | 1.43
Col | Row | 19.27 | 4.85 | 3.06 | 1.46 | 1.25 | 1.12
Col | Col | 526.31 | 35.86 | 12.22 | 2.81 | 1.32 | 1.13
Col | Both | 27.06 | 13.31 | 7.59 | 1.78 | 1.26 | 1.13
Both | Row | 7.62 | 3.39 | 2.58 | 1.64 | 1.42 | 1.33
Both | Col | 206.32 | 24.45 | 10.27 | 3.15 | 1.49 | 1.34
Both | Both | 79.19 | 11.16 | 6.09 | 2.01 | 1.43 | 1.34
Mix | 6.29 | 2.98 | 2.24 | 1.40 | 1.23 | 1.11

The idea of IM-Unpack is to use more low bit-width arithmetic operations to
compute a high bit-width operation. As we see in the description of IM-Unpack
algorithm, the number of row and column vectors increases, so the unpacked
matrices $\mathbf{A}_{ue}$ and $\mathbf{B}_{eu}$ will have a larger size
compared to $\mathbf{A}$ and $\mathbf{B}$, which obviously increases the
computational cost of low bit-width GEMMs. In this subsection, we evaluate how
much this cost would increase. For two matrices $\mathbf{A}$ and $\mathbf{B}$,
the complexity of a GEMM is $\mathcal{O}(ndh)$. Similarly, let
$n^{\prime},d^{\prime}$ be the size of $\mathbf{A}_{ue}$ and $h^{\prime}$ be
the number of rows of $\mathbf{B}_{eu}$. The cost of
$\mathbf{A}_{ue}\mathbf{S}_{uu}\mathbf{B}_{eu}^{\top}$ is
$\mathcal{O}(n^{\prime}d^{\prime}h^{\prime})$, we can directly measure the
unpack ratio

$r=(n^{\prime}d^{\prime}h^{\prime})/(ndh)$ (18)

to understand by how much the cost for low bit-width GEMMs increases. We uses
LLaMA-7B to study the unpack ratio $r$ when using different unpacking
strategies (Tab. 8). Note that since unpacking both requires keeping track of
the OB count in each row and column vector which is not as fast as the other
two strategies, we only use it for unpacking parameters $\mathbf{W}$ for
inference since it can be performed once when loading the model. The Mix in
Tab. 8 means that for each GEMM, we compare different strategies and choose
the optimal strategy that results in the smallest unpack ratio. We note that
the unpack ratios of computing $\mathbf{Y}$ and $\mathbf{P}$ are quite
reasonable, but the ratios of computing $\mathbf{O}$ is larger. This is
expected since the large outliers of the self-attention matrix $\mathbf{M}$
mainly concentrate in the diagonal [3]. We also study the unpack ratios of
each type of quantized GEMMs at different training phases, and show the
results of Mix strategy in Tab. 9. The ratios stay relatively unchanged as
training progresses. Also, we can observe similar high unpack ratio when
computing $\mathbf{O}$ and $\nabla_{\mathbf{V}}$ since these GEMMs involve
self-attention matrix $\mathbf{M}$. Lastly, we verify that we can unpack
matrices to arbitrarily low integer matrices (Tab. 10). The 2-bit setting is
the lowest bit width that can be used for symmetric signed integers
($\\{-1,0,1\\}$).

##  5 Limitations

To simplify the presentation, we used the simplest RTN quantization, which
might not deliver the optimal performance. More sophisticated techniques are
likely to further improve the results. For example, we may be able to remove
the demands of large $\beta$ for the set
$\\{\nabla_{\mathbf{Y}},\nabla_{\mathbf{P}},\nabla_{\mathbf{O}}\\}$ for ViT
training. The current unpacking strategies cannot handle the self-attention
matrix $\mathbf{M}$ efficiently since the outliers mainly concentrate on the
diagonal region rather than rows or columns; this needs further study.

##  6 Conclusion

Figure 8: Illustration of the bit representation of a matrix. Heavy-hitters
have higher order non-zero bits. When a matrix contains heavy-hitters, its bit
representation has a sparsity structure in the higher order bits as
illustrated.

In this paper, we verify the efficacy of integer GEMMs in both training and
inference for Transformer-based models in language modeling and vision. A
simple RTN quantization strategy works well compared to baselines. But in this
setting, the presence of large outliers/heavy-hitters makes it difficult to
make use of efficient low bit-width integer GEMMs since these outliers are
much larger than the representable range of low bit-width integers. We take a
“multi-resolution” view in how we extract a spectrum of bit-width tradeoffs.
This is loosely similar to sparsity but here, instead of making a zero versus
non-zero distinction between the entries, our heavy-hitters (which need higher
bit-width representations) are analogous to “non-sparse” entries (as
illustrated in Fig. 8). To address the challenge of high bit-width heavy-
hitters, we develop an algorithm to unpack integer matrices that contains
arbitrarily large values to slightly larger matrices with the property that
all values lie within the representable range of low bit-width integers and a
procedure to obtain the GEMM output of original matrices using only low bit-
width integer GEMMs on the unpacked matrices followed by some scaling (using
bit shifting) and accumulation. Our algorithm can greatly simplify the design
of hardware and improve the power efficiency by only supporting low bit-width
integer GEMMs for both training and inference.

## References

  * [1] Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mojan Javaheripi, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang.  Phi-2: The surprising power of small language models, 2023.
  * [2] Ron Banner, Yury Nahshan, and Daniel Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [3] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer, 2020.
  * [4] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa.  QuIP: 2-bit quantization of large language models with guarantees.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * [6] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  GPT3.int8(): 8-bit matrix multiplication for transformers at scale.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [8] Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu.  Towards accurate post-training quantization for vision transformer.  In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 5380–5388, New York, NY, USA, 2022. Association for Computing Machinery.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [10] Wikimedia Foundation.  Wikimedia downloads.
  * [11] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  OPTQ: Accurate quantization for generative pre-trained transformers.  In The Eleventh International Conference on Learning Representations, 2023.
  * [12] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [13] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * [14] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.  I-bert: Integer-only bert quantization.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5506–5518. PMLR, 18–24 Jul 2021.
  * [15] Zhikai Li and Qingyi Gu.  I-vit: Integer-only quantization for efficient vision transformer inference.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17065–17075, 2023.
  * [16] Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu.  Repq-vit: Scale reparameterization for post-training quantization of vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17227–17236, 2023.
  * [17] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou.  Fq-vit: Post-training quantization for fully quantized vision transformer.  In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 1173–1179, 2022.
  * [18] Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng.  LLM-FP4: 4-bit floating-point quantized transformers.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 592–605, Singapore, December 2023. Association for Computational Linguistics.
  * [19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [20] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.  Llm-qat: Data-free quantization aware training for large language models, 2023.
  * [21] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling.  Data-free quantization through weight equalization and bias correction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334, 2019.
  * [22] Shashi Narayan, Shay B. Cohen, and Mirella Lapata.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  ArXiv, abs/1808.08745, 2018.
  * [23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [25] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan.  Training deep neural networks with 8-bit floating point numbers.  Advances in neural information processing systems, 31, 2018.
  * [26] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt.  Stable and low-precision training for large-scale vision-language models.  arXiv preprint arXiv:2304.13013, 2023.
  * [27] Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi.  Training and inference with integers in deep neural networks.  In International Conference on Learning Representations, 2018.
  * [28] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  SmoothQuant: Accurate and efficient post-training quantization for large language models.  In Proceedings of the 40th International Conference on Machine Learning, 2023.
  * [29] Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun.  Ptq4vit: Post-training quantization framework for vision transformers with twin uniform quantization, 2022.
  * [30] Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan.  Towards unified int8 training for convolutional neural network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969–1979, 2020.

##  7 Appendices

In this section, we provide more details about design choices and experiment
setups as well as more experiments that are left out in the main text.

###  7.1 Why Using Percentiles?

Table 11: Standard deviation vs percentile when removing largest outliers. $\mathbf{X}$ has $2.25\times 10^{7}$ entries and $\mathbf{W}$ has $1.68\times 10^{7}$ entries.  Outliers Removed | $0$ | $10$ | $10^{2}$ | $10^{3}$
---|---|---|---|---
$\mathbf{W}$ | Standard Deviation | 0.0082 | 0.0082 | 0.0082 | 0.0082
95-Percentile | 0.0177 | 0.0177 | 0.0177 | 0.0177
$\mathbf{X}$ | Standard Deviation | 0.0330 | 0.0327 | 0.0320 | 0.0214
95-Percentile | 0.0280 | 0.0280 | 0.0280 | 0.0278

We need a way of mapping the actual range of values in a floating point matrix
to an integer range, and ensure most values fall into the desired range and
fill up the representable range as much as possible, so we need a statistic to
estimate the range of values in a FP matrix. We compared percentile and
standard deviation. We inspected different parameter matrices $\mathbf{W}$ and
the corresponding inputs $\mathbf{X}$ in the LLaMA-7B model [24]. While the
outlier problem in $\mathbf{W}$ is moderate, and both standard deviation and
percentile work well, the outliers in $\mathbf{X}$ is problematic and contains
a few values that are much larger than non-outliers. The estimation of
standard deviation might be severely impact the extreme outliers in
$\mathbf{X}$ as shown in Tab. 11: removing an extremely small subset of the
largest outliers can severely alter the estimates. On contrast, percentile is
more robust to the extreme outliers. As a result, we choose percentile as the
estimation of value range.

###  7.2 Baseline Comparison when Quantize Parameters Only

Table 12: Baseline comparison on LLaMA-7B and ViTs when only quantize
parameters. HS: HellaSwag, WG: WinoGrande

LLaMA-7B | Method | $\beta$ | $\overline{\text{Bits}}$ | ARC-c | ARC-e | BoolQ | HS | PIQA | WG
---|---|---|---|---|---|---|---|---|---
Full-Precision | - | 16 | 43.1 | 76.3 | 77.8 | 57.2 | 78.0 | 68.8
GPTQ | - | 4 | 37.4 | 72.7 | 73.3 | 54.9 | 77.9 | 67.9
LLM-FP4 | - | 4 | 40.4 | 74.9 | 74.2 | 55.8 | 77.8 | 69.9
QuIP | - | 2 | 22.3 | 42.8 | 50.3 | 34.0 | 61.8 | 52.6
RTN+HE | 5 | 2.5 | 39.3 | 72.8 | 69.9 | 53.4 | 74.9 | 66.4
| 7 | 2.9 | 42.6 | 73.9 | 72.3 | 55.9 | 77.0 | 67.4
| 11 | 3.5 | 43.9 | 76.1 | 77.3 | 56.3 | 77.3 | 69.3
| 15 | 4.0 | 43.0 | 75.7 | 77.5 | 57.0 | 78.0 | 69.2
| 31 | 5.0 | 42.7 | 76.1 | 76.1 | 57.3 | 77.3 | 69.3

ViT | Method | $\beta$ | $\overline{\text{Bits}}$ | Tiny | Small | Base | Large | Huge
---|---|---|---|---|---|---|---|---
Full-Precision | - | 32 | 75.5 | 81.4 | 85.1 | 85.8 | 87.6
PTQ4ViT | - | 3 | 18.3 | 36.2 | 21.4 | 81.3 | 78.9
RTN+HE | 3 | 1.8 | 0.5 | 8.3 | 63.6 | 81.9 | 83.3
| 5 | 2.4 | 38.2 | 69.0 | 81.1 | 84.9 | 86.7
| 7 | 2.9 | 63.6 | 76.7 | 83.6 | 85.4 | 87.2
| 15 | 4.0 | 73.4 | 80.5 | 84.8 | 85.7 | 87.6

One direction of quantization research focus on quantizing the parameters for
better storage and memory usage. We also evaluate how well RTN works on
storage and memory efficiency. After quantization, the quantized
$\mathbf{W}_{q}$ usually contains a few hundreds of distinct integers. Simply
representing $\mathbf{W}_{q}$ in plain integer format would not be efficient
and usually requires larger than 8 bits per value for memory. By inspecting
the value distribution of $\mathbf{W}_{q}$, we found that the fewer values
occur much more frequently than others, which create a clear opportunity for
compression. We simply apply Huffman Encoding (HE), which was also in [12] to
compress models for memory efficiency, to use shorter encoding for more
frequent values. As shown in Table 12, with RTN and HE, we are able to
significantly reduce the average bites per value with small or no performance
degradation and result in significantly better efficiency compared to
baselines [11, 4, 18, 29] for both Transformer based LLMs and Vision
Transformers.

###  7.3 Details of Training Experiments

We run all of our experiments on NVIDIA RTX 3090’s. The following are training
hyperparameters.

RoBERTa. The RoBERTa-Small is a 4-layer Transformer encoder whose model
dimension is 512, hidden dimension is 2048, and number of heads is 8. For
Small models, we train each model for 200K steps with batches of $256$
$512$-length sequences. We use an AdamW optimizer with 1e-4 learning rate,
10,000 warm-up steps, 0.01 weight decay, and linear decay. For Base models, we
train each model for 300K steps with batches of $128$ $512$-length sequences.
We use an AdamW optimizer with 5e-5 learning rate, 10,000 warm-up steps, 0.01
weight decay, and linear decay.

ViT. We use timm to train our ViT-Small models. The hyperparameters of all
experiments are the same: batch size 1024, optimizer AdamW, learning rate
0.001, weight decay 0.05, augmentation rand-m9-mstd0.5-inc1, mixup 0.8, cutmix
1.0.

###  7.4 Unpack Ratios of ViT-Large

Table 13: Averaged unpack ratios of each type of GEMMs in ViT-Large: linear layers (computing $\mathbf{Y}$), attention score (computing $\mathbf{P}$), and attention output (computing $\mathbf{O}$) when using different unpack strategies and integer bit length $b$ under quantization $\beta$ settings. AS: Attention Score, AO: Attention Output.  $\beta$ | 5 | 7 | 15
---|---|---|---
Integer Bits $b$ | 3 | 4 | 5 | 3 | 4 | 5 | 4 | 5 | 6
Linear ($\mathbf{Y}$) |  $\mathbf{X}$ | Row |  $\mathbf{W}$ | Row | 2.90 | 2.00 | 1.55 | 3.01 | 2.38 | 1.59 | 2.63 | 2.22 | 1.54
Row | Col | 10.97 | 2.32 | 1.56 | 12.34 | 4.12 | 1.65 | 10.46 | 4.31 | 1.62
Row | Both | 6.24 | 2.08 | 1.55 | 6.84 | 2.82 | 1.60 | 6.22 | 2.76 | 1.56
Col | Row | 2.33 | 1.39 | 1.20 | 3.38 | 1.51 | 1.26 | 3.06 | 1.46 | 1.25
Col | Col | 8.89 | 1.64 | 1.22 | 13.97 | 2.63 | 1.32 | 12.22 | 2.81 | 1.32
Col | Both | 4.99 | 1.44 | 1.20 | 7.99 | 1.76 | 1.27 | 7.59 | 1.78 | 1.26
Mix | 2.60 | 1.27 | 1.06 | 2.44 | 1.40 | 1.15 | 2.10 | 1.42 | 1.16
AS ($\mathbf{P}$) |  $\mathbf{Q}$ | Row |  $\mathbf{K}$ | Row | 1.84 | 1.07 | 1.00 | 2.01 | 1.35 | 1.00 | 1.99 | 1.40 | 1.00
Row | Col | 3.06 | 1.07 | 1.00 | 6.38 | 1.39 | 1.00 | 6.34 | 1.46 | 1.00
Col | Row | 1.34 | 1.01 | 1.00 | 2.50 | 1.04 | 1.00 | 2.49 | 1.05 | 1.00
Col | Col | 2.39 | 1.01 | 1.00 | 8.25 | 1.08 | 1.00 | 8.24 | 1.10 | 1.00
Mix | 1.33 | 1.01 | 1.00 | 1.91 | 1.04 | 1.00 | 1.90 | 1.04 | 1.00
AO ($\mathbf{O}$) |  $\mathbf{M}$ | Row |  $\mathbf{V}$ | Row | 2.84 | 2.07 | 1.65 | 3.07 | 2.24 | 1.80 | 2.56 | 2.11 | 1.78
Row | Col | 5.78 | 2.12 | 1.65 | 11.12 | 2.47 | 1.81 | 9.22 | 2.35 | 1.79
Col | Row | 3.98 | 2.26 | 1.64 | 4.69 | 2.57 | 1.81 | 3.58 | 2.33 | 1.77
Col | Col | 8.42 | 2.29 | 1.64 | 16.97 | 2.83 | 1.81 | 12.92 | 2.60 | 1.77
Mix | 2.25 | 1.61 | 1.32 | 2.55 | 1.77 | 1.42 | 2.22 | 1.70 | 1.42

Similar to Tab. 8 in the main text, we also evaluate the unpack ratios of ViT-
Large, which are shown in Tab. 13. The overall results are similar to what was
observed in unpack ratios of LLaMA-7B (Tab. 8).

###  7.5 More Empirical Results on LLM Quantization

Table 14: Baseline comparison on LLaMA-13B when quantize computation in all linear layers. Method | $\beta$ | Type | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
---|---|---|---|---|---|---|---|---
Full-Precision | - | BF16 | 48.0 | 79.5 | 80.6 | 60.0 | 79.2 | 72.1
SmoothQuant | - | INT8 | 45.5 | 76.3 | 76.5 | 58.0 | 78.0 | 72.1
| - | INT4 | 25.1 | 49.9 | 57.6 | 56.0 | 61.3 | 52.6
LLM-FP4 | - | FP4 | 39.9 | 71.7 | 71.9 | 53.3 | 74.8 | 66.7
RTN | 5 | INT | 37.6 | 70.0 | 69.1 | 51.9 | 72.4 | 64.6
| 7 | INT | 44.1 | 76.1 | 73.5 | 57.3 | 76.7 | 67.6
| 15 | INT | 46.9 | 78.8 | 79.4 | 59.0 | 78.2 | 72.5
| 31 | INT | 48.0 | 79.7 | 80.2 | 59.9 | 78.0 | 71.3
Table 15: Baseline comparison on LLaMA-13B when quantize all GEMMs in a Transformer. Method | $\beta$ | Type | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
---|---|---|---|---|---|---|---|---
Full-Precision | - | BF16 | 48.0 | 79.5 | 80.6 | 60.0 | 79.2 | 72.1
RTN | 5 | INT | 25.1 | 44.4 | 54.8 | 37.7 | 57.9 | 52.0
| 7 | INT | 38.0 | 66.9 | 70.1 | 53.3 | 72.5 | 64.2
| 15 | INT | 45.9 | 77.6 | 80.0 | 59.5 | 77.5 | 71.5
| 31 | INT | 47.9 | 79.3 | 80.0 | 60.5 | 78.6 | 70.9
Table 16: RTN performance on Mistral-7B and Phi-2 when quantize computation in all linear layers.  | Method | $\beta$ | ARC-c | ARC-e | BoolQ | HellaSwag | PIQA | WinoGrande
---|---|---|---|---|---|---|---|---
Mistral-7B | Full-Precision | - | 50.3 | 80.9 | 83.6 | 61.3 | 80.7 | 73.8
RTN | 5 | 38.1 | 70.5 | 69.9 | 53.9 | 73.3 | 61.4
| 7 | 44.9 | 75.0 | 76.0 | 58.7 | 77.8 | 68.6
| 15 | 48.8 | 79.7 | 80.3 | 60.8 | 79.6 | 73.2
| 31 | 50.3 | 80.1 | 83.5 | 61.5 | 80.7 | 74.4
Phi-2 | Full-Precision | - | 20.6 | 26.1 | 41.3 | 25.8 | 54.3 | 49.3
RTN | 5 | 22.1 | 26.7 | 41.5 | 25.6 | 52.3 | 48.1
| 7 | 21.3 | 25.8 | 40.9 | 25.8 | 53.9 | 49.5
| 15 | 21.3 | 27.3 | 45.4 | 25.8 | 53.4 | 48.8
| 31 | 21.0 | 25.8 | 40.8 | 25.7 | 53.0 | 50.7

To evaluate how well RTN works on the inference of different models and
different model sizes, beside the experiments shown in the main text, we also
run experiments on LLaMA-13B [24], Mistral-7B [13], and Phi-2 [1]. The results
are summarized in Tab. 14, Tab. 15, and Tab. 16. To minimize code change, we
only evaluate the quantization of linear layers, as in many quantization
works, for Mistral-7B and and Phi-2.

###  7.6 More Empirical Results on Training

Figure 9: Loss curves of T5-Large finetuning on 1/4 of XSum dataset for 1 epoch.  Table 17: Validation metrics of T5-Large finetuning on 1/4 of XSum dataset for 1 epoch.  Method | $\beta$ | Type | Loss | Rouge1 | Rouge2 | Rougel | Rougelsum
---|---|---|---|---|---|---|---
Full-Precision | - | BF16 | 1.65 | 36.12 | 13.00 | 29.21 | 29.20
RTN | 31 | INT | 1.66 | 36.03 | 13.83 | 29.03 | 29.04

To understand of how well RTN works on the training for larger models without
using too much compute, we finetune a T5-Large model on the first 50K instance
of XSum summarization dataset [22] using BF16 and RTN, and show the results in
Fig. 9. The validation metrics are shown in Tab. 17. We could draw a similar
conclusion as in the main text that RTN quantized training gives similar
results as BF16 training.