11institutetext: Heidelberg Collaboratory for Image Processing,
IWR, Heidelberg University, Germany
11email<EMAIL_ADDRESS>
# A Style-Aware Content Loss for
Real-time HD Style Transfer

Artsiom Sanakoyeu Both authors contributed equally to this work. Dmytro
Kotovenko††footnotemark:  Sabine Lang  Björn Ommer

###### Abstract

Recently, style transfer has received a lot of attention. While much of this
research has aimed at speeding up processing, the approaches are still lacking
from a principled, art historical standpoint: a style is more than just a
single image or an artist, but previous work is limited to only a single
instance of a style or shows no benefit from more images. Moreover, previous
work has relied on a direct comparison of art in the domain of RGB images or
on CNNs pre-trained on ImageNet, which requires millions of labeled object
bounding boxes and can introduce an extra bias, since it has been assembled
without artistic consideration. To circumvent these issues, we propose a
style-aware content loss, which is trained jointly with a deep encoder-decoder
network for real-time, high-resolution stylization of images and videos. We
propose a quantitative measure for evaluating the quality of a stylized image
and also have art historians rank patches from our approach against those from
previous work. These and our qualitative results ranging from small image
patches to megapixel stylistic images and videos show that our approach better
captures the subtle nature in which a style affects content.111Project page:
https://compvis.github.io/adaptive-style-transfer.

###### Keywords:

Style transfer, generative network, deep learning

##  1 Introduction

|  |  |  |
---|---|---|---|---
Figure 1: Evaluating the fine details preserved by our approach. Can you guess which of the cut-outs are from Monet’s artworks and which are generated? Solution is on p. 3.3. | [13] | [13] |  [13] on collection |  [24] on collection | Ours on collection
---|---|---|---|---|---
Content |  |  |  |  |
|  |  |  |  |
|  |  |  |  |
| (a) | (b) | (c) | (d) | (e)
Figure 2: Style transfer using different approaches on $1$ and a collection of
reference style images. (a) [13] using van Gogh’s ”Road with Cypress and Star”
as reference style image; (b) [13] using van Gogh’s ”Starry night”; (c) [13]
using the average Gram matrix computed across the collection of Vincent van
Gogh’s artworks; (d) [24] trained on the collection of van Gogh’s artworks
alternating target style images every SGD mini-batch; (e) our approach trained
on the same collection of van Gogh’s artworks. Stylizations (a) and (b) depend
significantly on the particular style image, but using a collection of the
style images (c), (d) does not produce visually plausible results, due to
oversmoothing over the numerous Gram matrices. In contrast, our approach (e)
has learned how van Gogh is altering particular content in a specific manner
(edges around objects also stylized, cf. bell tower)

A picture may be worth a thousand words, but at least it contains a lot of
very diverse information. This not only comprises _what_ is portrayed, e.g.,
composition of a scene and individual objects, but also _how_ it is depicted,
referring to the artistic style of a painting or filters applied to a photo.
Especially when considering artistic images, it becomes evident that not only
content but also style is a crucial part of the message an image communicates
(just imagine van Gogh’s Starry Night in the style of Pop Art). Here, we
follow the common wording of our community and refer to ’content’ as a synonym
for ’subject matter’ or ’sujet’, preferably used in art history. A vision
system then faces the challenge to decompose and separately represent the
content and style of an image to enable a direct analysis based on each
individually. The ultimate test for this ability is style transfer [13] –
exchanging the style of an image while retaining its content.

In contrast to the seminal work of Gatys et al. [13], who have relied on
powerful but slow iterative optimization, there has recently been a focus on
feed-forward generator networks [24, 47, 42, 44, 29, 7, 21]. The crucial
representation in all these approaches has been based on a VGG16 or VGG19
network [41], pre-trained on ImageNet [36]. However, a recent trend in deep
learning has been to avoid supervised pre-training on a million images with
tediously labeled object bounding boxes [46]. In the setting of style transfer
this has the particular benefit of avoiding from the outset any bias
introduced by ImageNet, which has been assembled without artistic
consideration. Rather than utilizing a separate pre-trained VGG network to
measure and optimize the quality of the stylistic output [13, 24, 47, 42, 44,
29, 7], we employ an encoder-decoder architecture with adversarial
discriminator, Fig. 3, to stylize the input content image and also use the
encoder to measure the reconstruction loss. In essence the stylized output
image is again run through the encoder and compared with the encoded input
content image. Thus, we learn a style-specific content loss from scratch,
which adapts to the specific way in which a particular style retains content
and is more adaptive than a comparison in the domain of RGB images [52].

Most importantly, however, previous work has only been based on a _single_
style image. This stands in stark contrast to art history which understands
”style as an expression of a collective spirit” resulting in a ”distinctive
manner which permits the grouping of works into related categories” [10]. As a
result, art history developed a scheme, which allows to identify groups of
artworks based on shared qualities. Artistic style consists of a diverse range
of elements, such as form, color, brushstroke, or use of light. Therefore, it
is insufficient to only use a single artwork, because it might not represent
the full scope of an artistic style. Today, freely available art datasets such
as Wikiart [25] easily contain more than 100K images, thus providing numerous
examples for various styles. Previous work [13, 24, 47, 42, 44, 29, 7] has
represented style based on the Gram matrix, which captures highly image-
specific style statistics, cf. Fig. 2. To combine several style images in [13,
24, 47, 42, 44, 29, 7] one needs to aggregate their Gram matrices. We have
evaluated several aggregation strategies and averaging worked the best, Fig.
2(c). But, obviously, neither art history, nor statistics suggests aggregating
Gram matrices. Additionally, we investigated alternating the target style
images in every mini-batch while training [24], Fig. 2(d). However, all these
methods cannot make proper use of several style images, because combining the
Gram matrices of several images forfeits the details of style, cf. the
analysis in Fig. 2. In contrast, our proposed approach allows to combine an
arbitrary number of instances of a style during training.

We conduct extensive evaluations of the proposed style transfer approach; we
quantitatively and qualitatively compare it against numerous baselines. Being
able to generate high quality artistic works in high-resolution, our approach
produces visually more detailed stylizations than the current state of the art
style transfer approaches and yet shows real-time inference speed. The results
are quantitatively validated by experts from art history and by adopting in
this paper a _deception rate_ metric based on a deep neural network for artist
classification.

###  1.1 Related Work

In recent years, a lot of research efforts have been devoted to texture
synthesis and style transfer problems. Earlier methods [18] are usually non-
parametric and are build upon low-level image features. Inspired by Image
Analogies [18], approaches [11, 30, 39, 40] are based on finding dense
correspondence between content and style image and often require image pairs
to depict similar content. Therefore, these methods do not scale to the
setting of arbitrary content images.

In contrast, Gatys et al. [13, 12] proposed a more flexible iterative
optimization approach based on a pre-trained VGG19 network [41]. This method
produces high quality results and works on arbitrary inputs, but is costly,
since each optimization step requires a forward and backward pass through the
VGG19 network. Subsequent methods [24, 42, 27] aimed to accelerate the
optimization procedure [13] by approximating it with feed-forward
convolutional neural networks. This way, only one forward pass through the
network is required to generate a stylized image. Beyond that, a number of
methods have been proposed to address different aspects of style transfer,
including quality [14, 49, 4, 47, 23], diversity [28, 44], photorealism [32],
combining several styles in a single model [45, 7, 3] and generalizing to
previously unseen styles [21, 29, 15, 38]. However, all these methods rely on
the fixed style representation which is captured by the features of a VGG [41]
network pre-trained on ImageNet. Therefore they require a supervised pre-
training on millions of labeled object bounding boxes and have a bias
introduced by ImageNet, because it has been assembled without artistic
consideration. Moreover, the image quality achieved by the costly optimization
in [13] still remains an upper bound for the performance of recent methods.
Other works like [48, 5, 34, 9, 1] learn how to discriminate different
techniques, styles and contents in the latent space.

Figure 3: Encoder-decoder network for style transfer based on style-aware
content loss.

Zhu et al. [52] learn a bidirectional mapping between a domain of content
images and paintings using generative adversarial networks. Employing cycle
consistency loss, they directly measure the distance between a backprojection
of the stylized output and the content image in the RGB pixel space. Measuring
distances in the RGB image domain is not just generally prone to be coarse,
but, especially for abstract styles, a pixel-wise comparison of backwards
mapped stylized images is not suited. Then, either content is preserved and
the stylized image is not sufficiently abstract, e.g., not altering object
boundaries, or the stylized image has a suitable degree of abstractness and so
a pixel-based comparison with the content image must fail. Moreover, the more
abstract the style is, the more potential backprojections into the content
domain exist, because this mapping is underdetermined (think of the many
possible content images for a single cubistic painting). In contrast, we spare
the ill-posed backward mapping of styles and compare stylized and content
images in the latent space which is trained jointly with the style transfer
network. Since both content and stylized images are run through our encoder,
the latent space is trained to only pay attention to the commonalities, i.e.,
the content present in both. Another consequence of the cycle consistency loss
is that it requires content and style images used for training to represent
similar scenes [52], and thus training data preparation for [52] involves
tedious manual filtering of samples, while our approach can be trained on
arbitrary unpaired content and style images.

##  2 Approach

Content | (a) Pollock | (b) El-Greco | (c) Gauguin | (d) Cézanne
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 4: $1$st row - results of style transfer for different styles. $2$nd
row - sketchy content visualization reconstructed from the latent space $E(x)$
using method of [33]. (a) The encoder for Pollock does not preserve much
content due to the abstract style; (b) only rough structure of the content is
preserved (coarse patches) because of the distinct style of El Greco; (c)
latent space highlights surfaces of the same color and that fine object
details are ignored, since Gauguin was less interested in details, often
painted plain surfaces and used vivid colors; (d) encodes the thick, wide
brushstrokes Cézanne used, but preserves a larger palette of colors.

To enable a fast style transfer that instantly transfers a content image or
even frames of a video according to a particular style, we need a feed-forward
architecture [24] rather than the slow optimization-based approach of [13]. To
this end, we adopt an encoder-decoder architecture that utilizes an encoder
network $E$ to map an input content image $x$ onto a latent representation
$z=E(x)$. A generative decoder $G$ then plays the role of a painter and
generates the stylized output image $y=G(z)$ from the sketchy content
representation $z$. Stylization then only requires a single forward pass, thus
working in real-time.

###  2.1 Training with a Style-Aware Content Loss

Previous approaches have been limited in that training worked only with a
single style image [13, 24, 21, 29, 47, 7, 42] or that style images used for
training had to be similar in content to the content images [52]. In contrast,
given a single style image $y_{0}$ we include a set $Y$ of related style
images $y_{j}\in Y$, which are automatically selected (see Sec. 2.2) from a
large art dataset (Wikiart). We do _not_ require the $y_{j}$ to depict similar
content as the set $X$ of arbitrary content images $x_{i}\in X$, which we
simply take from Places365 [51]. Compared to [52], we thus can utilize
standard datasets for content and style and need no tedious manual selection
of the $x_{i}$ and $y_{j}$ as described in Sect. 5.1 and 7.1 of [52].

To train $E$ and $G$ we employ a standard adversarial discriminator $D$ [16]
to distinguish the stylized output $G(E(x_{i}))$ from real examples $y_{j}\in
Y$,

$\mathcal{L}_{D}(E,G,D)=\mathop{\mathbb{E}}_{y\sim
p_{Y}(y)}\left[\log{D(y)}\right]+\mathop{\mathbb{E}}_{x\sim
p_{X}(x)}\left[\log{\left(1-D(G(E(x)))\right)}\right]$ (1)

However, the crucial challenge is to decide which details to retain from the
content image, something which is not captured by Eq. 1. Contrary to previous
work, we want to directly enforce $E$ to strip the latent space of all image
details that the target style disregards. Therefore, the details that need to
be retained or ignored in $z$ depend on the style. For instance, Cubism would
disregard texture, whereas Pointillism would retain low-frequency textures.
Therefore, a pre-trained network or fixed similarity measure [13] for
measuring the similarity in content between $x_{i}$ and $y_{i}$ is violating
the art historical premise that the manner, in which content is preserved,
depends on the style. Similar issues arise when measuring the distance after
projecting the stylized image $G(E(x_{i}))$ back into the domain $X$ of
original images with a second pair of encoder and decoder
$G_{2}(E_{2}(G(E(x_{i}))))$. The resulting loss proposed in [52],

$\mathcal{L}_{cycleGAN}=\mathop{\mathbb{E}}_{x\sim p_{X}(x)}\left[\left\lVert
x-G_{2}(E_{2}(G(E(x))))\right\rVert_{1}\right],$ (2)

fails where styles become abstract, since the backward projection of abstract
art to the original image is highly underdetermined.

Therefore, we propose a style-aware content loss that is being optimized,
while the network learns to stylize images. Since encoder training is coupled
with training of the decoder, which produces artistic images of the specific
style, the latent vector $z$ produced for the input image $x$ can be viewed as
its style-dependent sketchy content representation. This latent space
representation is changing during training and hence _adapts_ to the style.
Thus, when measuring the similarity in content between input image $x_{i}$ and
the stylized image $y_{i}=G(E(x_{i}))$ in the latent space, we focus only on
those details which are relevant for the style. Let the latent space have $d$
dimensions, then we define a style-aware content loss as normalized squared
Euclidean distance between $E(x_{i})$ and $E(y_{i})$:

$\mathcal{L}_{c}(E,G)=\mathop{\mathbb{E}}_{x\sim
p_{X}(x)}\left[\frac{1}{d}\left\lVert
E(x)-E(G(E(x)))\right\rVert_{2}^{2}\right]$ (3)

To show the additional intuition behind the style-aware content loss we used
the method [33] to reconstruct the content image from latent representations
trained on different styles and illustrated it in Fig. 4. It can be seen that
latent space encodes a sketchy, style-specific visual content, which is
implicitly used by the loss function. For example, Pollock is famous for his
abstract paintings, so reconstruction (a) shows that the latent space ignores
most of the object structure; Gauguin was less interested in details, painted
a lot of plain surfaces and used vivid colors which is reflected in the
reconstruction (c), where latent space highlights surfaces of the same color
and fine object details are ignored.

Since we train our model for altering the artistic style without supervision
and from scratch, we now introduce extra signal to initialize training and
boost the learning of the primary latent space. The simplest thing to do is to
use an autoencoder loss which computes the difference between $x_{i}$ and
$y_{i}$ in the RGB space. However, this loss would impose a high penalty for
any changes in image structure between input $x_{i}$ and output $y_{i}$,
because it relies only on low-level pixel information. But we aim to learn
image stylization and want the encoder to discard certain details in the
content depending on style. Hence the autoencoder loss will contradict with
the purpose of the style-aware loss, where the style determines which details
to retain and which to disregard. Therefore, we propose to measure the
difference after applying a weak image transformation on $x_{i}$ and $y_{i}$,
which is learned while learning E and G. We inject in our model a transformer
block $\mathbf{T}$ which is essentially a one-layer fully convolutional neural
network taking an image as input and producing a transformed image of the same
size. We apply $T$ to images $x_{i}$ and $y_{i}=G(E(x_{i}))$ before measuring
the difference. We refer to this as _transformed image loss_ and define it as

$\mathcal{L}_{\mathbf{T}}(E,G)=\mathop{\mathbb{E}}_{x\sim
p_{X}(x)}\left[\frac{1}{CHW}||\mathbf{T}(x)-\mathbf{T}(G(E(x))||_{2}^{2}\right],$
(4)

where $C\times H\times W$ is the size of image $x$ and for training $T$ is
initialized with uniform weights.

Fig. 3 illustrates the full pipeline of our approach. To summarize, the full
objective of our model is:

$\mathcal{L}(E,G,D)=\mathcal{L}_{c}(E,G)+\mathcal{L}_{t}(E,G)+\lambda\mathcal{L}_{D}(E,G,D),$
(5)

where $\lambda$ controls the relative importance of adversarial loss. We solve
the following optimization problem:

$E,G=\arg\;\min\limits_{E,G}\;\max\limits_{D}\mathcal{L}(E,G,D).$ (6)

###  2.2 Style Image Grouping

In this section we explain an automatic approach for gathering a set of
related style images. Given a single style image $y_{0}$ we strive to find a
set $Y$ of related style images $y_{j}\in Y$. Contrary to [52] we avoid
tedious manual selection of style images and follow a fully automatic
approach. To this end, we train a VGG$16$ [41] network $C$ from scratch on the
Wikiart [25] dataset to predict an artist given the artwork. The network is
trained on the $624$ largest (by number of works) artists from the Wikiart
dataset. Note that our ultimate goal is stylization and numerous artists can
share the same style, e.g., Impressionism, as well as a single artist can
exhibit different styles, such as the different stylistic periods of Picasso.
However, we do _not_ use any _style_ labels. Artist classification in this
case is the surrogate task for learning meaningful features in the artworks’
domain, which allows to retrieve similar artworks to image $y_{0}$.

Let $\phi(y)$ be the activations of the fc$6$ layer of the VGG$16$ network $C$
for input image $y$. To get a set of related style images to $y_{0}$ from the
Wikiart dataset $\mathcal{Y}$ we retrieve all nearest neighbors of $y_{0}$
based on the cosine distance $\delta$ of the activations $\phi(\cdot)$, i.e.

$Y=\\{y\;|\;y\in\mathcal{Y},\delta(\phi(y),\phi(y_{0}))<t\\},$ (7)

where $\delta(a,b)=1+\frac{\phi(a)\phi(b)}{||a||_{2}||b||_{2}}$ and $t$ is the
$10\%$ quantile of all pairwise distances in the dataset $\mathcal{Y}$.

##  3 Experiments

To compare our style transfer approach with the state-of-the-art, we first
perform extensive qualitative analysis, then we provide quantitative results
based on the _deception score_ and evaluations of experts from art history.
Afterwards in Sect. 3.3 we ablate single components of our model and show
their importance.
Implementation details: The basis for our style transfer model is an encoder-
decoder architecture, cf. [24]. The encoder network contains $5$ conv layers:
$1\times$conv-stride-1 and $4\times$conv-stride-2. The decoder network has $9$
residual blocks [17], $4$ upsampling blocks and $1\times$conv-stride-1. For
upsampling blocks we used a sequence of nearest-neighbor upscaling and conv-
stride-$1$ instead of fractionally strided convolutions [31], which tend to
produce heavier artifacts [35]. Discriminator is a fully convolutional network
with $7\times$conv-stride-2 layers. For a detailed network architecture
description we refer to the supplementary material. We set $\lambda=0.001$ in
Eq. 5. During the training process we sample $768\times 768$ content image
patches from the training set of Places365 [51] and $768\times 768$ style
image patches from the Wikiart [25] dataset. We train for $300000$ iterations
with batch size $1$, learning rate $0.0002$ and Adam [26] optimizer. The
learning rate is reduced by a factor of $10$ after $200000$ iterations.
Baselines: Since we aim to generate high-resolution stylizations, for
comparison we run style transfer on our method and all baselines for input
images of size $768\times 768$, unless otherwise specified. We did not not
exceed this resolution when comparing, because some other methods were
reaching the GPU memory limit. We optimize Gatys et al. [13] for $500$
iterations using L-BFGS. For Johnson et al. [24] we used the implementation of
[8] and trained a separate network for every reference style image on the same
content images from Places365 [51] as our method. For Huang et al. [21], Chen
et al. [4] and Li et al. [29] implementations and pre-trained models provided
by the authors were used. Zhu et al. [52] was trained on exactly the same
content and style images as our approach using the source code provided by the
authors. Methods [13, 24, 21, 4, 29] utilized only one example per style, as
they cannot benefit from more (cf. the analysis in Fig. 2).

###  3.1 Qualitative Results

Style |  Content |  (a) Ours |  (b) [13] |  (c) [52] |  (d) [4] |  (e) [21] |  (f) [29] |  (g) [24]
---|---|---|---|---|---|---|---|---

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

Figure 5: Results from different style transfer methods. We compare methods on
different styles and content images.

Full image stylization: In Fig. 5 we demonstrate the effectiveness of our
approach for stylizing different contents with various styles. Chen et al. [4]
work on the overlapping patches extracted from the content image, swapping the
features of the original patch with the features of the most similar patch in
the style image, and then averages the features in the overlapping regions,
thus producing an over-smoothed image without fine details (Fig. 5(d)). [21]
produces a lot of repetitive artifacts, especially visible on flat surfaces,
cf. Fig. 5(e, rows 1, 4–6). Method [29] fails to understand the content of the
image and applies different colors in the wrong locations (Fig. 5(f)). Methods
[24, 52] often fail to alter content image and their effect may be
characterized as shifting the color histogram, e.g., Fig. 5(g, rows 3, 7; c,
rows 1, 3–4). One reason for such failure cases of [52] is the loss in the RGB
pixel space based on the difference between a backward mapping of the stylized
output and the content image. Another reason for this is that we utilized the
standard Places365 [51] dataset and did not hand-pick training content images,
as is advised for [52]. Thus, artworks and content images used for training
differed significantly in their content, which is the ultimate test for a
stylization that truly alters the input and goes beyond a direct mapping
between regions of content and style images. The optimization-based method
[13] often works better than other baselines, but produces a lot of prominent
artifacts, leading to details of stylizations looking unnatural, cf. Fig. 5(b,
rows 4, 5, 6). This is due to an explicit minimization of the loss directly on
the pixel level. In contrast to this, our model can not only handle styles,
which have salient, simple to spot characteristics, but also styles, such as
El Greco’s Mannerism, with less graspable stylistic characteristics, where
other methods fail (Fig. 5, b–g, $5$th row).
Fine-grained style details: In Fig. 7 we show zoomed in cut-outs from the
stylized images. Interestingly, the stylizations of methods [13, 4, 21, 29,
20] do not change much across styles (compare Fig. 7(d, f–i, rows 1–3)). Zhu
et al. [52] produce more diverse images for different styles, but obviously
cannot alter the edges of the content (blades of grass are _clearly visible_
on all the cutouts in Fig. 7(e)). Fig. 7(c) shows the stylized cutouts of our
approach, which exhibit significant changes from one style to another. Another
interesting example is the style of Pollock, Fig. 7(row 8), where the style-
aware loss allows our model to properly alter content to the point of
discarding it – as would be expected from a Pollock action painting. Our
approach is able to generate high-resolution stylizations with a lot of style
specific details and retains those content details which are necessary for the
style.

Content | (a) Early period | (b) Stylized | (c) Late period | (d) Stylized
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 6:  Artwork examples of the early artistic period of van Gogh (a) and
his late period (c). Style transfer of the content image (1st column) onto the
early period is presented in (b) and the late period in (d).

Style |  (a) |  (b) |  (c) |  (d) [13] |  (e) [52] |  (f) [4] |  (g) [21] |  (h) [29] |  (i) [24]
---|---|---|---|---|---|---|---|---|---

|  |  |
---|---|---|---

Figure 7: Details from stylized images produced for different styles for a
fixed content image (a). (b) is our entire stylized image, (c) the zoomed in
cut-out and (d)-(i) the same region for competitors. Note the variation across
different styles along the _column_ for our method compared to other
approaches. This highlights the ability to adapt content (not just colors or
textures) where demanded by a style. Fine grained artistic details with sharp
boundaries are produced, while altering the original content edges.

Style transfer for different periods of van Gogh: We now investigate our
ability to properly model fine differences in style _despite_ using a group of
style images. Therefore, we take two reference images Fig. 6(a) and (c) from
van Gogh’s early and late period, respectively, and acquire related style
images for both from Wikiart. It can be clearly seen that the stylizations
produced for either period Fig. 6(b, d) are fairly different and indeed depict
the content in correspondence with the style of early (b) and late (d) periods
of van Gogh. This highlights that collections of style images are properly
used and do not lead to an averaging effect.
High-resolution image generation: Our approach allows us to produce high
quality stylized images in high-resolution. Fig. 8 illustrates an example of
the generated piece of art in the style of Berthe Morisot with resolution
$1280\times 1280$. The result exhibits a lot of fine details such as color
transitions of the oil paint and brushstrokes of different sizes. More HD
images are in the supplementary material.

Figure 8: High-resolution image (1280x1280 pix) generated by our approach. A
lot of fine details and brushstrokes are visible. A style example is in the
bottom left corner.

Real-time HD video stylization: We also apply our method to several videos.
Our approach can stylize HD videos ($1280\times 720$) at $9$ FPS. Fig. 9 shows
stylized frames from a video. We did not use a temporal regularization to show
that our method produces equally good results for consecutive frames with
varying appearance w/o extra constraints. Stylized videos are in the
supplementary material.

Style examples | Stylized frames
---|---
|
Figure 9: Results of our approach applied to the HD video of Eadweard
Muybridge ”The horse in motion” (1878). Every frame was _independently_
processed (no smoothing or post-processing) by our model in the style of
Picasso. Video resolution is $1920\times 1280$ pix. The full video is
available on YouTube: https://youtu.be/TtHJcL8Feu0.

###  3.2 Quantitative Evaluation

Style transfer deception rate: While several metrics [37, 2, 19] have been
proposed to evaluate the quality of image generation, until now no evaluation
metric has been proposed for an automatic evaluation of style transfer
results. To measure the quality of the stylized images, we introduce the
_style transfer deception rate_. We use a VGG16 network trained from scratch
to classify $624$ artists on Wikiart. Style transfer deception rate is
calculated as the fraction of generated images which were classified by the
network as the artworks of an artist for which the stylization was produced.
For fair comparison with other approaches, which used only one style image
$y_{0}$ (hence only one artist), we restricted $Y$ to only contain samples
coming from the same artist as the query example $y_{0}$. We selected $18$
different artists (i.e. styles). For every method we generated $5400$
stylizations ($18$ styles, $300$ per style). In Tab. 2 we report mean
deception rate for $18$ styles. Our method achieves $0.393$ significantly
outperforming the baselines. For comparison, mean accuracy of the network on
hold-out real images of aforementioned $18$ artists from Wikiart is $0.616$.
Human art history experts perceptual studies: Three experts (with a PhD in art
history with focus on modern and pre-modern paintings) have compared results
of our method against recent work. Each expert was shown $1000$ groups of
images. Each group consists of stylizations which were generated by different
methods based on the same content and style images. Experts were asked to
choose one image which best and most realistically reflects the current style.
The score is computed as the fraction of times a specific method was chosen as
the best in the group. We calculate a mean expert score for each method using
$18$ different styles and report them in Tab. 2. Here, we see that the experts
selected our method in around $50\%$ of the cases.
Speed and memory: Tab. 2 shows the time and memory required for stylization of
a single image of size $768\times 768$ px for different methods. One can see
that our approach and that of [24] and [52] have comparable speed and only
very modest demands on GPU memory, compared to modern graphics cards.

Table 1: Mean deception rate and mean expert score for different methods. The higher the better. Method | Deception rate | Expert score
---|---|---
Content images | 0.002 | -
AdaIn [21] | 0.074 | 0.060
PatchBased [4] | 0.040 | 0.132
Johnson et al. [24] | 0.051 | 0.048
WCT [29] | 0.035 | 0.044
CycleGan [52] | 0.139 | 0.044
Gatys et al. [13] | 0.147 | 0.178
Ours | 0.393 | 0.495

Table 2: Average inference time and GPU memory consumption, measured on a Titan X Pascal, for different methods with batch size $1$ and input image of $768\times 768$ pix. Method | Time | GPU memory
---|---|---
Gatys et al. [13] | 200 sec | 3887 MiB
CycleGan [52] | 0.07 sec | 1391 MiB
AdaIn [21] | 0.16 sec |  $8872$ MiB
PatchBased [4] | 8.70 sec | 4159 MiB
WCT [29] |  $5.22$ sec |  $10720$ MiB
Johnson et al. [24] |  0.06 sec |  671 MiB
Ours |  $0.07$ sec |  $1043$ MiB

###  3.3 Ablation Studies

Effect of different losses: We study the effect of different components of our
model in Fig. 10. Removing the style-aware content loss significantly degrades
the results, (c). We observe that without the style-aware loss training
becomes instable and often stalls. If we remove the transformed image loss,
which we introduced for a proper initialization of our model that is trained
from scratch, we notice mode collapse after $5000$ iterations. Training
directly with pixel-wise L$2$ distance causes a lot of artifacts (grey blobs
and flaky structure), (d). Training only with a discriminator neither exhibits
the variability in the painting nor in the content, (e). Therefore we conclude
that both the style-aware content loss and the transformed image loss are
critical for our approach.

Style | (a) Content | (b) Ours | (c) | (d) | (e) | (f)
---|---|---|---|---|---|---
|  |  |  |  |  |
Figure 10: Different variations of our method for Gauguin stylization. See Sect. 3.3 for details. (a) Content image; (b) full model ($\mathcal{L}_{c}$, $\mathcal{L}_{rgb}$ and $\mathcal{L}_{D}$); (c) $\mathcal{L}_{rgb}$ and $\mathcal{L}_{D}$; (d) without transformer block; (e) only $\mathcal{L}_{D}$; (f) trained with all of Gauguin’s artworks as style images. Please zoom in to compare. Style | Content | (a) | (b)
---|---|---|---
|  |  |
Figure 11: Encoder ablation studies: (a) stylization using our model; (b)
stylization using pre-trained VGG16 encoder instead of $E$.

Single vs collection of style images: Here, we investigate the importance of
the style image grouping. First, we trained a model with only one style image
of Gauguin, which led to mode collapse. Second, we trained with all of
Gauguin’s artworks as style images (without utilizing style grouping
procedure). It produced unsatisfactory results, cf. Fig. 10(f), because style
images comprised several distinct styles. Therefore we conclude that to learn
a good style transfer model it is important to group style images according to
their stylistic similarity.
Encoder ablation: To investigate the effect of our encoder $E$, we substitute
it with VGG16 [41] encoder (up to conv5_3) pre-trained on ImageNet. The VGG
encoder retains features that separate object classes (since it was trained
discriminatively), as opposed to our encoder which is trained to retain style-
specific content details. Hence, our encoder is not biased towards class-
discriminative features, but is style specific and trained from scratch. Fig.
11(a, b) show that our approach produces better results than with pre-trained
VGG16 encoder. ††footnotetext: Solution to Fig. 1: patches $3$ and $5$ were
generated by our approach, others by artists.

##  4 Conclusion

This paper has addressed major conceptual issues in state-of-the-art
approaches for style transfer. We overcome the limitation of only a single
style image or the need for style and content training images to show similar
content. Moreover, we exceed a mere pixel-wise comparison of stylistic images
or models that are pre-trained on millions of ImageNet bounding boxes. The
proposed style-aware content loss enables a real-time, high-resolution
encoder-decoder based stylization of images and videos and significantly
improves stylization by capturing how style affects content. †† This work has
been supported in part by a DFG grant, the Heidelberg Academy of Science, and
an Nvidia hardware donation.

## References

  * [1] Bautista, M.A., Sanakoyeu, A., Tikhoncheva, E., Ommer, B.: Cliquecnn: Deep unsupervised exemplar learning. In: Advances in Neural Information Processing Systems. pp. 3846–3854 (2016)
  * [2] Che, T., Li, Y., Jacob, A.P., Bengio, Y., Li, W.: Mode regularized generative adversarial networks. arXiv preprint arXiv:1612.02136 (2016)
  * [3] Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: An explicit representation for neural image style transfer. In: Proc. CVPR (2017)
  * [4] Chen, T.Q., Schmidt, M.: Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337 (2016)
  * [5] Collomosse, J., Bui, T., Wilber, M.J., Fang, C., Jin, H.: Sketching with style: Visual search with sketches and aesthetic context. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [6] Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A.A.: Generative adversarial networks: An overview. IEEE Signal Processing Magazine 35(1), 53–65 (2018)
  * [7] Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. Proc. of ICLR (2017)
  * [8] Engstrom, L.: Fast style transfer. https://github.com/lengstrom/fast-style-transfer/ (2016), commit 55809f4e
  * [9] Esser, P., Sutter, E., Ommer, B.: A variational u-net for conditional appearance and shape generation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2018)
  * [10] Fernie, E.: Art History and its Methods: A critical anthology. Phaidon, London (1995), p. 361
  * [11] Frigo, O., Sabater, N., Delon, J., Hellier, P.: Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 553–561 (2016)
  * [12] Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks. arXiv preprint arXiv:1505.07376 12 (2015)
  * [13] Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on. pp. 2414–2423. IEEE (2016)
  * [14] Gatys, L.A., Ecker, A.S., Bethge, M., Hertzmann, A., Shechtman, E.: Controlling perceptual factors in neural style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [15] Ghiasi, G., Lee, H., Kudlur, M., Dumoulin, V., Shlens, J.: Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830 (2017)
  * [16] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [18] Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. pp. 327–340. ACM (2001)
  * [19] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems. pp. 6629–6640 (2017)
  * [20] Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006). https://doi.org/10.1126/science.1127647, http://science.sciencemag.org/content/313/5786/504
  * [21] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)
  * [22] Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Transactions on Graphics (TOG) 36(4), 107 (2017)
  * [23] Jing, Y., Liu, Y., Yang, Y., Feng, Z., Yu, Y., Song, M.: Stroke controllable fast style transfer with adaptive receptive fields. arXiv preprint arXiv:1802.07101 (2018)
  * [24] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
  * [25] Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A., Winnemoeller, H.: Recognizing image style. arXiv preprint arXiv:1311.3715 (2013)
  * [26] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [27] Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: European Conference on Computer Vision. pp. 702–716. Springer (2016)
  * [28] Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Diversified texture synthesis with feed-forward networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [29] Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. In: Advances in Neural Information Processing Systems. pp. 385–395 (2017)
  * [30] Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. arXiv preprint arXiv:1705.01088 (2017)
  * [31] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [32] Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [33] Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2015)
  * [34] Mao, H., Cheung, M., She, J.: Deepart: Learning joint representations of visual arts. In: Proceedings of the 2017 ACM on Multimedia Conference. pp. 1183–1191. ACM (2017)
  * [35] Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016). https://doi.org/10.23915/distill.00003, http://distill.pub/2016/deconv-checkerboard
  * [36] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
  * [37] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: Advances in Neural Information Processing Systems. pp. 2234–2242 (2016)
  * [38] Shen, F., Yan, S., Zeng, G.: Meta networks for neural style transfer. arXiv preprint arXiv:1709.04111 (2017)
  * [39] Shih, Y., Paris, S., Barnes, C., Freeman, W.T., Durand, F.: Style transfer for headshot portraits. ACM Transactions on Graphics (TOG) 33(4), 148 (2014)
  * [40] Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Transactions on Graphics (TOG) 32(6), 200 (2013)
  * [41] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [42] Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.S.: Texture networks: Feed-forward synthesis of textures and stylized images. In: ICML. pp. 1349–1357 (2016)
  * [43] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [44] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In: Proc. CVPR (2017)
  * [45] Wang, H., Liang, X., Zhang, H., Yeung, D.Y., Xing, E.P.: Zm-net: Real-time zero-shot image manipulation network. arXiv preprint arXiv:1703.07255 (2017)
  * [46] Wang, X., Gupta, A.: Unsupervised learning of visual representations using videos. arXiv preprint arXiv:1505.00687 (2015)
  * [47] Wang, X., Oxholm, G., Zhang, D., Wang, Y.F.: Multimodal transfer: A hierarchical deep convolutional neural network for fast artistic style transfer. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [48] Wilber, M.J., Fang, C., Jin, H., Hertzmann, A., Collomosse, J., Belongie, S.: Bam! the behance artistic media dataset for recognition beyond photography. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [49] Wilmot, P., Risser, E., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893 (2017)
  * [50] Xue, Y., Xu, T., Zhang, H., Long, R., Huang, X.: Segan: Adversarial network with multi-scale $l\\_1$ loss for medical image segmentation. arXiv preprint arXiv:1706.01805 (2017)
  * [51] Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Advances in neural information processing systems. pp. 487–495 (2014)
  * [52] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision (2017)

##  Appendix 0.A Implementation Details

###  0.A.1 Network Architecture

We incarnate our approach using an encoder-decoder architecture with a
discriminator. Below we follow the naming convention similar to the one used
in [52].

Let:

  * •

cFs1-k denote $F\times F$ Convolution-InstanceNorm-ReLU layer with $k$ filters
and stride $1$;

  * •

cFs1-k-sigmoid denote $F\times F$ Convolution-InstanceNorm-Sigmoid layer with
$k$ filters and stride $1$;

  * •

cFs1-k-noact denote $F\times F$ Convolution-InstanceNorm layer with $k$
filters and stride $1$;

  * •

dF-k denote a $F\times F$ Convolution-InstanceNorm-ReLU layer with $k$ filters
and stride $2$;

  * •

dF-k-LReLU denote a $F\times F$ Convolution-InstanceNorm-LeakyReLU layer with
$k$ filters and stride $2$;

  * •

Rk denote a residual block that contains two $3\times 3$ convolutional layers
followed by InstanceNorm [43];

  * •

uk denote an upscaling block that contains nearest-neighbor upscaling by a
factor of $2$ followed by Convolution-InstanceNorm-ReLU layer with $k$ filters
and stride $1$.

All convolutional layers use reflection padding.

####  0.A.1.1 Encoder-decoder architecture

Encoder has InstanceNorm layer at the beginning and $5$ convolutional layers:
InstanceNorm, c3s1-32, d3-32, d3-64, d3-128, d3-256. Decoder contains 9
residual blocks, $4$ upscaling blocks and $1$ convolution $7\times 7$ with
sigmoid activation: R256$\times 9$, u256, u128, u64, u32, c7s1-3-sigmoid.

####  0.A.1.2 Transformer block $\mathbf{T}$

Transformer block $\mathbf{T}$ contains a convolutional layer followed by a
weight normalization layer with fixed norm $||W||=1$. Convolutional layer: $3$
kernels of size $10\times 10$, stride $1$, uniformly initialized.

####  0.A.1.3 Discriminator architecture

Discriminator is implemented as a fully convolutional network and tries to
classify if input image patches are real or fake. It contains $7$
convolutional layers. We use Leaky ReLU activations with slope $0.2$.
Architecture is defined as: d5-128-LReLU, d5-128-LReLU, d5-256-LReLU,
d5-512-LReLU, d5-512-LReLU, d5-1024-LReLU, d5-1024-LReLU,
c3s1-1-noact.

We use $4$ auxiliary classifiers (implemented as a convolutional layer with
$1$ filter each) to capture image details on different scales [22, 50] and to
alleviate artifacts: auxiliary classifiers were added after 1st, 2nd, 4th and
6th convolutional layers of the discriminator. We simply sum up all the losses
from different scales.

###  0.A.2  Style Image Grouping Details

As described in Sec. 2.2, the number of style images is not fixed and depends
on the neighborhood size of the query image $y_{0}$ and how frequent the style
is in the Wikiart [25] dataset. For different style examples in our
experiments it varied from $55$ to $1391$ related images. We observe: the more
style examples the better, as long as they are from the same style. Ablation
studies (Sec. 3.3) show that using too few style examples (e.g. one) leads to
mode collapse; at the same time using a lot of images, which are less related
to each other (e.g. all images of an artist), produces unsatisfactory
stylizations.

---
Figure 12: High-resolution ($2288\times 1280$ pix) Pissarro stylization by our
approach. The top row depicts $4$ randomly chosen from set $Y$ instances of
style obtained by the style image grouping.
---
Figure 13: High-resolution ($1712\times 1280$ pix) Roerich stylization by our
approach. The top row depicts $4$ randomly chosen from set $Y$ instances of
style obtained by the style image grouping.

###  0.A.3 Training Details

All networks were trained from scratch on randomly cropped image patches of
size $768\times 768$ pix. $\lambda$ in Eq. 5 was set to $0.001$. We trained
for $300000$ iterations with batch size $1$ and learning rate $2\times
10^{-4}$ using Adam [26] optimizer. We reduced the learning rate by a factor
of $10$ after $200000$ iterations.

In each iteration we alternatively update encoder-decoder and discriminator.
To balance out the discriminator and encoder-decoder training [6] we update
solely the discriminator, if it has accuracy $<0.8$, and update only encoder-
decoder otherwise. The discriminator accuracy is calculated using exponential
moving average during training.

##  Appendix 0.B Extra Qualitative Results

####  0.B.0.1 Altering the style of an existing artwork

Our method is able to change the style of an existing artwork, rendering it in
another style. It means, that our algorithm can also handle content images
which are artistic, which, to our knowledge, was never shown in previous work
before. We refer the reader to the results on the project page1.

####  0.B.0.2 Real-time HD video stylization

We stylized Eadweard Muybridge’s ”The horse in motion” video, 1878, in the
style of Picasso. The stylized video resolution is $1920\times 1280$ pix.
Additionally, we apply our method to the video ”Aix-en-Provence, France” by
Jan van Bekkum in resolution $1920\times 1280$ pix. We stitch together the
original video and $3$ stylizations in the style of Cézanne, Kandinsky and
Picasso produced by our approach. The videos are available on the project
page1.

####  0.B.0.3 More examples of high-resolution style transfer

In Fig. 12, 13 we show extra high-resolution images stylized by our approach.
Additional comparison of our method against baselines and more high-resolution
visual results are available on the project page1.

##  Appendix 0.C Extra ablation studies

(a) Style | (b) Content | (c) Ours | (d)
---|---|---|---
|  |  |
Figure 14: Investigation of the importance of an independent transformer
block. (a) style image (van Gogh); (b) content image; (c) our stylization; (d)
stylization with the loss $\mathcal{L}_{conv1}$ instead of
$\mathcal{L}_{\mathbf{T}}$.

####  0.C.0.1 Transformer block ablation

To demonstrate the need in an independent transformer block, we replaced the
transformed image loss $\mathcal{L}_{\mathbf{T}}$ with a loss
$\mathcal{L}_{conv1}$ using conv1 features of the encoder,

$\mathcal{L}_{conv1}=(E,G)=\mathop{\mathbb{E}}_{x\sim
p_{X}(x)}\left[\frac{1}{d_{conv1}}||\phi_{conv1}(x)-\phi_{conv1}(G(E(x))||_{2}^{2}\right],$
(8)

where $\phi_{conv1}(x)$ are the activations of the conv1 layer of the encoder
$E$ for input image $x$ and $d_{conv1}$ is the dimensianality of
$\phi_{conv1}(x)$. This produced worse results (cf. Fig. 14(d)), since the
loss $\mathcal{L}_{conv1}$ is then directly tied to the convolutional layer
used for our style-aware content representation in the encoder. In contrast,
the proposed transformer block can be learned independently from the early
layers of the encoder.


# Mitigating the Impact of Outlier Channels for Language
Model Quantization with Activation Regularization

###### Abstract

We consider the problem of accurate quantization for language models, where
both the weights and activations are quantized to 4 bits per parameter with
uniform quantization, the lowest bitwidth format natively supported by
existing GPU hardware. In this context, the key challenge is _activation
quantization_ : it is known that language models contain outlier channels
whose values on average are orders of magnitude higher than than other
channels, which prevents accurate low-bitwidth quantization with known
techniques. We systematically study this phenomena and find that these outlier
channels emerge early in training, and that they occur more frequently in
layers with residual streams. We then propose a simple strategy which
regularizes a layer’s _inputs_ via quantization-aware training (QAT) and its
_outputs_ via activation kurtosis regularization. We show that regularizing
both the inputs and outputs is crucial for preventing a model’s “migrating”
the difficulty in input quantization to the weights, which makes post-training
quantization (PTQ) of weights more difficult. When combined with weight PTQ,
we show that our approach can obtain a W4A4 model with integer quantization
that performs competitively to the standard-precision W16A16 baseline.111Code
is available at https://github.com/aninrusimha/qat-pretrain

Aniruddha Nrusimha1 Mayank Mishra2 Naigang Wang3

Dan Alistarh4,5 Rameswar Panda2 Yoon Kim1

1Massachusetts Institute of Technology 2MIT-IBM Watson AI Lab

3IBM Research 4IST Austria 5NeuralMagic

<EMAIL_ADDRESS>
##  1 Introduction

Large language models (LLM) have been shown to contain _outlier channels_ ,
i.e., feature dimensions whose values are orders of magnitude higher than the
others. These outlier channels are known to be crucial for strong model
performance (Kovaleva et al., 2021; Puccetti et al., 2022), but pose
significant challenges from a model compression perspective, for instance via
post-training quantization (PTQ) (Dettmers et al., 2022; Xiao et al., 2023;
Wei et al., 2022). Concretely, to enable the use of low-bitwidth integer
matrix multiplications—which can lead to significant speed-ups—both the
activations and the weights need to be quantized. However the presence of high
outlier values in the model activations results in high quantization errors,
and thus overall poor PTQ accuracy (see, e.g., Xiao et al. (2023)).

To mitigate the effect of outlier channels for activation quantization at the
per-tensor level, existing works have explored various approaches, including
keeping some of the computations in higher precision (Dettmers et al., 2022;
Ashkboos et al., 2023; Zhao et al., 2023), or “migrating” the difficulty of
quantizing outlier channels to other parts of the model (Xiao et al., 2023;
Wei et al., 2023; Liu et al., 2023). While the above strategies have been
effective for achieving INT8 activation quantization, INT4 quantization with
PTQ methods remains an open challenge, with current methods still facing
nontrivial degradations in perplexity (Wu et al., 2023; Shao et al., 2023;
Yuan et al., 2023).

Figure 1: (Top) Average of the absolute activation values of a KV projection
layer for a 1B language model trained with (a) standard training, (b) QAT with
learned clipping values in the input layer, and (c) QAT on the inputs and
kurtosis regularization on the layer’s outputs. For the QAT runs, we show the
learned clip value as a green 2d manifold. (Bottom) Parameter values of
individual weights in the KV projection of the same layer corresponding to
each model after training. QAT-only training results in the model’s weights’
becoming harder to quantize, whereas kurtosis regularization mitigates this.

In this work, we perform an empirical study of outlier channel phenomena from
a pretraining perspective. We find that dimensions with outlier channels
emerge relatively early in training (see fig. 1(a), top), suggesting that
their mitigation requires early intervention. These outlier channels are
particularly prevalent in the output projection layer of the first layer, as
well as the query-key-value projection layers of the other layers. Next, we
explore a simple strategy that regularizes a layer’s input and output. On the
input side, we show that a quantization-aware training (QAT) approach which
_learns_ the clipping values for each activation layer (Choi et al., 2018;
Bhalgat et al., 2020) is effective at controlling the number of outlier
channels, in addition to mitigating the effect of outliers through clipping
(see fig. 1(b), top). However, while this approach can train a W16A4 model
that has similar perplexity to a W16A16 model, post-training weight
quantization to W4A4 results in nontrivial perplexity degradations, due to the
model’s _weights_ now becoming more difficult to quantize (see fig. 1(b),
bottom). We thus additionally regularize the kurtosis of a layer’s _output_ ,
which discourages the creation of outliers wholesale. Specifically, this
discourages the layer’s _weights_ having pathologically large rows (fig. 1(c),
bottom).

Putting all these elements together, we show that we can train a language
model at moderate scale (1 billion parameter models trained on 20 billion
tokens) whose W4A4 perplexity is competitive to the standard-precision W16A16
baseline.

##  2 Background and Related Work

###  2.1 Uniform Quantization & Quantized Matmuls

We focus on uniform quantization, where the quantized values are evenly spaced
between an interval range. Formally, for a given matrix
$\mathbf{A}\in\mathbb{R}^{n\times n}$ that we wish to quantize to $b$ bits,
let $c^{-}$ and $c^{+}$ be the pre-defined (or learned) clipping values. The
quantization function $Q:\mathbb{R}^{n\times m}\to\mathbb{Z}^{n\times m}$ is
then given by,

$\displaystyle
Q({\mathbf{A}})=\operatorname{round}(s\times\operatorname{clamp}(\mathbf{A},c^{-},c^{+})+z),$

where $s=\frac{2^{b}-1}{c^{+}-c^{-}}$ is the scale factor and
$z=\operatorname{round}(s\times c^{-})$ is the (optional) zero-point offset.
This function, which can be generalized to different granularities of
$\mathbf{A}$ (e.g., rows, columns or subgroups) transforms the entries of
$\mathbf{A}$ into integers between $[0,2^{b}-1]$.

The quantized matrix $\mathbf{Q}_{\mathbf{A}}=Q(\mathbf{A})$ can be utilized
in two different ways. First, the value can be dequantized to its original
precision via $\widehat{\mathbf{A}}=\frac{1}{s}(\mathbf{Q}_{\mathbf{A}}-z)$
before multiplication. This method is typically used by pure weight
quantization schemes, which multiply in the precision the model was trained
in. Weight-only quantization can reduce a model’s memory footprint, and
insofar as LLM inference is often memory bound, it can also enable faster
inference by reducing the amount of time spent on memory operations during the
forward pass (Lin et al., 2023; Frantar & Alistarh, 2024). However, the fact
that the actual matmul is done in high precision is a fundamental limitation
of weight-only quantization.

Second, the quantized values can be directly used for the matrix
multiplication. Let $\mathbf{Q}_{\mathbf{U}}=Q(\mathbf{U})$,
$\mathbf{Q}_{\mathbf{V}}=Q(\mathbf{V})$ be the quantized versions of
$\mathbf{U}\in\mathbb{R}^{n\times k}$, $\mathbf{V}\in\mathbb{R}^{k\times m}$
with the respective scaling factors $s_{\mathbf{U}},s_{\mathbf{V}}$ and
offsets $z_{\mathbf{U}},z_{\mathbf{V}}$. We can approximate
$\mathbf{U}\mathbf{V}$ with

$\displaystyle\mathbf{U}\mathbf{V}$
$\displaystyle\approx\widehat{\mathbf{U}}\widehat{\mathbf{V}}=\frac{1}{s_{\mathbf{U}}s_{\mathbf{V}}}\times(\mathbf{Q}_{\mathbf{U}}-z_{\mathbf{U}})(\mathbf{Q}_{\mathbf{V}}-z_{\mathbf{V}}),$

where we can make use of low-precision matmuls for
$(\mathbf{Q}_{\mathbf{U}}-z_{\mathbf{U}})(\mathbf{Q}_{\mathbf{V}}-z_{\mathbf{V}})$.
In cases where the _rows_ of $\mathbf{U}$ and _columns_ of $\mathbf{V}$ are
quantized separately with the corresponding scaling vectors
$\mathbf{s}_{\mathbf{U}}\in\mathbb{R}^{n},\mathbf{s}_{\mathbf{V}}\in\mathbb{R}^{m}$
and offset vectors
$\mathbf{z}_{\mathbf{U}}\in\mathbb{Z}^{n},\mathbf{z}_{\mathbf{V}}\in\mathbb{Z}^{m}$,
we can still make use of integer matmuls since
$\widehat{\mathbf{U}}\widehat{\mathbf{V}}$ is given by

$\displaystyle\operatorname{diag}(\mathbf{s}_{\mathbf{U}})^{-1}(\mathbf{Q}_{\mathbf{U}}-\mathbf{z}_{\mathbf{U}}\otimes\mathbf{1}_{k})(\mathbf{Q}_{\mathbf{V}}-\mathbf{1}_{k}\otimes\mathbf{z}_{\mathbf{V}})\operatorname{diag}(\mathbf{s}_{\mathbf{V}})^{-1}$

where $\mathbf{1}_{k}\in\mathbb{Z}^{k}$ is a vector of 1s and $\otimes$ is the
outer product.222If the offset vectors are not integers we can expand the
expression and still use integer matmuls for
$\mathbf{Q}_{\mathbf{U}}\mathbf{Q}_{\mathbf{V}}$. For the cross terms we can
use the identity
$(\mathbf{z}_{\mathbf{u}}\otimes\mathbf{1}_{k})\mathbf{Q}_{\mathbf{V}}=\mathbf{z}_{\mathbf{u}}\otimes(\mathbf{1}_{k}^{\top}\mathbf{Q}_{\mathbf{V}})$,
and thus we can still make use of integer matmuls for most of the FLOPs.
Note, however, lower-precision matmuls cannot straightfowardly be used if the
$\mathbf{U}$ is quantized at the _column_ level.

This second strategy which makes use of lower-precision matmuls can
significantly improve inference latency and energy efficiency on supported
hardware. For example, INT4 tensor core matmuls can be up to four times faster
than FP16 tensor core matmuls on the NVIDIA Ampere
architecture,333https://developer.nvidia.com/blog/nvidia-ampere-architecture-
in-depth/ while from a hardware-efficiency perspective, dedicated hardware for
integer operations require much less area and energy usage than their
floating-point counterparts (Jouppi et al., 2021; van Baalen et al., 2023).

###  2.2 Challenges in LLM Quantization

In LLMs, the majority of FLOPs are spent on dense matmuls of the form
$\mathbf{X}\mathbf{W}$ where $\mathbf{X}\in\mathbb{R}^{L\times d_{in}}$ are
the input activations (for $L$ input tokens) and
$\mathbf{W}\in\mathbb{R}^{d_{in}\times d_{out}}$ are the model weights. For
the Transformer architecture in particular this corresponds to the key, query,
value projection layers, as well as the FFN layers. Given the sheer number of
FLOPs in LLMs, inference efficiency can be improved significantly through
lower-precision matmuls.

While there has been much work on post-training weight-only quantization for
pretrained LLMs (Frantar et al., 2022; Dettmers & Zettlemoyer, 2023; Lin et
al., 2023; Kim et al., 2023; Dettmers et al., 2023; Chee et al., 2023; Lee et
al., 2023; Egiazarian et al., 2024, inter alia), PTQ for activations remains
difficult due to the presence of outlier channels in LLMs trained with
standard precision (Dettmers et al., 2022; Xiao et al., 2023). Informally,
outlier channels are a set of input channels (i.e., columns of $\mathbf{X}$)
whose values are many orders of magnitudes higher than the others, and have
been shown to be crucial for performance (Kovaleva et al., 2021). If one were
just interested in quantizing $\mathbf{X}$ independently, outlier channels
could be managed by quantizing each column of $\mathbf{X}$ separately such
that the scaling factor associated with an outlier channel is commensurate.
However, as outlined in the previous section this would not enable the use of
lower-precision matmuls, which requires $\mathbf{X}$ to be quantized by (at
most) rows; unfortunately row-level (i.e., per-token) quantization results in
significant performance degradations (Xiao et al., 2023).

###  2.3 Quantization-Aware Training

Figure 2: Frequency of outlier channels over the course of training. (Left)
Proportion of outlier channels by layer depth. Layer 1 has highest occurrence
of outlier channels. (Middle) In layer 1 inputs to the attention projection
layer have the most outlier channels. (Right) This is generally not the case
for the other layers, where the input to the QKV project layer has the most
outlier channels.

Quantization-aware training (QAT) describes a class of techniques which aims
to enable better quantization by simulating quantization during training (Zhou
et al., 2016; Jacob et al., 2018; Zhang et al., 2018; Jung et al., 2019; Jain
et al., 2020, inter alia). While there are many methods for QAT, we use a
simple modified version of PACT (Choi et al., 2018) and LSQ (Bhalgat et al.,
2020), which learn the clip values $c^{-}$ and $c^{+}$ for the activations.
This approach uses the learned clip values to perform quantization during the
forward pass, and uses the straight-through estimator for the gradients with
respect to the clip values. While QAT has been studied extensively in the
context of (typically smaller) vision models, QAT for pretraining language
models with more than a billion parameters remains less explored.

Figure 3: Trajectory of a channel’s activations across 50B tokens of training.
We show each channel’s absolute activation value averaged across 500K tokens.

##  3 Motivating Study: Outlier Channels in Language Models

We first conduct a preliminary analysis to study the emergence of outlier
channels during pretraining, with both our own and open-source models. For our
own pretrained models, we use the standard “pre-LayerNorm” Transformer
architecture (Xiong et al., 2020), where given layer $l$’s input
$\mathbf{X}^{(l)}\in\mathbb{R}^{L\times d}$ we obtain the next layer
$\mathbf{X}^{(l+1)}$ via,

$\displaystyle\mathbf{Y}_{1}=\operatorname{LayerNorm}(\mathbf{X}^{(l)}),\hskip
11.38109pt\mathbf{Q},\mathbf{K},\mathbf{V}=\mathbf{Y}_{1}\mathbf{W}_{QKV},\hskip
11.38109pt\mathbf{Y}_{2}=\operatorname{softmax}(\mathbf{Q}\mathbf{K}^{\top}\odot\mathbf{M})\mathbf{V},$
$\displaystyle\mathbf{Z}=\mathbf{X}+\mathbf{Y}_{2}\mathbf{W}_{O},\hskip
11.38109pt\mathbf{Y}_{3}=\operatorname{LayerNorm}(\mathbf{X}+\mathbf{Y}_{2}\mathbf{W}_{O}),\hskip
11.38109pt\mathbf{Y}_{4}=\sigma(\mathbf{Y}_{2}\mathbf{W}_{1}),\hskip
11.38109pt\mathbf{X}^{(l+1)}=\mathbf{Z}+\mathbf{Y}_{4}\mathbf{W}_{2}.$

Here $\mathbf{W}_{QKV}\in\mathbb{R}^{d\times
3d},\mathbf{W}_{O}\in\mathbb{R}^{d\times
d},\mathbf{W}_{1}\in\mathbb{R}^{d\times
4d},\mathbf{W}_{2}\in\mathbb{R}^{4d\times d}$ are learnable matrices, and the
bias vectors are omitted for brevity. Our study focuses on the following
activations that have been previously found to contain outlier channels: QKV
Input ($\mathbf{Y}_{1}$), Attn Proj Input ($\mathbf{Y}_{2}$), MLP Input
($\mathbf{Y}_{3}$), MLP Proj Input ($\mathbf{Y}_{4}$). We train 1 billion
parameter (24-layer model with 1920 dimensions) on 50 billion tokens from the
SlimPajama dataset (Soboleva et al., 2023). We periodically collect activation
statistics for all layers by running model checkpoints on (the same) 500K
tokens from the C4 dataset.

First, we attempt to measure the prevalence of outlier channels aggregated by
layer type and depth. For the purposes of this analysis, we name a channel an
outlier if the average absolute value of the channel is over six times the
average absolute value of all the input activations. This definition of an
outlier channel is somewhat arbitrary, but similar definitions in the
literature based on the other metrics (Kovaleva et al., 2021) generate similar
results; we use this definition as opposed to definitions on the absolute
values (Dettmers et al., 2022) to enable comparison across different layers.
The results of this analysis are in fig. 2. Our results generally follow what
has been established in the literature: while outliers are distributed
throughout depth, the layers which tend to have the most outlier channels in
their input are those whose inputs are the residual stream of the network.
Interestingly, we find that outlier channels emerge early in training, and
rapidly become numerous. The proportion of outlier channels within a layer
then decreases gradually and eventually plateaus.

Figure 4: The distribution of activations over of a non-outlier channel (left)
and two outlier channels (middle, right) over training. Figure 5: Activation
development in two open-source models: Pythia 6.9b (Biderman et al., 2023) and
OLMo 7B (Groeneveld et al., 2024). We show activiations for a layer that reads
from the residual stream (QKV Input) and one that does not (MLP Proj Input).
Note that the OLMo data includes a step-0 checkpoint (i.e., at
initialization).

We next perform a more granular analysis, where we analyze the average
absolute value of channels over the training of a 1B model with 50B tokens.
This is shown in fig. 3. Within channels, we observe that the development of
outliers occurs early on during training. In most cases outliers primarily
occur in layers that take as input the residual stream, although there is
still significant variation in the average magnitude of channels in the input
to other layers. We take a closer examination of the development of some the
largest individual outlier channels for a particular layer in fig. 4. Channel
600, which is not an outlier channel, has channel values that are distributed
roughly as a Gaussian with a mean of zero. The outlier channels, in
comparison, have mean values that are significantly different from zero. This
initial examination suggests that outlier channels are not scaled differently
than non outlier channels, but have a shifted distribution. This potentially
indicates why scaling and shifting methods, like OmniQuant (Shao et al.,
2023), outperform scaling-only methods such as SmoothQuant (Xiao et al.,
2023).

#### Open-source Models.

To validate the generality of our observations, we perform our analysis on two
publicly available 7B models with public checkpoints, Pythia (Biderman et al.,
2023) and OLMo (Groeneveld et al., 2024). In fig. 5 we can see the development
of activation outliers early on in the training of both models, although the
outliers in OLMo take longer to develop. Furthermore, we confirm a pattern
found across the literature, that the primary place where outliers develop is
not between layers in a given attention or MLP block but in the residual
stream between blocks. That is, the types of layers that do or do not develop
outliers are the same in both our model and the pretrained models (e.g., QKV
Input activations have outlier channels, while MLP Proj Input activations do
not).

##  4 Mitigating Outlier Channels with Activation Regularization

Based on insights from the previous section, we propose a simple
regularization strategy for quantizing the activations of the linear layers,
where we use QAT on the input activations and simultaneously penalize the
kurtosis of the layer’s outputs.

###  4.1 Input Activations: QAT with Learned Clip Values

As evident from §2.1, the clip values $c^{-}$ and $c^{+}$ play a key role in
uniform quantization. Following PACT (Choi et al., 2018) and LSQ (Bhalgat et
al., 2020), we treat these quantization parameters as learnable parameters and
optimize them with gradient descent. Concretely, during the forward pass we
run the quantization/dequantization step, as shown in algorithm 1. For the
backward pass, we use a straight-through estimator to obtain
$\nabla{\mathbf{A}}$, $\nabla c^{+}$, $\nabla c^{-}$ from
$\nabla\widehat{\mathbf{A}}$ (the gradients with respect to the
quantized/dequantized layer). This is shown in algorithm 2. We will show in
our experiments that quantizing during training is crucial for 4-bit
quantization; just clamping the activations without quantization leads to poor
performance.

###  4.2 Output Activations: Kurtosis Regularization

In our initial experiments we found that QAT on a layer’s input is sufficient
to train a W16A4 model that matches the performance of a W16A16. However,
since we do not perform QAT for the weights, efficient deployment requires
post-training weight quantization to 4 bits. While existing work has shown
that weight-only PTQ to 4 bits (i.e., W16A16 $\to$ W4A16) can be done almost
losslessly (Frantar et al., 2022; Shao et al., 2023), we observed this to not
be the case with QAT models, with W16A4 $\to$ W4A4 resulting in nontrivial
perplexity degradations. This is due to the fact that a model can essentially
“migrate” the outlier channels to the corresponding rows of the weight matrix,
which makes per-column weight PTQ more difficult (as shown in fig. 1(b),
bottom).

Algorithm 1 QAT forward pass

0: $\mathbf{A}$, $c^{-}$, $c^{+}$, $b$, align_zero;

$s=\frac{2^{b}-1}{c^{+}-c^{-}}$

if align_zero then

$z=\operatorname{round}(s\times c^{-})$

else

$z=0$

end if

$\mathbf{Q}_{\mathbf{A}}=\operatorname{round}(s\times\text{ clamp
}(\mathbf{A},c^{-},c^{+})+z)$

$\widehat{\mathbf{A}}=\frac{1}{s}(\mathbf{Q}_{\mathbf{A}}-z)$

return $\widehat{\mathbf{A}}$

Figure 6: The forward and backward passes of QAT. Here $\mathbf{A}$ is the
activation tensor, $b$ is the bit width, $c^{-}$ and $c^{+}$ are the learned
clip values, and $\nabla\widehat{\mathbf{A}}$ is the gradient with respect to
$\widehat{\mathbf{A}}$. Algorithm 2 QAT backward pass

0: $\mathbf{A}$, $c^{-}$, $c^{+}$, $b$, $s$, $\nabla\widehat{\mathbf{A}}$;

$\mathbf{Q}=s\times(\mathbf{A}-c^{-})$

$\mathbf{E}=(\mathbf{{Q}}-\operatorname{round}(\mathbf{{Q}}))/(2^{b}-1)$

$\nabla\mathbf{A}_{ij}=\begin{cases}0&\text{if }\mathbf{A}_{ij}>c^{+}\text{ or
}\mathbf{A}_{ij}<c^{-}\\\
\nabla\widehat{\mathbf{A}}_{ij}&\text{otherwise}\end{cases}$

$\mathbf{C}^{+}_{ij}=\begin{cases}\nabla\widehat{\mathbf{A}}_{ij}&\text{if
}\mathbf{A}_{ij}>c^{+}\\\ -\
\mathbf{E}_{ij}\times\nabla\widehat{\mathbf{A}}_{ij}&\text{elif
}\mathbf{A}_{ij}>c^{-}\\\ 0&\text{otherwise}\end{cases}$

$\mathbf{C}^{-}_{ij}=\begin{cases}-\
\mathbf{E}_{ij}\times\nabla\widehat{\mathbf{A}}_{ij}&\text{if
}\mathbf{A}_{ij}<c^{+}\\\ \nabla\widehat{\mathbf{A}}_{ij}&\text{elif
}\mathbf{A}_{ij}<c^{-}\\\ 0&\text{otherwise}\end{cases}$

$\nabla c^{+}=\sum_{ij}\mathbf{C}^{+}_{ij}$, $\nabla
c^{-}=\sum_{ij}\mathbf{C}^{-}_{ij}$

return $\nabla\mathbf{A},\nabla c^{+},\nabla c^{-}$

One approach to mitigating these _outlier weights_ would be to directly
regularize the weights via QAT or some other approach (e.g.,
$\ell_{\infty}$-norm regularization). However, we found these direct
regularization approaches to result in much worse performance and/or unstable
training. We thus adopt a more indirect regularization strategy, exploiting
the fact that high input channel weights typically lead to a layer’s _outputs_
having outliers, i.e., the output distribution is heavy-tailed (see fig. 1).
Our approach thus regularizes the output distribution’s kurtosis. which
measures how heavy-tailed a distribution is. An estimate of the kurtosis of a
set of values $\mathbf{x}\in\mathbb{R}^{d}$ is given by,
$\operatorname{Kurtosis}(\mathbf{d})=\frac{\sum_{i}^{k}(\mathbf{x}_{i}-\mu)^{4}}{\sigma^{4}+\epsilon},$
where $\mu$ and $\sigma$ are respectively the empirical mean and standard
deviation of $\mathbf{x}$, and $\epsilon$ is a small term for numerical
stability. We multiply the sum of the kurtosis estimates for each token with
hyperparameter $\lambda$, and add the result to the cross-entropy loss. While
prior work has shown the benefits of regularizing the kurtosis of a layer’s
activation distribution to be close to that of a uniform distribution (Chmiel
et al., 2020), regularizing the output distribution’s kurtosis to make it less
heavy-tailed has not been explored before to our knowledge.

###  4.3 Post-training Weight Quantization

After training the model to W16A4 with activation regularization on both the
inputs/outputs, we experiment with two methods for quantizating the weights to
4 bits. The simplest baseline we use is round-to-nearest (RTN) quantization,
which for our purposes implies per-token (for activations)444While there are
more sophisticated activiation quantization approaches (Yuan et al., 2023;
Chee et al., 2023), these typically have additional overhead (for low-
precision matmuls) and are thus not as fast as simple RTN integer
quantization. or per-output-channel (for weights) uniform min-max
quantization. While the underperformance of RTN weight quantization versus
more sophisticated quantization strategies that use calibration data is widely
known, we deliberately include this simple data-agnostic baseline to show that
activation regularization results in weights that are also easier to quantize
(i.e., less perplexity degradation with RTN). Our second approach applies GPTQ
(Frantar et al., 2022), which uses a small amount of calibration data to
quantize the weights, and is still near the state-of-the-art for 4-bit weight
quantization.

##  5 Empirical Study

| Native Activations |  | 4-bit Activations
---|---|---|---
Weight Precision | 16 | 4 |  | 4 | 4 | 3 | 3
Weight Quantizer | None | GPTQ |  | GPTQ | RTN | GPTQ | RTN
Baseline | 23.57 | 24.10 |  | 113233 | 11855 | 11755 | 17187
Activation Clamping | 23.73 | 24.85 |  | 378 | 423 | 568 | 663
Kurtosis Regularization | 23.72 | 24.57 |  | 8720 | 8140 | 10235 | 19665
QAT | 24.30 | 25.32 |  | 25.32 | 27.76 | 32.56 | 46.47
QAT + Kurtosis Regularization | 24.10 | 24.57 |  | 24.57 | 24.90 | 26.83 | 30.46
Baseline | 25.70 | 26.16 |  | 8430 | 10028 | 9107 | 14498
Activation Clamping | 26.38 | 27.60 |  | 32378 | 6852 | 26120 | 15908
Kurtosis Regularization | 26.28 | 26.95 |  | 7319 | 6852 | 9066 | 15908
QAT | 26.72 | 27.86 |  | 27.87 | 32.70 | 64.61 | 58.81
QAT + Kurtosis Regularization | 26.11 | 26.56 |  | 26.56 | 27.13 | 30.12 | 33.46
Table 1: Perplexity of 1B models on C4 (top) and PTB (bottom). Native
activation are 16 bits for Baseline, Activation Clamping, Kurtosis
Regularization; and 4 bits for QAT,  QAT + Kurtosis Regularization.

###  5.1 Experimental Setup

We use the Megatron-LM (Shoeybi et al., 2020) codebase and train on the
SlimPajama dataset (Soboleva et al., 2023). While the trajectory analyses in
§3 were done for 50B tokens, due to limited compute we train for 20B tokens
for these experiments.

#### Baselines.

In order to isolate the contributions of each component of our method, we
compare against several baselines, on top of the standard-precision baseline.
The activation clamping baseline uses static, per-layer clipping values to
clamp the input activations. To advantage this approach as much as possible,
we use an “oracle” clipping values obtained from QAT to decide the per-layer
clipping values, which was found to be more effective than grid-searching on
the clipping values. In activation clamping the activations are not quantized
during training, and thus this baseline isolates the effect of QAT. The
_kurtosis regularization_ baseline applies kurtosis regularization just on the
outputs, without QAT. The _QAT_ -only baseline just applies QAT in the input
activations.

#### Hyperparameters.

All hyperparameters were tuned for our 1B W16A16 baseline and kept constant
throughout experiments, except for weight decay where we selected between
$\\{0.1,0.01\\}$ for all methods. We use a batch size of 1M tokens, learning
rate of 1.5e-4, cosine learning decay, and FP16 precision. For QAT we
initialize our clipping values to $\pm 4$ for clipping value initializations,
unless the layer’s input is bounded. We use the same learning rate but no
momentum or weight decay for clip values. For kurtosis we use 1e-5 as the
regularization strength.

#### Evaluation.

We evaluate the perplexity of each model on the C4 and PTB datasets. We test
models in three different weight quantization categories: 16 bits, 4 bits, and
3 bits. The 4-bit and 3-bit experiments test with both RTN and GPTQ. For
activations, we test in native precision (16 bits for non-QAT models, and 4
bits for the QAT models) as well as in 4 bits. For GPTQ we use a small amount
of C4 data for calibration.

###  5.2 Results

We report the results of our 1B experiments on the C4 and PTB dataset in table
1. We observe that our approach can learn a W4A4 model that has respectable
performance compared to the W16A16 baseline. We also observe that the gap
between the QAT model with and without kurtosis expands as weights are
quantized more and more. At full precision, the gap is less than 1%. At 4
bits, this expands to between 3% and 4%, and at 3 bits this gap widens to 21%.
All non-QAT method have catastrophic performance degradations with 4-bit
activations. Activation clamping is the only method that achieves less than
two orders of magnitude increase in perplexity. In table 2 we perform
experiments on downstream tasks for select models to validate our usage of
perplexity as a proxy for downstream performance. We observe that models with
similar perplexity exhibit similar downstream performance.

We also perform a suite of experiments at the 300M scale, where we just
experiment with the QAT baselines. This is shown in table 3. We largely
observe the same trends, with one exception: the gap between the QAT and
QAT+Kurtosis Regularization model is smaller than at the 1B scale.

Model | Setting | HellaSwag | PIQA | ARC-easy
---|---|---|---|---
Baseline | W16A16 | 32.13% | 65.51% | 48.32%
QAT | W16A4 | 31.79% | 65.56% | 47.85%
QAT + Kurtosis Regularization | W16A4 | 31.50% | 64.96% | 48.36%
Table 2: Downstream evaluation of our 1B models on HellaSwag, PIQA, and ARC-easy. | Native Activations |  | 4b Activations
---|---|---|---
Weight Precision | 16 | 4 |  | 4 | 4 | 3 | 3
Weight Quantizer | None | GPTQ |  | GPTQ | RTN | GPTQ | RTN
Baseline | 29.23 | 30.36 |  | 4288 | 3864 | 4820.5 | 3923.96
QAT | 30.25 | 31.30 |  | 31.30 | 32.55 | 36.47 | 44.73
QAT+Kurtosis Regularization | 29.95 | 30.83 |  | 30.83 | 31.73 | 35.47 | 45.04
Baseline | 32.61 | 34.12 |  | 2974 | 2896 | 3767 | 2950
QAT | 33.56 | 34.83 |  | 34.83 | 34.24 | 47.51 | 51.22
QAT+ Kurtosis Regularization | 33.14 | 34.23 |  | 34.23 | 34.55 | 40.74 | 52.63
Table 3: Perplexity of 300M models on C4 (top) and PTB (bottom). Native
activation are 16 bits for Baseline, Activation Clamping, Kurtosis
Regularization; and 4 bits for QAT, QAT + Kurtosis Regularization.

###  5.3 Analysis

#### Post-Training Quantization of Activations.

Our method shows that QAT from scratch is effective for training a model with
4-bit activations. However, given that most available pretrained models are
not trained with 4-bit activations, it would be ideal if we could take a
16-bit activation model and then finetune it with QAT to 4 bits. To test for
whether this is possible, we performed an extensive hyperparameter search for
QAT finetuning on the pretrained 300M baseline model, where we finetune with
QAT for 1B tokens. Even with extensive hyperparameter tuning, QAT finetuning
resulted in a W4A4 model with a 16% degradation in perplexity over the W16A16
baseline. Upon further investigation, we found that while our QAT-pretrained
models were able to learn to clip outliers without hurting performance, the
QAT finetuning models struggled to do so. Finetuning the model longer than 1
billion tokens did not improve results.

We also tried applying OmniQuant (Shao et al., 2023), a state-of-the-art
weight-and-activation method for PTQ, to go from W16A16 to W4A4. We found this
approach to not perform well, with a significant degradation in perplexity
with the 1B model (74.99 on C4 and 107.29 on PTB). Our degradation is larger
than what has been reported for pretrained models in the original paper, which
could potentially be due to our use of a smaller model (which are typically
harder to quantize). Given that the outlier channels seem to emerge early in
training (§3), these negative results highlight the importance of early-
training interventions for achieving 4-bit activation models.

#### Direct Approaches for Weight Regularization.

Our use of kurtosis regularization on the output activations to mitigate the
effect of “quantization difficulty migration” from the activations to the
weights is admittedly indirect. We also experimented with more direct methods
for controlling the outliers in the weights: regularizing the kurtosis of the
weights instead (at the tensor-level or at the column-level); and regularizing
the weight’s $l_{\infty}$ norm. Despite an extensive hyperparameter search,
these methods led to unstable training, and we were unable to get these models
to converge (unless the regularization-strength hyperparameter was so low that
there was effectively no regularization). QAT on the weights also proved
unsuccesful, with QAT-weight models underperforming baselines by a significant
margin.

#### Throughput.

Our QAT approach requires modifying the forward and backward passes, which
adds nontrivial overhead with an unoptimized, torch.compile-only
implementation. This is mainly due to the reduction step in the clip val
gradient in the backward pass. We thus implemented our own CUDA kernels that
perform a blockwise reduction followed by atomic additions to enable faster
throughput. The throughput of our custom kernels on a single H100 node (with
eight GPUs) is shown in table 4. We find that while there is still some
reduction in throughput, it is closer to the baseline setting than the
torch.compile implementation. Given that the numbers in table 4 are from a
single node, we anticipate that the actual throughput differences would be
even smaller when taking into account the necessary overheads of distributed
training.

##  6 Limitations & Discussion

There are several limitations to our study. While we experiment with language
modeling at moderate scale, we were unable to perform experiments on larger
models (and train for longer) due to limited compute resources. However, we
note that while the 300M parameter models did not benefit as much from the
kurtosis intervention on top of QAT, at 1B there was quite a large benefit;
this gives us optimism for the utility of our methods at larger scale.

Our study targets integer quantization to 4 bits to enable the use of INT4
matmuls, which is supported by the Ampere architecture GPUs. The more recent
GPU architectures (Hopper, Blackwell) unfortunately do not natively support
INT4 matmuls, which limit the applicability of our approach on these GPUs.
However, the latest Blackwell architecture supports FP4
computations,555https://www.nvidia.com/en-us/data-
center/technologies/blackwell-architecture/ and it is possible that QAT may
improve FP4-training and moreover enable even lower-precision quantization.

Finally, our study focuses on quantizing only the activations of inputs to
linear layers, since linear matmuls consumes the majority of FLOPs during LLM
inference (on moderate-length sequences). Future work could consider applying
QAT to quantize the activations involved in the attention computations, which
could be extremely useful in long-context settings.

Model size | Batch size | Baseline | QAT (torch.compile) | QAT (our custom CUDA kernel)
---|---|---|---|---
1B | 1M tokens | 41913 | 20195 | 37510
3B | 2M tokens | 15161 | 7519 | 13142
Table 4: Throughput in terms of tokens per second (TPS) on a single node with
eight H100s (higher is better). The baseline achieves approximately 50% mean
FLOPs utilization (MFU), while our kernel achieves 45%.

##  7 Conclusion

We study outlier channels in language models from a pretraining perspective.
We show that these channels emerge early in pretraining, and are moreover
particularly numerous in activations with residual streams. Based on these
findings, we propose a simple strategy for mitigating the effect of these
outlier channels through activation regularization. We regularize the input
activations with QAT plus learned clip values, and we further regularize the
output activations via the kurtosis. Our approach is able to learn a W4A4
language model at reasonable scale (1 billion parameters trained on 20B
tokens) that is competitive with the standard-precision W16A16 baseline.

## Acknowledgments

This study was supported by funds from an MIT-IBM Watson AI grant.

## References

  * Ashkboos et al. (2023) Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh.  Towards end-to-end 4-bit inference on generative large language models.  _arXiv preprint arXiv:2310.09259_ , 2023.
  * Bhalgat et al. (2020) Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization, 2020.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.  Pythia: A suite for analyzing large language models across training and scaling, 2023.
  * Chee et al. (2023) Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa.  Quip: 2-bit quantization of large language models with guarantees, 2023.
  * Chmiel et al. (2020) Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, Uri Weiser, et al.  Robust quantization: One model to rule them all.  _Advances in neural information processing systems_ , 33:5308–5317, 2020.
  * Choi et al. (2018) Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan.  Pact: Parameterized clipping activation for quantized neural networks, 2018.
  * Dettmers & Zettlemoyer (2023) Tim Dettmers and Luke Zettlemoyer.  The case for 4-bit precision: k-bit inference scaling laws.  In _International Conference on Machine Learning_ , pp. 7750–7774. PMLR, 2023.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.
  * Dettmers et al. (2023) Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.  Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023.
  * Egiazarian et al. (2024) Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh.  Extreme compression of large language models via additive quantization.  _arXiv preprint arXiv:2401.06118_ , 2024.
  * Frantar & Alistarh (2024) Elias Frantar and Dan Alistarh.  Marlin: a fast 4-bit inference kernel for medium batchsizes.  https://github.com/IST-DASLab/marlin, 2024.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  GPTQ: Accurate post-training compression for generative pretrained transformers.  _arXiv preprint arXiv:2210.17323_ , 2022.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  Olmo: Accelerating the science of language models, 2024.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2704–2713, 2018.
  * Jain et al. (2020) Sambhav Jain, Albert Gural, Michael Wu, and Chris Dick.  Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks.  _Proceedings of Machine Learning and Systems_ , 2:112–128, 2020.
  * Jouppi et al. (2021) Norman P Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, et al.  Ten lessons from three generations shaped google’s tpuv4i: Industrial product.  In _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_ , pp. 1–14. IEEE, 2021.
  * Jung et al. (2019) Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4350–4359, 2019.
  * Kim et al. (2023) Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer.  Squeezellm: Dense-and-sparse quantization, 2023.
  * Kovaleva et al. (2021) Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky.  Bert busters: Outlier dimensions that disrupt transformers.  _arXiv preprint arXiv:2105.06990_ , 2021.
  * Lee et al. (2023) Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park.  Owq: Lessons learned from activation outliers for weight quantization in large language models.  _arXiv preprint arXiv:2306.02272_ , 2023.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han.  Awq: Activation-aware weight quantization for llm compression and acceleration, 2023.
  * Liu et al. (2023) Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang.  Qllm: Accurate and efficient low-bitwidth quantization for large language models, 2023.
  * Puccetti et al. (2022) Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell’Orletta.  Outlier dimensions that disrupt transformers are driven by frequency.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 1286–1304, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.93.  URL https://aclanthology.org/2022.findings-emnlp.93.
  * Shao et al. (2023) Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo.  Omniquant: Omnidirectionally calibrated quantization for large language models.  _arXiv preprint arXiv:2308.13137_ , 2023.
  * Shoeybi et al. (2020) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.
  * Soboleva et al. (2023) Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023.  URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.
  * van Baalen et al. (2023) Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, et al.  Fp8 versus int8 for efficient deep learning inference.  _arXiv preprint arXiv:2303.17951_ , 2023.
  * Wei et al. (2022) Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu.  Outlier suppression: Pushing the limit of low-bit transformer language models.  In _Proceedings of NeurIPS_ , 2022.
  * Wei et al. (2023) Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu.  Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023.
  * Wu et al. (2023) Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He.  Understanding int4 quantization for language models: Latency speedup, composability, and failure cases.  In _Proceedings of the 40th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, pp. 37524–37539, 2023.
  * Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models, 2023.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pp. 10524–10533. PMLR, 2020.
  * Yuan et al. (2023) Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu.  Rptq: Reorder-based post-training quantization for large language models.  _arXiv preprint arXiv:2304.01089_ , 2023.
  * Zhang et al. (2018) Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * Zhao et al. (2023) Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci.  Atom: Low-bit quantization for efficient and accurate llm serving.  _arXiv preprint arXiv:2310.19102_ , 2023.
  * Zhou et al. (2016) Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  _arXiv preprint arXiv:1606.06160_ , 2016.