# Quantised Global Autoencoder: A Holistic Approach to Representing Visual
Data

Tim Elsner  Paula Usinger  Victor Czech  Gregor Kobsik  Yanjiang He  Isaak Lim
Leif Kobbelt
Visual Computing Institute
RWTH Aachen University
<EMAIL_ADDRESS>/ graphics.rwth-aachen.de

###### Abstract

In quantised autoencoders, images are usually split into local patches, each
encoded by one token. This representation is redundant in the sense that the
same number of tokens is spend per region, regardless of the visual
information content in that region. Adaptive discretisation schemes like
quadtrees are applied to allocate tokens for patches with varying sizes, but
this just varies the region of influence for a token which nevertheless
remains a local descriptor. Modern architectures add an attention mechanism to
the autoencoder which infuses some degree of global information into the local
tokens. Despite the global context, tokens are still associated with a local
image region. In contrast, our method is inspired by spectral decompositions
which transform an input signal into a superposition of global frequencies.
Taking the data-driven perspective, we learn custom basis functions
corresponding to the codebook entries in our VQ-VAE setup. Furthermore, a
decoder combines these basis functions in a non-linear fashion, going beyond
the simple linear superposition of spectral decompositions.
We can achieve this global description with an efficient transpose operation
between features and channels and demonstrate our performance on compression.

Figure 1: Our autoencoder produces a number of global tokens that represent
the image, with each token influencing the coloured region of the image. These
global tokens help to spread the unevenly distributed information content in
the image across the tokens. Example shows $256$ tokens, a ratio of $64$
pixels to one token in $[1,512]$, _i.e_. $288$ byte for an image from
ImageNet[5].

##  1 Introduction

The standard convolution operation on images only takes a local neighbourhood
of pixels into account. A vanilla convolutional autoencoder therefore produces
an encoding that stores aggregated local information in each latent vector, as
larger receptive fields are costly. With the lack of global information, each
part of an image needs the same amount of encodings to describe it properly.
Since images often have parts that contain much information, while other parts
do not have a high information content, encoding each part of the image with
the same amount of tokens is counter intuitive: Some tokens might contain
little information, while other regions might not be properly represented
because of a lack of tokens in the vicinity.
To bypass this problem, recent approaches introduced some degree of global
information, often using the attention mechanism[1] and the following
transformers[32, 6]. However, even in an attention-infused architecture,
latent variables still have a strong local bias. Similarly, even adaptive
resolution approaches[12] are improving a low-level grid-based representation,
already losing accuracy.
In classic image compression, approaches often relied on the Fourier transform
to produce an image decomposition where the first few global frequencies
provided an approximation of the compressed content. Our approach follows this
path, but instead of only providing a linear combination of fixed basis
functions to sum up, we learn a function that produces these elements. We
provide a strategy to learn a set of tokens with global information that are
encoded and decoded together in a holistic fashion, as shown in Fig. 1.

Our contributions are as follows:

  * •

We provide a new architecture recombined from simple parts, but with a small
twists to turn local into global information that we then quantise. We do not
require any artificial global information aggregators like attention.

  * •

We interpret the learned codebooks for the quantisation as Voronoi diagrams
that we optimise to evenly distribute information content among tokens.

  * •

We turn our approach into a hierarchical decomposition for better
interpretability with a simple regularisation.

  * •

We compare our compression rates and provide experiments to better understand
our representation.

Instead of encoding patches with often vastly different amounts of information
content using the same amounts of tokens, our global approach on compression
shows better performance, even outperforming methods that learn adaptive
hierarchical descriptions[12]. We demonstrate autoregressive generation as a
downstream task.
We build our approach around the quantisation mechanism introduced by the VQ-
VAE[31] framework to obtain quantised tokens. We introduce global information
into our tokens with a simple operation: After producing a feature map of the
image, we transpose image features with channels, then project each ”slice” of
the feature map down to a single token.
As we introduce global elements into the VQ-VAE framework, we call our
approach Quantised Global Variational Autoencoder (QG-VAE).

##  2 Related Work

We discuss Traditional Compression Methods for image compression and learned
methods based around Autoencoders, as we combine ideas from both domains. We
further elaborate the different approaches in learned autoencoder-based
approaches, then dive into Quantised Methods that try to increase the
information content of the representation, as we aim to produce a quantised
and finite representation with our approach.

###  2.1 Traditional Compression Methods

In traditional image compression, images are often decomposed into different
base frequency bands, then expressed as a linear combination. Exemplary,
Watson _et al_. use the discrete cosine transformation[35] that was later used
for the JPEG compression standard[34]. Similarly, the Laplacian Pyramid[3]
expresses an image as a set of frequency images ordered from coarse to fine,
finding application _e.g_. in texture synthesis [10]. As a related approach,
Eigenfaces[30] compute a basis set of faces, then express faces as a linear
combination, only storing coefficients of the basis functions.
While these methods are fast and do not require training a neural network,
they all only express images as a linear combination of components, never in a
context sensitive and holistic manner.

###  2.2 Autoencoders

Neural networks are trained to produce information to output values that
fulfil some objective. Reproducing the same output, called Autoencoder,
automatically produces a compact representation at the smallest part
(bottleneck) of the architecture[25, 11]. Variational autoencoders[14] extend
this concept by introducing a probabilistic take, decoding regions instead of
points from latent space to an output.

##### Quantised Methods

With the advent of transformers as new state of the art for autoregressive
text generation, van den Oord _et al_. introduced the VQ-VAE[31], which
produces quantised representations of an input by rounding to a nearest
codeword from a codebook while minimising the amount of rounding. Similar to
VAEs, this maps regions of the latent space that are rounded to the same
value, to the same output. This representation allows storing inputs as
indices in the codebook, ideal for compression and transformers for
generation[22, 9, 37]. Their quantisation approach forms the basis of our
approach.
While ideas exist that _e.g_. to reduce the complexity of the approach[20] or
to strengthen the architecture[28, 23], most related work focuses on either
improving information content or improving realism of the output. Tackling the
uneven distribution of information similar to us, Huang _et al_. introduce an
adaptive refinement of VQ-VAEs, called DQ-VAE[12], adding extra tokens for
refining details in high information content regions. Other ideas include
adding a stochastic element to the tokenisation [29]. A different branch
focuses on optimising codebook usage, distributing information evenly across
the codebook, _e.g_. directly through optimal transport[2], and strategies
like resetting unused codewords[40] and/or regularising the distribution of
codewords[16, 33, 36]. Our approach follows the codebook reset strategy for
simplicity.
In a different direction, the VQGAN family introduced by Esser and Rombach _et
al_. [7] and following work[38, 41] tries to compensate for the lack of
information from the compression by filling gaps with a GAN-like[8, 27]
network that ”sharpens” the output. Recently, TiTok[39] applies a full
transformer architecture based on the Vision Image Transformer[6] to achieve
even more compact codes with good quality.

##  3 Global Quantised Autoencoder

Figure 2: Our base architecture: We encode our data by increasing the input to
$C$ different feature maps in the channels (desired number of codewords) with
a U-Net [24], then compress each channel individually into a token. We do so
by transposing channel and collapsed feature dimension, then applying a single
linear layer / affine transformation $K$, split into different heads (green
and violet). For quantisation, we use the same idea as a naive VQ-VAE. We then
decode the result in the same manner.

Figure 3: We express re-initialising codewords while relieving overused
codewords as re-arranging a Voronoi diagram through moving the Voronoi
centres.

We propose to decompose an input into global tokens unlike a regular VQ-VAE
[31], where each token (mostly) corresponds to a local patch. Exemplary for
images, regions with lower amounts of information content do not waste tokens,
_e.g_. for an image of a person in front of a blue sky, regions with a blue
sky have less information than the face of the person, hence should not use
the same amount of tokens. Our approach works in five steps to tackle this
problem, the full pipeline shown in Fig. 2:

  1. 1.

The input is processed into a multi-channel feature map of the image, each
channel holding global information.

  2. 2.

Each individual channel of the feature map is processed into a single vector
that holds global information.

  3. 3.

Each vector is quantised into a discrete token.

  4. 4.

Every token is decoded back into a single channel of a feature map.

  5. 5.

The feature map is processed back to a full image.

Some examples are shown in Fig. 4, with more shown in Sec. G.

###  3.1 Background

To better motivate the setup of our autoencoder architecture, we first recap
some basics about the Fourier transform.
The core idea of this transform is to represent a (spatial) signal as a
superposition of frequency functions. If we restrict the signal to be discrete
(and periodic), the frequency spectrum is also discrete (and periodic) leading
to the so-called Discrete Fourier Transform (DFT):

$F(u,v)=\sum_{x=0}^{W-1}\sum_{y=0}^{H-1}I_{x,y}\cdot
e^{-i2\pi\left(\frac{ux}{W}+\frac{vy}{H}\right)},$ (1)

where $I_{x,y}$ describes a pixel of an image $I$ that has the size $W\times
H$. By using (complex) sine and cosine functions as basis functions of a
linear function space, each input signal can be effectively approximated by a
linear combination of frequency basis functions. The coefficients of this
linear combination are the frequency components (complex amplitudes). If we
reconstruct the signal from only a sub sequence of the frequency components,
we obtain a low frequency approximation of the input. More generally, we can
express the frequency decomposition $F$ for $n$ frequencies as a set of
independent frequency band computations with:

$F=\\{f(I,i)|i=0,..,n\\}$ (2)

These can be decoded back to an image by linear combination of an inverse
function $f^{-1}$:

$\hat{I}=\sum_{i=0,..,n}f^{-1}(f(I,i),i)$ (3)

While this is a generally good representation of arbitrary data, we rather
want to learn a frequency decomposition-like representation that is content
sensitive and more expressive. We do so by learning individual functions per
frequency, _i.e_. not only by modulating the frequency of a sine/cosine
function, but by training a neural network that applies an individual function
per frequency band. We further want to allow more complex relationships
between the frequency bands, _i.e_. not only sum up the frequencies as a
linear combination, but to process them in context to each other. With both
the global tokens and the non-trivial combination of frequency bands for
decoding, we describe our approach as A Holistic Approach to Representing
Visual Data and call it Quantised Global Variational Autoencoder (QG-VAE).
Hence, Eq. 2 and Eq. 3 become:

$F=E(I),\hat{I}=D(F)=D(E(I))$ (4)

for a learnable function $E$ and its inverse $D$ being holistic functions in
the sense that they process all frequency bands together.
We call the resulting factors of our decomposition pseudo frequency tokens, as
they differ to a regular frequency decomposition in two main aspects. While in
the Fourier transform, the coefficients are computed by evaluating a dot
product between the input and a fixed frequency basis functions, we replaces
this by a learned neural network and quantise these coefficients. While
learning an individual function per pseudo frequency would be ideal, this
would, for $C$ frequency bands, require learning $C$ different global
functions that process the input image. This is impractical for our purposes
due to a high number of parameters and hence difficulties in generalising
across multiple images. It also does not allow processing the different pseudo
frequencies in context to each other. Instead, we resort to a small change
that makes our approach feasible. Where DFT applies a different function to
the same image through the changed exponent to produce each frequency, we
instead use the same (learned) function $K$ at our core to produce the
different pseudo frequencies. We obtain these pseudo frequencies
$S_{i},i\in{0,1,...,C}$ by applying $K$ to $C$ different feature maps of the
input image, with $E$ producing a feature map of the input:

$E(I)=F_{map},S_{i}=\\{K(F_{map}^{i})~{}|~{}i~{}\in{0,1,...,C}\\}$ (5)

With $F_{map}^{i}$ describing a single channel of the feature map. In our
case, $K$ are always learned affine transformations turning $W\cdot H$
dimensional feature maps into $64$-dimensional tokens to quantise, with
$K^{-1}$ then transforming $64$-dimensional tokens back into a $W\times H$
dimensional feature map. This feature map is then transformed back into an
image by the decoder $D$.
Hence, while DFT applies a function with different parameters to the same
global image multiple times, we split this in two parts to save learnable
parameters. We first apply our learnable transformation $E$ to produce
different feature maps (instead of the same input for Fourier) that we then
put into the same learnable function (instead of applying a individually
modulated function for Fourier). This effectively splits the learnable
parameters in an individual part and a shared part, making our approach easier
to train.
We further discuss alternatives to our formulation and their pitfalls in Sec.
E (Appendix).

###  3.2 Quantised Global Autoencoder

Our approach follows the same principle for all modalities and variants. We
first apply a U-Net [24] to increase the number of channels of our image to
the number of codewords $C$ we want to obtain (Encoding), _e.g_. for RGB
images processing $[B\times 3\times W\times H]$ into $[B\times C\times W\times
H]$. For closeness to actual code and optimisation, we use a notation with
batch size $B$. Each of these $C$ feature maps (channels) then contains a full
image, with each pixel being enriched with global information. We then switch
from local to global features, compressing each feature map individually into
a token, obtaining $C$ tokens.
We reverse this compression back into an image with $C$ channels and apply a
second U-Net to re-combine these images into one output (Decoding).
The following paragraphs describe these steps in more detail.

##### Encoding and Decoding of the Data

To ensure we have all the information distributed across the feature maps that
each form their own pseudo frequency, we apply a U-Net [24] before and after
the switch from local to global features, referred to as Encoder $E$ and
Decoder $D$. These U-Nets, together with the affine transformation $K$, then
learn what corresponds to the projection to the basis functions. To keep
memory requirements reasonable, we omit some of the upscaling (encoder) or
downscaling (decoder) steps for larger images, _e.g_. turning our $[B\times
3\times W\times H]$ dimensional input into a $[B\times
C\times\frac{W}{2^{f}}\times\frac{H}{2^{f}}]$ feature map for a downscaling
factor $f$.

##### Switching from Local to Global Features

To process the multiple feature maps in parallel, we use a transpose operation
that swaps feature dimension of the combined width and height with the channel
dimension, see Fig. 2. For a processed image of size $[B\times C\times W\times
H]$, we collapse the two spatial dimensions into one feature dimension, then
transpose features and channels to obtain a $[B\times W\cdot H\times C]$
dimensional tensor. Now each of the $C$ features holds a full (flattened)
feature map in its channels. We then apply a $1$D convolution with kernel-size
$1$, effectively a single learned affine transformation. This reduces our
image-sized channels down to the space in which we apply quantisation,
typically $64$ dimensions. For this example, we obtain a $[B\times 64\times
C]$ dimensional tensor. Inspired by transformers, we optionally split the
affine transformation $K$ compressing the feature maps into the quantisation
space into multiple heads, by using multiple different linear layers, _e.g_.
one to produce codewords $1$ to $32$, another to produce codeword $32$ to
$64$, and so on.
To then quantise, we use the same approach as traditional VQ-VAEs [31]:
Rounding to the nearest codeword from our codebook, and then both regularising
so that the encoder produces an output close to the codewords ($L_{comm}$) and
so that the codewords move closer to what the encoder produces ($L_{code}$).
We use the same loss formulation that uses the distance function to nearest
codeword $d$ from the codebook $e$:

$L=L_{rec}+L_{comm}+0.25\cdot L_{code}\\\
=[\hat{D}(\hat{E}(I))-I]^{2}+d(\lfloor\hat{E}(I)\rfloor,e)+d(\hat{E}(I),\lfloor
e\rfloor)$ (6)

where $\hat{E}$ and $\hat{D}$ already contain the affine transformations into
quantisation space $K$ and $K^{\prime}$ and the encoder/decoder $E$ and $D$.

###  3.3 Optimising Codebook Useage

Similar to approaches like SVQ-VAE[29] and VQ-WAE [33], we want to improve the
codebook usage. The VQ-VAE losses do not penalise unused codewords, only
producing encoder outputs distant from the codebook. Hence, the autoencoder
wastes potential. While one single codebook already provides good results, we
use individual codebooks per token for better codebook usage and performance
with slightly longer convergence time. We use an individual codebook to
quantise each feature map $C$, because as opposed to a traditional VQ-VAE, we
no longer have the same meaning for tokens across the quantised domain:
Multiple similar patches of grass in a quantised image would traditionally be
mapped to the same token. However, in our case, the $i$th codebook index
should have a different meaning in the $n$th frequency band than in the $m$th
frequency band. Note that this does not affect the tokens we need to store, as
we still only need to store the list of indices of our tokens to store an
input.
While resetting unused codebook entries or regularising the distribution of
codewords [16, 36, 40] is nothing new, the individual nature of our entries
(each frequency band having its own codebook) is making this more complicated:
With individual codebooks per pseudo frequency, we only have few samples for
every codebook per batch, making regularisation unstable. We instead opt to
reset underused codewords to explicitly disencumber overused codewords. For
this, we consider the space to quantise as the space of a Voronoi diagram,
with different codewords being the Voronoi centres. We can compute codebook
resets by simply tracking the gradient magnitude of our codebook over multiple
batches as the gradient is a direct measure for how much an entry should be
changed to improve the result. Through a number of steps, usually $500$, we
sum up all the gradient magnitudes for each codeword. In result, this
basically gives each Voronoi cell an error rate. Unused codewords and their
associated Voronoi cells will have an accumulated gradient magnitude of $0$,
while overused codewords that are often used but are not chosen perfectly will
have a high amount of gradient magnitude. Hence, we move an unused codeword
(and hence, the associated Voronoi centre) to the position of the most
overused codeword with the largest gradient, and perturb it in a random
direction $d\cdot\epsilon$. Codebook and commitment loss will then separate
the two codewords further, splitting the overused codeword / one overused
Voronoi cell into two distinct ones. This balances the amount of information
content between the codewords and thus the information content in each Voronoi
cell of the encoding space with just a few lines of code.

###  3.4 Obtaining a Meaningful Decomposition of the Latent Space

While our approach already provides a decomposition into pseudo frequencies
that is somewhat structured, we can order this from important to unimportant.
This extra structure can be an additional help to understand our approach and
is yet again inspired by classic frequency decomposition.
In fields like texture synthesis, a coarse to fine structure for a frequency
decomposition is often achieved through the Laplacian pyramid $L$ with levels
$0$ to $n$, build from the Gaussian Pyramid $G$ and its levels $G_{i}$ for an
input $G_{0}=I$:

$G_{i+1}=\text{reduce}(G_{i})\quad\text{for}\quad i=0,1,\ldots,n-1\\\
L_{i}=G_{i}-\text{expand}(G_{i+1})\quad\text{for}\quad i=0,1,\ldots,n-1\\\
L_{n}=G_{n}$ (7)

The produced image pyramid thus contains a base image $G_{n}$ and multiple
frequency bands $L_{i-1}$ to $L_{0}$ that always ensure full reconstruction.
When applying the Laplacian pyramid for all image sizes, we are left with $i$
frequency bands. With this formulation, taking the first $k$ frequencies is
always the optimal solution in the sense that there is no other way of
combining $k$ different of these frequency bands to obtain a better $L2$
error.
Inspired by this, we re-formulate our reconstruction loss such that using the
first $k$ pseudo frequency bands should provide the best $L2$ loss:

$L_{rec}=\sum_{k\in[0,C]}~{}(f_{k}(x)-x)^{2}$ (8)

where $f_{k}$ is our trainable network only using the first $k$ pseudo
frequencies.
As training to properly decode this optimally for every input with every
different pseudo frequency is cumbersome, we instead do this in a
probabilistic way. We apply a dropout-like procedure on every third batch,
multiplying everything from the $k$th latent code onwards with $0$, where
$k\sim\\{0,1,2,\ldots,n\\}$:

$L_{rec}=(D(E(x)\circ d^{k})-x)^{2}$ (9)

where $d^{k}$ is a tensor where $d^{k}_{i}=1$ for $i<k$ and $d^{k}_{i}=0$
otherwise, nullifying every token with an index greater than $k$ of our
encoder output. With optimal training, this is close to the formulation of the
Laplacian pyramid: Our decoder outputs an average over the dataset when
decoding a latent space with no information ($k=0$), with each pseudo
frequency band adding the largest amount of detail possible. We visualise the
resulting decomposition in Fig. 6.
With this regularisation, we gain more interpretability, but do lose some
precision. Instead of enforcing a certain way for our U-Net to work (here
expressed as to being able to properly decode with only subsets of our pseudo
frequency bands given), we achieve the best performance by processing all
pseudo frequencies in a holistic sense.

Figure 4: Input, output, and absolute difference between the two, produced by
our autoencoder. $256$ tokens/$288$ byte for ImageNet[5] and CelebA[18], $64$
tokens/$72$ bytes for CIFAR-10[15]. All with $512$ tokens in the codebook.
Note how sharp the features, _e.g_. the horse rider, in CIFAR are.

##  4 Evaluation

We run all our benchmarks on a single Nvidia GeForce RTX 2080 Ti,
demonstrating that our approach requires no expensive hardware or excessive
amounts of VRAM. We further use a U-Net with residual blocks from [26] and
downscaling/upsampling instead of max pooling/transposed convolution to make
our code less intricate. We include an exemplary one-click Jupyter Notebook
with the right settings111Code can be found at https://www.graphics.rwth-
aachen.de/media/resource_files/demo.ipynb.

###  4.1 Benchmarks in Compression

To evaluate the compression ability at the core of our approach, we expand the
comparison of VQ-WAE [33]. We train our approach for the same number of epochs
($70$ for CelebA, $100$ for every other benchmark), then evaluate on the
testset. Details about our training can be found in Sec. A. Results can be
found in Tab. 1, showing the strong performance of our approach. We further
like to point out that our approach surpasses most of these benchmarks and
saturates earlier.
We interpret these results in the context of global versus local descriptions:
An image is no longer composed of patches that possibly have visible seams
that can be picked up by the NN-based metrics like FID and LPIPS. Our approach
is biased towards learning an accurate, holistic representation through its
global feature maps instead of local patches that form a token. We also
observe that features in produced images compared to a regular VQ-VAE might
still not be correct, _i.e_. offset or wrong, but they look much sharper than
at similar PSNR values. This heavily shows in the FID scores of CIFAR-10. For
reference, consider high-frequency details like the legs of the horse rider in
Fig. 4. Also note that to compute the FID values, the latent space of a model
trained on ImageNet is used, which does not contain close up features of
faces, and should hence be taken with a grain of salt on the CelebA benchmark.
We provide extra qualitative results in Sec. G, in particular on larger
datasets, resolutions, and number of tokens. We also do some qualitative
analysis and compare to a VQ-VAE directly to better show the visual
improvements of our approach.

MNIST[17], $8$-by-$8$ tokens versus $64$ tokens (ours)
---
Metric | VQ-VAE | SVQ-VAE | VQ-WAE | QG-VAE
SSIM $\uparrow$ | 0.98 | 0.99 | 0.99 | 0.997
PSNR $\uparrow$ | 33.37 | 36.25 | 35.71 | 38.29
LPIPS $\downarrow$ | 0.02 | 0.01 | 0.01 | 0.002
FID $\downarrow$ | 4.8 | 3.2 | 2.33 | 1.13
CIFAR-10[15], $8$-by-$8$ tokens versus $64$ tokens (ours)
Metric | VQ-VAE | SVQ-VAE | VQ-WAE | QG-VAE
SSIM $\uparrow$ | 0.70 | 0.80 | 0.80 | 0.86
PSNR $\uparrow$ | 23.14 | 26.11 | 25.93 | 26.18
LPIPS $\downarrow$ | 0.35 | 0.23 | 0.23 | 0.18
FID $\downarrow$ | 77.3 | 55.4 | 54.3 | 38.9
SVHN[21], $8$-by-$8$ tokens versus $64$ tokens (ours)
Metric | VQ-VAE | SVQ-VAE | VQ-WAE | QG-VAE
SSIM $\uparrow$ | 0.88 | 0.96 | 0.96 | 0.97
PSNR $\uparrow$ | 26.94 | 35.37 | 34.62 | 36.25
LPIPS $\downarrow$ | 0.17 | 0.06 | 0.07 | 0.06
FID $\downarrow$ | 38.5 | 24.8 | 23.4 | 14.16
CelebA[18], $16$-by-$16$ tokens versus $256$ tokens (ours)
Metric | VQ-VAE | SVQ-VAE | VQ-WAE | QG-VAE
SSIM $\uparrow$ | 0.82 | 0.89 | 0.89 | 0.94
PSNR $\uparrow$ | 27.48 | 31.05 | 30.60 | 31.44
LPIPS $\downarrow$ | 0.17 | 0.06 | 0.07 | 0.09
FID $\downarrow$ | 19.4 | 14.8 | 12.2 | 16.09
Table 1: Comparison of our approach (QG-VAE) to other VQ-VAE-based compression schemes in similar conditions, namely the vanilla VQ-VAE[31], SVQ-VAE[29], and VQ-WAE[33]. We build upon the data of [33]. Details given in Sec. A. Bold indicates best performance. | CIFAR-10[15] | CelebA64[18]
---|---|---
Metric | DQ | QG | DQ | QG
SSIM $\uparrow$ | 0.499 | 0.683 | 0.836 | 0.844
PSNR $\uparrow$ | 17.35 | 21.94 | 25.48 | 26.74
LPIPS $\downarrow$ | 0.469 | 0.357 | 0.190 | 0.184
FID $\downarrow$ | 140.3 | 88.35 | 40.74 | 34.66
Table 2: Comparison of training a hierarchical DQ-VAE[12] compared to ours for
$50$ epochs, always with our QG-VAE using the same number of tokens ($20$ for
CIFAR, $80$ for CelebA64).

####  4.1.1 Comparison to Adaptive Refinement

We further compare to the approach of Huang _et al_. [12]. They propose a
hierarchical approach to VQ-VAEs by additionally refining those regions that
have a high reconstruction error. We applied their code to CIFAR and CelebA,
then compare this explicit hierarchical decomposition against our approach in
Tab. 2. As we are concerned for the actual amount of information encoded, not
for the qualities of an additional post processing, _e.g_. through a GAN[7],
we chose to use both their approach and our approach without this sharpening.
We report better performance, concurring with what we also observed in other
experiments: With fewer tokens, our approach outperforms other approaches even
stronger. While their approach improves results, it can only make a binary
decision to refine an area, and due to the locality of the additional tokens,
is still bound to a (refined) grid layout.

###  4.2 Applications of our Latent Space

While compression is our main focus, we did some basic exploration of using
our tokenised $1$D latent space for generation. We theorise that our natively
linear latent code is beneficial for generation: Autoregressively predicting
token by token, and hence, patch by patch of in a $2$D grid, does not come
native for a model designed for 1D sequences. We provide some (early) results
on generating CIFAR images with an autoregressive transformer[13], hinting at
the potential for future work, in Sec. H.

Input

$0$ tokens

First $8$ tokens

First $32$ tokens

First $128$ tokens

Figure 5: Learned pseudo frequency decomposition on the CelebA dataset [18].
From left: Input, then from left to right, we allow more tokens in the latent
code to obtain more detail in the output. All images produced by using the
regularisation from Sec. 3.4.

Figure 6: A small ablation study on CIFAR-10[15], showing various settings and
the impact of our regularisation procedure to order the latent space.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 7: Learned feature maps of our approach: We normalised the individual
$C$ many $\frac{W}{2^{f}}$-by-$\frac{H}{2^{f}}$ feature maps from the step
before they would be compressed into pseudo frequency tokens by the affine
transformation, then visualised $16$ randomly chosen ones here, all from the
same image as Fig. 6, but not ordered, _i.e_. without Sec. 3.4.

###  4.3 Interpretation of our Holistic Representation

We generally believe our better compression rate to be rooted in holistic
instead of local descriptions of an input. We explicitly demonstrate how our
learned latent space produces global features instead of localised ones with
additional experiments, from which we can draw further conclusions about the
properties of our learned space.
First, we visualise the feature maps, _i.e_. the $C$ many different channels
of our $[B\times C\times\frac{W}{2^{f}}\times\frac{H}{2^{f}}]$ tensor before
compressing each $C$ dimension into a token by affine transformation and
quantisation in Fig. 7. This shows how each token captures different global
properties, following the a holistic strategy to not just encode an input
through local patches.
Second, we average the change for all possible replacements for a token in the
output in Fig. 1. We observe very global changes for single token changes,
compared to very local changes for VQ-VAE. Similarly, we can slightly change
an input image to provoke large changes in the learned latent space: For a
simple change of $20$-by-$20$ pixels in an $128$-by-$128$ pixel image, we
observe a change of $27$ tokens, while we have $\frac{128^{2}}{256}=64$ pixels
per token. An example can be found in Sec. F.
Third, we train an ordered latent space according to Sec. 3.4, with the first
$k$ tokens always holding the largest amount of content in the least squares
sense. Then, we show the learned decomposition in Fig. 6, showing how
gradually the image changes from an average to the target image.
Notably, even with this enforced decomposition, our approach has properties
that are not typical for a simple linear combination. As an example, we
consider _e.g_. Eigenfaces[30] that compute an ordered set of face
coefficients from high to low importance via PCA. Hence, adding new
coefficients of base elements provides additional detail, while our approach
is able to completely change attributes instead (_e.g_. gradually moving the
smile with more tokens in Fig. 6). However, as our ordering regularisation
means that each first $k$ tokens of pseudo frequency bands must also be
interpretable on their own, we observe some decay in reconstruction quality
compared to our holistic version without regularisation.
Fourth, we present a comparison of different stages of the training process
for a VQ-VAE and our approach in Sec. D (Appendix).
Lastly, we show a small ablation study to demonstrate the impact of the
different parameters Fig. 6, always showing deviation from the settings we
found to be ideal.

###  4.4 Limitations, Discussion, Future Work

While our approach has a number of advantages despite its simple nature and
lack of more complex components like attention, we also highlight a number of
smaller drawbacks: As tokens are no longer local, changes to a small region
will possibly change a large number of tokens, and changes to a single token
can impact whole regions. This makes _e.g_. inpainting-like operations more
difficult.
Further, our approach does generally require more parameters, as two U-Nets as
backbone are more expensive as _e.g_. convolution and pooling of a regular VQ-
VAE. However, especially for smaller resolutions (_e.g_. CIFAR), we find that
our approach performs similarly with the same number of parameters. Even with
those higher numbers of parameters, we find that a U-Net with at most $128$
channels is not extra ordinarily expensive given the shown performance, in
particular considering that our vanilla loss function identical to a regular
VQ-VAE[31] is much cheaper to compute than _e.g_. the mathematical
regularisation of [33]. We show the exact parameter combinations in Sec. A and
discuss details in Sec. B. Additionally, we do not optimally utilise our
codebooks: We do not ensure that our codebook resets are optimal, as we do not
ensure that we split a codebook entry in an optimal way (Fig. 3 shows an
optimal split). This can lead to potentially having multiple codewords that
are very similar, but are still each with a high gradient. We only reset
unused codebook entries instead of resetting very rarely used ones. Also, we
decided not to include attention[1], as to not dilute that the architecture
itself is able to produce a global representation. We leave this for future
work to keep our baseline approach simple.

##  5 Conclusion

We propose a holistic approach to autoencoding, capturing global features in
quantised tokens. Opposing to traditional decompositions, we process these
pseudo frequency tokens together. We believe that this holistic take on
autoencoding has the potential to spawn a whole new branch of autoencoders, as
we demonstrate success in multiple benchmarks with a rather simple
formulation. We do so without any powerful mathematical regularisations like
Wasserstein loss or recent architectural improvements like attention or
artificial sharpeners.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [2] Xiaoyu Bie, Dexiong Chen, Xiaodong Cun, and Xi SHEN.  Learning discrete representation with optimal transport quantized autoencoders, 2023.
  * [3] Peter J Burt and Edward H Adelson.  The laplacian pyramid as a compact image code.  In Readings in computer vision, pages 671–679. Elsevier, 1987.
  * [4] Aaron Defazio, Xingyu, Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky.  The road less scheduled, 2024.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  CoRR, abs/2010.11929, 2020.
  * [7] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  CoRR, abs/2012.09841, 2020.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [9] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696–10706, 2022.
  * [10] David J Heeger and James R Bergen.  Pyramid-based texture analysis/synthesis.  In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 229–238, 1995.
  * [11] Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  science, 313(5786):504–507, 2006.
  * [12] Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang.  Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization, 2023.
  * [13] Andrew Karpathy.  mingpt, 2023.
  * [14] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2022.
  * [15] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [16] Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumäe, and Antoine Laurent.  Robust training of vector quantized bottleneck models.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2020.
  * [17] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [18] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [19] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * [20] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen.  Finite scalar quantization: Vq-vae made simple.  arXiv preprint arXiv:2309.15505, 2023.
  * [21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.  Reading digits in natural images with unsupervised feature learning.  In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
  * [22] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  CoRR, abs/2102.12092, 2021.
  * [23] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2, 2019.
  * [24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015.
  * [25] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986.  Biometrika, 71(599-607):6, 1986.
  * [26] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * [27] Jürgen Schmidhuber.  Learning factorial codes by predictability minimization.  Neural Computation, 4(6):863–879, 1992.
  * [28] Dario Serez, Marco Cristani, Vittorio Murino, Alessio Del Bue, and Pietro Morerio.  Enhancing hierarchical vector quantized autoencoders for image synthesis through multiple decoders.  In International Conference on Image Analysis and Processing, pages 393–405. Springer, 2023.
  * [29] Marek Strong, Jonas Rohnke, Antonio Bonafonte, Mateusz Łajszczak, and Trevor Wood.  Discrete acoustic space for an efficient sampling in neural text-to-speech.  arXiv preprint arXiv:2110.12539, 2021.
  * [30] M.A. Turk and A.P. Pentland.  Face recognition using eigenfaces.  In Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 586–591, 1991.
  * [31] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [33] Tung-Long Vuong, Trung Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Phung.  Vector quantized wasserstein auto-encoder, 2023.
  * [34] G.K. Wallace.  The jpeg still picture compression standard.  IEEE Transactions on Consumer Electronics, 38(1):xviii–xxxiv, 1992\.
  * [35] Andrew B Watson et al.  Image compression using the discrete cosine transform.  Mathematica journal, 4(1):81, 1994.
  * [36] Hanwei Wu and Markus Flierl.  Vector quantization-based regularization for autoencoders.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6380–6387, 2020.
  * [37] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [38] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu.  Vector-quantized image modeling with improved VQGAN.  CoRR, abs/2110.04627, 2021.
  * [39] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen.  An image is worth 32 tokens for reconstruction and generation.  arXiv preprint arXiv:2406.07550, 2024.
  * [40] Chuanxia Zheng and Andrea Vedaldi.  Online clustered codebook.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22798–22807, 2023.
  * [41] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen.  Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%.  arXiv preprint arXiv:2406.11837, 2024.

##  A Benchmark Parameters

While modern optimisers such as schedule free Adam[4] worked better in our
early experiments, for comparability, we train all our examples with
AdamW[19], using a learning rate of $0.0002$ with weight decay of $0.01$.
While the exact number of parameters of the architecture in [33] is not
specified, we argue that our architecture is vastly different and hence
difficult to compare. However, exemplary for CIFAR, a classic VQ-VAE with the
same number of channels performs worse, with similar numbers for the other
approaches. Our approach works similarly good when using significantly less
parameters when using a higher downscaling factor, _i.e_. the encoder U-Net
producing a $[B\times C\times\frac{W}{8}\times\frac{H}{8}]$ as we demonstrate
for cases in ImageNet and CelebA in Fig. 15 and Fig. 14.
We omit the perplexity score in our benchmarks, as we do not, as in VQ-
VAE[31], have the same tokens in different locations, but tokens that can mean
very different things in different pseudo frequency bands.
We specify the exact settings in Tab. 3. Note that for CelebA, for
comparability, we follow [33] in using a $64$-by-$64$ version of CelebA.

Used in | Dataset | Codewords | Feature Map / $f$ | Heads | Blocks | Channels | Reset time | Epochs
---|---|---|---|---|---|---|---|---
Default | MNIST[15] | $64$ |  $32$-by-$32/f=0$ | $8$ | $2$ | $256$ | $100$ | $100$
Default | CIFAR-10[15] | $64$ |  $32$-by-$32/f=0$ | $8$ | $1$ | $128$ | $100$ | $100$
Default | SVHN[21] | $64$ |  $32$-by-$32/f=0$ | $8$ | $1$ | $128$ | $100$ | $100$
Default | CelebA(64)[18] | $256$ |  $16$-by-$16$ / $f=2$ | $64$ | $2$ | $256$ | $1000$ | $70$
Default | CelebA(256*)[18] | $256$ |  $32$-by-$32$ / $f=3$ | $64$ | $2$ | $128$ | $1350$ | $55$
Fig. 16 | ImageNet$64$[5] | 256/1024 |  $16$-by-$16$ / $f=2$ | $64$ | $2$ | $128$ | $1000$ | $100$
Default | ImageNet$128$[5] | $256$ |  $32$-by-$32$ / $f=2$ | $64$ | $2$ | $128$ | $1000$ | $100$
Table 3: The different training parameters used for our experiments, with
’Default’ meaning these settings were used throughout the paper if not stated
otherwise. ’Codewords’ describes the number of codewords that encode one
input, ’Feature Map’ the resulting width/height after the U-Net, ’Heads’ shows
the number of different affine transformations to turn a feature map into the
space in which we quantise, ’Blocks’ and ’Channels’ describe the number of
residual blocks and maximum number of channels for our U-Net, and ’Reset time’
describes after how many batches we reset our unused codebook entries.
*We trained on the original CelebA images embedded in $256$ pixel black boxes to avoid adapting our U-Net. As we have no local tokens, adding a static black frame around our image does not harm quality.

##  B Implementation Details

As the number of codewords grow, _e.g_. from a very large input image in full
HD resolution, the number of codebooks grows with it. While this has not been
an issue for any of our benchmarks or even our large test images on
ImageNet[5] at $256$ pixels, we recommend using either fewer dimensions for
the quantisation dimension or using partially shared codebooks between _e.g_.
$16$ frequencies at a time, similar to the heads of the affine transformation
that project an $[B\times C\times W\times H]$-sized feature map to the
quantisation dimension. We observed this to work well in early experiments,
but omitted this due not not processing any extra large images and slightly
weaker codebook resets.

##  C Additional Feature Maps

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 8: Learned feature maps of our approach when ordering the latent space
as discussed in Sec. 3.4: We normalised the individual $C$ many
$\frac{W}{2^{f}}$-by-$\frac{H}{2^{f}}$ feature maps from the step before they
would be compressed into pseudo frequency tokens by the affine transformation,
then visualised the first $16$ ones, all from the same image as Fig. 6.

We provide the ordered feature maps of Fig. 6, similarly to Fig. 7, in Fig. 8.
Note how these feature maps are visibly more orthogonal to each other.

##  D Training Progress

VQ-VAE, Input, QG-VAE | VQ-VAE, Input, QG-VAE | VQ-VAE, Input, QG-VAE | VQ-VAE, Input, QG-VAE
---|---|---|---
|  |  |

(a)

(b)

(c)

(d)

Figure 9: Slightly different formation of features during training. The figure
is meant to show the different ways how the two approaches learn, not to
compare the image quality. Therefore, all images in a row are taken during
roughly the same PSNR, with increasing PSNR from top to bottom.

Differences during the way an image forms during training are observable, we
hence output a number of processes in Fig. 9. We can observe that our approach
learns to form the image slightly differently.

##  E Alternative Formulations

While our approach of first reducing an image size, then translating between
feature and channel dimension to turn local into global features, works well,
we also considered some alternative ways to obtain global tokens:

##### Fully Convolutional Architecture

For an input image, we can compress it down by applying convolution and
pooling layers until we reach a single feature with $C$ many channels to
represent an image, then quantise and decode the result. However, as the
quantisation usually needs about $64$ dimensions to work properly (see [31]),
this would require $C*64$ many channels at feature size $1$-by-$1$. For _e.g_.
$1024$ tokens (which we need for proper ImageNet reconstruction quality in
$256$ pixels), this would result in $65,536$ channels, with the previous layer
thus requiring at least to be $[B\times 16,384\times 2\times 2]$. In turn, the
operation to reduce from $16,384$ to $65,536$ channels would require over a
billion parameters, making this approach not feasible for any larger image.

##### Fully Attention Based

Again, for larger images like ImageNet with $256$ pixels, this would require
computation of $n^{2}=65,536$ many attention scores. While this is certainly
possible _e.g_. on an H$100$ GPU, this is often not feasible for labs with in-
expensive hardware.

##### Fully Linear Architecture

Staying with the ImageNet example, even when applying a few layers of
convolution and pooling first, when combining _e.g_. $16$-by-$16$ feature
dimensions with $256$ channels into $64\cdot 1024$ (to have enough space for
quantisation and to produce $1024$ tokens), this again would require over $4$
billion parameters. Again, this is not feasible for both hardware requirements
and generalisation.

##  F Smaller Experiments

We can apply local changes, like adding a ring to a volcano, to provoke large
changes in token space, as can be seen in Fig. 10 (left shows input, right
shows output).

Figure 10: Even the smallest change can change the course of the
representation: More than $10$ percent of the tokens are changed by adding the
ring to the volcano. Left: Input, Right: Output.

##  G Additional Results, Qualitative Analysis, Comparison Against VQ-VAE

We provide additional examples of our autoencoder quality, in particular on
larger datasets and higher resolutions. In Fig. 15, we show results of
CelebA[18] in original size (opposing to the $64$-by-$64$ results used for the
benchmarks here and in originally in [33]). To keep our U-Net simple, we embed
the $178$-by-$218$ in black $256^{2}$ images. Conveniently, due to global
instead of local tokens, our approach does not suffer from unused boundary
regions that we throw away at the end.

##### ImageNet

While we did not extensive train and fine-tune our approach on ImageNet[5] due
to a lack of benchmarks and resources (_e.g_. no comparison to VQ-WAE[33]
possible due to undisclosed code), we do provide some results on (unoptimised)
settings to demonstrate that we can also perform well on larger datasets and
with larger numbers of tokens. We train on ImageNet in $64$-by-$64$ pixels
with $256$ and $1024$ tokens. We also train our approach on $128$-by-$128$
pixels with $256$ tokens, shown in Fig. 14.
We show further results in ImageNet in $128$ pixels in Fig. 14, comparing
against an already improved VQ-VAE with codebook regularisation that avoids
unused codebook entries. In result, our approach is better at producing
sharper features, even if they are at times slightly misplaced.

##  H Generative Performance of VQ-VAE versus QG-VAE Tokens

While generation is not our focus, and these are only early results without
fine tuning or architecture optimisation, we still provide a small
quantitative measurement of applying our learned representation in generation.
We also measured the performance of applying a generative transformer[13] on
tokens generated by our approach. We compare to a VQ-VAE[31], while for
reasons of fairness, we capped both at the same reconstruction quality, and
applied the transformer only in unconditional form. We show the resulting FID
plots in Fig. 11, and additional randomly selected result in Fig. 12.
While the initial improvements are smaller and FID is higher, we argue that
this is a result of our more complex latent space: Simply repeating a token
_e.g_. for blue sky is not beneficial for our representation. However, with
the later better results, we argue that this shows how our learned, natively
linear sequence of tokens is beneficial for generation tasks when compared to
artificially cutting a $2$-dimensional grid of tokens into strips. We argue
that further research in this direction, in particular on larger datasets,
could be promising.

Figure 11: Generation quality when training the same transformer on our pseudo
frequency tokens of CIFAR-10[15] versus training on tokens produced by a VQ-
VAE[31]. Our approach performs better, producing lower FID. We compute on
$10,000$ unconditional samples each. Figure 12: Results of applying a
generative transformers for unconditional generation on our learned latent
space. Left: VQ-VAE for reference, right: QG-VAE (ours). The transformer
applied to tokens produced by ours has fewer ’empty’ backgrounds, which we
attribute to our holistic and global approach to image representation: Our
tokens prohibit the ’easy’ solution of just placing multiple uniform coloured
tokens in sequence.

Input | VQ-VAE | QG-VAE (ours) | Input | VQ-VAE | QG-VAE (ours)
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

Input | VQ-VAE | QG-VAE (ours) | Input | VQ-VAE | QG-VAE (ours)
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

Figure 14: Examples on ImageNet $128$, always input, output (VQ-VAE[31]),
output (QG-VAE). Note how small details, like the spider legs or global
structures like the fence behind the bike, are better represented for our
approach instead of just being blurred. Each image has only $256$ tokens
($288$ bytes). Figure 15: Additional results of CelebA, but in full size, with
inputs (top) and outputs (bottom). We use twice as many tokens here as in the
$64$-by-$64$ benchmarks in Tab. 1 ($512$). We report a PSNR of $\sim 28.2$.

Figure 16: Additional results on ImageNet64, with inputs always above the
outputs, with $256$ tokens ($288$ bytes per image, upper half of the figure).
We report a PSNR of $\sim 28.4$ after $4$ epochs. For $1024$ tokens, we obtain
a PSNR of $\sim 31.4$ after 4 epochs (lower half of the figure).