# Nonlinear Transform Coding

Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh,
Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, George Toderici  Google
Research
Mountain View, CA 94043, USA
{jballe, philchou, dminnen, saurabhsingh, nickj, eirikur, sjhwang,
<EMAIL_ADDRESS>
###### Abstract

We review a class of methods that can be collected under the name nonlinear
transform coding (NTC), which over the past few years have become competitive
with the best linear transform codecs for images, and have superseded them in
terms of rate–distortion performance under established perceptual quality
metrics such as MS-SSIM. We assess the empirical rate–distortion performance
of NTC with the help of simple example sources, for which the optimal
performance of a vector quantizer is easier to estimate than with natural data
sources. To this end, we introduce a novel variant of entropy-constrained
vector quantization. We provide an analysis of various forms of stochastic
optimization techniques for NTC models; review architectures of transforms
based on artificial neural networks, as well as learned entropy models; and
provide a direct comparison of a number of methods to parameterize the
rate–distortion trade-off of nonlinear transforms, introducing a simplified
one.

Figure 1: Linear transform code (left), and nonlinear transform code (right)
of a banana-shaped source distribution, both obtained by empirically
minimizing the rate–distortion Lagrangian (eq. 13). Lines represent
quantization bin boundaries, while dots indicate code vectors. While LTC is
limited to lattice quantization, NTC can more closely adapt to the source,
leading to better compression performance (RD results in fig. 3; details in
section III).

##  I Introduction

There is no end in sight for the world’s reliance on multimedia communication.
Digital devices have been increasingly permeating our daily lives, and with
them comes the need to store, send, and receive images and audio ever more
efficiently. Almost universally, transform coding (TC) has been the method of
choice for compressing this type of data source.

In his 2001 article for IEEE Signal Processing Magazine [1], Vivek Goyal
attributed the success of TC to a divide-and-conquer paradigm: the practical
benefit of TC is that it separates the task of decorrelating a source, from
coding it. Any source can be optimally compressed in theory using vector
quantization (VQ) [2]. However, in general, VQ quickly becomes computationally
infeasible for sources of more than a handful dimensions, mainly because the
codebook of reproduction vectors, as well as the computational complexity of
the search for the best reproduction of the source vector grow exponentially
with the number of dimensions. TC simplifies quantization and coding by first
mapping the source vector into a latent space via a decorrelating invertible
transform, such as the Karhunen–Loève Transform (KLT), and then separately
quantizing and coding each of the latent dimensions.

Much of the theory surrounding TC is based on an implicit or explicit
assumption that the source is jointly Gaussian, because this assumption allows
for closed-form solutions. If the source is Gaussian, all that is needed to
make the latent dimensions independent is decorrelation. When speaking of TC,
it is almost always assumed that the transforms are linear, even if the source
is far from Gaussian. As an example, consider the banana-shaped distribution
in fig. 1: While linear transform coding (LTC) is limited to lattice
quantization, nonlinear transform coding (NTC) can more closely adapt to the
source, leading to better compression performance.

Until a few years ago, one of the fundamental constraints in designing
transform codes was that determining nonlinear transforms with desirable
properties, such as improved independence between latent dimensions, is a
difficult problem for high-dimensional sources. As a result, not much
practical research had been conducted in directly using nonlinear transforms
for compression. However, this premise has changed with the recent resurgence
of artificial neural networks (ANNs). It is well known that, with the right
set of parameters, ANNs can approximate arbitrary functions [3]. It turns out
that in combination with stochastic optimization methods, such as stochastic
gradient descent (SGD), and massively parallel computational hardware, a
nearly universal set of tools for function approximation has emerged. These
tools have also been used in the context of data compression [4, 5, 6, 7, 8,
9]. Even though these methods were developed from scratch, they have rapidly
become competitive with modern conventional compression methods such as HEVC
[HEVC], which are the culmination of decades of incremental engineering
efforts. This demonstrates, as it has in other fields, the flexibility and
ease of prototyping that universal function approximation brings over
designing methods manually, and the power of developing methods in a data-
driven fashion.

This paper reviews some of the recent developments in data-driven lossy
compression; in particular, we focus on a class of methods that can be
collectively called _nonlinear transform coding_ (NTC), providing insights
into its capabilities and challenges. We assess the empirical rate–distortion
(RD) performance of NTC with the help of simple example sources: the Laplace
source and the two-dimensional distribution of fig. 1. To this end, we
introduce a novel variant of entropy-constrained vector quantization (ECVQ)
algorithm [10]. Further, we provide insights into various forms of
optimization techniques for NTC models and review ANN-based transform
architectures, as well as entropy modeling for NTC. A further contribution of
this paper is to provide a direct comparison of a number of methods to
parameterize the RD trade-off, and to introduce a simplified method.

In the next section, we first review stochastic gradient optimization of the
RD Lagrangian, a necessary tool for optimizing ANNs for lossy compression. We
introduce _variational_ ECVQ, illustrating this type of optimization. VECVQ
also serves as a baseline to evaluate NTC in the subsequent section. In that
section, we discuss various approaches for approximating the gradient of the
RD objective and review ANN architectures. Section IV reviews entropy modeling
via learned forward and backward adaptation, and illustrates its performance
gains on image compression. Section V compares several ways of parameterizing
the transforms to continuously traverse the RD curve with a single set of
transforms. The last two sections discuss connections to related work and
conclude the paper, respectively.

##  II Stochastic rate–distortion optimization

Consider the following lossy compression scenario. Alice is drawing vectors
$\bm{x}\in\operatorname{\mathbb{R}}^{N}$ from some data source, whose
probability density function we denote $p_{\textrm{source}}$. Here, Alice is
concerned with compressing each vector into a bit sequence, communicating this
sequence to Bob, who then uses the information to reconstruct an approximation
to $\bm{x}$. Each possible vector $\bm{x}$ is approximated using a codevector
$\bm{c}_{k}\in C$, where $C=\\{\bm{c}_{k}\in\operatorname{\mathbb{R}}^{N}\mid
0\leq k<K\\}$ is called the codebook. Once the codevector index $k=e(\bm{x})$
for a given $\bm{x}$ is determined using the encoder $e(\cdot)$, Alice
subjects it to lossless entropy coding, such as Huffman coding or arithmetic
coding, which yields a bit sequence of nominal length $s(k)$. In what follows,
we’ll assume that the performance of this entropy coding method is optimized
to closely approximate the theoretical limit, i.e., that Alice and Bob share
an estimate of the marginal probability distribution of $k$, also called an
entropy model, $P(k)$, and that $s(k)\approx-\log P(k)$. To the extent that
$P(k)$ approximates $M(k)=\operatorname{\mathbb{E}}_{\bm{x}\sim
p_{\textrm{source}}}\delta(k,e(\bm{x}))$, the true marginal distribution of
$k$ (where $\delta$ denotes the Kronecker delta function), $s(k)$ is close to
optimal, since codes of length $-\log M(k)$ would achieve the lowest possible
average rate, the entropy of $k$. Since Alice and Bob also share knowledge of
the codebook, Bob can decode the index $k$ and finally look up the
reconstructed vector $\bm{c}_{k}$.

Figure 2: Top: A near-optimal entropy-constrained scalar quantizer of a
standard Laplacian source, found using the VECVQ algorithm (eq. 5). Bottom: an
entropy-constrained vector quantizer of a banana-shaped source, found using
the same algorithm.

To optimize the efficiency of this scheme, Alice and Bob seek to
simultaneously minimize the cross entropy of the index under the entropy model
(the rate) as well as the distortion between $\bm{x}$ and the reconstructed
vector, quantified by some distortion measure $d$:

$L=\operatorname{\mathbb{E}}_{\bm{x}\sim p_{\textrm{source}}}\bigl{[}-\log
P(k)+\lambda\,d(\bm{x},\bm{c}_{k})\bigr{]},$ (1)

with $k=e(\bm{x})$ as determined by the encoder $e(\cdot)$, choosing a
codebook index for each possible source vector. Many authors formulate this as
a minimization problem over one of the terms given a hard constraint on the
other [11]. In this paper, we consider the Lagrangian relaxation of the
distortion-constrained problem, with the Lagrange multiplier $\lambda$ on the
distortion term determining the trade-off between rate and distortion.

The top panel of fig. 2 illustrates a lossy compression method for a simple,
one-dimensional Laplacian source, optimized for squared error distortion
(i.e., $d(\bm{x},\bm{c})=\|\bm{x}-\bm{c}\|_{2}^{2}$). The source distribution
is plotted in blue. The codebook vectors are represented by the horizontal
locations of the black stalks, while the height of each stalk is proportional
to the likelihood of that code vector under the entropy model $P$. Dotted
lines delineate the quantization bins, i.e., the intervals for which all
source values get mapped to a given codebook value (the one within the
respective interval). For Laplacian sources, the minimizer of $L$ has been
studied by [12]. It is characterized by equal-width quantization bins and
equidistant code vectors, except for the center bin (coinciding with the mode
of the source distribution); both characteristic features are present in the
figure up to small deviations. The bottom panel of the same figure visualizes
a vector quantizer for a banana-shaped source distribution. The boundaries
between quantization bins are shown as pink lines, while the code vectors are
rendered as discs. Note the presence of hexagon-like bins, which are a feature
of optimal VQ for squared-error distortions.

###  II-A Variational entropy-constrained vector quantization

To generate both of the results in fig. 2, we used a novel algorithm for
entropy-constrained vector quantization based on directly minimizing eq. 1. To
begin, without loss of generality, we parameterize the entropy model as

$P(k)=\frac{e^{a_{k}}}{\sum_{j=0}^{K-1}e^{a_{j}}}.$ (2)

Then, denoting model parameters $\Theta=\\{a_{k},\bm{c}_{k}\mid 0\leq k<K\\}$,
we define the sample loss

$\ell_{\Theta}(k,\bm{x})=-\log P(k)+\lambda\,d(\bm{x},\bm{c}_{k})$ (3)

and the encoder function

$e_{\Theta}(\bm{x})=\operatorname*{arg\,min}_{k}\ell_{\Theta}(k,\bm{x}),$ (4)

where we have made explicit their dependence on the parameters $\Theta$. We
express eq. 1 as

$L_{\text{VQ}}=\operatorname{\mathbb{E}}_{\bm{x}}\ell_{\Theta}(e_{\Theta}(\bm{x}),\bm{x})=\operatorname{\mathbb{E}}_{\bm{x}}\min_{k}\ell_{\Theta}(k,\bm{x}),$
(5)

which we now wish to minimize over $\Theta$ using stochastic gradient descent
(SGD). SGD relies on a Monte Carlo approximation of the expectation, and the
fact that expectations and derivatives are both linear operators, whose order
can be exchanged. Thus

$\displaystyle\frac{\partial}{\partial\Theta}L_{\text{VQ}}$
$\displaystyle=\operatorname{\mathbb{E}}_{\bm{x}}\frac{\partial}{\partial\Theta}\min_{k}\ell_{\Theta}(k,\bm{x}),$
(6) which can be approximated by the sample expectation
$\displaystyle\frac{\partial}{\partial\Theta}L_{\text{VQ}}$
$\displaystyle\approx\frac{1}{B}\sum_{\\{\bm{x}_{b}\sim p_{\text{source}}\mid
0\leq b<B\\}}\frac{\partial\ell_{\Theta}(k_{b},\bm{x}_{b})}{\partial\Theta},$
(7)

with $k_{b}=e_{\Theta}(\bm{x}_{b})$. This is an unbiased estimator of the
derivative of $L_{\text{VQ}}$ based on averaging the derivatives of $\ell$
over a batch of $B$ source vector samples.

Minimization of $L_{\text{V}Q}$ will fit the entropy model to the marginal
distribution of $k$, $M(k)=\operatorname{\mathbb{E}}_{\bm{x}\sim
p_{\textrm{source}}}\delta(k,e(\bm{x}))$, as well as adjust the codebook
vectors to minimize distortion. To see this, add and subtract the expected
negative log likelihood of $k$ under the marginal to eq. 5:

$L_{\text{VQ}}=D_{\text{KL}}[M\;\|\;P]+\operatorname{\mathbb{E}}_{\bm{x}}\bigl{[}-\log
M(k)+\lambda\,d(\bm{x},\bm{c}_{k})\bigr{]}.$ (8)

Since the second term is constant wrt. the parameters of $P$ almost
everywhere, minimizing $L_{\text{VQ}}$ results in fitting $P$ to $M$ by
minimizing their Kullback–Leibler (KL) divergence. Similarly, since the first
term is constant wrt. the codebook almost everywhere, each $\bm{c}_{k}$ is
adjusted to minimize the distortion between it and all source vectors getting
mapped to $k$. Note that since the KL divergence is non-negative,
$L_{\text{VQ}}$ can be interpreted as an upper bound on the second term, which
is the rate–distortion objective for the optimal choice of entropy model. This
can be likened to variational Bayesian inference, in which a _variational_
approximation ($P$) to an unobserved true distribution ($M$) is found by
minimizing an upper bound on the true objective. We therefore name this method
variational entropy-constrained vector quantization (VECVQ).111The VECVQ
algorithm inherits from two methods. The first is the ECVQ algorithm of [13,
14, 10], which minimizes eq. 1 using a clustering algorithm instead of
gradient descent. The second is the online $K$-means algorithm of [15], which
minimizes the distortion part of eq. 1,
$\operatorname{\mathbb{E}}_{\bm{x}}\bigl{[}d(\bm{x},\bm{c}_{k})\bigr{]}$,
using gradient descent. Both ECVQ and the online $K$-means algorithms derive
in turn from the generalized Lloyd algorithm [16, 17, 18, 19]. Note that,
since $P$ as defined in eq. 2 can represent arbitrary distributions, the
variational approximation here is capable of recovering the true marginal,
i.e., the KL divergence can converge to zero.

Figure 3: Left: rate–distortion performance of different quantizers for
standard Laplace source. Both VECVQ and NTC with optimized offset (“dithering
proxy”) recover the optimal entropy-constrained scalar quantizer established
by [12]. NTC with randomized offset (“dithered quantization”) is slightly
suboptimal at lower rates, as predicted by theory. The NTC trained with the
straight-through proxy is unstable at low rates. Using the dithering proxy
with explicit soft quantization recovers the optimal quantizer as well. $R(D)$
indicates the information-theoretic rate–distortion function,
$R(D)=\inf_{p(\hat{x}\mid x)}\\{I(x;\hat{x})$ s.t.
$\operatorname{\mathbb{E}}[d(x,\hat{x})]\leq D\\}$ (achievable only in the
limit of large blocksizes, not with a scalar quantizer). Right:
rate–distortion performance of different quantizers for banana source. NTC
closely matches the performance of VECVQ; the straight-through variant
diverges at low rates. The linear TC trained for the same objective performs
significantly worse. Constraining it to the KLT is not necessarily optimal, as
pointed out by [1].

In the left panel of fig. 3, we plot the operational rate–distortion function
of the optimal entropy-constrained scalar quantizer due to [12], as well as
the empirical rate–distortion function of the VECVQ algorithm for the same
Laplace source. The plot shows that the algorithm recovers the theoretical
optimum. Since it is constrained only by the size of the codebook, we can use
the algorithm as an empirical lower bound on the rate–distortion objective of
more constrained compression methods, such as nonlinear transform coding, even
for source distributions for which no theoretical optimum is presently known.
As an example, consider the more complex banana distribution in the right
panel: the nonlinear transform coders trained with the dithering proxy (to be
discussed in the next section) perform ever so slightly worse than VECVQ.

##  III Nonlinear transform coding

It is easy to modify eq. 1 to accommodate nonlinear transform coding. Rather
than explicitly enumerating the codebook vectors, we consider mapping the
source vectors into a latent space $\operatorname{\mathbb{R}}^{M}$ and back
via a pair of transforms. Quantization and compression takes place in this
latent space. Specifically, we define the analysis transform as a parametric
function $\bm{y}=g_{a}(\bm{x})$, implemented by a neural network with
parameters $\bm{\phi}$, and the synthesis transform as a function
$\bm{\tilde{x}}=g_{s}(\bm{\tilde{y}})$, with parameters $\bm{\theta}$. We can
write the rate–distortion objective as:

$L_{\text{NTC}}=\operatorname{\mathbb{E}}_{\bm{x}}\bigl{[}-\log
P\bigl{(}\lfloor
g_{a}(\bm{x})\rceil\bigr{)}+\lambda\,d\bigl{(}\bm{x},g_{s}(\lfloor
g_{a}(\bm{x})\rceil)\bigr{)}\bigr{]},$ (9)

where $\lfloor\cdot\rceil$ denotes uniform scalar quantization (rounding to
integers). $P$ is now a probability distribution over a space of integer
vectors, which take the role of the codebook index $k$ in eq. 1.

Figure 4: A near-optimal nonlinear transform code of a standard Laplacian
source, obtained by minimizing the dithering rate–distortion proxy (eq. 13).

As an example, consider the nonlinear transform code illustrated in fig. 4.
Again, we plot the effective codebook vectors and quantization boundaries on
top of the source distribution. However, unlike the example in fig. 2, this
quantization scheme is defined indirectly via the analysis and synthesis
transforms, as illustrated in the bottom panel of fig. 4. The analysis
transform maps the space of source values to the latent space (blue curve). In
this space, uniform quantization is applied, rounding values between half-
integers to full integers. These integer values are then mapped back into the
source space using the synthesis transform (orange curve).

There are a few key observations here: First, the analysis transform
determines the effective quantization bins. In particular, its intersections
with the dotted lines, corresponding to half-integers in the latent space,
give rise to quantization bins of varying size in the source space. Second,
the synthesis transform determines the effective codebook vectors. Notably,
the full behavior of the synthesis transform as determined by the optimization
procedure does not matter – only its values at integer locations are relevant.
Third, since the transforms are not constrained to be exact inverses of each
other, using uniform quantization in the latent space is sufficient to enable
codebook vectors to be located anywhere in the corresponding quantization bins
(technically, even outside of it). Nonlinear transform coding thus generalizes
companding [20, 21], which permits implementing non-uniform quantization using
uniform quantizers; with nonlinear transforms, the quantization step size can
be fixed to one without loss of generality (parameterizing the model for
different rate–distortion trade-offs is discussed in section V).

Figure 1 illustrates LTC and NTC of the banana distribution also shown in fig.
2 (bottom), and evaluated in fig. 3 (right). Both the linear and nonlinear
transform codes are designed to minimize eq. 9 under their respective
constraints. Because the linear transform coder is constrained to affine
transformations, the method effectively amounts to a lattice quantizer in the
source space (left panel). Note that the linear transform is not orthogonal
and hence is _not_ the KLT, as would be optimal if the source distribution
were Gaussian.

The nonlinear transform coder has more flexibility, and can adapt the shapes
of its quantization bins to better fit the source distribution (right panel).
Note that in both cases, since invertibility of the transforms is not
enforced, codebook vectors do not necessarily appear in consistent locations
relative to their bins. Their optimal locations, for squared error
distortions, are at the conditional mean of their respective cells. Both
coders reflect this by shifting the codebook vectors closer to the high-
probability regions of the source distribution. This may come at the expense
of reconstruction accuracy in low-probability regions – in the nonlinear
example, some low-probability codebook vectors even lie outside of their
respective bins – because the behavior of the method in these regions often
does not contribute much to the overall objective. Another reason for this
trade-off may be limitations in the parameterizations of the transforms. This
is further examined in section III-B.

Figure 5: Location-dependent Jacobian matrices $\nabla g_{a}(\bm{x})$ (left;
arrows visualize local Jacobian inverse) and $\nabla g_{s}(\bm{y})$ (right;
arrows visualize local Jacobian) of a model optimized for squared error
distortion on the banana source. The transforms form a local orthogonalization
of the source density.

Although the optimized nonlinear analysis and synthesis transforms illustrated
in fig. 1 (right) are globally nonlinear, they are of course differentiable,
and hence can be viewed as locally approximately linear. We find that in high-
probability regions of the data distribution, they locally resemble KLTs in
that they are approximately orthogonalized (fig. 5). Specifically, at each
point $\bm{x}$, the columns of the inverse Jacobian matrix of the analysis
transform appear approximately orthogonal to each other, as shown in the left
panel of the figure. Likewise, the columns of the Jacobian matrix of the
synthesis transform are approximately orthogonal, as shown in the right panel.
In the neighborhood of some point $\bm{x}_{0}$,
$(\bm{y}-\bm{y}_{0})\approx\nabla g_{a}(\bm{x}_{0})\cdot(\bm{x}-\bm{x}_{0})$
approximates a linear orthogonal analysis transform, and
$(\bm{\tilde{x}}-\bm{\tilde{x}}_{0})\approx\nabla
g_{s}(\bm{y}_{0})\cdot(\bm{y}-\bm{y}_{0})$ approximates its inverse, where
$\bm{y}_{0}=g_{a}(\bm{x}_{0})$ and $\bm{\tilde{x}}_{0}=g_{s}(\bm{y}_{0})$.

While more can be said about the local properties of the analysis and
synthesis transforms (see supplement), for now, let us consider the transforms
as “black boxes” that simply serve to approximate the optimal transforms.

###  III-A Optimization and proxy rate–distortion loss

Note that in the VECVQ loss given in eq. 5, the encoder function is defined by
exhaustively minimizing over all possible codes. As such, it can be folded
into a minimum over the sample loss $\ell_{\Theta}$, which is differentiable
with respect to almost all $\bm{x}$. If we were to choose another encoder
function with trainable parameters of its own, we would not be able to obtain
a gradient of the loss function with respect to them that is useful for SGD.
The gradient would be zero for almost all $\bm{x}$, because $e$ is integer
valued. This problem also appears in eq. 9 due to the quantizer. Derivatives
of the loss with respect to any parameter of the analysis transform are zero
almost everywhere. However, when employing dithered quantization (i.e.,
randomizing the quantization offset) [22], this problem can be avoided [23].

Consider uniformly sampling one random quantization offset per latent
dimension $\bm{o}\in[-\frac{1}{2},\frac{1}{2})^{M}$, and formulating the
following loss function as an expectation over it:

$\operatorname{\mathbb{E}}_{\bm{o}}L_{\text{NTC},\bm{o}}=\operatorname{\mathbb{E}}_{\bm{x},\bm{o}}\Bigl{[}-\log
P\bigl{(}\lfloor g_{a}(\bm{x})-\bm{o}\rceil;\bm{o}\bigr{)}\\\
+\lambda\,d\bigl{(}\bm{x},g_{s}(\lfloor
g_{a}(\bm{x})-\bm{o}\rceil+\bm{o})\bigr{)}\Bigr{]},$ (10)

where $L_{\text{NTC},\bm{o}}$ is the loss for a given offset $\bm{o}$, and
$P(\cdot;\bm{o})$ is an entropy model conditioned on $\bm{o}$. (Note that all
else being equal, the marginal distribution of the quantized latents changes
with the offset.) This loss function is differentiable with respect to the
parameters of $g_{a}$. To see this, let us consider both terms separately. For
the rate term, we have

$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x},\bm{o}}\log P\bigl{(}\lfloor
g_{a}(\bm{x})-\bm{o}\rceil;\bm{o}\bigr{)}$ $\displaystyle=$
$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x},\bm{o}}\sum_{\bm{k}\in\operatorname{\mathbb{Z}}^{M}}\delta\bigl{(}\lfloor
g_{a}(\bm{x})-\bm{o}\rceil=\bm{k}\bigr{)}\log P(\bm{k};\bm{o})$
$\displaystyle=$
$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x}}\idotsint_{-\frac{1}{2}}^{\frac{1}{2}}\;\mathrm{d}\bm{o}\sum_{\bm{k}\in\operatorname{\mathbb{Z}}^{M}}$
$\displaystyle\hskip
60.00009pt\delta\bigl{(}\|g_{a}(\bm{x})-\bm{o}-\bm{k}\|_{\infty}\leq\tfrac{1}{2}\bigr{)}\log
P(\bm{k};\bm{o})$ $\displaystyle=$
$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x}}\idotsint_{-\infty}^{\infty}\;\mathrm{d}\bm{v}\,\delta\bigl{(}\|g_{a}(\bm{x})-\bm{v}\|_{\infty}\leq\tfrac{1}{2}\bigr{)}\log
p(\bm{v})$ $\displaystyle=$
$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x}}\idotsint_{-\frac{1}{2}}^{\frac{1}{2}}\;\mathrm{d}\bm{u}\log
p\bigl{(}g_{a}(\bm{x})+\bm{u}\bigr{)}$ $\displaystyle=$
$\displaystyle-\operatorname{\mathbb{E}}_{\bm{x},\bm{u}}\log
p\bigl{(}g_{a}(\bm{x})+\bm{u}\bigr{)},$ (11)

where $\delta$ is the Kronecker delta function, we define
$\bm{v}=\bm{k}+\bm{o}$ and
$p(\bm{v})=P(\lfloor\bm{v}\rceil;\bm{v}-\lfloor\bm{v}\rceil)$, and consider
$\bm{u}\in[-\frac{1}{2},\frac{1}{2})^{M}$ uniformly distributed. It is easy to
show that $p$ is non-negative and integrates to one, and hence represents a
probability density function. We can thus interpret $p$ as a continuous
equivalent of an entropy model for the “noisy” latents $g_{a}(\bm{x})+\bm{u}$.
For the distortion term, we have

$\operatorname{\mathbb{E}}_{\bm{x},\bm{o}}d\bigl{(}\bm{x},g_{s}(\lfloor
g_{a}(\bm{x})-\bm{o}\rceil+\bm{o})\bigr{)}=\\\
\operatorname{\mathbb{E}}_{\bm{x},\bm{u}}d\bigl{(}\bm{x},g_{s}(g_{a}(\bm{x})+\bm{u})\bigr{)},$
(12)

since dithered quantization and additive uniform noise have the same marginal
distribution (i.e., integrating out $\bm{o}$ and $\bm{u}$ is equivalent). For
a proof, refer to [22].

Hence, denoting $\bm{\tilde{y}}=g_{a}(\bm{x})+\bm{u}$, we can now write

$\operatorname{\mathbb{E}}_{\bm{o}}L_{\text{NTC},\bm{o}}=\operatorname{\mathbb{E}}_{\bm{x},\bm{u}}\bigl{[}-\log
p(\bm{\tilde{y}})+\lambda\,d\bigl{(}\bm{x},g_{s}(\bm{\tilde{y}})\bigr{)}\bigr{]},$
(13)

which can be directly minimized via SGD as in eq. 7.

Analogously to eq. 8, we can interpret eq. 13 as a variational upper bound on
the true marginal:

$\operatorname{\mathbb{E}}_{\bm{o}}L_{\text{NTC},\bm{o}}=D_{\text{KL}}[m\;\|\;p]\\\
+\operatorname{\mathbb{E}}_{\bm{x},\bm{u}}\bigl{[}-\log
m(\bm{\tilde{y}})+\lambda\,d\bigl{(}\bm{x},g_{s}(\bm{\tilde{y}})\bigr{)}\bigr{]},$
(14)

where
$m(\bm{v})=\operatorname{\mathbb{E}}_{\bm{x},\bm{u}}\delta(\bm{v},g_{a}(\bm{x})+\bm{u})$
is the marginal distribution of the noisy latents.222This dithering objective
of rate–distortion optimized nonlinear transform coding is equivalent to
$\beta$-variational autoencoders [5, 24, 25], up to choice of parameterized
distributions. As such, minimizing eq. 13 results in fitting the continuous
entropy model $p$ to the marginal. Note that, unlike in the case of VECVQ, the
KL divergence may not converge to zero, as for high-dimensional source
distributions such as images, the entropy model will generally not be capable
of representing the marginal accurately. Section IV talks about this in more
detail.

Figure 6: The same instance of NTC as in fig. 4, obtained by minimizing
$\operatorname{\mathbb{E}}_{\bm{o}}L_{\text{NTC},\bm{o}}$. For this figure, a
sub-optimal offset was chosen post hoc. Top: visualization of effective
quantizer. Center: analysis and synthesis transforms giving rise to the
quantizer. Note that the transforms themselves are identical to the ones in
fig. 4, but the quantization in the latent space is performed with an offset
$o=.35$. Bottom: $L_{\text{NTC},o}$ as a function of $o$.

There is one caveat with using dithered quantization for compression itself:
it is not necessarily optimal, since
$\operatorname{\mathbb{E}}_{\bm{o}}L_{\text{NTC},\bm{o}}\geq\min_{\bm{o}}L_{\text{NTC},\bm{o}}$
(fig. 3, left panel, verifies this empirically). If we do not wish to use
dithered quantization, we can still use eq. 13 as a proxy loss for transform
coding with a fixed quantization offset known to both Alice and Bob. A simple
algorithm for stochastic optimization of an NTC model is:

  1. 1.

Minimize eq. 13.

  2. 2.

Determine which offsets $\bm{o}$ minimize $L_{\text{NTC},\bm{o}}$. If the
continuous entropy model $p$ is accurate enough, this can be done without re-
estimating the discrete entropy models, since
$P(\bm{k};\bm{o})=p(\bm{k}+\bm{o})$.

Note that the offsets themselves cannot be determined via gradient descent on
$L_{\text{NTC}}$, for the same reason that $g_{a}$ cannot be determined in
this way. We must therefore use some other method. While [5] explicitly
perform a grid search over $\bm{o}$, some follow-up papers have resorted to a
simple heuristic: guided by the result that for Laplacian distributions, it is
optimal to pick an offset that aligns the mode of the source distribution with
a codebook vector, one can simply pick an offset for each latent dimension
such that one of the quantization bins is centered on the mode (or, in case
that is computationally intractable, the median) of the entropy model $p$ [26,
27]. For an entropy model with fixed mode, such as a zero-mean Gaussian, this
implies that the offset may as well be fixed a priori.

A suboptimal choice of offset for an NTC encoding a Laplace source is
illustrated in fig. 6, along with a plot of $L_{\text{NTC},o}$ as a function
of $o$ for the same source. Note that optimizing the dithering proxy loss
leads to the transforms becoming increasingly curved around the central
quantization bin, to accommodate arbitrary choices of $o$. Because the
analysis transform becomes increasingly flat around the center, and the
synthesis transform increasingly steep, the effective code vector and
quantization bin around the mode of the distribution is skewed towards the
near-optimal quantizer illustrated in fig. 4. It could be argued that, to
minimize the loss function, this should happen in all bins. However, we
haven’t observed this empirically, presumably due to ANNs naturally favoring
smoother functions, and the other bins not contributing enough to the value of
the loss function.

Figure 7: Transforms for Laplace source with explicit soft quantization due to
[28]. Top: ANN transforms including the explicit soft quantization. Bottom:
ANN transforms excluding the explicit soft quantization. With this technique,
the ANNs themselves can implement smoother (i.e., in some sense, simpler)
functions.

[28] discuss augmenting the transforms with a soft quantization function (and
making appropriate modifications to the entropy model), which explicitly
implements the curvature observed in fig. 4. The soft quantization function
has a _temperature_ parameter, interpolating between the identity function and
hard quantization. By explicitly modeling this behavior, the technique
relieves the ANN itself from implementing it (fig. 7), and represents a more
controlled approach to bridging the gap between quantization and additive
uniform noise while retaining near-optimal performance at all rates (fig. 3).
The temperature parameter allows explicitly trading off the bias of the proxy
loss with the variance of the gradients. The method also suggests that the
hard quantization offset after training can simply be chosen to be consistent
with the soft quantization offset during training. However, it requires
careful choice of an annealing schedule for the temperature. For simplicity,
the experiments in this paper use the mode-centering approach described above.

Figure 8: Instabilities observed with straight-through proxy objective at low
rates for the Laplace (top) and banana (bottom) distribution. While the
synthesis transform tends to be smoother, the analysis transform begins
oscillating around the locations of bin boundaries, leading effectively to
discontiguous quantization bins.

Other authors choose to retain the dithering proxy for the rate term, but use
a _straight-through_ gradient estimate for the distortion term (effectively
computing the distortion loss with constant-offset quantization during
training, but replacing the gradient expression of the quantization operation
with the identity function [29, 30, 31, 32]. We have found this approach to
yield reasonable results at higher rates, but at low rates, the ad-hoc nature
of this approach leads to problematic behaviors of the transforms (RD
performance plotted in fig. 3, illustration in fig. 8).

###  III-B Nonlinear transforms

ANNs are known as universal function approximators, and as such we permitted
ourselves to ignore their details in the examples above. However, it is
crucial to take into account their limitations, some of which arise as a
function of their architecture. This is particularly important for complex or
high-dimensional source distributions, such as natural images. With higher
complexity and rates (larger values of $\lambda$), the optimal transforms
generally are more complex and require neural networks with an increasing
number of parameters.333It could be argued that in the high-rate limit, the
transforms should collapse to identity functions. However, we haven’t observed
this effect for image compression models and practically interesting
rate–distortion trade-offs, suggesting that this is only the case for
extremely high rates.

In general, neural networks are compositions of layers (parametric functions
$\operatorname{\mathbb{R}}^{A}\to\operatorname{\mathbb{R}}^{B}$), wherein each
layer typically consists of a linear transformation such as matrix
multiplication or convolution, followed by the addition of a bias vector,
followed in turn by a nonlinear function, which is typically applied
separately on each vector dimension:

$\bm{v}=g(\bm{r}),\text{ with }\bm{r}=\bm{W}\bm{u}+\bm{b},$ (15)

where $\bm{u}\in R^{A}$ is the input vector to the layer, $\bm{v}\in R^{B}$
are the layer’s outputs or _activations_ , and
$\bm{W}\in\operatorname{\mathbb{R}}^{B\times A}$ and
$\bm{b}\in\operatorname{\mathbb{R}}^{B}$ are the layer’s parameters. For the
NTC examples above, we used neural networks with 4 fully connected (i.e., non-
convolutional) layers, the first three using the $\operatorname{softplus}$
nonlinearity ($g(x)=\ln(1+e^{x})$ elementwise), while the last layer omits the
nonlinearity in order not to constrain the range of the transform to positive
values. The _approximation capacity_ , i.e., the capability of the neural
network to approximate increasingly complex functions, grows with the number
of units per layer ($A$, $B$), as well as the depth of the network (the number
of layers). Above, we chose $A=B=100$ (except that we set $A=N$ for the first,
and $B=M=N$ for the last layer in $g_{a}$; analogous for $g_{s}$), which we
found empirically to be large enough for all chosen values of $\lambda$.

For practical sources such as images, video, or audio, imposing special
structure in the transforms may have significant benefits in terms of
computational complexity, training data efficiency, or both. Generally, NTC
models for this type of data use combinations of architectural constraints,
most commonly _convolutionality_ in $g_{a}$ and $g_{s}$, as well as
_downsampling_ in $g_{a}$ and _upsampling_ in $g_{s}$, making the transforms
share certain characteristics with multi-scale filterbanks, and leading to
latent vectors with a tensor structure, consisting of one or more
spatial/temporal dimensions, as well as one _channel_ dimension (akin to
subbands). A detailed example of such an architecture is described by [5].

Figure 9: Rate–distortion performance of an NTC model with GDN vs. a
collection of pointwise nonlinearities on the Kodak testset [Kodak] (after
[33], with more nonlinearities: ReLU, leaky ReLU, tanh, softplus, ELU [34],
SELU [35], Swish [36]). We compare networks with a different number of hidden
units $A,B$ per layer. At low rates, the approximation capacity of all
networks is sufficient, and performance converges. At high rates, the capacity
of smaller networks saturates earlier; in this regime, the performance benefit
of GDN vs. other activation functions becomes measurable.

It has been observed that spatially local normalization as a nonlinearity is
beneficial in terms of the trade-off between number of units and RD
performance in image compression. In particular, a computationally optimized
version of generalized divisive normalization (GDN) [37] as used in recent
models is defined as

$v_{i}=\frac{r_{i}}{\beta_{i}+\sum_{j}\gamma_{ij}\,|r_{j}|},$ (16)

where $\bm{r}$ are the linear responses of the layer, $\bm{v}$ represents the
vector of normalized responses (the activations), and the vector $\bm{\beta}$
and matrix $\bm{\gamma}$ represent parameters of the transformation (both non-
negative). The computation is typically replicated across spatial dimensions,
as linear filtering is in convolutions, and $i,j$ only index the channel
dimension. [38] show that the originally more complex form of GDN can be
simplified to resemble a weighted $\ell^{1}$-norm (plus a constant), as in eq.
16, with negligible RD performance loss, but minimizing computationally costly
exponentiations. Since ANNs can be understood as universal function
approximators, the benefit of a particular architectural constraint may only
become evident when the network is at its approximation capacity. [33] finds
that this is the case at higher rates; i.e., for a constant network
architecture, the superiority of GDN vs. pointwise nonlinearities disappears
at lower rates (fig. 9). [38] also discuss the trade-off between computational
complexity and RD performance resulting from other architectural choices in
further detail, such as the number of channels per layer or the number of
decoder layers, and provide an algorithm to semi-automatically determine these
hyperparameters.

##  IV Learned entropy models

In linear transform coding with a Gaussian source assumption, the
probabilistic model $P$ in eq. 9 is typically considered to be a distribution
factorized over each latent dimension, since the KLT factorizes the source.
However, as pointed out by [1], this is not necessarily a good model for real-
world sources, and decorrelating the source is not generally RD optimal; this
is also illustrated in fig. 3.

In NTC, any differentiable density model $p$ can be used to minimize eq. 13 in
principle; after determining a quantization offset, it can be translated into
a corresponding probability mass function $P(\cdot;\bm{o})$, and used for
entropy coding. However, to fully benefit from the function approximation
capabilities of ANNs, techniques have been developed that allow jointly
optimizing the transforms with an ANN-based density model, such that both
components of the model are adapted to each other as best as possible. Key to
this approach is conditioning. Generally, entropy coding methods such as
arithmetic coding process one dimension at a time. Thus, we must be able to
write the entropy model as a chain of conditionals:

$P(\bm{\hat{y}}\mid\bm{\hat{z}})=\prod_{i}P(\hat{y}_{i}\mid\bm{\hat{y}}_{:i},\bm{\hat{z}}),$
(17)

where $\bm{\hat{y}}$ is the quantized latent representation,
$\bm{\hat{y}}_{:i}$ indicates the vector comprising the dimensions of
$\bm{\hat{y}}$ preceding the $i$th (according to some predetermined ordering),
and $\bm{\hat{z}}$ is another (optional) vector that must be known to both
Alice and Bob. In the simplest case, $P$ is assumed factorized, and the chain
collapses to a product of independent scalar densities. Conditioning on some
other vector $\bm{\hat{z}}$ typically requires transmitting the vector as side
information, and thus corresponds to forward adaptation (FA) of the density
model. Conditioning on the preceding dimensions of $\bm{\hat{y}}$ can be done
without additional side information, but requires interleaving the computation
of the probabilities with the decoding of $\bm{\hat{y}}$; this is backward
adaptation (BA). FA and BA have long been used in conventional image
compression. Context-adaptive arithmetic coding is an example of BA; mode
selection and signaling is an example of FA [39].

$\bm{x}$$\bm{\hat{x}}$$g_{a}$$g_{s}$$\bm{y}$$\bm{\hat{y}}$$h_{a}$$h_{m}$$h_{s}$$\bm{z}$$\bm{\hat{z}}$$\bm{\mu}$$\bm{\sigma}$$+$$\lfloor\cdot\rceil$ECED$+$$+$$\lfloor\cdot\rceil$ECED$+$$\bm{o}$$\bm{o}$$-$$-$$\bm{\hat{y}}_{:i}$channel
Figure 10: Illustration of a nonlinear transform coder using both learned
forward and backward adaptation. $\bm{x}$ is the source vector, $\bm{\hat{x}}$
the reconstruction. $\bm{y}$ is a latent representation tensor, and
$\bm{\hat{y}}$ its uniformly quantized counterpart. $\bm{z}$ and
$\bm{\hat{z}}$ are an analogous hierarchical latent representation computed
via a transform $h_{a}$, representing side information. While the entropy
model on $\bm{\hat{z}}$ is predetermined, the entropy model on $\bm{\hat{y}}$
is here assumed conditionally independent Gaussian with mean tensor $\bm{\mu}$
and standard deviation tensor $\bm{\sigma}$. Both tensors are computed as
functions of previously decoded values of $\bm{\hat{y}}$ (backward adaptation)
and the side information $\bm{\hat{z}}$ (forward adaptation) using the ANNs
$h_{m}$ and $h_{s}$. While Bob begins with entropy decoding $\bm{\hat{z}}$
(ED), and then uses it to decode $\bm{\hat{y}}$, Alice must have access to the
entropy model on $\bm{\hat{y}}$ to entropy encode it (EC). Thus, in addition
to computing the upper half of the diagram, she must also compute the section
in the shaded box. The quantization offset of $\bm{y}$ is assumed aligned with
the conditional mean of the Gaussian, making the entropy model only dependent
on $\bm{\sigma}$. To enable reliable cross-platform decoding of
$\bm{\hat{y}}$, $h_{s}$ may be computed using a learned integer transform and
translated into an arithmetic code via lookup tables.

Learned forward adaptation was first described in the context of image
compression [26], inspired by the observation that the magnitudes of spatially
nearby elements of the latent tensor in a convolutional NTC for images tend to
be correlated. As illustrated in fig. 10, $\bm{y}=g_{a}(\bm{x})$ is further
processed by an ANN $h_{a}$ to produce a side information vector $\bm{z}$. The
entropy model for $\bm{\hat{y}}$ is conditioned on the decoded $\bm{\hat{z}}$.
[26] assume elements of $\bm{y}$ to be zero-mean Gaussians, conditionally
independent wrt. $\bm{\hat{z}}$; a non-zero mean model is introduced by [27].
The entropy of $\bm{\hat{z}}$ is small enough to warrant the improved fit of
$P(\bm{\hat{y}}\mid\bm{\hat{z}})$, effectively lowering the rate. Figure 11
compares the rate–distortion performance of several learned image compression
models with JPEG [JPEG] and BPG, a variant of HEVC [HEVC], in terms of peak
signal-to-noise ratio (PSNR) as well as MS-SSIM, a popular perceptual image
quality metric [40]. The introduction of FA into NTC leads to a significant
improvement of RD performance with respect to both metrics.

[27] are among the first authors to discuss learned backward adaptation. They
introduce a spatially autoregressive model, where $\bm{\hat{y}}$ is processed
one spatial location at a time, producing a distribution for each channel
vector conditioned on previously decoded spatial locations. Combined with FA,
the model produces further RD gains over the FA-only model (fig. 11), and
outperforms BPG.

A downside of BA compared to FA is that it impedes computational parallelism:
the system must alternate between computing conditional probabilities and
entropy decoding. On the other hand, with FA, Alice may effectively convey
more information to Bob than necessary.444We can write the excessive
information as $H(\bm{\hat{z}}\mid\bm{\hat{y}})$, i.e. the conditional entropy
of the side information $\bm{\hat{z}}$ given the latents $\bm{\hat{y}}$. It is
also referred to as _bits-back_ cost [41, 42]; practical algorithms to
asymptotically get these “bits back” have been proposed by [43] and more
recently by [44]. Note that in all models compared in fig. 11, the side
information only amounts to a fraction of the total bit rate. Thus, since
$H(\bm{\hat{z}})\geq H(\bm{\hat{z}}\mid\bm{\hat{y}})$, the bits-back cost in
these models is negligible. To alleviate the computational bottleneck in BA,
[32] introduce a model that iterates over channel slices rather than spatial
locations, which is more amenable to parallelization using GPUs and, along
with further modeling improvements, presents a significant improvement over
traditional methods.

Figure 11: Image compression performance of NTC on the Kodak testset [Kodak],
comparing different learned entropy models to JPEG and BPG, which is a popular
variant of HEVC, a relatively recent and popular commercial method. The
_factorized_ and _FA_ models replicate [26]; the _FA + spatial BA_ model is
due to [27]; the _FA + channel BA_ model is provided by [32]. We compare
models that are approximately equivalent in terms of optimization procedure
and architecture of analysis and synthesis transforms. Conditional entropy
models lead to significant improvements over factorized models. With
sophisticated entropy modeling, learned image compression compares favorably
to BPG in terms of PSNR, which BPG is optimized for. Regarding MS-SSIM, note
that even the factorized model optimized for mean squared error (MSE) performs
relatively closely to BPG, and when optimized for MS-SSIM, far outperforms it.
We attribute this to the fact that regardless of the distortion measure,
nonlinear transforms are better suited to model the source distribution (fig.
1), and that MS-SSIM captures certain characteristics of the source that are
also relevant perceptually (fig. 14).

A problem frequently encountered with conditional entropy models is numerical
determinism. To make image compression models practically relevant, they need
to be implemented on a wide variety of hardware platforms. However, when
probabilities are computed using floating-point arithmetic, numerical round-
off errors can lead to catastrophic decoding failures due to the sensitivity
of entropy coding with respect to discrepancies in the probability model
between sender and receiver. The exact numerical round-off at each layer of an
ANN depends on the hardware representation of floating point numbers, as well
as the mode of parallelism, because round-off errors are not associative. This
problem is typically handled in linear transform coders by using lookup tables
to model probabilities [39, e.g., ]. [45] provide a solution for ANN-based
entropy modeling, where ANNs are trained using floating-point arithmetic, but
use integer arithmetic when deployed. This enables reliable decoding on
arbitrary hardware platforms for the above-mentioned class of entropy models.

##  V RD traversal with $\lambda$-parameterization

The loss function in eq. 13 is optimized in expectation over the source
distribution. The resulting transform thus jointly minimizes the rate and the
expected distortion $d$ between the source and the reconstruction, for a fixed
trade-off predetermined by the choice of $\lambda$. In many linear transform
coders, the system is parameterized by the quantization step size, such that
only one set of transforms is needed to continuously traverse a range of RD
trade-offs. Using this approach with a single set of trained nonlinear
transforms was first explored by [46]. However, this is not generally optimal.

With NTC, the transforms and/or the entropy model can be made more general
functions of $\lambda$. One method to “condition” an ANN, first introduced in
the context of stylization tasks [47], is to insert additional computations
between layers, such as affine transformations:

$\bm{w}=h_{f}(\lambda)\odot\bm{v}+h_{b}(\lambda),$ (18)

where $\bm{v}$ are the outputs of a layer, $\bm{w}$ are the inputs to the next
layer, and $\odot$ represents elementwise multiplication. In this context,
$h_{f}$ and $h_{b}$ are parametric functions of $\lambda$ that can be computed
themselves via ANNs. The parameters of these ANNs in turn are optimized for
the RD objective (eq. 13) as well. Parameterizing the entropy model and
transforms this way was proposed earlier by [48, 49]. We simplify this
approach here by noting that, since $h_{f}$ and $h_{b}$ are functions of a
scalar, they may be conveniently defined via first-order splines (i.e.,
piecewise linear functions). Additionally, we propose to remove the
$\lambda$-parameterization of the entropy model, and for transforms using GDN,
to simply treat its parameters ($\bm{\beta}$ and $\bm{\gamma}$) as functions
of $\lambda$ instead of using affine transformations.

Figure 12: Performance comparison of $\lambda$-parameterization techniques
implementing eq. 18 using first-order splines, and parameterizing only the
transforms, not the entropy model. All models have the same architecture and
were optimized using the dithering proxy either for MSE (left panel) or MS-
SSIM (right panel). Results are shown for the Kodak testset [Kodak]. As in
fig. 9, differences between models become more evident at high rates.
Optimization using the straight-through proxy gives consistent results (not
shown). _Full_ : RD performance of separate models for each $\lambda$; _latent
scaling_ : only the output of the $g_{a}$ and the input to $g_{s}$ are scaled
with a single scalar each ($h_{b}$ is zero); _latent affine_ : each output
channel of $g_{a}$ and each input channel of $g_{s}$ are subjected to a scalar
affine transformation, as in eq. 18; _layer affine_ : each channel of each
layer of $g_{a}$ and $g_{s}$ is subjected to a scalar affine transformation;
_layer scaling_ : ditto, except that $h_{b}$ is zero; _GDN parameters_ :
rather than adding a scaling between layers, the parameters of each instance
of GDN in the transforms ($\bm{\beta}$ and $\bm{\gamma}$) are represented as
first-order splines dependent on $\lambda$. We note that a simple affine
transformation of the latent space, corresponding to varying the quantization
interval, is not sufficient to maintain comparable performance with the full
model. Scaling the activations of each layer appears sufficient, while
reparameterizing GDN as a function of $\lambda$ yields slightly better
performance. Figure 13: Performance comparison of $\lambda$-parameterization
using splines vs. ANNs, and of optionally applying it to the entropy model. As
in fig. 9, differences between models become more evident at high rates.
Comparing models implementing eq. 18 using ANNs vs. first-order splines, we
find that the performance of ANNs is consistently worse than that of splines,
despite a larger number of parameters, suggesting that optimization of ANNs
may be numerically more difficult in this context. Comparing models using
splines and either a $\lambda$-parameterized entropy model, or a forward-
adaptive entropy model independent of $\lambda$, we find that the benefit of
parameterizing the EM is rather small. Results shown are for the Kodak testset
[Kodak] and are optimized using the dithering proxy; the straight-through
proxy yields consistent results.

We carried out experiments with an NTC model with FA following the
experimental setup of [26], but with 160 channels per layer. When implementing
$h_{f}$ and $h_{b}$ with ANNs, we used two-layer networks with 128 hidden
units for each scalar element produced by the functions. For the spline
implementation, we used a first-order spline with 25 parameters. First, we
found that the optimization of ANN-based parameterization is numerically more
difficult than first-order splines (fig. 13). Furthermore, we removed the
$\lambda$-parameterization of the entropy model and noted that it is not
crucial to RD performance (same figure). Along with the practical requirement
that $h_{s}$ needs to be implemented with integer arithmetic for cross-
platform stability, this suggests that making the entropy model explicitly
dependent on $\lambda$ may not be worth the complexity of implementation.

Figure 12 compares $\lambda$-parameterizations of only the transforms $g_{a}$
and $g_{s}$, and using splines, in terms of RD performance. As for the
experiments with different nonlinearities, differences between the
parameterizations emerge at high rates, since the network capacities saturate
in that regime. We find that a scaling or affine transformation of the latent
space alone, roughly equivalent to parameterizing the quantization step size
and offset, are not sufficient to achieve an RD performance equivalent to the
family of full, non-parameterized models. However, any layer-wise
parameterization appears close enough. This is explainable by the fact that
the RD family of optimal entropy-constrained scalar quantizers cannot in
general be parameterized by a scaling of the quantization offset (a notable
exception being the Laplace source discussed above). GDN reparameterization
performs the best in an RD sense, but also requires slightly more model
parameters compared to the other methods, since eq. 18 requires two length-$B$
vectors, but $\bm{\gamma}$ is a $B\times B$ matrix.

##  VI Related work

Due to the resurgence of ANNs and data-driven computing in recent years, the
field of data compression has received an influx of new ideas. While transform
coding as a concept has been around for decades [50], one could observe a
recent convergence of it with the idea of autoencoders [51]. Autoencoders,
likewise, have been discussed for decades, but largely in a separate
community. One notable step towards this convergence was the fusion of
variational Bayesian methods with autoencoders, which introduced a
probabilistic interpretation, making the connection to information-theoretic
quantities such as entropy [52, 53]; another was the use of a dithering-based
loss for optimization of nonlinear transform codes [23].

As is often the case in lossy compression, the field of nonlinear transform
coding has been driven forward by the need to compress digital images. Early
image compression models using ANNs include the work of [7], who do not use
entropy modeling; [8], who use a context-based adaptive entropy coder not
jointly optimized with the transforms; and [5, 6], who jointly optimize the
transforms with continuous entropy models, the latter with a different
formulation than what we use here. [9] combine an autoencoder with VQ in the
latent space over small blocks of coefficients, utilizing a soft quantization
proxy. More recent work using soft notions of quantization includes the work
of [54, 28].

Beyond the use of convolutional filtering, up- or downsampling, and special
nonlinearities [5] as discussed earlier, many authors exploit properties of
the image distribution by way of introducing special structure into the
transforms, such as multi-scale architectures [8, 55, 56]; non-local, or
“attention”-based network architectures [57, 58]; or iteration built into the
transforms [59, 60]. Recently, the topic of extending nonlinear transform
codes to video signals has received much attention, and the space of possible
network architectures suitable for this application has been explored,
including spatiotemporal convolutions, optical flow networks, as well as
multi-scale linear filtering [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 29].
[45] develop integer architectures for learned entropy models, in order to
guarantee reliable decoding on arbitrary hardware platforms, and [38] discuss
selecting architecture parameters, such as the number of layers, or number of
channels per layer, while taking into account the RD performance.

Notable work in the space of learned entropy models includes [72], which use
block-based forward adaptation, and several other concurrent publications
performing learned backward adaptation [27, 73, 74]. More recent work on
learned backward adaptation includes the papers by [75, 76].

The first work exploring the RD trade-off with a single set of nonlinear
transforms is due to [46], who explore varying the quantization step size, as
in linear TC (corresponding most closely to “latent affine” in fig. 12). [77]
use the same method for coding point-cloud geometries. Layer-wise
parameterization as in eq. 18 was introduced for image compression by [48]. It
was generalized by [49] for a range of other tasks including compression. Our
technical contribution is to simplify the parameterization and to adapt it for
GDN.

Figure 14: Reconstructions of kodim15 [Kodak] compressed by BPG (left, 0.0738
bpp) and by a learned NTC model [32] optimized for MS-SSIM (right, 0.0713
bpp). The combination of NTC and an MS-SSIM loss function, which is designed
to model texture masking effects in the human visual system, leads to
significantly better texture retention in the sweater and fewer geometric
distortions on the person’s face.

While many authors have explored ANN-based compression in the context of
existing, commercially viable image and video compression methods [78, 79, 80,
81, 82, 83, 84], other authors begin with nonlinear transforms, and explore
incorporating concepts traditionally used in linear TC, such as energy
compaction [85], wavelets [86, 87], or trellis coded quantization [88].
Lossless image compression based on learned entropy models has been explored
by, e.g., [74, 89]. Still others explore the intersection between learned
image compression and other vision tasks such as content and semantic analysis
[90, 91, 92, 93], inpainting [94], super-resolution [95], quality enhancement
[96], or encryption [97].

Another topic of active research is the question of more “perceptual” image
compression. [26] discuss optimization of NTC models for squared error vs. MS-
SSIM. [98] provide a more in-depth discussion, with an even larger set of
different perceptual distortion measures. Interestingly, [99] provide a method
to optimize NTC models for non-differentiable perceptual metrics. [100, 101,
102] study the perceived image quality of learned image compression models
optimized for metrics such as squared error and MS-SSIM with the help of human
rating experiments. Other authors have explored augmenting the fixed
distortion measure with ANN-based losses that have shown visually convincing
results in image generation tasks, most notably generative adversarial
networks (GANs) [103, 104, 105, 106, 107]. [108] formulate theoretical limits
for the three-way trade-off between the rate, the reconstruction quality of an
image compression method, as well as divergence measures between the source
distribution and the marginal distribution of image reconstructions, the
latter of which are related to adversarial losses.

While image compression dominates the literature on NTC, other applications
have emerged as well, such as compression of point clouds [109, 77],
volumetric data [110], ANN features for tasks like large-scale image retrieval
[31], and compression of ANN parameters themselves [30]. Further reviews of
the current state of the literature, specifically with respect to image and
video compression, are given by [111, 88].

##  VII Conclusion

We have presented an overview of nonlinear transform coding, which heavily
relies on ANNs as universal function approximators and stochastic optimization
of the rate–distortion Lagrangian. We introduced the VECVQ algorithm, a
stochastically optimized version of entropy-constrained VQ, as well as a novel
method of $\lambda$-parameterization. Furthermore, we provide the first direct
comparison of different $\lambda$-parameterizations for image compression
models.

Most of the desirable properties of NTC stem from the use of stochastic RD
optimization and ANNs. Given enough computing power, NTC models are quickly
adaptible to arbitrary data sources, including domain-specific imagery (e.g.,
medical, astronomical) or new modalities of multimedia, since many parameters
of the system can be found using end-to-end optimization rather than manual
experimentation. This can lower prototyping times from years to weeks.

NTC models can also be directly optimized for any differentiable distortion
measure (or, more generally, distortion loss). Figure 14 illustrates the
visual difference of a model optimized for MS-SSIM vs. squared error, which
has long been noted as perceptually flawed [112]. This adaptability of NTC
will dovetail with the development of better and more general perceptual
losses, of which hybrid adversarial losses are an example [103, 104, 105, 106,
107].

With the rapidly increasing availability of parallelized computation, we
believe NTC will fundamentally change the landscape of practical data
compression.

## Acknowledgements

The authors would like to thank the anonymous reviewers, as well as Lucas
Theis and Aaron Wagner for their helpful comments on the draft and insightful
discussions.

## References

  * [1] Vivek K. Goyal  “Theoretical Foundations of Transform Coding”  In _IEEE Signal Processing Magazine_ 18.5, 2001  DOI: 10.1109/79.952802
  * [2] Allen Gersho and Robert M. Gray  “Vector Quantization and Signal Compression”  Kluwer, 1992
  * [3] Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus and Shimon Schocken  “Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function”  In _Neural Networks_ 6.6, 1993  DOI: 10.1016/S0893-6080(05)80131-5
  * [4] George Toderici et al.  “Variable Rate Image Compression with Recurrent Neural Networks”  In _4th Int. Conf. on Learning Representations (ICLR)_ , 2016  arXiv:1511.06085
  * [5] Johannes Ballé, Valero Laparra and Eero P. Simoncelli  “End-to-end Optimized Image Compression”  In _5th Int. Conf. on Learning Representations (ICLR)_ , 2017  OPENREVIEW: rJxdQ3jeg
  * [6] Lucas Theis, Wenzhe Shi, Andrew Cunningham and Ferenc Huszár  “Lossy Image Compression with Compressive Autoencoders”  In _5th Int. Conf. on Learning Representations (ICLR)_ , 2017  OPENREVIEW: rJiNwv9gg
  * [7] George Toderici et al.  “Full Resolution Image Compression with Recurrent Neural Networks”  In _2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017  DOI: 10.1109/CVPR.2017.577
  * [8] Oren Rippel and Lubomir Bourdev  “Real-Time Adaptive Image Compression”  In _Proc. of the 34th Int. Conf. on Machine Learning_ 70, Proc. of Machine Learning Research, 2017
  * [9] Eirikur Agustsson et al.  “Soft-to-hard Vector Quantization for End-to-end Learning Compressible Representations”  In _Advances in Neural Information Processing Systems 30_ , 2017
  * [HEVC] “ITU-R Rec. H.265 & ISO/IEC 23008-2: High Efficiency Video Coding”, 2013
  * [10] Philip A. Chou, Tom Lookabaugh and Robert M. Gray  “Entropy-Constrained Vector Quantization”  In _IEEE Trans. on Acoustics, Speech, and Signal Processing_ 37.1, 1989  DOI: 10.1109/29.17498
  * [11] Thomas M. Cover and Joy A. Thomas  “Elements of Information Theory”  Wiley, 2006
  * [12] Gary J. Sullivan  “Efficient scalar quantization of exponential and Laplacian random variables”  In _IEEE Trans. on Information Theory_ 42.5, 1996  DOI: 10.1109/18.532878
  * [13] T. Berger  “Optimum Quantizers and Permutation Codes”  In _IEEE Trans. on Information Theory_ 18.6, 1972  DOI: 10.1109/TIT.1972.1054906
  * [14] N. Farvardin and J. Modestino  “Optimum Quantizer Performance for a Class of Non-Gaussian Memoryless Sources”  In _IEEE Trans. on Information Theory_ 30.3, 1984  DOI: 10.1109/TIT.1984.1056920
  * [15] Léon Bottou and Yoshua Bengio  “Convergence Properties of the K-Means Algorithms”  In _Advances in Neural Information Processing Systems 7_ , 1994
  * [16] S.. Lloyd  “Least Squares Quantization in PCM” (Previously an unpublished Bell Lab tech. note, 1957.)  In _IEEE Trans. on Information Theory_ 28.2, 1982  DOI: 10.1109/TIT.1982.1056489
  * [17] J. Max  “Quantizing for Minimum Distortion”  In _IRE Trans. on Information Theory_ 6.1, 1960  DOI: 10.1109/TIT.1960.1057548
  * [18] James MacQueen  “Some Methods for Classification and Analysis of Multivariate Observations”  In _Proc. of the 5th Berkeley Symp. on Mathematical Statistics and Probability_ , 1967, pp. 281–297
  * [19] Y. Linde, A. Buzo and Robert M. Gray  “An Algorithm for Vector Quantizer Design”  In _IEEE Trans. on Communications_ 28.1, 1980  DOI: 10.1109/TCOM.1980.1094577
  * [20] W.. Bennett  “Spectra of Quantized Signals”  In _Bell System Technical Journal_ 27.3, 1948  DOI: 10.1002/j.1538-7305.1948.tb01340.x
  * [21] Allen Gersho  “Asymptotically Optimal Block Quantization”  In _IEEE Trans. on Information Theory_ 25.4, 1979  DOI: 10.1109/TIT.1979.1056067
  * [22] Leonard Schuchman  “Dither Signals and Their Effect on Quantization Noise”  In _IEEE Trans. on Communication Technology_ 12.4, 1964  DOI: 10.1109/TCOM.1964.1088973
  * [23] Johannes Ballé, Valero Laparra and Eero P. Simoncelli  “End-to-End Optimization of Nonlinear Transform Codes for Perceptual Quality”  In _2016 Picture Coding Symp. (PCS)_ , 2016  DOI: 10.1109/PCS.2016.7906310
  * [24] Irina Higgins et al.  “$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework”  In _5th Int. Conf. on Learning Representations (ICLR)_ , 2017  OPENREVIEW: Sy2fzU9gl
  * [25] Alexander A. Alemi et al.  “Fixing a Broken ELBO”  In _Proc. of the 35th Int. Conf. on Machine Learning_ 80, Proc. of Machine Learning Research, 2018
  * [26] Johannes Ballé et al.  “Variational Image Compression with a Scale Hyperprior”  In _6th Int. Conf. on Learning Representations (ICLR)_ , 2018  OPENREVIEW: rkcQFMZRb
  * [27] David Minnen, Johannes Ballé and George Toderici  “Joint Autoregressive and Hierarchical Priors for Learned Image Compression”  In _Advances in Neural Information Processing Systems 31_ , 2018
  * [28] Eirikur Agustsson and Lucas Theis  “Universally Quantized Neural Compression” accepted at NeurIPS, 2020  arXiv:2006.09952
  * [29] Eirikur Agustsson et al.  “Scale-Space Flow for End-to-End Optimized Video Compression”  In _2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020  DOI: 10.1109/CVPR42600.2020.00853
  * [30] Deniz Oktay, Johannes Ballé, Saurabh Singh and Abhinav Shrivastava  “Scalable Model Compression by Entropy Penalized Reparameterization”  In _8th Int. Conf. on Learning Representations (ICLR)_ , 2020  OPENREVIEW: HkgxW0EYDS
  * [31] Saurabh Singh et al.  “End-to-End Learning of Compressible Features” accepted at ICIP, 2020  arXiv:2007.11797
  * [32] David Minnen and Saurabh Singh  “Channel-Wise Autoregressive Entropy Models for Learned Image Compression” accepted at ICIP, 2020  arXiv:2007.08739
  * [Kodak] Eastman Kodak  “Kodak Lossless True Color Image Suite (PhotoCD PCD0992)”, 1993  URL: http://r0k.us/graphics/kodak/
  * [33] Johannes Ballé  “Efficient Nonlinear Transforms for Lossy Image Compression”  In _2018 Picture Coding Symp. (PCS)_ , 2018  DOI: 10.1109/PCS.2018.8456272
  * [34] Djork-Arné Clevert, Thomas Unterthiner and Sepp Hochreiter  “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”  In _4th Int. Conf. on Learning Representations (ICLR)_ , 2016  arXiv:1511.07289
  * [35] Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter  “Self-Normalizing Neural Networks”  In _Advances in Neural Information Processing Systems 30_ , 2017
  * [36] Prajit Ramachandran, Barret Zoph and Quoc V. Le  “Searching for Activation Functions”, 2017  arXiv:1710.05941
  * [37] Johannes Ballé, Valero Laparra and Eero P. Simoncelli  “Density Modeling of Images Using a Generalized Normalization Transformation”  In _4th Int. Conf. on Learning Representations (ICLR)_ , 2016  arXiv:1511.06281
  * [38] Nick Johnston, Elad Eban, Ariel Gordon and Johannes Ballé  “Computationally Efficient Neural Image Compression”, 2019  arXiv:1912.08771
  * [39] Detlev Marpe, Heiko Schwarz and Thomas Wiegand  “Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC Video Compression Standard”  In _IEEE Trans. on Circuits and Systems for Video Technology_ 13.7, 2003  DOI: 10.1109/TCSVT.2003.815173
  * [JPEG] “ITU-R Rec. T.81 & ISO/IEC 10918-1: Digital compression and coding of continuous-tone still images”, 1992
  * [40] Zhou Wang, Eero P. Simoncelli and Alan C. Bovik  “Multiscale Structural Similarity for Image Quality Assessment”  In _The Thirty-Seventh Asilomar Conf. on Signals, Systems & Computers, 2003_, 2003  DOI: 10.1109/ACSSC.2003.1292216
  * [41] C.. Wallace  “Classification by minimum-message-length inference”  In _Advances in Computing and Information — ICCI ’90_ , 1990  DOI: 10.1007/3-540-53504-7_63
  * [42] Geoffrey E. Hinton and Richard S. Zemel  “Autoencoders, Minimum Description Length and Helmholtz Free Energy”  In _Advances in Neural Information Processing Systems 6_ , 1993
  * [43] Brendan J. Frey and Geoffrey E. Hinton  “Efficient Stochastic Source Coding and an Application to a Bayesian Network Source Model”  In _The Computer Journal_ 40.2/3, 1997  DOI: 10.1093/comjnl/40.2_and_3.157
  * [44] James Townsend, Thomas Bird and David Barber  “Practical lossless compression with latent variables using bits back coding”  In _7th Int. Conf. on Learning Representations (ICLR)_ , 2019  OPENREVIEW: ryE98iR5tm
  * [45] Johannes Ballé, David Minnen and Nick Johnston  “Integer Networks for Data Compression with Latent-Variable Models”  In _6th Int. Conf. on Learning Representations (ICLR)_ , 2018  OPENREVIEW: S1zz2i0cY7
  * [46] Thierry Dumas, Aline Roumy and Christine Guillemot  “Autoencoder Based Image Compression: Can the Learning be Quantization Independent?”  In _2018 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018  DOI: 10.1109/ICASSP.2018.8462263
  * [47] Vincent Dumoulin, Jonathon Shlens and Manjunath Kudlur  “A Learned Representation for Artistic Style”  In _5th Int. Conf. on Learning Representations (ICLR)_ , 2017  OPENREVIEW: BJO-BuT1g
  * [48] Yoojin Choi, Mostafa El-Khamy and Jungwon Lee  “Variable Rate Deep Image Compression with a Conditional Autoencoder”  In _2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , 2019  DOI: 10.1109/ICCV.2019.00324
  * [49] Alexey Dosovitskiy and Josip Djolonga  “You Only Train Once: Loss-Conditional Training of Deep Networks”  In _8th Int. Conf. on Learning Representations (ICLR)_ , 2020  OPENREVIEW: HyxY6JHKwr
  * [50] N. Ahmed and K.. Rao  “Orthogonal Transforms for Digital Signal Processing”  Springer, 1975
  * [51] Geoffrey E. Hinton and R.. Salakhutdinov  “Reducing the Dimensionality of Data with Neural Networks”  In _Science_ 313.5786, 2006  DOI: 10.1126/science.1127647
  * [52] Diederik P. Kingma and Max Welling  “Auto-Encoding Variational Bayes”  In _2nd Int. Conf. on Learning Representations (ICLR)_ , 2014  arXiv:1312.6114
  * [53] Danilo J. Rezende, Shakir Mohamed and Daan Wierstra  “Stochastic Backpropagation and Approximative Inference in Deep Generative Models”  In _Proc. of the 31st Int. Conf. on Machine Learning_ 32, Proc. of Machine Learning Research, 2014
  * [54] David Alexandre, Chih-Peng Chang, Wen-Hsiao Peng and Hsueh-Ming Hang  “Learned Image Compression with Soft Bit-Based Rate-Distortion Optimization”  In _2019 IEEE Int. Conf. on Image Processing (ICIP)_ , 2019  DOI: 10.1109/ICIP.2019.8803120
  * [55] Ken M. Nakanishi, Shin-ichi Maeda, Takeru Miyato and Daisuke Okanohara  “Neural Multi-scale Image Compression”  In _Computer Vision – ACCV 2018_ , 2018  DOI: 10.1007/978-3-030-20876-9_45
  * [56] Chunlei Cai, Li Chen, Xiaoyun Zhang and Zhiyong Gao  “Efficient Variable Rate Image Compression with Multi-Scale Decomposition Network”  In _IEEE Trans. on Circuits and Systems for Video Technology_ 29.12, 2019  DOI: 10.1109/TCSVT.2018.2880492
  * [57] Haojie Liu, Tong Chen, Qiu Shen and Zhan Ma  “Practical Stacked Non-local Attention Modules for Image Compression”  In _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) Wksps._ , 2019
  * [58] Zhengxue Cheng, Heming Sun, Masaru Takeuchi and Jiro Katto  “Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules”  In _2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020  DOI: 10.1109/CVPR42600.2020.00796
  * [59] Jianrui Cai and Lei Zhang  “Deep Image Compression with Iterative Non-Uniform Quantization”  In _2018 25th IEEE Int. Conf. on Image Processing (ICIP)_ , 2018  DOI: 10.1109/ICIP.2018.8451411
  * [60] Alexander G. Ororbia et al.  “Learned Neural Iterative Decoding for Lossy Image Compression Systems”  In _2019 Data Compression Conf. (DCC)_ , 2019  DOI: 10.1109/DCC.2019.00008
  * [61] Chao-Yuan Wu, Nayan Singhal and Philipp Krähenbühl  “Video Compression Through Image Interpolation”  In _Computer Vision – ECCV 2018_ , 2018  DOI: 10.1007/978-3-030-01237-3_26
  * [62] Jun Han, Salvator Lombardo, Christopher Schroers and Stephan Mandt  “Deep Probabilistic Video Compression”  In _7th Int. Conf. on Learning Representations (ICLR)_ , 2019  OPENREVIEW: r1l-e3Cqtm
  * [63] Guo Lu et al.  “DVC: An End-to-End Deep Video Compression Framework”  In _2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019  DOI: 10.1109/CVPR.2019.01126
  * [64] Zhibo Chen, Tianyu He, Xin Jin and Feng Wu  “Learning for Video Compression”  In _IEEE Trans. on Circuits and Systems for Video Technology_ 30.2, 2020  DOI: 10.1109/TCSVT.2019.2892608
  * [65] Oren Rippel et al.  “Learned Video Compression”  In _2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , 2019  DOI: 10.1109/ICCV.2019.00355
  * [66] Amirhossein Habibian, Ties Rozendaal, Jakub M. Tomczak and Taco S. Cohen  “Video Compression with Rate-Distortion Autoencoders”  In _2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , 2019  DOI: 10.1109/ICCV.2019.00713
  * [67] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer and Christopher Schroers  “Neural Inter-Frame Compression for Video Coding”  In _2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , 2019  DOI: 10.1109/ICCV.2019.00652
  * [68] Salvator Lombardo, Jun Han, Christopher Schroers and Stephan Mandt  “Deep Generative Video Compression”  In _Advances in Neural Information Processing Systems 32_ , 2019
  * [69] Adam Golinski et al.  “Feedback Recurrent Autoencoder for Video Compression”, 2020  arXiv:2004.04342
  * [70] Ren Yang, Fabian Mentzer, Luc Van Gool and Radu Timofte  “Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model”, 2020  arXiv:2006.13560
  * [71] Guo Lu et al.  “An End-to-End Learning Framework for Video Compression” to appear  In _IEEE Trans. on Pattern Analysis and Machine Intelligence_ , 2020  DOI: 10.1109/TPAMI.2020.2988453
  * [72] David Minnen et al.  “Image-Dependent Local Entropy Models for Learned Image Compression”  In _2018 25th IEEE Int. Conf. on Image Processing (ICIP)_ , 2018  DOI: 10.1109/ICIP.2018.8451502
  * [73] Jooyoung Lee, Seunghyun Cho and Seung-Kwon Beack  “Context-adaptive Entropy Model for End-to-end Optimized Image Compression”  In _7th Int. Conf. on Learning Representations (ICLR)_ , 2019  OPENREVIEW: HyxKIiAqYQ
  * [74] Fabian Mentzer et al.  “Conditional Probability Models for Deep Image Compression”  In _2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2018  DOI: 10.1109/CVPR.2018.00462
  * [75] Mu Li et al.  “Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression”  In _IEEE Trans. on Image Processing_ 29, 2020  DOI: 10.1109/TIP.2020.2985225
  * [76] Zongyu Guo et al.  “3-D Context Entropy Model for Improved Practical Image Compression”  In _2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Wksps. (CVPRW)_ , 2020  DOI: 10.1109/CVPRW50498.2020.00066
  * [77] André F.. Guarda, Nuno M.. Rodrigues and Fernando Pereira  “Deep Learning-Based Point Cloud Geometry Coding: RD Control Through Implicit and Explicit Quantization”  In _2020 IEEE Int. Conf. on Multimedia & Expo Wksps. (ICMEW)_, 2020  DOI: 10.1109/ICMEW46912.2020.9106022
  * [78] Thorsten Laude and Jörn Ostermann  “Deep Learning-Based Intra Prediction Mode Decision for HEVC”  In _2016 Picture Coding Symp. (PCS)_ , 2016  DOI: 10.1109/PCS.2016.7906399
  * [79] Feng Jiang et al.  “An End-To-End Compression Framework Based on Convolutional Neural Networks”  In _IEEE Trans. on Circuits and Systems for Video Technology_ 28.10, 2018  DOI: 10.1109/TCSVT.2017.2734838
  * [80] Jiahao Li et al.  “Fully Connected Network-Based Intra Prediction for Image Coding”  In _IEEE Trans. on Image Processing_ 27.7, 2018  DOI: 10.1109/TIP.2018.2817044
  * [81] Chuanmin Jia et al.  “Content-Aware Convolutional Neural Network for In-Loop Filtering in High Efficiency Video Coding”  In _IEEE Trans. on Image Processing_ 28.7, 2019  DOI: 10.1109/TIP.2019.2896489
  * [82] Dong Liu, Haichuan Ma, Zhiwei Xiong and Feng Wu  “CNN-based DCT-like Transform for Image Compression”  In _MMM 2018: MultiMedia Modeling_ , 2018  DOI: 10.1007/978-3-319-73600-6_6
  * [83] Yueyu Hu, Wenhan Yang, Mading Li and Jiaying Liu  “Progressive Spatial Recurrent Neural Network for Intra Prediction”  In _IEEE Trans. on Multimedia_ 21.12, 2019  DOI: 10.1109/TMM.2019.2920603
  * [84] Philipp Helle et al.  “Intra Picture Prediction for Video Coding with Neural Networks”  In _2019 Data Compression Conference (DCC)_ , 2019  DOI: 10.1109/DCC.2019.00053
  * [85] Zhengxue Cheng, Heming Sun, Masaru Takeuchi and Jiro Katto  “Energy Compaction-Based Image Compression Using Convolutional Autoencoder”  In _IEEE Trans. on Multimedia_ 22.4, 2020  DOI: 10.1109/TMM.2019.2938345
  * [86] Pinar Akyazi and Touradj Ebrahimi  “Learning-Based Image Compression Using Convolutional Autoencoder and Wavelet Decomposition”  In _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) Wksps._ , 2019
  * [87] Chuxi Yang, Yan Zhao and Shigang Wang  “Deep Image Compression in the Wavelet Transform Domain Based on High Frequency Sub-Band Prediction”  In _IEEE Access_ 7, 2019  DOI: 10.1109/ACCESS.2019.2911403
  * [88] Dong Liu et al.  “Deep Learning-Based Video Coding: A Review and a Case Study”  In _ACM Computing Surveys_ 53.1, 2020  DOI: 10.1145/3368405
  * [89] Fabian Mentzer et al.  “Practical Full Resolution Learned Lossless Image Compression”  In _2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019  DOI: 10.1109/CVPR.2019.01088
  * [90] Mu Li et al.  “Learning Convolutional Networks for Content-Weighted Image Compression”  In _2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2018  DOI: 10.1109/CVPR.2018.00339
  * [91] Joaquim Campos, Simon Meierhans, Abdelaziz Djelouah and Christopher Schroers  “Content Adaptive Optimization for Neural Image Compression”  In _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) Wksps._ , 2019
  * [92] Mohammad Akbari, Jie Liang and Jingning Han  “DSSLIC: Deep Semantic Segmentation-Based Layered Image Compression”  In _2019 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019  DOI: 10.1109/ICASSP.2019.8683541
  * [93] Sihui Luo et al.  “DeepSIC: Deep Semantic Image Compression”  In _ICONIP 2018: Neural Information Processing_ , 2018  DOI: 10.1007/978-3-030-04167-0_9
  * [94] Mohammad Haris Baig, Vladlen Koltun and Lorenzo Torresani  “Learning to Inpaint for Image Compression”  In _Advances in Neural Information Processing Systems 30_ , 2017
  * [95] Sheng Cao, Chao-Yuan Wu and Philipp Krähenbühl  “Lossless Image Compression through Super-Resolution”, 2020  arXiv:2004.02872
  * [96] Jooyoung Lee, Seunghyun Cho and Munchurl Kim  “A Hybrid Architecture of Jointly Learning Image Compression and Quality Enhancement with Improved Entropy Minimization”, 2019  arXiv:1912.12817
  * [97] Xintao Duan, Jingjing Liu and En Zhang  “Efficient Image Encryption and Compression Based on a VAE Generative Model”  In _Journal of Real-Time Image Processing_ 16, 2019  DOI: 10.1007/s11554-018-0826-4
  * [98] Keyan Ding, Kede Ma, Shiqi Wang and Eero P. Simoncelli  “Comparison of Image Quality Models for Optimization of Image Processing Systems”, 2020  arXiv:2005.01338
  * [99] Li-Heng Chen et al.  “ProxIQA: A Proxy Approach to Perceptual Optimization of Learned Image Compression”, 2019  arXiv:1910.08845
  * [100] Giuseppe Valenzise, Andrei Purica, Vedad Hulusic and Marco Cagnazzo  “Quality Assessment of Deep-Learning-Based Image Compression”  In _2018 IEEE 20th Int. Wksp. on Multimedia Signal Processing (MMSP)_ , 2018  DOI: 10.1109/MMSP.2018.8547064
  * [101] Zhengxue Cheng et al.  “Perceptual Quality Study on Deep Learning Based Image Compression”  In _2019 IEEE Int. Conf. on Image Processing (ICIP)_ , 2019  DOI: 10.1109/ICIP.2019.8803824
  * [102] Joao Ascenso, Pinar Akyazi, Fernando Pereira and Touradj Ebrahimi  “Learning-Based Image Coding: Early Solutions Reviewing and Subjective Quality Evaluation”  In _Optics, Photonics and Digital Technologies for Imaging Applications VI_ 11353, Proc. SPIE, 2020  DOI: 10.1117/12.2555368
  * [103] Shibani Santurkar, David Budden and Nir Shavit  “Generative Compression”  In _2018 Picture Coding Symp. (PCS)_ , 2018  DOI: 10.1109/PCS.2018.8456298
  * [104] Eirikur Agustsson et al.  “Generative Adversarial Networks for Extreme Learned Image Compression”  In _2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , 2019  DOI: 10.1109/ICCV.2019.00031
  * [105] Michael Tschannen, Eirikur Agustsson and Mario Lucic  “Deep Generative Models for Distribution-Preserving Lossy Compression”  In _Advances in Neural Information Processing Systems 31_ , 2018
  * [106] Shinobu Kudo, Shota Orihashi, Ryuichi Tanida and Atsushi Shimizu  “GAN-based Image Compression Using Mutual Information Maximizing Regularization”  In _2019 Picture Coding Symp. (PCS)_ , 2019  DOI: 10.1109/PCS48520.2019.8954548
  * [107] Fabian Mentzer, George Toderici, Michael Tschannen and Eirikur Agustsson  “High-Fidelity Generative Image Compression” accepted at NeurIPS, 2020  arXiv:2006.09965
  * [108] Yochai Blau and Tomer Michaeli  “Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff”  In _Proc. of the 36th Int. Conf. on Machine Learning_ 97, Proc. of Machine Learning Research, 2019
  * [109] Maurice Quach, Giuseppe Valenzise and Frederic Dufaux  “Learning Convolutional Transforms for Lossy Point Cloud Geometry Compression”  In _2019 IEEE Int. Conf. on Image Processing (ICIP)_ , 2019  DOI: 10.1109/ICIP.2019.8803413
  * [110] Danhang Tang et al.  “Deep Implicit Volume Compression”  In _2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020  DOI: 10.1109/CVPR42600.2020.00137
  * [111] Siwei Ma et al.  “Image and Video Compression with Neural Networks: A Review”  In _IEEE Trans. on Circuits and Systems for Video Technology_ 30.6, 2020  DOI: 10.1109/TCSVT.2019.2910119
  * [112] Bernd Girod  “What’s wrong with mean squared error?”  In _Digital Images and Human Vision_ MIT Press, 1993, pp. 207–220
  * [113] Robert M. Gray and David L. Neuhoff  “Quantization”  In _IEEE Trans. on Information Theory_ 44.6, 1998  DOI: 10.1109/18.720541

##  VIII Supplemental materials

###  VIII-A Local properties of nonlinear transforms

In this section, we appeal to high-rate quantization theory [21, 113] to claim
that an optimal nonlinear analysis transform has an inverse Jacobian whose
columns tend towards orthogonality, at least in the limit of high rate. In
this sense, at high rates, the optimal nonlinear analysis transform is locally
orthogonal, approximately. Similarly the optimal nonlinear synthesis transform
has a Jacobian whose columns tend toward orthogonality. To simplify the
technical argument, we first assume that the dimension $M$ of the latent space
is the same as the source dimension, and that the transforms are at least
locally invertible. We also assume here that the distortion measure is the
squared error. Later we comment on how to lift these assumptions.

Let $\bm{x}$, ${\bm{y}}$, ${\bm{k}}$, and $\bm{\hat{x}}$ denote source,
latent, quantized latent, and reproduction vectors, respectively, all with
dimension $M$. Let ${\bm{y}}=g_{a}(\bm{x})$ denote the analysis transform, let
${\bm{k}}=\lfloor{\bm{y}}\rceil$ denote uniform scalar quantization of
${\bm{y}}$ by component-wise rounding, and let $\bm{\hat{x}}=g_{s}({\bm{k}})$
denote the synthesis transform. Let $q({\bm{k}})$ denote a probability mass
function on the integer vectors ${\bm{k}}$ (typically fully-factorized across
the components of ${\bm{k}}$). Assume $g_{a}$, $g_{s}$, and $q$ are optimal
for some very large Lagrange multiplier $\lambda$. That is, they minimize
$L=\lambda D+R$, where

$\displaystyle D$ $\displaystyle=\operatorname{\mathbb{E}}_{\bm{x}\sim
p}\,\Bigl{\|}\bm{x}-g_{s}\bigl{(}\lfloor
g_{a}(\bm{x})\rceil\bigr{)}\Bigr{\|}^{2},$ (19) $\displaystyle R$
$\displaystyle=\operatorname{\mathbb{E}}_{\bm{x}\sim p}\,\Bigl{[}-\log
q\bigl{(}\lfloor g_{a}(\bm{x})\rceil\bigr{)}\Bigr{]},$ (20)

and $p(\bm{x})$ is the density of $\bm{x}$.

Further, let $S_{\bm{k}}=g_{a}^{-1}({\bm{k}}+[-0.5,0.5)^{M})$ be the
quantization cell containing all source vectors $\bm{x}$ that map to
${\bm{k}}=\lfloor g_{a}(\bm{x})\rceil$, let
$\bm{x}_{\bm{k}}=g_{a}^{-1}({\bm{k}})\in S_{\bm{k}}$ be the source vector that
maps exactly to ${\bm{k}}=g_{a}(\bm{x})$, and let $\bm{\hat{x}}_{\bm{k}}$ be
the centroid of $S_{\bm{k}}$. Note that
$\bm{\hat{x}}_{\bm{k}}=g_{s}({\bm{k}})$ since the synthesis transform $g_{s}$
is optimal.

Since $\lambda$ is large, we are in the high-rate regime. In this regime, the
cells $S_{\bm{k}}$ are small. Thus, for all $\bm{x}\in S_{\bm{k}}$,
$g_{a}(\bm{x})\approx g_{a}(\bm{x}_{\bm{k}})+\nabla
g_{a}(\bm{x}_{\bm{k}})\cdot(\bm{x}-\bm{x}_{\bm{k}})$, where $\nabla g_{a}$ is
the $M\times M$ Jacobian of $g_{a}$. In other words,
${\bm{y}}\approx{\bm{k}}+\nabla
g_{a}(\bm{x}_{\bm{k}})\cdot(\bm{x}-\bm{x}_{\bm{k}})$, or
$\bm{x}\approx\bm{x}_{\bm{k}}+\nabla
g_{a}(\bm{x}_{\bm{k}})^{-1}\cdot({\bm{y}}-{\bm{k}})$. Thus
$S_{\bm{k}}\approx\bm{x}_{\bm{k}}+\nabla
g_{a}(\bm{x}_{\bm{k}})^{-1}\cdot[-0.5,0.5)^{M}$ is the image of the hypercube
$[-0.5,0.5)^{M}$ under the linear map $\nabla g_{a}(\bm{x}_{\bm{k}})^{-1}$,
centered on $\bm{x}_{\bm{k}}$. In other words, $S_{\bm{k}}$ is a hyper
($M$-dimensional) parallelepiped, whose edge vectors are given by the columns
of $\nabla g_{a}(\bm{x}_{\bm{k}})^{-1}$, and whose volume is $\det(\nabla
g_{a}(\bm{x}_{\bm{k}})^{-1})$.

Now, as in [113, eq. (25)], approximate the partial distortion in cell
$S_{\bm{k}}$ as

$\displaystyle D_{\bm{k}}$
$\displaystyle=\frac{1}{M}\int_{S_{\bm{k}}}\|\bm{x}-\bm{\hat{x}}_{\bm{k}}\|^{2}p(\bm{x})\;\mathrm{d}\bm{x}$
$\displaystyle\approx
p(\bm{\hat{x}}_{\bm{k}})m(S_{\bm{k}})V(S_{\bm{k}})^{1+2/M},$ (21)

where $V(S_{\bm{k}})$ is the volume of $S_{\bm{k}}$ and $m(S_{\bm{k}})$ is its
_normalized second moment of inertia_ [21, 113], defined by

$m(S_{\bm{k}})=\frac{1}{M}\frac{1}{V(S_{\bm{k}})^{1+2/M}}\int_{S_{\bm{k}}}\|\bm{x}-\bm{\hat{x}}_{\bm{k}}\|^{2}\;\mathrm{d}\bm{x}.$
(22)

Note that $m(S_{\bm{k}})$ is normalized such that it is a function of only the
shape of $S_{\bm{k}}$, not its scale [21], i.e.,

$\displaystyle m(\alpha S_{\bm{k}})$
$\displaystyle=\frac{1}{M}\frac{1}{V(\alpha
S_{\bm{k}})^{1+2/M}}\int_{S_{\bm{k}}}\|\alpha\bm{x}-\alpha\bm{\hat{x}}_{\bm{k}}\|^{2}\;\mathrm{d}(\alpha\bm{x})$
$\displaystyle=\frac{1}{M}\frac{\alpha^{M+2}}{\alpha^{M(1+2/M)}V(S_{\bm{k}})^{1+2/M}}\int_{S_{\bm{k}}}\|\bm{x}-\bm{\hat{x}}_{\bm{k}}\|^{2}\;\mathrm{d}\bm{x}$
$\displaystyle=m(S_{\bm{k}}).$ (23)

It is intuitively obvious that among hyper parallelepipeds, $m(S_{\bm{k}})$ is
minimized by a hyper cube, for which the value is $1/12$.

The overall distortion can thus be approximated as

$\displaystyle D=\sum_{\bm{k}}D_{\bm{k}}$
$\displaystyle\approx\sum_{\bm{k}}p(\bm{\hat{x}}_{\bm{k}})m(S_{\bm{k}})V(S_{\bm{k}})^{1+2/M}$
$\displaystyle\approx\int
p(\bm{x})m(\bm{x})V(\bm{x})^{2/M}\;\mathrm{d}\bm{x},$ (24)

where $V(\bm{x})$ is the volume of the cell containing $\bm{x}$, and likewise
$m(\bm{x})$ is the normalized second moment of inertia of the cell containing
$\bm{x}$.

The overall rate can also be approximated, as

$\displaystyle R$
$\displaystyle\approx-\frac{1}{M}\sum_{\bm{k}}p(\bm{\hat{x}}_{\bm{k}})V(S_{\bm{k}})\log\bigl{(}q(\bm{\hat{x}}_{\bm{k}})V(S_{\bm{k}})\bigr{)}$
$\displaystyle\approx-\frac{1}{M}\int
p(\bm{x})\log\bigl{(}q(\bm{x})V(\bm{x})\bigr{)}\;\mathrm{d}\bm{x}$ (25)
$\displaystyle=\frac{1}{M}\Bigl{(}D_{\text{KL}}[p\;\|\;q]+h[p]\Bigr{)}-\frac{1}{M}\int
p(\bm{x})\log V(\bm{x})\;\mathrm{d}\bm{x},$

where $q(\bm{x})$ denotes the density such that
$q(\bm{x})=q(\bm{k})/V(S_{\bm{k}})$ for all $\bm{x}\in S_{\bm{k}}$,
$D_{\text{KL}}[p\;\|\;q]$ is the Kullback–Leibler divergence between $p$ and
$q$, and $h[p]$ is the differential entropy of $p$.

Putting these together,

$\displaystyle L$ $\displaystyle\approx\operatorname{\mathbb{E}}_{\bm{x}\sim
p}\bigr{[}\lambda m(\bm{x})V(\bm{x})^{2/M}-\tfrac{1}{2}\log
V(\bm{x})^{2/M}\bigl{]}$ (26)
$\displaystyle\phantom{\approx}+\tfrac{1}{M}D_{\text{KL}}[p\;\|\;q]+\mathrm{const}.$
(27)

We now claim that a linear orthogonal transform $g_{a}$ (and its corresponding
$q$) will nearly minimize the two terms (26) and (27). To see this, observe
that if we could minimize them independently over $q$, $m$, and $V$, we would
choose $q(x)=p(x)$, $m(x)=1/12$, and $V(x)=((2\ln 2)\lambda m(x))^{-2/M}=((\ln
2)\lambda/6)^{-2/M}$. The corresponding transform $g_{a}$ would be a linear
orthogonal transform, which has hyper cubical quantization cells all the same
size. Unfortunately, this is generally not a viable solution for transform
codes, since $q$ is typically constrained to be factorized along the
components of $g_{a}$. Hence, $D_{\text{KL}}[p\;\|\;q]$ will generally not be
zero (unless $p$ itself can be factorized along the components of $g_{a}$, for
example if $p$ is Gaussian and the orthonormal transform is oriented along the
principal axes of the Gaussian). Thus when $q$ is so constrained, there is
generally a trade-off between minimizing the two terms.

In the limit of high rate, as $\lambda$ becomes large, (26) dominates (27).
The former is minimized by the choices given above, corresponding to an
orthogonal transform with hyper cubical quantization cells all the same size.
Thus, orthogonal transforms are near-optimal in the high-rate case. (This is a
special case of the usual high-rate quantization theory result that the
optimal entropy-constrained vector quantizer is formed by tesselating
polytopes with minimum normalized moment of inertia [21].)

For realistic bit rates, of course, (27) matters. For a given transform
$g_{a}$ (and hence $m$ and $V$), the optimal $q=q(g_{a})$ will minimize
$D_{\text{KL}}[p\;\|\;q]$ subject to the constraint that $q$ is factorizable
over the components of $g_{a}$. Since both terms depend on $g_{a}$, they can
no longer be independently minimized. Thus $m(\bm{x})$ may no longer be
minimal, meaning that the quantization cells may no longer be cubical.
Furthermore, $V(\bm{x})$ may no longer be constant, meaning that the codewords
may no longer have a uniform density. As $\lambda$ changes the target bit
rate, so it will change the character of the transform, as the quantization
cells evolve from hyper cubes at high rates to other shapes at lower rates.
Through such evolution, however, (26) always biases the solution toward
quantization cells that have as small normalized moment of inertia as
possible, i.e., towards hyper cubes. It is for this reason that the local
Jacobian and inverse Jacobian matrices of an optimal nonlinear transform
$g_{a}$ tend towards orthogonality, especially at high rates, even as globally
the transform becomes warped.

Now let us comment on how one might lift some of our assumptions. In practice,
of course, the latent dimension $M$ is frequently chosen lower than the source
dimension, say $M_{0}$. In this case, the analysis transform $g_{a}^{(M)}$
cannot be invertible, and its Jacobian is not square and thus has no inverse.
However, suppose that $g_{a}^{(M_{0})}$ is the analysis transform optimal for
the case where the latent dimension is the same as the source dimension, at a
particular level of distortion $D$, as described above. Further suppose all
but $M$ of the latent variables of $g_{a}^{(M_{0})}$ consistently quantize to
zero. Then the source distribution must be largely confined to an
$M$-dimensional manifold, at least within distortion $D$. Letting
$g_{a}^{(M)}$ be the analysis transform obtained from $g_{a}^{(M_{0})}$ by
removing the latent variables that consistently quantize to zero, it is clear
that the Jacobian of $g_{a}^{(M)}$ is the same as the Jacobian of
$g_{a}^{(M_{0})}$, with some rows removed. More generally, as long as $M$ is
not chosen below the dimension essential for reproducing the source at a given
level of distortion, the inverse image of a quantization cube
${\bm{k}}+[-0.5,0.5)^{M}$ in the latent space is a quantization cell
$S_{\bm{k}}$ in the $M$-dimensional manifold. Within the manifold, the
normalized moment of inertia of $S_{\bm{k}}$ should still tend to be
minimized, and hence the $M$ rows of the Jacobian of $g_{a}^{(M)}$ should
still tend to be orthogonal, using arguments similar to those above.

The other assumption that one might lift is the assumption that the distortion
measure is the squared error. The basic conclusions above should hold for any
doubly differentiable distortion loss (e.g., most any perceptual distortion
function, as well as hybrid adversarial losses), after a local coordinate
transformation. To proceed, suppose for all $\bm{x}$,
$d(\bm{x},\bm{\tilde{x}})$ as a function of $\bm{\tilde{x}}$ is twice
continuously differentiable in $\bm{\tilde{x}}$ and that it is minimized when
$\bm{\tilde{x}}=\bm{x}$. Then

$d(\bm{x},\bm{\tilde{x}})\approx
d(\bm{x},\bm{x})+(\bm{\tilde{x}}-\bm{x})^{\top}H(\bm{x})(\bm{\tilde{x}}-\bm{x})$
(28)

is the Taylor approximation of $d(\bm{x},\bm{\tilde{x}})$ in $\bm{\tilde{x}}$
around $\bm{x}$, where $H(\bm{x})$ is the symmetric positive definite Hessian
of $d(\bm{x},\cdot)$ at $\bm{x}$. Assume the Hessian is constant in the
vicinity of $\bm{x}$, namely $H(\bm{x})=H$, and that it can be factored into
its square roots as $H=R^{T}R$. Then (with the inessential assumption that
$d(\bm{x},\bm{x})=0$), the distortion in the vicinity of $\bm{x}$ can be
expressed as the squared error

$d(\bm{x},\bm{\tilde{x}})\approx||R(\bm{\tilde{x}}-\bm{x})||^{2}.$ (29)

Thus, at high rates, the conclusions of this section will hold in the vicinity
of $\bm{x}$, after a local coordinate transformation by $R$, the square root
of the Hessian of $d(\bm{x},\cdot)$ at $\bm{x}$. (When
$d(\bm{x},\bm{\tilde{x}})$ as a function of $\bm{\tilde{x}}$ is not minimized
by $\bm{\tilde{x}}=\bm{x}$, as might be the case for hybrid adversarial
losses, an additional local coordinate transformation may be necessary.) For
perceptual distortion measures whose Hessians are constant over the entire
space, such a trick can be used for traditional linear transforms. However, in
the usual case where the Hessian varies over the space (e.g., to reflect
perceptual masking effects), nonlinear transforms are uniquely able to build
the perceptual measure directly into the optimal transform.