# Reducing The Amortization Gap of Entropy Bottleneck In End-to-End Image
Compression

Muhammet Balcilar  InterDigital, Inc.
Rennes, France
<EMAIL_ADDRESS>Bharath Damodaran  InterDigital, Inc.
Rennes, France
<EMAIL_ADDRESS>Pierre Hellier  InterDigital, Inc.
Rennes, France
<EMAIL_ADDRESS>
###### Abstract

End-to-end deep trainable models are about to exceed the performance of the
traditional handcrafted compression techniques on videos and images. The core
idea is to learn a non-linear transformation, modeled as a deep neural
network, mapping input image into latent space, jointly with an entropy model
of the latent distribution. The decoder is also learned as a deep trainable
network, and the reconstructed image measures the distortion. These methods
enforce the latent to follow some prior distributions. Since these priors are
learned by optimization over the entire training set, the performance is
optimal in average. However, it cannot fit exactly on every single new
instance, hence damaging the compression performance by enlarging the bit-
stream. In this paper, we propose a simple yet efficient instance-based
parameterization method to reduce this amortization gap at a minor cost. The
proposed method is applicable to any end-to-end compressing methods, improving
the compression bitrate by $1\%$ without any impact on the reconstruction
quality.

###### Index Terms:

Neural Image Compression, Entropy Model, Amortization Gap.

##  I Introduction

Image and video compression is still a topic of utmost importance, despite
decades of hard work. The pandemic, as well as the rise of the metaverse, lead
to a huge volume of video transmitted over the network, and limiting the
amount of data matters to reduce the energy consumption. Classical methods,
based on handcrafted extracted features such as DCT, have prevailed over the
last three decades. Since roughly five years, methods based on machine
learning have emerged, and their performances now challenge those of
traditional techniques.

These end-to-end deep compression methods are a special case of Variational
Autoencoder (VAE) model as in [1], where the encoder, decoder and entropy
model are learned jointly. The output of the encoder is referred to as the
latent variable, and the entropy model prior is trained on the distribution of
observed latent variable over the training dataset [2, 3, 4, 5, 6, 7, 8]. It
was shown that minimizing the evidence lower bound (ELBO) of this special VAE
is equivalent to minimizing jointly the mean square error (MSE) of the
reconstruction image and the entropy of latents w.r.t the priors [4].

At test time, these codecs encode the quantized latent variable losslessly
w.r.t the trained priors into a bit-stream. It is typically performed by any
entropy coder such as range or arithmetic coding [9]. The prior distribution
being known by the decoder (or reconstructed by side information [4, 5, 6, 7,
8]), the latent can be reconstructed from the transmitted bitstream, and
consequently used to reconstruct the original data. At the moment, various
priors have been proposed so far: fully-factorized [3], zero-mean gaussian
[4], gaussian [5, 6] or mixture of gaussian [7], where some predict priors
using an autoregressive schema [5, 6, 7, 8].

All prior models are learned by amortizing its parameters over the entire
training set. Consequently, the learned prior is optimal in average, but sub-
optimal for a given specific data instance. This is due to the fact that the
observed latent distribution differs from the learned prior distribution. This
problem is referred to as the amortization gap [10]. Methods proposed to solve
this problem are two folds: firstly, enforce that the latent of the given
instance obeys the priors [11, 12, 13, 14, 15]; secondly, modify the priors to
better fit the given instance’s latents [16, 17, 18, 19]. The first class of
methods does not need any update on the receiver side, but has limited gain.
The second class of methods can update the encoder/decoder in addition to
entropy model as well resulting more gain, at the additional cost of
transmitting these updates to the receiver. However, all methods require post-
training to overfit on a given data instance, which increases the encoding
time significantly.

In this paper, we propose two main contributions: first, we define the
amortization gap of entropy model in compressing perspective and report the
amortization gap of some recent neural image compression model over benchmark
datasets. Second, we propose simple yet efficient methods for factorized and
hyperprior entropy models to adjust the priors to fit on any new instance to
be compressed. Our solution does not need post-training and does not add
computational complexity. We show that a gain of at least $1\%$ can be
expected from sota end-to-end compression method without any impact on the
reconstruction quality.

##  II Neural Image Compression

In this section, we introduce the mathematical notations for the end-to-end
variational autoencoders and the models of the entropy model. The encoder
$\mathbf{y}=g_{a}(\mathbf{x};\mathbf{\phi})$, where $\mathbf{\phi}$ are the
parameters of the corresponding neural network, transforms each input
$\mathbf{x}\in\mathbf{R}^{n\times n\times 3}$ into a lower dimensional latent
$\mathbf{y}\in\mathbf{R}^{m\times m\times o}$ and quantize it to obtain
$\mathbf{\hat{y}}=\mathbf{Q}(\mathbf{y})$.

Recent methods have proposed the two following possible solutions:

  * •

The fully factorized model [3], where the the quantized latents are compressed
losslessly by the entropy coder using factorized entropy model
$p_{f}(\mathbf{\hat{y}}|\Psi)$.

  * •

The hyperprior model, now used in most settings [4, 7], where a side
information is extracted so as to remove spatial structure from the latent
information so that the model generalizes better. The side information
$\mathbf{z}=h_{a}(\mathbf{y};\Phi)$ where $\mathbf{z}\in\mathbf{R}^{k\times
k\times f}$ (and its quantization $\mathbf{\hat{z}}=\mathbf{Q}(\mathbf{z})$)
are also learned. In that case, $\mathbf{\hat{y}}$ is encoded with the
hyperprior entropy model $p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z}};\Theta)$,
and $\mathbf{\hat{z}}$ is encoded with factorized entropy model
$p_{f}(\mathbf{\hat{z}}|\Psi)$.

The decoder $\mathbf{\hat{x}}=g_{s}(\mathbf{\hat{y}};\mathbf{\theta})$
reconstructs the image $\mathbf{\hat{x}}$ from the transmitted quantized
latent variables, or reconstructed latent in the case of the hyperprior model.
In the general case, the parameters
$\mathbf{\phi},\mathbf{\theta},\Psi,\Theta$ of $g_{a},g_{s},p_{f},p_{h}$ are
obtained by minimizing the following rate-distortion loss.

$\small\mathcal{L}=\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{x}\sim
p_{x}\\\ \epsilon\sim
U\end{subarray}}\left[-log(p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z}},\Theta))-log(p_{f}(\mathbf{\hat{z}}|\Psi))+\lambda
d(\mathbf{x},\mathbf{\hat{x}})\right],$ (1)

where $d(.,.)$ is any distortion loss such as MSE, $\lambda$ is the trade-off
parameter to control compression ratio and quality, $\mathbf{Q}(.)$ is
continuous relaxation at train time as $\mathbf{Q}(x)=x+\epsilon$,
$\epsilon\sim U(-0.5,0.5)$.

At test time, the quantized latent variables are compressed losslessly by the
entropy coder as follows:

  * •

For the fully factorized model [3], each $k\times k$ slice of side latent has
a trainable cumulative distribution function (cdf) in entropy model shown by
$\bar{p}_{\Psi}^{(c)}(.),~{}c=1\dots f$, and probability mass function (pmf)
for a given value of $x$ is derived as
$\hat{p}_{\Psi}^{(c)}(x)=\bar{p}_{\Psi}^{(c)}(x+0.5)-\bar{p}_{\Psi}^{(c)}(x-0.5)$.
Thus, the entropy model applies as follows;

$p_{f}(\mathbf{\hat{z}}|\Psi)=\prod_{c=1}^{f}\prod_{i,j=1}^{k,k}\hat{p}_{\Psi}^{(c)}({\mathbf{\hat{z}}_{i,j,c}})$
(2)

  * •

In the case of the hyperprior model, the entropy model of the latent
$\mathbf{\hat{y}}$ is conditioned with the side information
$\mathbf{z}=h_{a}(\mathbf{y};\Phi)$. Thus, $\mathbf{\hat{y}}$ is encoded with
the hyperprior entropy model
$p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z}};\Theta)$, and $\mathbf{\hat{z}}$ is
encoded with factorized entropy model $p_{f}(\mathbf{\hat{z}}|\Psi)$.

Let us describe in more details the special case of the hyperprior model. Each
latent point is modeled as $1d$ Gaussian distribution and its pmf is
$\hat{N}(x;\mu,\sigma)=\bar{N}(x+0.5;\mu,\sigma)-\bar{N}(x-0.5;\mu,\sigma)$
while $\bar{N}(.;\mu,\sigma)$ is the cdf of $1d$ Gaussian distribution. The
hyperprior entropy model is written
$p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z}},\Theta)=\prod_{i}\hat{N}(\mathbf{\hat{y}}_{i};\mu_{i},\sigma_{i})$at
train time where $\mu,\sigma=h_{s}(\mathbf{\hat{z}};\Theta)$ as in [4] or
$\mu_{i},\sigma_{i}=h_{s}(\mathbf{\hat{z},\mathbf{\hat{y}}_{<i}};\Theta)$ in
autoregressive prediction in [5, 7, 8]. $h_{s}$ is a trainable model
implemented as a neural network with parameter $\Theta$. However, this
implementation is not effective at test time due to the necessity of
recalculating the pmf table at receiver side for each latent points $i$. Thus,
it is common practice to use $s$ number of predefined integer resolution pmf
tables under zero means but different scale parameters (logarithmic
distributed scale values between $\sigma_{min}$ to $\sigma_{max}$) [4, 5, 6,
7, 8]. As long as $\mathbf{\tilde{y}}_{i}=Q(\mathbf{y}_{i}-\mu_{i})$,
$\mathbf{\hat{y}}_{i}=\mathbf{\tilde{y}}_{i}+\mu_{i}$, $\sigma_{c}$ is $c$-th
predefined scale and $\mathcal{N}(\sigma_{c})$ is a set of latent indices
whose winning scale is $\sigma_{c}$, the hyperprior entropy model is
implemented as follows at test time:

$p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z},\Theta})=\prod_{c=1}^{s}\prod_{i\in\mathcal{N}(\sigma_{c})}\hat{N}(\mathbf{\tilde{y}}_{i};0,\sigma_{c})$
(3)

##  III Proposed Method

In this section, we define the amortization gap of the entropy models and
propose solutions to reduce it.

###  III-A Amortization Gap of the Entropy Model

At training time, parameters $\mathbf{\phi,\theta},\Phi,\Theta$ and $\Psi$ are
estimated by optimizing $\mathcal{L}$ over the data distribution
$\mathbf{x}\sim p_{x}$. The parameters may be optimal in average for entire
dataset, but not any specific instance $\mathbf{x}$, which is known as an
amortization gap [10]. Actually, the amortization gap in compression schema
may occur for each trainable blocks in the model. However, we are here only
interested in the gap for the entropy models.

The amortization gap of the entropy model is the difference between the
optimal entropy model and the learned entropy model (see Fig. 1). It
quantifies the expected gain in bit length, if the entropy models’ pmfs are
optimal on every input instance. This gap can be calculated for the factorized
entropy model as follows:

$\mathcal{G}_{f}=-log(p_{f}(\mathbf{\hat{z}}|\Psi))+\log(p^{*}_{f}(\mathbf{\hat{z}}))$
(4)

$p^{*}_{f}(\mathbf{\hat{z}})$ refers to the optimal entropy model within the
same entropy family $\mathcal{P}_{f}$. For the hyperprior entropy model, it
reads:

$\mathcal{G}_{h}=-log(p_{h}(\mathbf{\hat{y}}|\mathbf{\hat{z}},\Theta))+log(p^{*}_{h}(\mathbf{\hat{y}}))$
(5)

$p^{*}_{h}(\mathbf{\hat{y}})$ refers to the optimal entropy model within the
same entropy family $\mathcal{P}_{h}$. Eqn. 5 and 4 shows that this gap can be
bridged by using the optimal entropy model on each instance. To find such an
instance-specific optimal entropy model, one does not need to optimize the
log-likelihood, since the normalized histogram is the optimal pmf [20].

Thus, replacing learned pmf tables with histogram of actual latents for each
instance removes the amortization gap of the entropy model. However, it
introduces the additional cost of transmitting these histograms for each
instance, enlarging the bit stream which is not practically feasible.

###  III-B Explicit Parameterization

Here, we propose an efficient solution to bridge the amortization gap of the
entropy model at a negligible additional transmission cost by an explicit
parameterization. More specifically, when an input image has to be encoded, we
parameterize the distribution of the latents $\tilde{p}_{f|h}(.,\beta)$ for
factorized and hyperprior entropy models to closely approximate the optimal
pmf. Our approximation is illustrated in Fig. 1. This low-level approximation
can be hopefully transmitted at a negligible extra cost (signaling cost of
$\beta$), and aims at improving the encoding using $\tilde{p}_{f|h}(.,\beta)$.

We propose two variants for approximating $\tilde{p}_{f|h}(.,\beta)$. The
first one is generic, uses Gaussian mixture model and is described in section
III-B1. It is dedicated to the factorized entropy model, since this modeling
is flexible. The second one is a simplified version, where the central bin is
spread on neighbouring bins, and is described in section III-B2. This approach
is dedicated to the hyperprior entropy model, since the discrepancy between
learned and actual pmf is smaller.

Figure 1: Learned pmfs ($\hat{p}_{\Psi},\hat{N}$), reparameterized pmfs
($\tilde{p}_{f},\tilde{p}_{h}$) and normalized frequencies ($h_{f},h_{h}$) for
a certain image’s selected latent under factorized (left) and hyperprior
(right) entropy models. Our reparameterization fits better on the normalized
frequencies, leading to improved compression.

####  III-B1 Truncated Gaussian Mixture on Discrete Support

Since the factorized entropy model is a non-parametric distribution model [3,
4], the pmf is flexible enough to have any shape. One of the most successful
parametric distribution model is Gaussian Mixture Model (GMM) which can
approximate any smooth density function [21] with a cost of three parameters
per component. Thus, we propose to use GMM as a tool to re-model the latent
distribution of the factorized entropy model.

In our case, the function to be approximated is defined on integer center and
support domain of GMM is truncated such that $[x_{min}\dots x_{max}]$. Thus,
we can write our mixture model’s pmf for factorized entropy model
$\tilde{p}_{f}(x;\beta^{(c)}))$ as:

$\tilde{p}_{f}(x;\beta^{(c)})=\frac{\sum_{k=1}^{K}\pi_{k}N(x;\mu_{k},\sigma_{k})}{\sum_{z=x_{min}}^{x_{max}}\sum_{k=1}^{K}\pi_{k}N(z;\mu_{k},\sigma_{k})}$
(6)

Here $\beta^{(c)}=\\{\pi_{k},\mu_{k},\sigma_{k}\\}_{k=1\dots K}$ denotes the
set of parameters to be inferred in (6) for $c$-th latent band. Since there is
no closed form solution of $\beta^{(c)*}$ that maximize (6), optimization is
used for estimation. In practice, given the small number of parameters,
convergence is fast. Fig 1 (left) displays the result of pmf approximation
using a mixture of two Gaussians ($\scriptstyle K=2$). In practice, the tuning
of parameter $K$ leads to a trade-off between approximation accuracy and
transmission cost.

For the hyperprior entropy model, since the latent distribution has zero-mean
and is uni-modal, we relax eqn. 6 by setting $K=1$ and $\mu=0$, also called as
zero-mean truncated Gaussian approximation. This is illustrated in Fig.
1(right) where $\tilde{p}_{h}(x;\beta^{(c)})$ obtained by equation (6) under
$K=1$ and $\mu=0$.

####  III-B2 Difference of the Center Bins Probability

We propose in this section a simpler alternative for the hyperprior entropy,
since the latter is parametric and by construction has a zero-mean Gaussian
shape. Hence, in that special case, we propose an alternative at minimal
transmission cost. We use the heuristic of computing the error of center bins
probability and spread this error to the other bins proportionally seems
arguable strong closed form alternative. In this approach, reparameterized
$c$-th pmf of hyperprior entropy model can be written in (7) where the
parameter $\beta$ is the error of the center bins probability between learned
one and the actual one in the normalized histogram such as
$\beta^{(c)}=\hat{N}(0;0,\sigma_{c})-h_{h}^{(c)}(0)$.

$\small\tilde{p}_{h}(x;\beta^{(c)})=\begin{cases}\hat{N}(x;0,\sigma_{c})-\beta^{(c)}&\mbox{if
}x=0\\\
\hat{N}(x;0,\sigma_{c})(1+\frac{\beta^{(c)}}{1-\hat{N}(0;0,\sigma_{c})})&\mbox{if
}x\neq 0\end{cases}$ (7)

####  III-B3 Quantization of the Parameters

According to selected re-parameterization method, 4 different kind of
parameters ($\mu,\sigma,\pi$ of Gaussian and differences of center bin
probability) should be explicitly encoded into the bitstream as an extra
parameters. In order to do that, we apply quantization on predefined number of
bins (8-bit by default). Since the range of these parameters are different, we
prepared shared quantization tables for each parameters. We discretize
$\sigma$ in range of $[0.002,20]$ with logarithmic bin width, $\pi$ in range
of $[0,1]$, $\mu$ in range of $[x_{min},x_{max}]$ (minimum symbol to maximum
symbol in the cdf table) and differences of center bin probability in range of
$[-0.03,0.03]$ with uniform bin width using 256 quantization centers for 8-bit
quantization.

##  IV Experimental Results

We used CompressAI library [22] to test our contributions on already
implemented $6$ SOTA deep compression methods, as well as on a very recent
method [8]. We used two datasets to evaluate our method: Kodak test set[23]
and Clic-2021 Challenge’s Professional test set [24], consisting of 24 and 60
images respectively.

Analysis of amortization gap: We measured the amortization gaps of pre-trained
SOTA methods and our gains on Kodak test set [23] on the lowest bpp. Results
are given in Table I. Regardless of the baseline method, the factorized
entropy model’s amortization gap is quite large (7.6-11.8%), compared to the
hyperpriors one (1.9-4.5%). This observation is also predictable by Fig 1
where it clearly shows that mismatch between $h_{f}$ and $\hat{p}_{\Psi}$ is
much more bigger than $h_{h}$ and $\hat{N}$. This can be explained easily: the
hyperprior entropy model uses instance specific information, and fully
factorized model does not. The hyper-prior methods encode very small amount of
the data (0.6-5.9%) with less effective entropy model (factorized entropy),
but vast majority of them (94.1-99.4%) is encoded by effective entropy model
(hyperprior entropy). Thus, in average their amortization gap (1.9-4.7%) is
smaller compared to the fully-factorized method (9.5%). From the different
version of same method, it can be seen that when the amount of side
information decreases, correspondingly the hyperprior gap increases. For
instance, the hyperprior gap is 3.4% where there is 5.9% side information but
increases to 4.5% when there is 3.5% side information.

Reduction of Amortization Gap: Our proposed methods managed to save
significant amount of bits on all studied methods in Table I. The magnitude of
saving is higher with the factorized entropy model compared to hyper-prior
model. We can reduce the factorized entropy gap from 9.5% to 2.71%, with the
gain of 6.79% when all information is encoded by this entropy model in
bmshj2018-factorized model. However, when the amount of information to be
encoded is less with this entropy model, our gain also reduces. For instance,
when only 2.3% of the information is encoded by this entropy model in
cheng2020-attn, our gain is 2.77%. The fixed extra parameter cost in our
proposal makes the efficiency of our method proportionally less when the
amount of information to be encoded decreases. The hyperprior entropy encodes
large percentage of the information, and our gain ranges from 1-1.8%, which
almost fills in average 45% gap exists in hyperprior entropy over different
methods.

TABLE I: Ratio of the encoded information, the amortization gap of the entropy models and our gain for each entropy model relative to the original bit-length for the methods trained lowest bpp objective. The gap and the gain is much more higher in factorized entropy. Our solution reduces the gap significantly both factorized and hyperprior entropy model. In total, proposed method saves more than 1% of file size from sota model. Results are averaged over Kodak Test set.  Model | Factorized Entropy | Hyperprior Entropy | Total
---|---|---|---
| Ratio | Gap | Gain | Ratio | Gap | Gain | Gap | Gain
| (in %) | (in %) | (in %) | (in %) | (in %) | (in %) | (in %) | (in %)
bmshj2018-factorized [3] | 100 | 9.5 | 6.79 | - | - | - | 9.5 | 6.79
bmshj2018-hyperprior [4] | 3.5 | 10.0 | 4.65 | 96.5 | 4.5 | 1.84 | 4.7 | 1.98
mbt2018-mean [5] | 5.9 | 11.4 | 4.27 | 94.1 | 3.4 | 1.24 | 3.8 | 1.43
mbt2018 [5] | 2.3 | 9.9 | 4.09 | 97.7 | 2.5 | 0.99 | 2.7 | 1.06
cheng2020-anchor [7] | 1.2 | 11.8 | 3.68 | 98.8 | 2.3 | 1.06 | 2.5 | 1.09
cheng2020-attn [7] | 2.3 | 9.5 | 2.77 | 97.7 | 1.9 | 0.88 | 2.1 | 0.92
InvCompress [8] | 0.6 | 7.6 | 2.18 | 99.4 | 3.0 | 1.32 | 3.0 | 1.33
Figure 2: Experimental results on Kodak and Clic Test Set for
bhshj2018-factorized model in [3] trained on 7 different psnr objectives.
Proposed method based on truncated GMM saves 6.8% original bit length for
Kodak, 11.5% for Clic test test on lower psnr and outperformed post-train
based high computational demanding alternatives.

In order to measure the performance of proposed method on different PSNR
targets, we plugged our method on the pre-trained model in [3] for the
factorized entropy and the best neural compressing model cheng2020-anchor [7]
provided in [22] for the hyperprior entropy.The performances are measured with
Kodak and Clic-2021 Challenge’s Professional test set. According to results in
Fig 2, the amortization gap of factorized entropy varies from 8.5% to 9.5% in
Kodak dataset where the proposed method gains from 5.3% to 6.8% in file size.
In Click-2021 dataset, the gap (9.5%-12.5%) and our gain (8%-11.5%) are even
bigger. Fig. 3 reports the results of the hyperprior entropy, our method saves
more than 1% of original file size in lower bit-rate and save around 0.5% in
highest bit-rate. The simplest approach that parameterizes the new probability
by the difference between center bin’s probability in Eq.7 gives competitive
result even better in higher psnr with zero-mean Gaussian parameterization.

Comparison with competitors: To compare our method with the existing method,
we also implemented two instance based post-training methods: post-train
encoder (trains the encoder for given test image) [12], post-train latent
(learns more effective instance’s latent directly without training
encoder)[13]. According to our test, we have found that [13] is faster (but
still needs significant time to train) than [12] but with less performance as
can be seen in Fig 2. Our proposal reaches better results and outperforms
significantly compared to these two approaches even without giving any
significant computational complexity.

Figure 3: Experimental results on Kodak and Clic Test Set for cheng2020-anchor
model in [7] trained on 6 different psnr objectives. The gap and our gain
decrease on higher psnr target. Proposed two methods save more than 1% of file
size for both test sets at lower psnr. Our simple center bin difference
solution gives comparable results even better at higher psnr.

Computational Complexity: To test the computational complexity, we use python
library of PAPI [25] and count floating point operations per pixel in
flops/pxl both encoding and decoding time. As the results are shown in Table
II, our proposal gives negligible extra complexity both encoding and decoding
time. Even in the fully-factorized model which uses relatively costly GMM
model fitting with $K=2$, our extra complexity demanding is less than %0.3 in
encoding time. Since the hyperprior model uses zero-mean gaussian fitting and
has good initial guess with learned standard deviation, extra encoding
operation even goes below %0.05. Since in the decoding time, extra process is
to re-arrange pmf table with encoded extra parameters, there is almost no
differences in decoding complexity. Note that, our alternatives which applies
finetuning in encoding time as in [13, 12] needs enormous extra computational
complexity in encoding. Their extra computational demand is depend on the
maximum iteration of finetuning, where each iteration is not less than a
single decoding (one forward pass) complexity in any model (most of the time
3-4 times longer than one forward-pass because of necessity of back-
propagation of gradients). In our test, we did not get meaningful result from
[13, 12] without applying 1000 iteration which has 3000-4000 times bigger
computational demand.

Implementation Details: In order to reproducibility, we implement the
proposals as a collection of classes of python end-to-end image compression
library CompressAI [22] and integrate to it as a part of this well-known
library. Using our classes, the one can easily apply proposal gap reduction to
any model and decrease the file size without any impact on the reconstruction
quality. In our models, the re-parameterization method which can be either
zero-mean gaussian, GMM with K=1,2,3 or difference of center bin approaches is
hyperparameter for both factorized and hyperprior entropy models. In practice,
a table-wise selection mechanism is used to determine if the learned pmf
should be replaced by the approximated one. This results necessity of
signaling receiver by 1-bit if the cdf table is replaced or not. Instead of
testing all pmf tables in both entropy models, we define another
hyperparameter which shows how many cdf tables are targeded. The number of
bits to encode single extra parameter explicitly can also be seen as
hyperparameter which is common for both entropy models. We use GMM for $K=2$
and targetted top 64 pmf tables in bmshj2018-factorized model. For
cheng2020-anchor, we use GMM for K=1 and targeted 32 pmf for factorized
entropy (side information), while zero mean gaussian and targetted 32 pmf for
hyperprior entropy (main information). We always use 8-bit quantization on the
extra parameters.

TABLE II: Necessary number of floating point operations per pixel (flops/pix) in encoding and decoding for studied two neural models. Our results are obtained over Kodak test set with our most efficient settings in terms of compression performance.  Model | Encoding | Decoding
---|---|---
| Baseline | Ours | Baseline | Ours
bmshj2018-factorized [3] | 83,797 | 84,024 | 85,400 | 85,412
cheng2020-anchor [7] | 348,142 | 348,298 | 519,626 | 519,639

##  V Conclusion

We have proposed here to improve the efficiency of entropy models in deep
neural image compression algorithm. First, we have defined the amortization
gap for entropy models, and we measured experimentally the gap for sota
methods. Then, we have proposed an effective and computationally-friendly
method to fill the gap, which differs from previously published methods. We
have shown experimentally that a gain above $1\%$ was obtained, on different
dataset and on $7$ various SOTA compression methods. Our method can actually
be applied to any end-to-end deep compression technique, without any impact on
the reconstruction quality. One known limitation of deep methods is
reconstruction failure, due to architecture discrepancy (the architecture used
at encoding and decoding differ). This is explained by the need of recomputing
the pmf table at decoding, and small errors occur because of floating-value
approximations [26]. In this regards, our method based on center-bin
difference will be robust since it needs only integer divide operation. Future
work may explore the link between explicit parameterization of the entropy
model, and fine-tuning of the encoder. We may expect to increase the
reconstruction quality, at constant bit-rate.

## References

  * [1] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in _ICLR_ , 2013.
  * [2] L. Theis, W. Shi, A. Cunningham, and F. Huszár, “Lossy image compression with compressive autoencoders,” in _ICLR_ , 2017.
  * [3] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in _ICLR_ , 2017.
  * [4] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in _ICLR_ , 2018.
  * [5] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31, 2018.
  * [6] D. Minnen and S. Singh, “Channel-wise autoregressive entropy models for learned image compression,” in _ICIP_ , 2020, pp. 3339–3343.
  * [7] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in _CVPR_ , 2020.
  * [8] Y. Xie, K. L. Cheng, and Q. Chen, “Enhanced invertible encoding for learned image compression,” in _Proceedings of the ACM International Conference on Multimedia_ , 2021.
  * [9] J. Rissanen and G. Langdon, “Universal modeling and coding,” _IEEE Transactions on Information Theory_ , vol. 27, no. 1, pp. 12–23, 1981.
  * [10] C. Cremer, X. Li, and D. Duvenaud, “Inference suboptimality in variational autoencoders,” _ICML_ , 2018.
  * [11] C. Aytekin, X. Ni, F. Cricri, J. Lainema, E. Aksu, and M. Hannuksela, “Block-optimized variable bit rate neural image compression,” in _CVPR workshop_ , June 2018.
  * [12] G. Lu, C. Cai, X. Zhang, L. Chen, W. Ouyang, D. Xu, and Z. Gao, “Content adaptive and error propagation aware deep video compression,” in _ECCV_ , vol. 12347, 2020, pp. 456–472.
  * [13] J. Campos, S. Meierhans, A. Djelouah, and C. Schroers, “Content adaptive optimization for neural image compression,” in _CVPR Workshops_ , June 2019\.
  * [14] Y. Yang, R. Bamler, and S. Mandt, “Improving inference for neural image compression,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 573–584.
  * [15] T. Guo, J. Wang, Z. Cui, Y. Feng, Y. Ge, and B. Bai, “Variable rate image compression with content adaptive optimization,” in _CVPR Workshops_ , June 2020.
  * [16] J. P. Klopp, L.-G. Chen, and S.-Y. Chien, “Utilising low complexity cnns to lift non-local redundancies in video coding,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 6372–6385, 2020.
  * [17] Y.-H. Lam, A. Zare, F. Cricri, J. Lainema, and M. M. Hannuksela, “Efficient adaptation of neural network filter for video compression,” in _ACM International Conference on Multimedia_ , ser. MM ’20, 2020, p. 358–366. [Online]. Available: https://doi.org/10.1145/3394171.3413536
  * [18] N. Zou, H. Zhang, F. Cricri, H. R. Tavakoli, J. Lainema, M. Hannuksela, E. Aksu, and E. Rahtu, “L2c – learning to learn to compress,” in _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_ , 2020, pp. 1–6.
  * [19] T. van Rozendaal, I. A. Huijben, and T. S. Cohen, “Overfitting for fun and profit: Instance-adaptive data compression,” in _ICLR_ , 2021.
  * [20] M. Mezard and A. Montanari, _Information, physics, and computation_. Oxford University Press, 2009.
  * [21] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_ , 2016.
  * [22] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” _arXiv preprint arXiv:2011.03029_ , 2020.
  * [23] E. Kodak, “Kodak Lossless True Color Image Suite (PhotoCD PCD0992).” [Online]. Available: http://r0k.us/graphics/kodak
  * [24] “CLIC: Challenge on learned image compression,” http://compression.cc
  * [25] D. Terpstra, H. Jagode, H. You, and J. Dongarra, “Collecting performance data with papi-c,” in _Tools for High Performance Computing 2009_. Springer, 2010, pp. 157–173.
  * [26] J. Ballé, N. Johnston, and D. Minnen, “Integer networks for data compression with latent-variable models,” in _ICLR_ , 2019.