# Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression

Yue Lv<EMAIL_ADDRESS>Tsinghua Shenzhen International Graduate
School, Tsinghua UniversityShenzhenChina ,  Jinxi Xiang<EMAIL_ADDRESS>Tencent AI LabShenzhenChina ,  Jun Zhang<EMAIL_ADDRESS>Tencent AI
LabShenzhenChina ,  Wenming Yang  yang.wenming@sz. tsinghua.edu.cn Tsinghua
Shenzhen International Graduate School, Tsinghua UniversityShenzhenChina ,
Xiao Han<EMAIL_ADDRESS>Tencent AI LabShenzhenChina and  Wei Yang
<EMAIL_ADDRESS>Tencent AI LabShenzhenChina

(2023)

###### Abstract.

The latest advancements in neural image compression show great potential in
surpassing the rate-distortion performance of conventional standard codecs.
Nevertheless, there exists an indelible domain gap between the datasets
utilized for training (i.e., natural images) and those utilized for inference
(e.g., artistic images). Our proposal involves a low-rank adaptation approach
aimed at addressing the rate-distortion drop observed in out-of-domain
datasets. Specifically, we perform low-rank matrix decomposition to update
certain adaptation parameters of the client’s decoder. These updated
parameters, along with image latents, are encoded into a bitstream and
transmitted to the decoder in practical scenarios. Due to the low-rank
constraint imposed on the adaptation parameters, the resulting bit rate
overhead is small. Furthermore, the bit rate allocation of low-rank adaptation
is _non-trivial_ , considering the diverse inputs require varying adaptation
bitstreams. We thus introduce a dynamic gating network on top of the low-rank
adaptation method, in order to decide which decoder layer should employ
adaptation. The dynamic adaptation network is optimized end-to-end using rate-
distortion loss. Our proposed method exhibits universality across diverse
image datasets. Extensive results demonstrate that this paradigm significantly
mitigates the domain gap, surpassing non-adaptive methods with an average BD-
rate improvement of approximately $19\%$ across out-of-domain images.
Furthermore, it outperforms the most advanced instance adaptive methods by
roughly $5\%$ BD-rate. Ablation studies confirm our method’s ability to
universally enhance various image compression architectures. Our project is
available at https://github.com/llvy21/DUIC.

Image Compression; Parameter-Efficient; Instance Adaptation; Dynamic Network

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29–November 3, 2023;
Ottawa, ON, Canada.††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29–November 3, 2023, Ottawa, ON,
Canada††price: 15.00††isbn: 979-8-4007-0108-5/23/10††doi:
10.1145/3581783.3612187††submissionid: 2176††ccs: Computing methodologies
Image compression

##  1\. Introduction

Image compression is a fundamental technology for media storage and
transmission in the digital information era. Common compression standards are
JPEG (Wallace, 1992), JPEG2000 (Skodras et al., 2001), BPG (Bellard, 2015),
and the latest VVC (Bross et al., 2021). With the development of deep
learning, neural image codecs have led to several breakthroughs in this area.
Recent studies are on par with, or even outperform the most advanced VVC codec
in terms of rate-distortion performance (Cheng et al., 2020; Qian et al.,
2021; He et al., 2022, 2021; Wang et al., 2022).

Figure 1. (upper left) Three state-of-the-art neural image codecs from
Cheng2020 (Cheng et al., 2020), Entroformer (Qian et al., 2021), and EVC (Wang
et al., 2022) perform fairly well on the Kodak dataset. (upper right) But
their performance drops significantly on an out-of-domain dataset containing
artistic images. We argue that neural codecs trained on natural images are
imperfect from the perspective of generalization. We aim to solve this
bottleneck with a dynamic adaptation design without updating all the
parameters of the codec for each specialized domain.  Our proposed instance
adaptation method based on Cheng2020 shows its robustness on out-of-domain
images.

It is a common practice to train neural image codecs on natural images, such
as CLIC (Toderici et al., 2020) and OpenImage (Kuznetsova et al., 2020), and
then evaluated on natural images benchmarks, such as Kodak (Company, 1993),
Tecnick (Asuni and Giachetti, 2014), and DIV2K (Agustsson and Timofte, 2017).
Apart from natural images, there exists a diverse, relatively unexplored space
of artistic drawing, comics, and gaming which offer depictions of the world
through artwork and are transmitted through the internet every day. Extending
neural compression models trained with neural images to these domains is non-
trivial. To highlight this issue, we showcase three representative models from
Cheng2020 (Cheng et al., 2020), Entroformer (Qian et al., 2021), and EVC (Wang
et al., 2022) in Fig. 1. The results show that neural image codecs are
comparable with VVC on the Kodak dataset, and consistently outperform the BPG.
However, the RD performance degrades dramatically on an artistic dataset from
BAM (Wilber et al., 2017), legging behind the BPG in general. Although neural
codecs have proven to be very successful, a model trained to minimize expected
RD cost over natural images is unlikely to be optimal for both optimization
and generalization. The problem of generalization would be critical when the
testing distribution differs from the training distribution.

To enhance the compression quality of out-of-domain images during inference,
instance-adaptive compression methods have been suggested, as shown in Fig. 2.
One line of work focuses on content adaptation, which adjusts the neural
codecs at inference time by refining the latent representation extracted from
the encoder. The values of the latent of the image are directly changed during
the encoder transformation stage, making the modified latent representation
fit the decoder (Campos et al., 2019; Guo et al., 2020b; Yang et al., 2020).
This method is attractive because it does not require any extra information to
be encoded in the bitstream, and no alterations will be made to the receiver’s
end. However, the benefits in terms of performance are restricted since
neither the encoder nor the decoder can be adjusted.

Another approach, which is complementary to latent adaptation, is decoder
adaptation by fine-tuning specific parameters of the decoder. The scenario is
that the decoder is frozen when deployed at the client’s end, but we can
adjust certain parameters by transmitting specific bitstreams. Fine-tuning
more network parameters brings better performance for sure, but it also
requires more rate cost. Rozendaal et al.(van Rozendaal et al., 2021b, a)
fine-tuned all the parameters and compress the model updates in spike-and-slab
prior. Although their approach achieves some degree of dynamic fine-tuning of
the number of parameters, the way of updating parameters lacks efficiency and
fails to apply to images for a huge extra bit cost. Existing methods (Lam et
al., 2020; van Rozendaal et al., 2021a, b; Zou et al., 2021) overlook the
importance of designing a parameter-efficient approach to maximize compression
performance while minimizing extra bitrate. Essentially, fine-tuning
parameters should also be treated as a rate-distortion problem. Tsubota et
al.(Tsubota et al., 2023) was the first to propose updating the model in a
parameter-efficient approach and introduced 768 extra parameters. Although
extra parameters were optimized with rate-distortion loss, the number of
updated parameters cannot be tailored to the content. This implies that the
rate is fixed without flexibility.

Figure 2. Different approaches for compression adaptation.

We propose a universal adaptation method for neural image compression to
efficiently update the decoder parameters during inference. Our approach
involves utilizing a plug-and-play adaptation block, which is composed of two
low-rank matrices, in order to minimize the additional bit costs associated
with updating the model. To optimize the adaptive parameters of an instance up
to its theoretical upper limit, we have developed a flexible and dynamic
decoder gate network that can adapt based on the number and position of blocks
to be plugged in. This approach enables the model to automatically determine
the bit rate budget of each layer, resulting in significant rate-distortion
gains.

Our contributions can be summarized as follows:

  * •

We propose to update the decoder weights using additional low-rank
decomposition matrices at inference time, which are compressed and transmitted
along with the image.

  * •

Our proposed dynamic gating network has the ability to learn the optimal
number and positions of adaptation automatically, facilitating efficient
domain transfer.

  * •

This technique effectively minimizes the domain gap between training and
inference, while also providing a versatile approach toward instance-adaptive
compression. Empirical evidence indicates that our approach can be extended to
various other neural image compression models.

Figure 3. Simplified pipeline of the dynamic universal instance adaptive image
compression. Steps 1-3 are executed on the server and step 4 is executed on
the client. 1) We first refine the latent representation $\boldsymbol{y}$ with
all the parameters fixed. 2) Then we adapt the model by dynamically choosing
the layers to update. Incremental model update parameters $\Delta\mathbf{W}$
and the gate networks $\boldsymbol{G_{\psi}}$ are optimized in this step. 3)
The latent code $\boldsymbol{y}$, instance incremental matrix
$\Delta\mathbf{W}$, and the gate output $\boldsymbol{g}$ are quantized and
encoded as content stream and model stream, respectively. 4) Finally, the
client decodes the bit stream and updates the decoder according to the gate
binary decisions to reconstruct the latent code.

##  2\. Related Work

###  2.1. Neural image compression

Neural compression is based on an autoencoder that tries to reconstruct the
input instance from quantized latents, cooperated with a prior that is used to
losslessly compress these latents. The transform coding paradigm can be
formulated by:

(1) $\displaystyle\boldsymbol{y}$
$\displaystyle=g_{a}(\boldsymbol{x};\boldsymbol{\phi})$ (2)
$\displaystyle\boldsymbol{\hat{y}}$ $\displaystyle=Q(\boldsymbol{y})$ (3)
$\displaystyle\boldsymbol{\hat{x}}$
$\displaystyle=g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta})$

where $\boldsymbol{x}$ is input original image; $\boldsymbol{y}$ is latent
code generated by analysis transform $g_{a}$ with the optimized parameter
$\phi$; $\boldsymbol{\hat{y}}$ is quantized latent code;
$\boldsymbol{\hat{x}}$ is compress reconstructed images by synthesis transform
$g_{s}$ with the parameter $\theta$. A lossy neural image compression network
is optimized with rate-distortion loss:

(4) $\mathcal{L}=\mathcal{R}+\lambda\cdot\mathcal{D}=-\log
p(\boldsymbol{\hat{y}})+\lambda\cdot\mathcal{D}(\boldsymbol{\hat{x}},\boldsymbol{x}),$

where $\mathcal{R}$ is the bit rate; $\mathcal{D}$ measures the image
reconstruction distortion; $\lambda$ is the Lagrange multiplier controlling
R-D trade-off.

To improve the compression performance, recent studies focus on exploring more
powerful non-linear neural network modules for image transformation, such as
residual networks (Cheng et al., 2020; He et al., 2022), self-attention (Chen
et al., 2021), ConvNext (Duan et al., 2023), and transformer (Lu et al., 2021;
Zhu et al., 2022b; Zou et al., 2022; Xiang et al., 2023).

Another research direction is to develop accurate entropy models to reduce
redundancy, such as hierarchical (Ballé et al., 2018), unidirectional auto-
regressive model (Minnen et al., 2018; Minnen and Singh, 2020), checkerboard
auto-regressive (He et al., 2021, 2022), and bi-directional auto-regressive
(Xiang et al., 2023). Hierarchical prior employs no context information and
the performance is limited. Unidirectional auto-regressive is a context model
that predicts the next code with all the decoded neighbor pixels serially and
achieves superior performance. To accelerate the auto-regressive, channel-wise
(Minnen and Singh, 2020), spatial-wise (He et al., 2021), spatial and channel
(He et al., 2022), and random decode sequence (Xiang et al., 2023) were
proposed to prompt the parallel coding. Some other works introduce codebooks
and vector-quantize the latent code to compress(Zhu et al., 2022a) or use
hierarchical VAE to compress latent code(Duan et al., 2023).

Although the above-mentioned designs push the boundaries of compression
further, the learning-based components are not fully investigated on out-of-
domain datasets to verify their robustness.

###  2.2. Instance Adaptive Image Compression

To address the issue of the domain gap, several notable studies have been
conducted. One approach involves the fine-tuning of the encoder during
inference. This method aims to correct the latent code without introducing
additional bit transmission. For example, Campos et al. (Campos et al., 2019)
solved it through an iterative procedure similar to during training with
gradient descent applied on latents. Guo et al. (Guo et al., 2020b) further
added side information after the latent representation has been refined. Yang
et al. (Yang et al., 2020) extended bits-back coding by encoding some side
information using an entropy model.

On the other hand, certain studies have focused on fine-tuning the decoder
during inference. For example, Lam et al.(Lam et al., 2020) update the
deviation parameters of the convolutional layer in the decoder and compress
the deviation parameters with the image for transmission; Zou et al.(Zou et
al., 2021) use a set of overfittable multiplicative parameters (OMP) to
overfit a video. The set of OMP is transmitted together with the images; Liu
et al. (Liu et al., 2021) downsample the video and then overfit a video super-
resolution network and transmit the parameters of the super-resolution network
together with the video; Tsubota et al. (Tsubota et al., 2023) propose to
update the model in a parameter-efficient approach using an adapter. Besides,
a few studies fine-tune the whole model. Rozendaal et al. (van Rozendaal et
al., 2021b, a) overfit the whole model for the input data, using spike-and-
slab probability distribution to compress the entire decoder and entropy model
since the client isn’t affected by the encoder update.

Instance-adaptive image compression is connected with parameter-efficient
transfer learning. To optimize compression performance and minimize bitstream
overhead, we should adopt a parameter-efficient approach due to the need to
compress and transmit model update parameters as extra bits. Parameter
efficient transfer learning origins from the pre-trained large language models
(Devlin et al., 2018; Radford et al., 2019). Studies were proposed to adapt a
large-scale pre-trained model to downstream tasks by updating only a small
number of parameters. For example, adapter tuning (Houlsby et al., 2019)
inserts bottleneck structure between existing layers; prefix tuning (Li and
Liang, 2021) and prompt tuning (Lester et al., 2021) attach additional prefix
tokens to the input; diff pruning (Guo et al., 2020a) incremental update the
pre-trained weights as a sparse matrix. LoRA (Hu et al., 2021) utilizes the
low intrinsic dimension in the model and parameterizes incremental weight as a
low-rank matrix by the product of two small matrices achieves comparable or
even better performance than full fine-tuning.

##  3\. Method

###  3.1. Overview

The proposed method is illustrated in Fig. 3. With a pre-trained neural image
compression model, we freeze the base model and incorporate additional image-
specific parameters to the decoder.

To begin, the encoder extracts the latent code from the input image, which is
then refined with rate-distortion loss following a simplified approach of Yang
et al. (Yang et al., 2020). The latent code $\boldsymbol{y}$ is optimized by
gradient descent with frozen all the pre-trained weights. After latent
refinement, the weight of the decoder is updated in an adaptive, parameter-
efficient approach. Both the latent code and model update are encoded and
transmitted from the server to clients in the content stream and model stream.

During the decompression process, the client decodes both the content and
model streams and subsequently updates a small portion of the decoder with
instance-specific parameters. This enables the reconstruction of image latent
code using the updated decoder.

###  3.2. Low-Rank Adaptation

For a pre-trained weight matrix of a specific layer
$\mathbf{W}^{k}\in\mathbb{R}^{c_{out}\times c_{in}\times h\times w}$, we aim
to adapt it for universal image compression by modifying its weight in an
efficient way:

(5)
$\boldsymbol{h}^{k+1}=(\mathbf{W}^{k}+\Delta\mathbf{W}^{k})\boldsymbol{h}^{k}$

where $\boldsymbol{h}^{k},\boldsymbol{h}^{k+1}$ are the $k$-th, and $(k+1)$-th
layer of features. We leverage the low-rank property of $\Delta\mathbf{W}^{k}$
to update instance-specific parameters in an efficient manner, ensuring
maximum reconstruction quality with minimal bit overhead. We freeze
$\mathbf{W}^{k}$ and incrementally update it with two learnable matrices
$\mathbf{A}$ and $\mathbf{B}$ by low-rank decomposition:

(6) $\Delta\mathbf{W}^{k}=\mathbf{B}\mathbf{A}$

where $\mathbf{A}\in\mathbb{R}^{r\times c_{in}}$ is initialized with random
Gaussian, $\mathbf{B}\in\mathbb{R}^{c_{out}\times r}$ is initialized as zero.
This initialization ensures $\Delta\mathbf{W}^{k}=0$ at the beginning of
adaptation. The rank $r\ll\min\\{c_{in},c_{out}\\}$.

We note ${\Delta\boldsymbol{\theta}}=\\{\mathbf{A},\mathbf{B}\\}$. During the
model update, $\Delta\boldsymbol{\theta}$ is quantized with quantization
interval $w$. As there is no prior for $\Delta\boldsymbol{\theta}$, we use a
factorized density model $\boldsymbol{\pi}$ to encode it into bitstream:

(7)
$p_{\Delta\hat{\boldsymbol{\theta}}\mid\boldsymbol{\pi}}(\Delta\hat{\boldsymbol{\theta}}\mid\boldsymbol{\pi})=\prod_{i}\left(p_{\Delta\theta_{i}\mid\boldsymbol{\pi}}(\boldsymbol{\pi})*\mathcal{U}\left(-\frac{w}{2},\frac{w}{2}\right)\right)\left(\Delta\hat{\theta}_{i}\right)$

where $\Delta\theta_{i}$ denotes the $i$-th element of
$\Delta\boldsymbol{\theta}$ with $i$ as the position of each element;
$\mathcal{U}\left(-\frac{w}{2},\frac{w}{2}\right)$ is a uniform noise as a
stand-in for quantization. We specify $\boldsymbol{\pi}$ as a logistic
distribution. As such, the probability estimation for
$\Delta\hat{\boldsymbol{\theta}}$ is:

(8)
$p_{\boldsymbol{\pi}}\left(\Delta\hat{\theta}_{i}\right)=c\left(\Delta\hat{\theta}_{i}+\frac{w}{2}\right)-c\left(\Delta\hat{\theta}_{i}-\frac{w}{2}\right)$

with $c()$ as the cumulative distribution function of $\boldsymbol{\pi}$
calculated by logistic distribution:

(9) $c(d)=\frac{1}{2}+\frac{1}{2}\tanh{\left(\frac{d-\mu}{2s}\right)}$

where $w$, $\mu$, $s$ are predefined parameters. We set $w=0.01$, $\mu=0$,
$s=0.05$ by default.

Combining with the rate of latents $\mathcal{R}(\hat{\boldsymbol{{y}}})$ and
distortion of reconstructed images
$\mathcal{D}(\boldsymbol{\hat{x}},\boldsymbol{x})$, the low-rank adaptation is
optimized end-to-end with:

(10) $\displaystyle\begin{aligned}
\mathcal{L}&=\mathcal{R}(\hat{\boldsymbol{{y}}})+\mathcal{R}(\Delta{\hat{\boldsymbol{\theta}}})+\lambda\cdot\mathcal{D}(\boldsymbol{\hat{x}},\boldsymbol{x}\mid\boldsymbol{\theta},\Delta\hat{\boldsymbol{\theta}})\\\
&=-\log p(\hat{\boldsymbol{{y}}})-\log
p_{\boldsymbol{\pi}}(\Delta\hat{\boldsymbol{{\theta}}})+\lambda\cdot\mathcal{D}(g_{s}(\hat{\boldsymbol{{y}}};\boldsymbol{\theta},\Delta\hat{\boldsymbol{\theta}}),\boldsymbol{x})\end{aligned}$

with $\lambda$ as the multiplier for the trade-off between rate and
distortion.

The low-rank adaptation of Eq. (5) can be readily applied to various decoder
layers, such as convolutional layers and attention layers. In typical
compression models (Cheng et al., 2020; He et al., 2022), one decoder layer,
such as a $3\times 3$ convolutional layer, contains a large number of
parameters, specifically $C\times C\times 3\times 3$, where $C$ denotes the
number of channels. By utilizing the low-rank adaptation technique, the number
of learnable parameters can be significantly reduced to $C\times r\times 2$,
resulting in substantial savings in terms of extra bitrate.

###  3.3. Dynamic Gating Network

Figure 4. Dynamic gate network structure.

Eq. (10) delineates the rate-distortion loss for optimizing a single layer in
low-rank adaptation. It is a logical progression to expand the use of low-rank
adaptation across multiple layers:

(11)
$\mathcal{L}=\mathcal{R}(\hat{\boldsymbol{y}})+\sum_{k=1}^{K}\mathcal{R}(\Delta\hat{\boldsymbol{\theta}}_{k})+\lambda\cdot\mathcal{D}(\hat{\boldsymbol{x}},\boldsymbol{x}\mid\boldsymbol{\theta},\Delta\hat{\boldsymbol{\theta}}_{1},\ldots,\Delta\hat{\boldsymbol{\theta}}_{K})$

where $K$ is the total number of layers with low-rank adaptation in the
decoder. Quantifying the number of layers utilizing low-rank adaptation
presents a challenge due to (1) the varying adaptation levels required for
each instance and (2) the differing rate-distortion enhancements with diverse
bitstream allocation strategies.

To augment the flexibility and adaptability of our instance adaptive image
compression model, we devise a dynamic architecture that enables adaptive
control over the number and location of plug-in blocks inspired by (Wang et
al., 2018). In contrast to a static network that employs the same architecture
during inference, our proposed dynamic instance adaptive network can
dynamically adjust its own structure to suit various raw inputs during
processing.

We explicitly assign a gating network $G$ to each incremental matrix. The
gating module maps the previous layer outputs to the binary decision to skip
or execute the weight update. We note the number of the insertable positions
for an incremental update in synthesis transform as
$k\in\\{1,2,3,\cdots,K\\}$, for the $k$-th input $\boldsymbol{h}^{k}$, the
incremental update is controlled by the gating signal $g^{k}\in\\{0,1\\}$,
then we define the output of the gated incremental layer as

(12)
$\boldsymbol{h}^{k+1}=(\mathbf{W}^{k}+g^{k}\Delta\mathbf{W}^{k})\boldsymbol{h}^{k}$

During the forward pass, we apply hard-gating to the gate outputs, which are
relaxed to soft-gating during backpropagation:

(13) $g^{k}=\text{sg}\left[\mathbbm{1}(G^{k}(\boldsymbol{h}^{k})\geq
0.5)\right]+G^{k}(\boldsymbol{h}^{k})-\text{sg}\left[G^{k}(\boldsymbol{h}^{k})\right]$

where $\mathbbm{1}$ is the indicator function; $G^{k}$ is the gating network;
sg stands for the stop gradient operator that is defined as an identity at
forward computation time and has zero partial derivatives.

We implement a dynamic adaptive gate network (shown in Fig. 4) that comprises
a convolution layer, ReLU activation, adaptive average pooling, full
connection, and softmax layer. The gating network maps the input to a binary
output $\\{0,1\\}$ that dynamically controls the layers of adaptation.

We optimize Eq. (11) with the dynamic gating network as:

(14) $\displaystyle\mathcal{L}=$
$\displaystyle\mathcal{R}(\hat{\boldsymbol{y}})+\sum_{k=1}^{K}g^{k}\cdot\mathcal{R}(\Delta\hat{\boldsymbol{\theta}}_{k})$
$\displaystyle+\lambda\cdot\mathcal{D}(\hat{\boldsymbol{x}},\boldsymbol{x}\mid\boldsymbol{\theta},g^{1}\Delta\hat{\boldsymbol{\theta}}_{1},\ldots,g^{K}\Delta\hat{\boldsymbol{\theta}}_{K})$

The algorithm for dynamic low-rank adaptation in image compression is
presented in Algorithm 1. The latent representation $\boldsymbol{y}$ is
initially refined for $N_{1}$ steps, following established practices (Yang et
al., 2020; Tsubota et al., 2023). Subsequently, the decoder parameters are
adapted with low-rank constraints for additional $N_{2}$ steps. It is
recommended to warm up each low-rank layer $\Delta\boldsymbol{\theta}_{k}$ for
the $N_{w}$ steps to prevent deterioration in reconstruction performance
during the early stages.

Input: input image $\boldsymbol{x}$, latent refinement step $N_{1}$, model
adaptation step $N_{2}$, warm-up step $N_{w}$.

Output: reconstructed image $\boldsymbol{\hat{x}}$, model updates
$\\{G^{1},\Delta\boldsymbol{\theta}_{1},\ldots
G^{K},\boldsymbol{\theta}_{K}\\}$.

1 $n\leftarrow 0$;

2 $\boldsymbol{y}\leftarrow g_{a}(\boldsymbol{x};\boldsymbol{\phi})$;

/* Content adaptation, refine the latent $\boldsymbol{y}$ */

3 while  _$n <N_{1}$_ do

4 $\boldsymbol{\hat{y}}\leftarrow Quant(\boldsymbol{y})$;

5 $\mathcal{L}(\boldsymbol{y})\leftarrow-\log
p(\boldsymbol{\hat{y}})+\lambda\cdot\mathcal{D}(\boldsymbol{x},g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta}))$;

6
$\boldsymbol{y}\leftarrow\boldsymbol{y}-\alpha\frac{\partial\mathcal{L}}{\partial\boldsymbol{y}}$;

7 $n\leftarrow n+1$;

8

9 end while

10$n\leftarrow 0$;

11 $\boldsymbol{\hat{y}}\leftarrow Quant(\boldsymbol{y})$;

/* Decoder adaptation ${G^{1},\ldots G^{K}}$ and
$\boldsymbol{\Delta\theta}^{1},\ldots\boldsymbol{\Delta\theta}^{K}$ */

12 while  _$n <N_{2}$_ do

$\boldsymbol{g}\leftarrow\boldsymbol{1}$ if $n\leq N_{w}$ else
$G_{\psi}(g_{s},\boldsymbol{\hat{y}};\boldsymbol{\theta},\boldsymbol{\Delta\theta})$;
// warm up

13 $\mathcal{L}\leftarrow-\log p(\boldsymbol{\hat{y}})-\log
p_{\boldsymbol{\pi}}(\boldsymbol{\Delta\hat{\theta}})+\lambda\cdot\mathcal{D}(\boldsymbol{x},g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta},\boldsymbol{g}\boldsymbol{\Delta\hat{\theta}}))$;

$\boldsymbol{\Delta\theta}\leftarrow\boldsymbol{\Delta\theta}-\beta\frac{\partial\mathcal{L}}{\partial\boldsymbol{\Delta\theta}}$;
// optimize model updates

$\psi\leftarrow\psi-\gamma\frac{\partial\mathcal{L}}{\partial\psi}$; //
optimize gate network

14 $n\leftarrow n+1$;

15

16 end while

17

Algorithm 1 Dynamic Low-Rank Adaptation for Image Compression

##  4\. EXPERIMENTS

###  4.1. Experimental Setup

Datasets. Our method was evaluated on universal image benchmark datasets
constructed in (Tsubota et al., 2023; Wilber et al., 2017). The dataset
includes a diverse range of image types, including (i) 24 natural images from
the Kodak dataset; (ii) 100 comics images (BAM); (iii) 100 vector art images
(BAM); (iv) 100 pixel-style gaming images (self-collected). It is noteworthy
that all image codecs were pre-trained on natural images. Therefore, we
consider natural images as in-domain evaluations and other images as out-of-
domain evaluations.

Training Recipe. To assess the suggested adaptation approach, we select
several pre-trained image codecs, namely Cheng2020 (Cheng et al., 2020),
Entroformer (Qian et al., 2021), and EVC (Wang et al., 2022). We prespecify
$r=2$. The latent representation is first refined for $N_{1}=2,000$ steps,
followed by an additional $N_{2}=2,000$ steps of optimization of the decoder’s
updated parameter. The learning rate for latent refinement and model update is
set at $10^{-3}$, while for the gate network, it is $10^{-5}$.

###  4.2. Rate-Distortion Performance

Figure 5. The RD performance with BPG, VVC, and existing instance-adaptive image compression methods Tsubota et al. (Tsubota et al., 2023) and Yang et al. (Yang et al., 2020) based on the same pre-trained codec (Cheng et al., 2020). BD rates are shown in Table 1. Table 1. Our method is compared with existing adaptive approaches, as well as BPG and VVC. The evaluation is measured in terms of BD-rate (%) using VVC as the anchor. Smaller values indicate superior performance. Method | Nature image | Vector art | Comic | Pixel art | Average
---|---|---|---|---|---
BPG | 9.57 | 18.92 | 11.98 | 18.29 | 14.69
VVC | 0.00 | 0.00 | 0.00 | $\boldsymbol{0.00}$ | 0.00
Cheng2020 (Cheng et al., 2020) | 1.61 | 14.68 | 11.39 | 25.04 | 13.18
Cheng2020 + Yang et al. (Yang et al., 2020) | -6.90 | 3.32 | 4.06 | 12.75 | 3.31
Cheng2020 + Tsubota et al. (Tsubota et al., 2023) | -7.94 | 0.38 | 1.55 | 10.55 | 1.14
Cheng2020 + Ours | -$\boldsymbol{8.08}$ | -$\boldsymbol{2.01}$ | -$\boldsymbol{1.13}$ | 6.36 | -$\boldsymbol{1.21}$

We compare our method with open-source instance-adaptive image compression
methods (Yang et al., 2020; Tsubota et al., 2023) using the same pre-trained
image codec cheng2020 (Cheng et al., 2020). In our comparison, we also include
traditional codecs such as BPG and VVC, as well as a baseline method without
adaptation. To evaluate their performance, we calculate the peak signal-to-
noise ratio (PSNR) and measure the data rate in bits per pixel (bpp).
Subsequently, we plot the rate-distortion (RD) curves. The experiment results
are shown in Fig. 5. We observe that for Kodak images, all three adaptation
methods achieved comparable performance, resulting in approximately 1dB
improvement in PSNR without domain gaps in natural images. For out-of-domain
images such as vector art, comic, and pixel-style images, our method
outperforms other adaptation methods and compensated for performance
degradation. Notably, for out-of-domain comics and vector arts, our method
even surpasses VVC. Nevertheless, the domain gap between pixel-style images
and natural images proved too wide to fully compensate. Consequently, our
method still significantly lags behind VVC, although it clearly outperforms
other instance adaptive compression methods. Table 1 presents a comparison of
the Bjøntegaard delta (BD) bit-rate. In order to compute the BD rate, we use
the VVC-intra as the anchor. Our method demonstrates superior performance in
both in-domain and out-of-domain images when compared to other instance-
adaptive compression methods.

###  4.3. Ablation Studies

Low-Rank Adaptation v.s. Other Adaptations. We investigate the decoder model
update of the same convolution layer using the same pre-trained image codec
(Cheng et al., 2020). We conduct a comparative study of our approach with
other adaptation methods, namely bias-only (Lam et al., 2020), adapter
(Tsubota et al., 2023), fully fine-tuning (van Rozendaal et al., 2021b), and
Singular Value Decomposition (SVD) fine-tuning (Han et al., 2023) on out-of-
domain images. We update the $3\times 3$ conv layer in the decoder’s
penultimate layer using pre-trained weight matrix
$\mathbf{W}^{k}\in\mathbb{R}^{c_{out}\times c_{in}\times h\times w}$, where
$c_{out}=c_{in}=128$ or $192$, $h=w=3$ (i.e., kernel size). We compress the
bias, singular value, adapter, and rank-composition matrices using a logistic
distribution, while the fine-tuned incremental parameters are compressed using
a spike-and-slab prior. We compare various model update approaches and their
corresponding mathematical formulas, the number of parameters, and BD-rate.
The number of updated parameters in the penultimate layer is calculated based
on Cheng2020 (Cheng et al., 2020) from compressai (Bégaint et al., 2020) at
quality $q=6$.

The experimental results are displayed in Fig. 6 and Table 2. The findings
indicate that parameter-efficient adaptations, such as SVD, adapter, and low-
rank, demonstrate significantly better RD performance. Among these methods,
updating the weight with low-rank decomposition yields the most significant
improvement. In contrast, the bias-only method yields limited improvement, as
it only applies to scenarios with minor disturbances. Fine-tuning all
parameters in a convolution layer is not feasible due to high rate overhead
and accompanying quantification errors. Additionally, updating the singular
value of the pre-trained weight matrix provides limited enhancement with fixed
orthonormal bases.

The adapter and low-rank weight updating techniques share some similarities.
Both involve introducing a small number of learnable parameters, with the
number of parameters being the same. They both use the matrix $\mathbf{A}$ to
map the input representation to a low-dimensional space and then the matrix
$\mathbf{B}$ to map back to the high-dimensional space. However, the adapter
is executed serially on the hidden layer outputs, perturbing the outputs,
while the low-rank update approach perturbs the pre-trained weights, resulting
in more intense linear changes and stronger instance adaptation capabilities.
Additionally, the adapter typically has larger values for its extra
parameters, as they act on the output of one layer. In summary, the low-rank
model update technique outperforms the adapter with equivalent update
parameters.

Table 2. Compare different model update approaches within the same convolution layer with BD-rate (%) and the number of updated parameters on vector art, comic, and pixel datasets. Update Approaches | Formulas | Updated params |  $\\#$ of params | Vector | Comic | Pixel | Average
---|---|---|---|---|---|---|---
baseline | $\boldsymbol{h}\leftarrow\mathbf{W}\boldsymbol{h}+\mathbf{b}$ | / | 0 | 0.00 | 0.00 | 0.00 | 0.00
fine-tune | $\boldsymbol{h}\leftarrow(\mathbf{W}+\Delta\mathbf{W})\boldsymbol{h}+(\mathbf{b}+\Delta\mathbf{b})$ |  $\Delta\mathbf{W}$, $\Delta\mathbf{b}$ | 332k | 2.76 | 3.31 | 3.18 | 3.08
bias-only | $\boldsymbol{h}\leftarrow\mathbf{W}\boldsymbol{h}+(\mathbf{b}+\Delta\mathbf{b})$ | $\Delta\mathbf{b}$ | 192 | -0.05 | -0.25 | -1.44 | -0.58
SVD | $\boldsymbol{h}\leftarrow\boldsymbol{U}(\boldsymbol{S}+\Delta\boldsymbol{S})\boldsymbol{V}^{T}\boldsymbol{h}+\mathbf{b}$ | $\Delta\boldsymbol{S}$ | 192 | -3.05 | -2.48 | -2.52 | -2.68
adapter | $\boldsymbol{h}\leftarrow\mathbf{W}\boldsymbol{h}+\mathbf{b}+\mathbf{B}\mathbf{A}(\mathbf{W}\boldsymbol{h}+\mathbf{b})$ | $\mathbf{B}\mathbf{A}$ | 768 | -4.64 | -3.99 | -4.09 | -4.24
low-rank | $\boldsymbol{h}\leftarrow(\mathbf{W}+\mathbf{B}\mathbf{A})\boldsymbol{h}+\mathbf{b}$ | $\mathbf{B}\mathbf{A}$ | 768 | -6.38 | -5.97 | -7.55 | -6.63
Figure 6. Comparison of model update approaches including SVD (Han et al.,
2023), adapter (Tsubota et al., 2023), fine-tune all (van Rozendaal et al.,
2021b), bias-only (Lam et al., 2020) and our low-rank update on pixel-style
images.

Dynamic v.s. Fixed Low-rank Adaptation. To verify the effectiveness of our
dynamic gate networks, we manually control the number of model update layers
and compare that with our dynamic adaptive update. In the experiment, we use
the baseline model without model updating as the anchor for computing BD-rate,
then we update 1, 2, 4, and 8 layers while our dynamic networks update non-
deterministic layers with different image sizes.

Experiment results are shown in Table 3 and reveal that our dynamic model
update outperforms pre-specified update layers manually and achieves optimal
performance. We can observe that the optimal number of update layers is
limited by the input sizes. Though the larger number of update layers brings
more improvement in reconstruction quality, the overall RD loss saturates at
about four layers of update. Meanwhile, as the image size shrinks, updating
fewer weights brings more performance gains, as the additional model streams
take up an increasingly large percentage or even over the content streams. It
indicates that when the image size is smaller than a certain threshold, the
optimal update decision is not to update for the extra rate overhead from
model updates does not provide unequal performance gain.

Table 3. Compare different model update layers with BD-rate ($\%$) on vector arts. The best results are highlighted in bold, while the second-best ones are denoted by underlines. $\\#$ of layers | h=256, w=256 | h=384, w=384 | original size
---|---|---|---
0 | 0.0 | 0.0 | 0.0
1 | -3.51 | -4.51 | -5.36
2 | -3.87 | -5.90 | -6.92
4 | -2.77 | -6.60 | -8.26
8 | -1.28 | -5.39 | -7.51
dynamic | -$\boldsymbol{6.22}$ | -$\boldsymbol{7.84}$ | -$\boldsymbol{8.93}$

Dynamic Gating Mechanism. To illustrate the intrinsic dynamic gating
mechanism, we conduct a statistical analysis of the frequency of gate opening
of each layer for the model update. We perform our analysis based on Cheng et
al. (Cheng et al., 2020), which uses residual blocks and sub-pixel convolution
in the decoder to reconstruct the image. We assign a dynamic network to each
residual block for updating of the pre-trained weight and sort them according
to the used order. A total of 11 dynamic networks are inserted.

The experimental results, as shown in Fig. 7, indicate that blocks with larger
numbers (closer to the model output) are updated more frequently, which is
consistent with prior experience. The back convolution layers of the decoder
have a small receptive field, making them effective for perceptual detail
learning, while the top hidden layers have a larger field and are better
suited for learning abstract and semantic features. The dynamic networks tend
to update the layers closer to the reconstructed images due to their ability
to construct low-level and pixel information, which is essential for in-domain
image reconstruction.

Figure 7. The frequency of each convolution layer is selected for model
updates. The index indicates the used order.

Application to Other Network Architectures. As depicted in Fig. 1, neural
codecs trained on natural images exhibit imperfections in terms of
generalization. The gap between training and inference is a common issue
encountered by neural codecs. To address this challenge, our dynamic low-rank
model adaptation framework is universally applicable to various network
architectures. It can effectively reduce the domain gap between training and
inference, thereby enhancing the performance of neural codecs.

We demonstrate the proposed dynamic adaptation without latent refinement when
applied to Entroformer (Qian et al., 2021) and EVC (Wang et al., 2022). We
adapt the model based on their open-source pre-trained weight. Experiment
results are shown in Fig. 8. We have confirmed the universality of our
approach and demonstrated the existence of a domain gap in many learning-based
compression methods. Empirical evidence indicates that the problem under
investigation is not the result of individual model defects, but rather a
common domain gap between training and inference. Our dynamic network
addresses this issue by minimizing the domain gap during inference, through
minor updates to the deployed neural codecs on the client, rather than the
laborious process of fine-tuning all weights for each domain and deploying
multiple decompression codecs.

Figure 8. Dynamic adaptation on vector arts based on Entroformer (Qian et al.,
2021) and EVC(Wang et al., 2022).

###  4.4. Adaptation Efficiency

Image adaptation techniques, including the one proposed in this study, require
additional encoding time and hardware resources to adapt the model. However,
this may pose practical limitations in scenarios where compression speed is
crucial and computational resources are scarce. To expand the utility, we
propose a one-step adaptation approach with one forward during inference.
Please refer to the Supplementary for the one-step adaptation setup.

Table 4 shows the correlation between the adaptation performance and the
inference time using our method. Notably, adaptive image compression
techniques yield superior rate-distortion (RD) performance, albeit at the
expense of increased inference time. Our evaluation of inference time was
conducted on a single NVIDIA Tesla V100 GPU.

Table 4.  Correlation between the adaptation performance and the inference time on vector art images. Steps | 1 | 100 | 500 | 1000 | 2000 | 3000 | 4000
---|---|---|---|---|---|---|---
Time | 0s | 13s | 55s | 110s | 219s | 322s | 427s
BD-rate | -1.07 | -3.77 | -7.47 | -9.43 | -11.14 | -17.17 | -17.49

##  5\. CONCLUSION

This study presents a dynamic low-rank instance-adaptive image compression
framework that aims to mitigate the domain gap and prevent performance
degradation on out-of-domain images. The proposed approach involves
incremental updates to the pre-trained weight through the use of low-rank
decomposition matrices to minimize the rate overhead. Moreover, a dynamic
network is designed to enable adaptive control over the number and position of
plug-in blocks. However, a major limitation of this study is the additional
encoding time required for adaptation. It should be noted that the fast one-
step adaptation falls short of achieving full adaptation. These findings offer
valuable insights for future research on real-time inference for universal
instance adaptive neural image compression.

###### Acknowledgements.

This work was partly supported by the National Natural Science Foundation of
China (Nos. 62171251& 62311530100), the Special Foundations for the
Development of Strategic Emerging Industries of Shenzhen (Nos.
JCYJ20200109143010272 & CJGJZD20210408092804011) and Oversea Cooperation
Foundation of Tsinghua.

## References

  * (1)
  * Agustsson and Timofte (2017) Eirikur Agustsson and Radu Timofte. 2017.  Ntire 2017 challenge on single image super-resolution: Dataset and study. In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_. 126–135.
  * Asuni and Giachetti (2014) Nicola Asuni and Andrea Giachetti. 2014.  TESTIMAGES: a Large-scale Archive for Testing Visual Devices and Basic Image Processing Algorithms.. In _STAG_. 63–70.
  * Ballé et al. (2018) Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018.  Variational image compression with a scale hyperprior. In _International Conference on Learning Representations_.
  * Bégaint et al. (2020) Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. 2020.  CompressAI: a PyTorch library and evaluation platform for end-to-end compression research.  _arXiv preprint arXiv:2011.03029_ (2020).
  * Bellard (2015) Fabrice Bellard. 2015\.  BPG Image Format.  https://bellard.org/bpg.
  * Brancati et al. (2022) Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guillaume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, et al. 2022\.  Bracs: A dataset for breast carcinoma subtyping in h&e histology images.  _Database_ 2022 (2022), baac093.
  * Bross et al. (2021) Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. 2021.  Overview of the versatile video coding (VVC) standard and its applications.  _IEEE Transactions on Circuits and Systems for Video Technology_ 31, 10 (2021), 3736–3764.
  * Campos et al. (2019) Joaquim Campos, Simon Meierhans, Abdelaziz Djelouah, and Christopher Schroers. 2019.  Content adaptive optimization for neural image compression.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ (2019).
  * Chen et al. (2021) Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. 2021.  End-to-end learnt image compression via non-local attention optimization and improved context modeling.  _IEEE Transactions on Image Processing_ 30 (2021), 3179–3191.
  * Cheng et al. (2020) Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. 2020\.  Learned image compression with discretized gaussian mixture likelihoods and attention modules. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7939–7948.
  * Company (1993) Eastman Kodak Company. 1993\.  Kodak lossless true color image.  http://r0k.us/graphics/kodak.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of NAACL-HLT_ (2018), 4171–4186.
  * Duan et al. (2023) Zhihao Duan, Ming Lu, Zhan Ma, and Fengqing Zhu. 2023\.  Lossy Image Compression with Quantized Hierarchical VAEs. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 198–207.
  * Guo et al. (2020a) Demi Guo, Alexander M Rush, and Yoon Kim. 2020a.  Parameter-efficient transfer learning with diff pruning.  _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ (2020), 4884–4896.
  * Guo et al. (2020b) Tiansheng Guo, Jing Wang, Ze Cui, Yihui Feng, Yunying Ge, and Bo Bai. 2020b.  Variable rate image compression with content adaptive optimization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 122–123.
  * Guo et al. (2022) Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. 2022.  HCSC: Hierarchical Contrastive Selective Coding.  _arXiv preprint arXiv:2202.00455_ (2022).
  * Han et al. (2023) Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. 2023\.  SVDiff: Compact Parameter Space for Diffusion Fine-Tuning.  _arXiv preprint arXiv:2303.11305_ (2023).
  * He et al. (2022) Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. 2022\.  Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5718–5727.
  * He et al. (2021) Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. 2021.  Checkerboard context model for efficient learned image compression. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14771–14780.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP. In _International Conference on Machine Learning_. PMLR, 2790–2799.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021\.  LoRA: Low-Rank Adaptation of Large Language Models. In _International Conference on Learning Representations_.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020\.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision_ 128, 7 (2020), 1956–1981.
  * Lam et al. (2020) Yat-Hong Lam, Alireza Zare, Francesco Cricri, Jani Lainema, and Miska M Hannuksela. 2020.  Efficient adaptation of neural network filter for video compression. In _Proceedings of the 28th ACM International Conference on Multimedia_. 358–366.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021\.  The power of scale for parameter-efficient prompt tuning.  _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ (2021), 3045–3059.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ (2021), 4582–4597.
  * Liu et al. (2021) Jiaming Liu, Ming Lu, Kaixin Chen, Xiaoqi Li, Shizun Wang, Zhaoqing Wang, Enhua Wu, Yurong Chen, Chuang Zhang, and Ming Wu. 2021\.  Overfitting the data: Compact neural video delivery via content-aware feature modulation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4631–4640.
  * Lu et al. (2021) Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. 2021.  Transformer-based image compression.  _2022 Data Compression Conference (DCC)_ (2021), 469–469.
  * Minnen et al. (2018) David Minnen, Johannes Ballé, and George D Toderici. 2018\.  Joint autoregressive and hierarchical priors for learned image compression.  _Advances in neural information processing systems_ 31 (2018).
  * Minnen and Singh (2020) David Minnen and Saurabh Singh. 2020.  Channel-wise autoregressive entropy models for learned image compression. In _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 3339–3343.
  * Qian et al. (2021) Yichen Qian, Ming Lin, Xiuyu Sun, Zhiyu Tan, and Rong Jin. 2021.  Entroformer: A transformer-based entropy model for learned image compression.  _International Conference on Learning Representations_ (2021).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Skodras et al. (2001) Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. 2001.  The JPEG 2000 still image compression standard.  _IEEE Signal processing magazine_ 18, 5 (2001), 36–58.
  * Toderici et al. (2020) George Toderici, Lucas Theis, Nick Johnston, Eirikur Agustsson, Fabian Mentzer, Johannes Balle, Wenzhe Shi, and Radu Timofte. 2020\.  http://compression.cc.
  * Tsubota et al. (2023) Koki Tsubota, Hiroaki Akutsu, and Kiyoharu Aizawa. 2023\.  Universal Deep Image Compression via Content-Adaptive Optimization with Adapters. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2529–2538.
  * van Rozendaal et al. (2021a) Ties van Rozendaal, Johann Brehmer, Yunfan Zhang, Reza Pourreza, and Taco S Cohen. 2021a.  Instance-adaptive video compression: Improving neural codecs by training on the test set.  _arXiv preprint arXiv:2111.10302_ (2021).
  * van Rozendaal et al. (2021b) Ties van Rozendaal, Iris AM Huijben, and Taco S Cohen. 2021b.  Overfitting for fun and profit: Instance-adaptive data compression.  _9th International Conference on Learning Representations_ (2021).
  * Wallace (1992) Gregory K Wallace. 1992\.  The JPEG still picture compression standard.  _IEEE transactions on consumer electronics_ 38, 1 (1992), xviii–xxxiv.
  * Wang et al. (2022) Guo-Hua Wang, Jiahao Li, Bin Li, and Yan Lu. 2022\.  Evc: Towards real-time neural image compression with mask decay.  _The Eleventh International Conference on Learning Representations_ (2022).
  * Wang et al. (2018) Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. 2018.  Skipnet: Learning dynamic routing in convolutional networks. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 409–424.
  * Wilber et al. (2017) Michael J Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann, John Collomosse, and Serge Belongie. 2017\.  Bam! the behance artistic media dataset for recognition beyond photography. In _Proceedings of the IEEE international conference on computer vision_. 1202–1211.
  * Xiang et al. (2023) Jinxi Xiang, Kuan Tian, Jun Zhang, Han Xiao, and Wei Yang. 2023.  MIMT: Masked Image Modeling Transformer for Video Compression. In _The Eleventh International Conference on Learning Representations_.
  * Xie et al. (2021) Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. 2021\.  Enhanced invertible encoding for learned image compression. In _Proceedings of the 29th ACM international conference on multimedia_. 162–170.
  * Yang et al. (2020) Yibo Yang, Robert Bamler, and Stephan Mandt. 2020.  Improving inference for neural image compression.  _Advances in Neural Information Processing Systems_ 33 (2020), 573–584.
  * Zhu et al. (2022a) Xiaosu Zhu, Jingkuan Song, Lianli Gao, Feng Zheng, and Heng Tao Shen. 2022a.  Unified multivariate gaussian mixture for efficient neural image compression. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 17612–17621.
  * Zhu et al. (2022b) Yinhao Zhu, Yang Yang, and Taco Cohen. 2022b.  Transformer-based transform coding. In _International Conference on Learning Representations_.
  * Zou et al. (2021) Nannan Zou, Honglei Zhang, Francesco Cricri, Ramin G Youvalari, Hamed R Tavakoli, Jani Lainema, Emre Aksu, Miska Hannuksela, and Esa Rahtu. 2021.  Adaptation and attention for neural video coding. In _2021 IEEE International Symposium on Multimedia (ISM)_. IEEE, 240–244.
  * Zou et al. (2022) Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. 2022\.  The devil is in the details: Window-based attention for image compression. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 17492–17501.

##  Appendix A Supplementary Material

###  A.1. One-Step Adaptation

Instance adaptive image compression techniques require additional encoding
time to adapt the model, resulting in better rate-distortion (RD) performance.
Nonetheless, this approach may not be suitable for certain practical usage
scenarios, especially those that prioritize compression speed. To broaden the
scope of instance adaptive image compression applications, this section
introduces a one-step method that eliminates the need for iterative
overfitting of instance parameters during inference, while still slightly
enhancing compression performance.

Directly obtaining instance parameters without iteration is not feasible, but
we can obtain an approximation by analyzing similar images. To achieve this,
we collected 7,000 images from various sources, including the BAM dataset
(Wilber et al., 2017), which were not previously used in evaluating the
method.

After that, we use open-source pre-trained HCSC (Guo et al., 2022) to extract
features: we take the final layer outputs of a pre-trained HCSC and embed
these images into 2048-dimensional feature space. Then we unsupervised
clustered these embedding features with $N_{clusters}=128$ and save the
cluster center $c_{n},n\in\\{1,2,\cdots,N_{clusters}\\}$. Images in the same
clusters usually have similar image textures. For each cluster
$\mathcal{C}_{n}$, we fine-tune incremental matrices
$\boldsymbol{\Delta\theta}^{n}$ on the penultimate layer in the decoder.

At inference time, for each input $\boldsymbol{x}$, we extract its embedding
feature $\mathcal{F}(\boldsymbol{x})$ and find the nearest cluster center $n$:

(15) $n=\arg\min d(\mathcal{F}(\boldsymbol{x}),c_{n})$

The corresponding parameters $\boldsymbol{\Delta\theta}^{n}$ of
$\mathcal{C}_{n}$ are quantized and decoded to bit rate and then transmitted
to the client to enhance the reconstruction equality:

(16)
$\hat{\boldsymbol{x}}=g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta},\boldsymbol{\Delta\hat{\theta}}^{n})$

Our evaluation of one-step adaptation includes in-domain and out-of-domain
images using the Cheng et al. (Cheng et al., 2020) baseline as an anchor for
computing BD-rate. To compare, we incrementally update the penultimate layer
in the decoder for each cluster. The experiment results, presented in Table 5,
demonstrate that performance can be improved even without adaptation during
inference time, on both in-domain and out-of-domain images. In strict terms,
one step is a complete process containing end-to-end forward propagation and
backward propagation. Our one-step adaptation just forwards the encoding
process and then matches the adaptation parameters. One-step adaptation
reduces the inference time and server performance requirements, making these
methods more practical.

Table 5. One-step adaptation results in BD-rate (%). Method | Kodak | Vector Art | Comic | Pixel Art
---|---|---|---|---
Cheng et al. | 0.0 | 0.0 | 0.0 | 0.0
Cheng et al. \+ Ours | -0.24 | -1.07 | -1.79 | -0.50

###  A.2. Extend to Pathology Images

Digital pathology images are limited in their application in areas such as
remote collaborative diagnosis and AI-assisted diagnosis due to their ultra-
high resolution and expensive save cost. Meanwhile, despite the neural codecs
developing rapidly, most of them are trained on natural images, and there is
no neural codec specifically for pathology images. We extend this study to
pathology images in an attempt to fill this blank.

We evaluate our method on pathology images from BRACS (Brancati et al., 2022)
dataset. We compare our method with open-source instance-adaptive image
compression methods. The evaluation measured in terms of BD-rate (%) using VVC
as the anchor. The experiment results are presented in Table 6.

Figure 9. The RD performance with BPG, VVC, and existing instance-adaptive image compression methods on pathology images. Table 6. Comparison on pathology images with BPG, VVC and other existing adaptive approaches. The evaluation is measured in terms of BD-rate (%) using VVC as the anchor. Method | BD-rate (%)
---|---
BPG | 10.56
VVC | 0.00
Cheng2020 (Cheng et al., 2020) | 11.63
Cheng2020 + Yang et al. (Yang et al., 2020) | 0.42
Cheng2020 + Tsubota et al. (Tsubota et al., 2023) | -3.02
Cheng2020 + Ours | -$\boldsymbol{5.04}$

During inference time adaptation, tiny updates to the deployed neural codecs
on the client greatly compensates the domain gap. Our instance adaptive image
compression model avoids preparing and deploying multiple task-specific
decompression codecs. Images from any domain can be compressed by only one
base compression model while alleviating the performance degradation and
achieving outstanding performance.

###  A.3. Visualization of the Model Update

Figure 10. Visualization of reconstruction with and without extra parameters
$\boldsymbol{\Delta\hat{\theta}}$ using a pixel-style image as an example. It
shows our approach provides domain-specific image texture to reduce
reconstruction error significantly.

We reconstruct the refined quantized latent code with and without extra
parameters $\Delta\boldsymbol{\hat{\theta}}$ to demonstrate its effectiveness.
Required bits are calculated as $-\log_{2}p(\boldsymbol{\hat{y}})$ to show the
bit allocation. We choose four patches that require more bits in the original
images and then reconstruct the latent code without the model update as
$\boldsymbol{\hat{x}}=g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta})$ and
with extra parameters $\Delta\boldsymbol{\hat{\theta}}$ as
$\boldsymbol{\hat{x}}=g_{s}(\boldsymbol{\hat{y}};\boldsymbol{\theta},\boldsymbol{\Delta}\boldsymbol{\hat{\theta}}$),
respectively. Then we visualize the reconstruction error
$\boldsymbol{\hat{x}}-\boldsymbol{x}$. As shown in Fig. 10, we can observe
that the error distributes mainly at edges and our approach reduced the
reconstruction errors significantly and provides domain-specific image texture
during adaptation.

###  A.4. Visualization Comparison with Other Approaches

The qualitative results of the baseline method Cheng et al. (Cheng et al.,
2020) without adaptation, other instance-adaptive image compression methods
(Yang et al., 2020; Tsubota et al., 2023), and ours are shown in Fig. 11. We
conducted a comparative analysis of these techniques utilizing the same pre-
trained model on three distinct out-of-domain image categories: vector art,
comic, and pixel style. Our approach demonstrated superior visual quality in
comparison to both the baseline method and other instance-adaptive codecs.

Figure 11. Visualization and PSNR/bpp performance comparison with existing
instance-adaptive image compression methods Yang et al. (Yang et al., 2020)
and Tsubota et al. (Tsubota et al., 2023) based on the same pre-trained
baseline cheng et al.(Cheng et al., 2020) from compressai (Bégaint et al.,
2020). The PSNR/bpp metrics are under each patch. Zoom in for better
visibility.

Our approach in the vector art patch yields superior visual quality when
reconstructing patterns and text, resulting in clearer boundaries and higher
fidelity. In the case of the comic example, the baseline method without
adaptation struggles with reconstructing dense diagonal lines. While other
adaptation methods reduce the reconstruction error, they still exhibit some
artifacts. Our method successfully reconstructs dense lines with adaptation.
We used a pixel-style image from Terraria as a sample, and the reconstruction
results demonstrate that the baseline method without adaptation results in
smoothing and chromatic aberration. Furthermore, it fails to reconstruct
jagged edges and instead produces smooth edges, which do not accurately
capture the characteristics of pixel-style images. However, our method, with
model adaptation, results in better restoration of color and pixel-level
edges. We randomly select a breast histopathological image from BRACS
(Brancati et al., 2022) as a sample to show that our method retain more nuclei
details than the method without adaptation.

###  A.5. Generalization on Invertible Neural Networks

To show the generalization potential of our dynamic instance image compression
method, we implement instance adaptive compression based on invertible neural
network (INN) (Xie et al., 2021) backbone. Reconstruct network is updated by
low-rank decomposition matrices to enhance compression performance. The
evaluation measured in terms of BD-rate (%) using baseline as the anchor is
presented in Table 7 and Fig. 12

Table 7. Evaluation results based on INN-based method (Xie et al., 2021). Method | Kodak | Vector art | Comic | Pixel art
---|---|---|---|---
INN | 0.0 | 0.0 | 0.0 | 0.0
INN + Ours | -8.42 | -10.43 | -13.94 | -18.08
Figure 12. The RD performance of INN-based image compression methods with and
without instance adaptation. BD rates are shown in Table 7.