# Prompt-ICM: A Unified Framework towards
Image Coding for Machines with Task-driven Prompts

Ruoyu Feng1* Jinming Liu2* Xin Jin3 Xiaohan Pan1 Heming Sun2 Zhibo Chen 1,†
1University of Science and Technology of China 2Waseda University 3Eastern
Institute of Advanced Study

###### Abstract

Image coding for machines (ICM) aims to compress images to support downstream
AI analysis instead of human perception. For ICM, developing a unified codec
to reduce information redundancy while empowering the compressed features to
support various vision tasks is very important, which inevitably faces two
core challenges: 1) How should the compression strategy be adjusted based on
the downstream tasks? 2) How to well adapt the compressed features to
different downstream tasks? Inspired by recent advances in transferring large-
scale pre-trained models to downstream tasks via prompting, in this work, we
explore a new ICM framework, termed Prompt-ICM. To address both challenges by
carefully learning task-driven prompts to coordinate well the compression
process and downstream analysis. Specifically, our method is composed of two
core designs: a) compression prompts, which are implemented as importance maps
predicted by an information selector, and used to achieve different content-
weighted bit allocations during compression according to different downstream
tasks; b) task-adaptive prompts, which are instantiated as a few learnable
parameters specifically for tuning compressed features for the specific
intelligent task. Extensive experiments demonstrate that with a single feature
codec and a few extra parameters, our proposed framework could efficiently
support different kinds of intelligent tasks with much higher coding
efficiency.

††footnotetext: * First two authors contributed equally.
† Corresponding author.

Figure 1: Different pipelines of image coding for machines (ICM): (a). Using
the compressed images to support downstream tasks; (b). One-to-one feature-
based ICM pipeline; (c). General features based ICM pipelines which ignores
the explicit interaction between the compression task and downstream tasks.
(d). Compared with (c). we further consider using task-driven prompts (those
colours) to better coordinate the compression process and downstream analysis.

##  1 Introduction

In modern society, intelligent multimedia applications have played an
irreplaceable role in our daily life, such as smart cities, intelligent
surveillance, and the Internet of Things (IoT). With the fast development of
machine vision technologies, there will be more and more images that need to
be compressed and transmitted over the Internet to serve intelligent analysis.
One of the key technologies is lossy compression, which aims to save storage
resources and transmission bandwidth. In the past decades, hand-crafted image
and video codecs [76, 63, 79, 73, 8] have significantly improved coding
efficiency.

Recently, learned-based codecs [5, 6, 59, 16, 37, 44, 43, 57, 58] have shown
strong potential, which not only outperform traditional hand-crafted codecs in
PSNR, but also can be optimized according to perception-related metrics (e.g.,
MS-SSIM [78], LPIPS[84]) to generate more realistic images. However, these
codecs are mainly designed to satisfy human perception. When facing AI task
analysis, existing image coding methods (even the learned-based ones) are
still questionable. Due to the fundamental differences between the information
needs of intelligent tasks and human vision, and the existence of various,
perhaps even unknown tasks, utilizing existing codecs to compress images for
downstream tasks is likely to yield suboptimal outcomes.

Therefore, a new task of compressing images for machine vision, called image
coding for machines (ICM) [23, 40], has emerged to build a joint efficient and
analytical framework. Such a framework is capable of obtaining and compressing
general representations to effectively support intelligent analytics for
massive and diverse applications.

The existing ICM methods can be divided into three branches: Figure 1 (a)
shows the first branch that uses task-specific codecs to compress images [40,
41], and then perform intelligent analysis based on reconstructed images.
These codecs are typically optimized for their respective task losses along
with rate losses, in an end-to-end manner. As shown in Figure 1 (b), methods
in the second branch [24, 25, 15, 14, 71, 56, 2] firstly extract specific
features for individual compression, and finally use the reconstructed
features to complete the intelligent task analysis. Note that, such two
branches have a large limitation: the different intelligent tasks need to use
their corresponding codecs for compression, respectively, _i.e_., lack of
generalization. And the lack of generalization might cause a significant extra
cost of computation and storage for different downstream tasks. To overcome
this defect, the third branch has been explored (as shown in Figure 1 (c)
[26]). This method comprises a generic feature extractor and its associated
feature codec, which reconstructs the general features for all subsequent
tasks. However, it suffers from suboptimal efficiency due to its disregard for
task-specific characteristics during compression and downstream analysis. In
other words, the potential benefits of an optimized compression scheme
tailored to individual downstream tasks have not been fully leveraged.

In this paper, we tend to explore a new framework for image coding for
machines (ICM) that circumvents the aforementioned issues. Inspired by the
recent successes of parameter efficient tuning for transferring large-scale
pre-trained models to downstream tasks [36, 60, 10, 83, 88, 87], we design a
new ICM framework, termed as Prompt-ICM, from a new aspect of carefully
learning task-driven prompts to coordinate well the compression process and
downstream analysis, as shown in Figure 1 (d). This framework consists of two
core designs. The first design is compression prompts, which refer to
importance maps that represent the positional importance distribution
conditioned on the extracted features and the corresponding intelligent task.
More specifically, the compression prompts are predicted by a lightweight
information selector (IS) module and utilized in conjunction with a spatially
variable-rate feature compression model to achieve content-weighted bit
allocation during the compression process, tailored to the specific task
requirements. The second component of our framework is task-adaptive prompts,
which incorporate a few additional learnable parameters to analyze the
compressed features for the specific downstream task. Together with
compression prompts, enabling our Prompt-ICM framework to utilize a unified
codec that efficiently supports diverse intelligent tasks with superior
compression performance.

The main contributions of this paper are summarised as follows:

  * $\bullet$

To the best of our knowledge, we are the first to investigate and formulate
the coordination of the interaction between compression and downstream
analytics in a unified framework. Our proposed Prompt-ICM can support
different kinds of intelligent tasks based on only a single codec.

  * $\bullet$

We propose the compression prompts for content-weighted compression according
to the demands of downstream tasks. As a by-product contribution, we design an
effective sub-component, a lightweight information selector (IS) module, to
predict importance maps as compression prompts.

  * $\bullet$

Furthermore, we propose task-adaptive prompt tuning to transfer compressed
features for downstream tasks, achieving significant performance improvement
with a few parameters, which is more practical for ICM applications.

##  2 Related Work

###  2.1 Image Compression

Image compression aims to represent original pixel samples using a compact and
high-fidelity format. Traditional hand-crafted image codecs typically involve
intra prediction, discrete cosine transformation or wavelet transformation,
quantization, and entropy coding [76, 63, 79, 73, 8]. Learned-based codecs [5,
6, 59, 16, 37, 44, 43, 57, 58] make use of neural networks to learn to
minimize distortion between pairs of source images and reconstructed images,
while maximizing the likelihood of the quantized latent representation for low
bitrate in an end-to-end manner. Furthermore, the utilization of learned-based
compression models offers a significant advantage in terms of versatility
through the joint optimization of perceptual metrics such as MS-SSIM[78],
LPIPS[84], and adversarial loss[58]. Despite a potential decrease in signal
fidelity, compression models optimized with these metrics can produce more
realistic images. However, since the rate-distortion trade-off is controlled
by a Lagrange multiplier $\lambda$, most existing methods are limited in that
a fixed value of $\lambda$ corresponds to a single point in the rate-
distortion curve. Recent works[17, 20, 37, 74, 82] propose different
approaches to support variable rates using a single model. Song _et al_. [72]
propose to perform spatial bit allocation according to a quality map that is
the same size as the original image.

###  2.2 Image Coding for Machines

Image coding for machines (ICM) targets at compressing and transmitting source
images to support downstream intelligent tasks, such as image
classification[36, 32, 52, 29, 22], object detection[69, 67, 68, 48, 47],
instance segmentation[31, 51, 7], and semantic segmentation[53, 3, 12, 13, 81,
86]. A natural way is joint optimization [1, 34, 45, 40, 77] of image
compression models and the downstream intelligent tasks. Another branch of
intuitive methods compresses the features[24, 25, 15, 14, 71, 56, 2] of
corresponding tasks instead of images for both coding efficiency and computing
offloading. Recently, Feng _et al_. [26] propose to learn features that are
both general and compact based on joint optimization of self-supervised
learning and entropy constraint. And all intelligent tasks are performed based
on the extracted features. Nevertheless, this method doesn’t consider the
coordination between the compression process and downstream transferring,
lacking targeted adjustments for different tasks. Differently, this paper aims
to design a unified framework that contains the advantages of the above
methods and avoids the corresponding disadvantages. More specifically, we
explore the coordination between general feature compression and downstream
task transferring and propose a unified framework that can adapt to different
kinds of machine vision tasks based on a single compression model with a few
learnable parameters.

###  2.3 Parameter Efficient Tuning for Large-scale Pre-trained Models

Parameter efficient tuning (PET) is first introduced in NLP [33, 62, 46, 42,
50, 30] since it’s inefficient to fully fine-tune all parameters of large-
scale pre-trained models [9, 65, 66, 64, 35] on each downstream intelligent
task. In computer vision, parameter efficient tuning is first introduced to
large-scale pre-trained visiaon-language models[64, 35] via prompt-based
tuning[88, 87], which introduces additional learnable prompts attached to the
input during the training stage, while keeping the pre-trained models fixed.
Zhang _et al_. [83] and Gao _et al_. [27] design lightweight adapters to
predict the adapted feature residuals to modulate representation space. Jia
_et al_. [36] adapt visual prompts for supervised pre-trained vision
transformers. Bahng _et al_. [4] explore visual prompts in input pixel space
for adapting pre-trained models. Nie _et al_. [60] inserts several lightweight
prompt blocks into backbones to adjust feature representation. This paper
considers a more practical scenario of multiple downstream intelligent tasks
supported for ICM. In combination with PET methods, our framework can support
different downstream tasks more efficiently.

Figure 2:  The framework of our proposed Prompt-ICM (taking Swin-T as an
example). Downstream transferring is performed via task-driven prompt tuning.
By tuning the lightweight information selector and task-adaptive prompts,
Prompt-ICM could efficiently support various downstream tasks, e.g.,
classification, segmentation, and detection.

##  3 Approach

###  3.1 Formulation of General ICM

In this section, we mathematically define the problem of general ICM from a
new perspective, which is formulated by Equation (1)-(4). To begin with, the
input image $\boldsymbol{x}$ is firstly analyzed by the pre-trained feature
extractor $\boldsymbol{FE}_{i}$ with parameters
$\theta_{\mathrm{\boldsymbol{FE}_{\textit{i}}}}$ to extract the feature
$\boldsymbol{f}_{i}$ for task $i$:

$\boldsymbol{f}_{i}=\boldsymbol{FE}_{i}\left(\boldsymbol{x};\,\theta_{\mathrm{\boldsymbol{FE}_{\textit{i}}}}\right).$
(1)

Note that when $\boldsymbol{FE}_{i}$ is None, $\boldsymbol{f}_{i}$ refers to
the raw image $\boldsymbol{x}$ for subsequent operations, as Figure 1 (a)
shows.

After that, a lossy codec $\boldsymbol{C}_{i}$ with parameters
$\theta_{\boldsymbol{C}_{i}}$ is used to compress the features
$\boldsymbol{f}_{i}$ for task $i$:

$\hat{\boldsymbol{f}}_{i}=\boldsymbol{C}_{i}\left(\boldsymbol{f}_{i};\,\theta_{\boldsymbol{C}_{i}}\right).$
(2)

Then the reconstructed feature $\hat{\boldsymbol{f}}_{i}$ is sent to the
remaining networks $\boldsymbol{T}_{i}$ with parameters
$\theta_{\boldsymbol{T}_{i}}$ to acquire prediction results $o_{i}$:

$o_{i}=\boldsymbol{T}_{i}\left(\hat{\boldsymbol{f}}_{i};\,\theta_{\boldsymbol{T}_{i}}\right).$
(3)

Generally, the optimization function of the ICM framework for downstream
transferring can be described as:

$\displaystyle
argmin_{\Phi=\left\\{\theta_{\boldsymbol{FE}_{i}},\theta_{\boldsymbol{C}_{i}},\theta_{\boldsymbol{T}_{i}}\right\\}}\alpha\mathcal{L}_{i}+R.$
(4)

where the Lagrange multiplier $\alpha$ controls the trade-off between bitrate
$R$ and loss $\mathcal{L}_{i}$ for task $i$.

###  3.2 Overview

In contrast to general ICM, we propose a new ICM framework called Prompt-ICM.
Firstly, similar to Figure 1 (c), we use a single general feature extractor
with fixed parameters $\varphi_{\boldsymbol{FE}}$111In this paper, we use
$\theta$ to represent learnable parameters, while using $\varphi$ to represent
fixed parameters for downstream tasks. to extract the general feature for all
downstream tasks instead of extracting different features for different tasks.
The Equation (1) in our framework can be revised as follows:

$\boldsymbol{f}=\boldsymbol{FE}\left(\boldsymbol{x};\,\varphi_{\mathrm{\boldsymbol{FE}}}\right).$
(5)

This will significantly reduce the extra cost of computation and storage for
different downstream tasks.

However, the framework corresponding to Figure 1 (c) does not take into
account the task-specific characteristics during compression, which may result
in inefficient coding for specific tasks. To mitigate the aforementioned
issue, we propose a lightweight information selector module
($\boldsymbol{IS}$) with tunable parameters $\theta_{\boldsymbol{IS}}$. The
module generates importance maps as compression prompts, which are used to
guide the spatial bit allocation of the codec. The general feature extractor
and customized compression prompts enable us to employ a single controllable
feature codec $\boldsymbol{C}$ with parameters $\varphi_{\boldsymbol{C}}$ for
all various downstream tasks instead of designing distinct codecs for
different tasks. Therefore, Equation (2) in our framework can be revised as:

$\boldsymbol{m}_{i}=\boldsymbol{IS}_{i}\left(\boldsymbol{f};\,\theta_{\boldsymbol{IS}_{i}}\right).$
(6)
$\hat{\boldsymbol{f}}_{i}=\boldsymbol{C}\left(\boldsymbol{f},\boldsymbol{m}_{i};\,\varphi_{\boldsymbol{C}}\right).$
(7)

The next step is to send the reconstructed feature $\hat{\boldsymbol{f}}_{i}$
to $\boldsymbol{T}_{i}$ to acquire prediction. Nevertheless, fine-tuning
$\boldsymbol{T}_{i}$ for each downstream task is parameter-consuming. In this
paper, we utilize the task-adaptive prompts $\boldsymbol{p}_{i}$ with a few
learnable parameters $\theta_{\boldsymbol{p}_{i}}$ to conduct efficient
transferring. Then the Equation (3) in our framework can be revised as
follows:

$o_{i}=\boldsymbol{T}_{i}\left(\hat{\boldsymbol{f}}_{i},\boldsymbol{p}_{i};\,\varphi_{\boldsymbol{T}^{{}^{\prime}}_{i}},\theta_{{\boldsymbol{p}_{i}}},\theta_{{\boldsymbol{h}_{i}}}\right).$
(8)

where the parameters of $\boldsymbol{T}_{i}$ are divided to two parts:
$\theta_{{\boldsymbol{h}_{i}}}$ denotes the parameters of the task head, while
remaining fixed parameters are represented as
$\varphi_{\boldsymbol{T}^{{}^{\prime}}_{i}}$.

Notably, when transferring to downstream tasks, we only need to fine-tune the
information selector, the task head, and task-adaptive prompts. The
optimization function can be revised as follows:

$\displaystyle
argmin_{\Phi^{{}^{\prime}}=\left\\{\theta_{\boldsymbol{IS}_{i}},\theta_{{\boldsymbol{h}_{i}}},\theta_{\boldsymbol{p}_{i}}\right\\}}\alpha\mathcal{L}_{i}+R,$
(9)

where the number of trainable parameters $\Phi^{{}^{\prime}}$ in Equation (9)
is far fewer than $\Phi$ in Equation (4).

Figure 3: Two variants of information selector modules that differ in the
input features for generating compression prompts.

###  3.3 General Feature Extraction

With a large-scale pre-trained vision model, the whole network is first
divided into multiple sub-layers as stages. We follow the regular concepts of
stage partitioning[32, 52]. Considering that the features would be consumed by
a variety of kinds of intelligent tasks, _e.g_., image classification, object
detection, and semantic segmentation, the features extracted at stage 1 (with
a 4x down-sampling factor) are taken as the general features to promise
completeness of information and integrity of the content’s spatial layout.
Formally, consider $s_{1}$ to be the first stage of a pre-trained vision model
$\boldsymbol{S}=\\{s_{j}\\}_{j=1}^{n}$ with $n$-stages. Given an input image
$\boldsymbol{x}$, we feed it into $s_{1}$ to obtain the feature map
$\boldsymbol{f}$ with 4$\times$ down-sampling, i.e., $\boldsymbol{FE}=s_{1}$.

In this paper, we take the Swin Transformer[52] as the base model for its
strong representation capability and functionalities resulting from the
hierarchical design.

###  3.4 Compression Prompts

During each inference, we use compression prompts $\boldsymbol{m}$ generated
by a lightweight information selector ($\boldsymbol{IS}$) to guide content-
weighted feature compression corresponding to the current task.

Generation of Compression Prompts. As shown in Figure 3 (a), $\boldsymbol{IS}$
module is used to extract importance maps as compression prompts. Besides, we
can perform additional forward propagation at the encoding side to obtain
multi-scale features $\boldsymbol{F}=\\{\boldsymbol{f}^{k}\\}_{k=1}^{n}$ which
contain richer and hierarchical information. Note that $\boldsymbol{f}^{1}$ in
$\boldsymbol{F}$ corresponds to $\boldsymbol{f}$ in Section 3.3. Then, the
Equation (6) can be revised as
$\boldsymbol{m}_{i}=\boldsymbol{IS}_{i}\left(\boldsymbol{F};\,\theta_{\boldsymbol{IS}_{i}}\right)$.
And the $\boldsymbol{IS}$ module can aggregate these features from multiple
semantic levels to better generate compression prompts, as shown in Figure 3
(b).

Content-weighted Feature Compression. To make use of the compression prompts,
we design a controllable feature codec by adjusting previous learned lossy
compression methods to enable compression prompts $\boldsymbol{m}_{i}$ to
guide content-weighted feature compression for task $i$. Notably, to ensure
the irrelevance of any specific task, the training of the codec is independent
of the rest of the network in the whole framework.

In previously learned lossy image compression, the goal is to simultaneously
minimize the bitrate and the distortion. Such an objective can be formulated
as minimizing:

$R+{\lambda}D.$ (10)

where $R$ represents the bitrate, $D$ denotes the distortion between the
original features and the reconstructed features, and $\lambda$ is the
Lagrange multiplier that controls the rate-distortion trade-off. In our
framework, we go beyond previous learned-based codecs in that one model
controlled by a fixed value of $\lambda$ corresponds to a single point in the
rate-distortion curve, and build a feature codec that can conduct bit
allocation according to manually set compression prompts $\boldsymbol{m}$.
Thus the Equation (10) is newly written as:

$R+{\boldsymbol{\Lambda}}\cdot{\boldsymbol{D}},$ (11)

where
${\boldsymbol{\Lambda}}=\left\\{{\lambda}_{h,w}\right\\}_{h=1,w=1}^{H,W}$
denotes the importance of each position, and
$\boldsymbol{\Lambda}=\boldsymbol{m}$.
$\boldsymbol{D}=\left\\{{D}_{h,w}\right\\}_{h=1,w=1}^{H,W}$ represents the
distortion in each position of the feature $\boldsymbol{f}$ and the
reconstructed feature $\hat{\boldsymbol{f}}$.

More specifically, we design the compression framework derived from the Mean &
Scale (M&S) Hyperprior model [59] and spatial variable-rate image compression
[72]. Since round-based quantization is non-differential, the additive uniform
noise[5] is added to the latent variables for rate estimation during training.

The overall R-D (rate-distortion) loss function for the training of the codec
is formulated as:

$\displaystyle\mathcal{L}_{f}=$
$\displaystyle\mathcal{R}(\hat{\boldsymbol{y}})+\mathcal{R}({\hat{\boldsymbol{z}}})+\boldsymbol{\Lambda}\cdot{\boldsymbol{D}}(\boldsymbol{f},\hat{\boldsymbol{f}})$
(12) $\displaystyle=$
$\displaystyle\mathop{\mathbb{E}}[-\log_{2}(p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}({\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}))]+\mathop{\mathbb{E}}[-\log_{2}(p_{\hat{\boldsymbol{z}}|{\boldsymbol{\psi}}}({\hat{\boldsymbol{z}}|{\boldsymbol{\psi}}}))]$
$\displaystyle+\sum_{h=1}^{H}\sum_{w=1}^{W}{\lambda}_{h,w}\frac{(f_{h,w}-\hat{f}_{h,w})^{2}}{HW},$

where
$p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})$
denotes the probability distribution of the latent variable
$\hat{\boldsymbol{y}}$ which is a compact representation of $\boldsymbol{f}$
and generated by the encoder of the codec. $\hat{\boldsymbol{y}}$ and side
information $\hat{\boldsymbol{z}}$ which provides hyper-prior information for
$\hat{\boldsymbol{y}}$ are encoded as bitstreams for transmission and storage.
The decoded bitstreams are used to generate the reconstructed feature
$\hat{\boldsymbol{f}}$.
$p_{\hat{\boldsymbol{z}}|{\boldsymbol{\psi}}}(\hat{\boldsymbol{z}}|{\boldsymbol{\psi}})$
denotes the probability distribution of side information
$\hat{\boldsymbol{z}}$, $\boldsymbol{\psi}$ denotes the factorized density
model to encode $\hat{\boldsymbol{z}}$, $H$ and $W$ denote the height and
width of the feature, and ${\lambda}_{h,w}$ denotes the Lagrange multiplier of
the corresponding position. The detailed network architecture, compression
process, and formulations are reported in the supplementary.

After the training stage of the feature codec, its parameters are fixed. When
transferring to downstream task $i$, the task-oriented adjustment is achieved
by fine-tuning the lightweight $\boldsymbol{IS}_{i}$ for generating
$\boldsymbol{m}_{i}$, resulting in a unified codec for various intelligent
tasks.

Figure 4: Classification results on different datasets at various bitrates.
Two traditional codecs HEVC-intra [73], VVC-intra [8], and one learned-based
codec _Cheng20_ [16] are compared with our method.

Figure 5: Results of semantic segmentation on Cityscapes (the first one) and
object detection and instance segmentation on COCO 2017 (second and third).

###  3.5 Task-adaptive Prompts

Task-adaptive prompts are instantiated as a few learnable parameters
specifically for tuning compressed features for image analysis. They are
injected into the pre-trained models on the decoder side, and fine-tuned to
fit the specific downstream tasks. It should be noted that the parameters of
task-adaptive prompts are much smaller than those of the original task model.

As Figure 2 shows, after obtaining the reconstructed feature
$\hat{\boldsymbol{f}}_{i}$ of task $i$, task-adaptive prompts
$\boldsymbol{p}_{i}=\\{\boldsymbol{p}_{i}^{k}\\}_{k=2}^{n}$ are introduced to
adjust features during the forward propagation in the rest n-1 stages,
corresponding to Equation (8).

The overall loss function for transferring to downstream task $i$ is given by:

$\vspace{-1mm}\mathcal{L}=\mathcal{R}(\hat{\boldsymbol{y}})+\mathcal{R}({\hat{\boldsymbol{z}}})+\alpha\mathcal{L}_{i}(o_{i},{gt}_{i}),$
(13)

where the $\mathcal{R}(\hat{\boldsymbol{y}})$ and
$\mathcal{R}({\hat{\boldsymbol{z}}})$ denote rate of the latent variables
$\hat{\boldsymbol{y}}$ and side information $\hat{\boldsymbol{z}}$,
$\mathcal{L}_{i}(\cdot,\cdot)$, $o_{i}$, and ${gt}_{i}$ denote the loss
function, output, and ground truth of the current task, respectively, and
$\alpha$ is the Lagrange multiplier to achieve the trade-off between the task
loss and bitrates. Note that in the downstream transferring, only parameters
of the information selector, task-adaptive prompts, and the task head are
learnable, while the feature extractor, feature compression model, and pre-
trained stages are all fixed, thus achieving efficient downstream task
transferring.

Thanks to compression prompts for content-weighted information selection and
task-adaptive prompt tuning, Prompt-ICM achieves both coding efficiency and
parameter efficiency for the downstream transfer to heterogeneous tasks with
only a single feature codec, resulting in a simple yet unified framework for
image coding for machines.

##  4 Experiments

###  4.1 Datasets

For training of the feature codec, we use ImageNet[21] as the training
database. As for the verification of downstream task transferring, we
experiment on four image classification datasets and two dense prediction
datasets. The four image classification datasets are CUB-200-2011[75],
Stanford Dogs[38], Stanford Cars[28], and Oxford Flowers[61], respectively.
The two datasets for dense prediction are COCO 2017[49] and Cityscapes[19].
COCO 2017 is a dataset for dense prediction tasks of object detection and
instance segmentation that contains 118K training images, 5K validation
images, and 20K test-dev images. Cityscapes is a fundamental and challenging
dataset, specifically for semantic segmentation. It has 5,000 high-quality
images with pixel-level annotations in total, with 2975 for training, 500 for
validation, and 1525 for testing, respectively.

Figure 6: Comparisons with SOTA ICM methods in terms of object detection and
instance segmentation on COCO 2017.

###  4.2 Implementation Details

Large-scale Pre-trained Backbones. We performed experiments using the Swin
Transformer[52] model pre-trained on the ImageNet-21K dataset[21]. For image
classification experiments, we used the Swin-Base model, while for dense
prediction experiments, we used the Swin-Tiny model. Additionally, we
conducted experiments using the Vision Transformer (ViT) [22], and the results
are included in the supplementary.

Controllable Feature Compression. We train the controllable feature codec for
2M iterations with a batch size of 8. Adam[39] optimizer is employed, and the
learning rate is 3e-5 and decreases to 3e-6 after 1.8M iterations. The
manually set compression prompts $\boldsymbol{m}$, i.e.,
${\boldsymbol{\Lambda}}$ in Equation (12) is uniformly sampled from [0.5, 32],
resulting in a bpp range of [0.02, 1.0] on the Kodak dataset. During the
training stage of the codec, to ensure the variety of possible compression
prompts, we randomly generate each instance in a mini-batch by using one of
the four different ways (1) a uniform map (2) a gradation map between two
randomly selected values (3) a kernel density estimation map of a Gaussian
mixture with random mean, variance, and a number of mixtures (4) a map
consisting of various blocks in a grid manner.

Downstream Transferring via Task-Driven Prompt Tuning. For compression
prompts, we take multi-scale feature aggregation as input to the information
selector by default. For task-adaptive prompts, visual prompt tuning (VPT)
[36] is instantiated for image classification, and Pro-Tuning [60] is
instantiated for dense prediction, _i.e_. object detection, instance
segmentation, and semantic segmentation. We follow the default settings in
their original papers.

###  4.3 Effectiveness and Superiority

Evaluation Protocol. We evaluate the rate-distortion performance across
various intelligent tasks. The distortion component is represented by metrics
specific to each task. The rate component is determined by bits per pixel
(bpp), which is computed as $\frac{b}{h\times w}$, where $h$ and $w$ denote
the height and width of the source image, respectively, and $b$ refers to the
total bits utilized by the coded feature bitstream.

Comparison Approaches. We mainly compare our method with the most advanced
codecs, including traditional codecs (HEVC[73], VVC[8]) and a learned-based
codec _Cheng20_ [16]. We take the results obtained by feeding uncompressed raw
images into the task model as the baseline, or said, performance upper bound.
For all subsequent evaluations of the compared approaches, reconstructed
images are fed into the task model, which has been previously trained with
uncompressed raw images, to obtain the corresponding results.

Table 1: The comparison of learnable parameters between the method of Prompt-ICM and the method of full tuning on different tasks. | Trainable
---
Parmeters (M)
| Cla.
---
(Swin-B)
| Det. & Ins.
---
(Swin-T)
| Sem.
---
(Swin-T)
Full Tuning | 87.61 | 47.49 | 59.64
Prompt-ICM | 0.87 | 25.31 | 37.46
|  |  |

Figure 7:  Ablation studies on generation of compression prompts using
features of different stages.

Figure 8: Ablation studies on generated compression prompts (GCP) based on
different datasets. “w/ GCP” represents using generated compression prompts to
guide content-weighted coding, while “w/o GCP” refers to the coding with
manual compression prompts and cannot allocate bits adaptively.

Image Classification. For training, we follow the default optimization
settings in [36]. When evaluating classification tasks, we resize and crop
each input image to $224\times 224$ before inputting them into the model. As
shown in Figure 4, the R-D performance of our method exceeds compared methods
by a significant margin. Surprisingly, our method can perform well at
extremely low bitrates (0.03$\sim$0.1 bpp). Meanwhile, as shown in Table 1,
Prompt-ICM only requires 0.87M learnable parameters to be updated during
transferring, while the full-tuning scheme requires 87.61M learnable
parameters. Last but not least, all results of Prompt-ICM are achieved by a
unified feature codec, which is critical to practical application scenarios.

Dense Prediction. For semantic segmentation on Cityscapes, we utilize
UperNet[80] implemented in mmseg[18] as the base framework. AdamW[54]
optimizer with the learning rate of 6e-5 and a weight decay of 0.01 is
employed. We use a batch size of 16 for 80K training iterations with the crop
size of 512${\times}$512\. For object detection and instance segmentation
tasks on COCO 2017, Mask R-CNN[31] with FPN[85] is utilized as the detector
implemented in mmdet[11]. We follow the common protocol that the image scale
is in [800, 1333] pixels during both the training and inference stages by
default. AdamW[54] optimizer (initial learning rate of 1e-4, weight decay of
0.05, and batch size of 16) is used with 12 epochs. As shown in Figure 5,
Prompt-ICM outperforms all other methods on the three dense prediction tasks.
Thanks to the conditional compression prompts that help with the content-
weighted compression process, Prompt-ICM can allocate more bits to those task-
related regions for the dense prediction tasks, which is further confirmed in
Section 4.5. Meanwhile, as shown in Table 1, the required numbers of learnable
parameters for object detection and instance segmentation, and semantic
segmentation are 25.31M and 37.46M, while those of full tuning are 47.49M and
59.64M. It shows that our task-adaptive prompts enable the proposed Prompt-ICM
framework to achieve significant parameter savings when transferring to dense
prediction tasks.

Figure 9:  Ablation studies on task-adaptive prompts (TAP). “w/o TAP”
corresponds to fully tuning all parameters of pre-trained backbones and the
task head on the decoder side.

Figure 10:  Visualisation results of compression prompts on different tasks,
including image classification, object detection, and semantic segmentation.
Positions with higher brightness in the compression prompts mean that they are
more important.

Comparison with SOTA ICM Methods. We compare our proposed method with Omni-
ICM[26] and preprocessing scheme [55], which also only use a single codec for
completing ICM tasks. As shown in Figure 6, our method achieves the best
performance and significantly outperforms others with much fewer learnable
parameters. Additional comparisons on other datasets are reported in the
supplementary due to limited space.

###  4.4 Ablation Study

####  4.4.1 Study on Compression Prompts

Generation of Compression Prompts. We study the effect of various combinations
of features for generating compression prompts on the classification of the
CUB-200-2011 dataset. As illustrated in Figure 7, it can be inferred that the
inclusion of all features from stage 1 to stage 3 results in the most superior
performance. This observation implies that information derived from multiple
semantic levels is advantageous for localizing importance. Moreover, the
results indicate that the performance remains relatively stable regardless of
the feature combination employed. Thus, the choice of a particular combination
should be based on the available computing power and specific requirements to
balance the trade-off between performance and complexity.

Content-weighted Feature Compression. To verify that our generated compression
prompts adaptively conduct content-weighted feature compression for a specific
task, we compare our method to the scheme without using the generated
compression prompts. By manually setting the compression prompts to a value
between 0 and 1 instead of the compression prompts generated by the
information selector, we can implement a codec without information selection.
Figure 8 shows that our generated compression prompts lead to a significant
improvement in rate-distortion performance on different datasets.

####  4.4.2 Study on Task-adaptive Prompts

Figure 9 further presents the ablation study about task-adaptive prompts.
Combined with the study on learnable parameters shown in Table 1, we can infer
that task-adaptive prompt tuning achieves even better performances than the
scheme of full tuning (w/o TAP), while tuning the task-adaptive prompts only
needs a few parameters (0.87M for task-adaptive prompt tuning vs. 87.61M for
full tuning) to be updated during downstream transferring. These findings
further provide evidence that our proposed Prompt-ICM approach possesses
excellent properties of both coding efficiency and parameter efficiency.

###  4.5 Vision Analysis and Insights

We visualize the compression prompts for different tasks as shown in Figure
10. It can be inferred that compression prompts are mainly concentrated on
objects and edges that are closely related to the current task. During the
compression process, compression prompts instruct the codec to allocate more
bits to those important regions and fewer bits to less important ones. More
specifically, for the classification task, heads of dogs and birds are more
critical to classification results, while human legs, regions of road, trees,
and the sea are unimportant to task inference. This phenomenon is reasonable
and intuitive. For dense prediction tasks, including semantic segmentation and
object detection tasks, the importance is broader and more concentrated on the
boundaries of objects, which are essential to precise localization and
identification. By jointly observing the visualization results of different
tasks, it can be inferred that the information selector pays different degrees
of attention to tasks of different granularities, pays more attention to
discriminative patterns for image-level tasks, and pays more attention to
local details for dense prediction tasks.

##  5 Conclusion

We present Prompt-ICM, a unified framework that makes use of large-scale pre-
trained models to support a variety of downstream intelligent tasks. By
introducing compression prompts to guide feature compression and task-adaptive
prompts for compressed feature tuning, Prompt-ICM can well transfer to
different intelligent tasks based on only one feature codec. Our experiments
demonstrate the significant superiority of our framework in a wide range of
vision-intelligent tasks.

## References

  * [1] Mohammad Akbari, Jie Liang, and Jingning Han.  Dsslic: Deep semantic segmentation-based layered image compression.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2042–2046. IEEE, 2019.
  * [2] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky.  Neural codes for image retrieval.  In ECCV, pages 584–599. Springer, 2014.
  * [3] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  TPAMI, 39(12):2481–2495, 2017.
  * [4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Exploring visual prompts for adapting large-scale models.  arXiv preprint arXiv:2203.17274, 1(3):4, 2022.
  * [5] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  End-to-end optimized image compression.  In ICLR, 2017.
  * [6] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston.  Variational image compression with a scale hyperprior.  In ICLR, 2018.
  * [7] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In ICCV, pages 9157–9166, 2019.
  * [8] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm.  Overview of the versatile video coding (vvc) standard and its applications.  TCSVT, 2021.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 33:1877–1901, 2020.
  * [10] Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides.  Conv-adapter: Exploring parameter efficient transfer learning for convnets.  arXiv preprint arXiv:2208.07463, 2022.
  * [11] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [12] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  TPAMI, 40(4):834–848, 2017.
  * [13] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, pages 801–818, 2018.
  * [14] Zhuo Chen, Kui Fan, Shiqi Wang, Lingyu Duan, Weisi Lin, and Alex Chichung Kot.  Toward intelligent sensing: Intermediate deep feature compression.  TIP, 29:2230–2243, 2019.
  * [15] Zhuo Chen, Kui Fan, Shiqi Wang, Ling-Yu Duan, Weisi Lin, and Alex Kot.  Lossy intermediate deep learning feature compression and evaluation.  In ACM MM, pages 2414–2422, 2019.
  * [16] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto.  Learned image compression with discretized gaussian mixture likelihoods and attention modules.  In CVPR, pages 7939–7948, 2020.
  * [17] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee.  Variable rate deep image compression with a conditional autoencoder.  In ICCV, pages 3146–3154, 2019.
  * [18] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [19] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, pages 3213–3223, 2016.
  * [20] Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, and Yihui Feng.  G-vae: A continuously variable rate deep image compression framework.  arXiv preprint arXiv:2003.02012, 2020.
  * [21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. Ieee, 2009.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2020.
  * [23] Lingyu Duan, Jiaying Liu, Wenhan Yang, Tiejun Huang, and Wen Gao.  Video coding for machines: A paradigm of collaborative compression and intelligent analytics.  TIP, 29:8680–8695, 2020.
  * [24] Ling-Yu Duan, Vijay Chandrasekhar, Jie Chen, Jie Lin, Zhe Wang, Tiejun Huang, Bernd Girod, and Wen Gao.  Overview of the mpeg-cdvs standard.  TIP, 25(1):179–194, 2015.
  * [25] Ling-Yu Duan, Yihang Lou, Yan Bai, Tiejun Huang, Wen Gao, Vijay Chandrasekhar, Jie Lin, Shiqi Wang, and Alex Chichung Kot.  Compact descriptors for video analysis: The emerging mpeg standard.  IEEE MultiMedia, 26(2):44–54, 2018.
  * [26] Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen.  Image coding for machines with omnipotent feature learning.  In ECCV, pages 510–528. Springer, 2022.
  * [27] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  arXiv preprint arXiv:2110.04544, 2021.
  * [28] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei.  Fine-grained car detection for visual census estimation.  In AAAI, volume 31, 2017.
  * [29] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  NeurIPS, 34:15908–15919, 2021.
  * [30] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  ICLR, 2021.
  * [31] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [33] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, pages 2790–2799. PMLR, 2019.
  * [34] Yueyu Hu, Shuai Yang, Wenhan Yang, Ling-Yu Duan, and Jiaying Liu.  Towards coding for human and machine vision: A scalable image coding approach.  In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2020.
  * [35] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, pages 4904–4916. PMLR, 2021.
  * [36] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  ECCV, 2022.
  * [37] Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici.  Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks.  In CVPR, pages 4385–4393, 2018.
  * [38] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR workshop on fine-grained visual categorization (FGVC), volume 2. Citeseer, 2011.
  * [39] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [40] Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youvalari, and Esa Rahtu.  Image coding for machines: An end-to-end learned approach.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1590–1594. IEEE, 2021.
  * [41] Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youvalari, Hamed Rezazadegan Tavakoli, and Esa Rahtu.  Learned image coding for machines: A content-adaptive approach.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2021.
  * [42] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [43] Mu Li, Wangmeng Zuo, Shuhang Gu, Jane You, and David Zhang.  Learning content-weighted deep image compression.  TPAMI, 2020.
  * [44] Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao, and David Zhang.  Learning convolutional networks for content-weighted image compression.  In CVPR, pages 3214–3223, 2018.
  * [45] Xin Li, Jun Shi, and Zhibo Chen.  Task-driven semantic coding via reinforcement learning.  TIP, 2021.
  * [46] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190, 2021.
  * [47] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  ECCV, 2022.
  * [48] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [49] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [50] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  arXiv preprint arXiv:2107.13586, 2021.
  * [51] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In CVPR, pages 8759–8768, 2018.
  * [52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, pages 10012–10022, 2021.
  * [53] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, pages 3431–3440, 2015.
  * [54] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [55] Guo Lu, Xingtong Ge, Tianxiong Zhong, Jing Geng, and Qiang Hu.  Preprocessing enhanced image compression for machine vision.  arXiv preprint arXiv:2206.05650, 2022.
  * [56] Siwei Ma, Xiang Zhang, Shiqi Wang, Xinfeng Zhang, Chuanmin Jia, and Shanshe Wang.  Joint feature and texture coding: Toward smart video representation via front-end intelligence.  TCSVT, 29(10):3095–3105, 2018.
  * [57] Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool.  Conditional probability models for deep image compression.  In CVPR, pages 4394–4402, 2018.
  * [58] Fabian Mentzer, George Toderici, Michael Tschannen, and Eirikur Agustsson.  High-fidelity generative image compression.  arXiv preprint arXiv:2006.09965, 2020.
  * [59] David Minnen, Johannes Ballé, and George Toderici.  Joint autoregressive and hierarchical priors for learned image compression.  In NeurIPS, 2018.
  * [60] Xing Nie, Bolin Ni, Jianlong Chang, Gaomeng Meng, Chunlei Huo, Zhaoxiang Zhang, Shiming Xiang, Qi Tian, and Chunhong Pan.  Pro-tuning: Unified prompt tuning for vision tasks.  arXiv preprint arXiv:2207.14381, 2022.
  * [61] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [62] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  Adapterhub: A framework for adapting transformers.  arXiv preprint arXiv:2007.07779, 2020.
  * [63] Majid Rabbani and Rajan Joshi.  An overview of the jpeg 2000 still image compression standard.  Signal processing: Image communication, 17(1):3–48, 2002.
  * [64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763. PMLR, 2021.
  * [65] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [66] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [67] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, pages 779–788, 2016.
  * [68] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In CVPR, pages 7263–7271, 2017.
  * [69] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 28:91–99, 2015.
  * [70] Jorma Rissanen and Glen Langdon.  Universal modeling and coding.  IEEE Transactions on Information Theory, 27(1):12–23, 1981.
  * [71] Saurabh Singh, Sami Abu-El-Haija, Nick Johnston, Johannes Ballé, Abhinav Shrivastava, and George Toderici.  End-to-end learning of compressible features.  In 2020 IEEE International Conference on Image Processing (ICIP), pages 3349–3353. IEEE, 2020.
  * [72] Myungseo Song, Jinyoung Choi, and Bohyung Han.  Variable-rate deep image compression through spatially-adaptive feature transform.  In ICCV, pages 2380–2389, 2021.
  * [73] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.  Overview of the high efficiency video coding (hevc) standard.  TCSVT, 22(12):1649–1668, 2012.
  * [74] George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell.  Full resolution image compression with recurrent neural networks.  In CVPR, pages 5306–5314, 2017.
  * [75] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [76] Gregory K Wallace.  The jpeg still picture compression standard.  IEEE transactions on consumer electronics, 38(1):xviii–xxxiv, 1992\.
  * [77] Shurun Wang, Shiqi Wang, Wenhan Yang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao.  Towards analysis-friendly face representation with scalable feature and texture compression.  TMM, 2021.
  * [78] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  TIP, 13(4):600–612, 2004.
  * [79] Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra.  Overview of the h. 264/avc video coding standard.  TCSVT, 13(7):560–576, 2003.
  * [80] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In Proceedings of the European conference on computer vision (ECCV), pages 418–434, 2018.
  * [81] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  NeurIPS, 34:12077–12090, 2021.
  * [82] Fei Yang, Luis Herranz, Joost Van De Weijer, José A Iglesias Guitián, Antonio M López, and Mikhail G Mozerov.  Variable rate deep image compression with modulated autoencoder.  IEEE Signal Processing Letters, 27:331–335, 2020.
  * [83] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  arXiv preprint arXiv:2111.03930, 2021.
  * [84] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595, 2018.
  * [85] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, pages 2881–2890, 2017.
  * [86] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, pages 6881–6890, 2021.
  * [87] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, pages 16816–16825, 2022.
  * [88] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  IJCV, 130(9):2337–2348, 2022.

##  Appendix A Controllable Feature Compression

Figure 11: The architecture of our spatially variable-rate feature compression
network.

Spatially Variable-Rate Feature Compression. As shown in Figure 11, the
feature $\boldsymbol{f}$ is input to the encoder $g_{a}$ and the corresponding
compression prompt $\boldsymbol{m}$ is input to the condition network $q_{a}$,
obtaining the latent variable $\boldsymbol{y}$. Then quantization is
performed. The process can be written by:

$\begin{split}\boldsymbol{y}=g_{a}(\boldsymbol{f},{\Psi}_{a}),&\quad\text{where}\quad{\Psi}_{a}=q_{a}(\boldsymbol{m}),\\\
&\hat{\boldsymbol{y}}=Q(\boldsymbol{y}),\end{split}$ (14)

where the $\boldsymbol{m}$ denotes compression prompts, and $Q$ denotes the
quantizer.

Since the hard rounding quantization operation of $Q$ is non-differential, an
additive noise alters quantization [5] during training. As for the inference
stage, after real round-based quantization, entropy coding techniques (e.g.,
Huffman coding and arithmetic coding[70]) can losslessly compress the
quantized discrete latent variable $\hat{\boldsymbol{y}}$ if the probability
distribution
$p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})$
is given. And we use $\hat{\boldsymbol{y}}$ to denote both
$\hat{\boldsymbol{y}}$ of the hard quantized latent variable and
$\tilde{\boldsymbol{y}}$ of the noised latent variable for simplicity.

Next, the latent variable $\boldsymbol{y}$ is input into the hyper-encoder
$h_{a}$ and the quantizer $Q$, obtaining the side-information. This process is
formulated as:

$\begin{split}\boldsymbol{z}&=h_{a}(\boldsymbol{y}),\\\
\hat{\boldsymbol{z}}&=Q(\boldsymbol{z}).\end{split}$ (15)

Additive noise is also performed for $\boldsymbol{z}$ during training as an
alternative of real round-based quantization for differentiability. And
entropy estimation of $\hat{\boldsymbol{z}}$ is performed by a learned
factorized entropy prior $\boldsymbol{\psi}$, formulated as:

$\displaystyle
p_{\hat{\boldsymbol{z}}|{\boldsymbol{\psi}}}(\hat{\boldsymbol{z}}|{\boldsymbol{\psi}})=\prod_{i}(p_{z_{i}|{\boldsymbol{\psi}}}({\boldsymbol{\psi}})\ast\mathcal{U}(-\frac{1}{2},\frac{1}{2}))({\hat{z}}_{i}),$
(16)

where $z_{i}$ denotes the $i$-th element of $\boldsymbol{z}$, and $i$
specifies to the position of each signal.

Then the side-information $\hat{\boldsymbol{z}}$ contains both the hyper prior
for estimating probability distributions of latent variable $\boldsymbol{y}$
and the conditioned information. It is then fed into the hyper-decoder $h_{s}$
and the condition generator $q_{g}$, which can be written as:

$\begin{split}p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})&\leftarrow
h_{s}(\hat{\boldsymbol{z}}),\\\
\boldsymbol{w}&=q_{g}(\hat{\boldsymbol{z}}),\end{split}$ (17)

where
$p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})$
denotes the estimated distribution conditioned on $\hat{\boldsymbol{z}}$ and
$\boldsymbol{w}$ represents the spatial conditioned information for feature
reconstruction. More specifically, the conditional probability distribution
$p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})$
after decoding $\hat{\boldsymbol{z}}$ is modeled by a mean and scale Gaussian
distribution, which is:

$p_{\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}}}(\hat{\boldsymbol{y}}|\hat{\boldsymbol{z}})\sim{\mathcal{N}(\boldsymbol{{\mu}},{\boldsymbol{\sigma}}^{2})}.$
(18)

For feature reconstruction, the decoder $g_{s}$ and condition network $q_{s}$
operates on the latent variable $\hat{\boldsymbol{y}}$ and spatial conditional
information $\boldsymbol{w}$, which can be formulated as:

$\hat{\boldsymbol{f}}=g_{s}(\hat{\boldsymbol{y}},{\Psi}_{s}),\quad\text{where}\quad{\Psi}_{s}=q_{s}(\boldsymbol{w}).$
(19)

Implementation Details. As illustrated in Figure 11, we design the compression
framework derived from the Mean & Scale (M&S) Hyperprior model [59] and
spatial variable-rate image compression [72]. Residual blocks are used to
increase the receptive field and representation capability[16]. Besides, the
spatial feature transform (SFT) blocks and spatial feature transform residual
block (SFT Resblk) shown in Figure 11 are derived from [72] to modulate
features during non-linear transform process.

##  Appendix B Feature Aggregation Information Selector

The goal of the information selector is to generate compression prompts that
adaptively assign importance factors to each location condition on task
requirements and feature contents. Since we choose to use the features
extracted at stage 1 of Swin Transformer as general features, its network
depth is shallow, so the features contain less semantic information.
Naturally, we can take use of multi-scale features that contain both detailed
spatial layout information and high-level semantic information as input of the
information selector to generate more proper compression prompts. Figure 12
illustrates the architecture of the information selector with feature
aggregation. Experimental results demonstrate the effectiveness of our
proposed simple and lightweight information selector with feature aggregation.

Figure 12:  The architecture of information selector with feature aggregation.

##  Appendix C Comparison with SOTA ICM Methods

Figure 13:  Comparison with Omni-ICM[26] on CUB-200-2011[75].  Figure 14:
Results based on CLIP-Adapter on Stanford-Car.  Figure 15:  Examples of manual
compression prompts.

Figure 16:  Classification results on different datasets under various
bitrates by using ViT-B as the backbone.

Figure 17:  Visualization of compression prompts by ViT-based Prompt-ICM on
different datasets. From left to right, the corresponding datasets are
CUB-200-2011, Stanford Cars, Stanford Dogs, and Oxford Flowers. Best viewed in
color.

In addition to the comparison with dense prediction tasks, we compare our
proposed method with Omni-ICM [26] on CUB-200-2011[75] fine-grained
classification task. Our Prompt-ICM can achieve far superior performance than
Omni-ICM, which shows that our framework has robustness on both image
classification and dense prediction tasks. We infer that the main reason for
the failure of Omni-ICM is because the features learned by contrastive
learning in [26] cannot transfer well to fine-grained classification tasks.
However, our framework uses more general features, and the proposed task-
driven prompts can help us better transfer to downstream tasks, thus obtaining
a satisfying performance.

##  Appendix D Extension of Task-Adaptive Prompts

Note that the the task-adaptive prompts in Prompt-ICM is not constrained to
any specific prompt tuning techniques. In the main text, we choose VPT[36] and
Pro-Tuning[60] as our instantiation choices. Additionally, we have also
instantiated the task-adaptive prompts with CLIP-Adapter [27] to exhibit the
versatility of our framework. As illustrated in Figure 14, our Prompt-ICM
framework retains its superiority over other methods, which demonstrate the
adaptability and compatibility of our framework.

##  Appendix E Manual Compression Prompts

The manual prompts mentioned in Section 4.2 of the main text are visualized in
Figure 15. Specifically, during the training stage of the codec, to ensure the
variety of possible compression prompts, we randomly generate each instance in
a mini-batch by using one of the four different ways (1) a uniform map (2) a
gradation map between two randomly selected values (3) a kernel density
estimation map of a Gaussian mixture with random mean, variance, and a number
of mixtures (4) a map consisting of various blocks in a grid manner. During
the inference stage, to achieve each point in the rate-distortion curve
discussed in Section 4.4.1, we employ uniform maps with a range of [0, 1] as
compression prompts, which were set manually.

##  Appendix F Extension to ViT

Feature Extraction and Feature Compression. We also extend our Prompt-ICM to
Vision Transformers (ViT) [22]. More specifically, with a normal ViT
consisting of 12 self-attention blocks, we take features extracted at block 6
as the general features. At the downstream transferring stage, the extracted
features are fed into the information selector to generate the compression
prompts. Then, features and compression prompts are input into the feature
compression model. The architecture of the compression model for ViT is almost
the same as that of Swin. The only difference is that the stride of all
convolutions whose original stride is not 1 is changed to 1, since the ViT
features are already 16x down-sampling. As for task-adaptive prompts, we
follow the visual prompt tuning (VPT)[36] and take experiments to evaluate the
effectiveness on image classification. We conduct experiments on four image
classification datasets including CUB-200-2011[75], Stanford Cars[28],
Stanford Dogs[38], and Oxford Flowers[61].

Experimental Results. As shown in Figure 16, Prompt-ICM can extend well to
normal Vision Transformer architecture and substantially outperforms the
compared methods. It can be inferred that Prompt-ICM are not limited to a
certain backbone, and can achieve excellent performance.

Compression Prompts Visualization. For the ViT-based model, Figure 17 shows
the compression prompts of the four datasets. As we can see, essential
patterns for recognition are allocated by more importance. For example, the
heads of birds, cars and dogs are more important to distinguish the image
compared to other patterns, while the flower bud is regarded as the key
information for flower classification.

##  Appendix G Limitation

The current state of development in parameter efficient tuning (PET)
techniques constrains the upper boundary of Prompt-ICM’s performance for dense
prediction tasks. Nevertheless, as the PET field advances, this limitation is
anticipated to be mitigated.