# SEMANTIC SEGMENTATION IN LEARNED COMPRESSED DOMAIN

###### Abstract

Most machine vision tasks (e.g., semantic segmentation) are based on images
encoded and decoded by image compression algorithms (e.g., JPEG). However,
these decoded images in the pixel domain introduce distortion, and they are
optimized for human perception, making the performance of machine vision tasks
suboptimal. In this paper, we propose a method based on the compressed domain
to improve segmentation tasks. i) A dynamic and a static channel selection
method are proposed to reduce the redundancy of compressed representations
that are obtained by encoding. ii) Two different transform modules are
explored and analyzed to help the compressed representation be transformed as
the features in the segmentation network. The experimental results show that
we can save up to 15.8% bitrates compared with a state-of-the-art compressed
domain-based work while saving up to about 83.6% bitrates and 44.8% inference
time compared with the pixel domain-based method.

Index Terms—  Image compression, Compressed domain, Channel selection, Deep
learning.

##  1 Introduction

With the development of deep learning, machine vision tasks have been widely
implemented in the past few years and have played important roles in our
lives. For example, semantic segmentation is applied to medical image
analysis, autonomous vehicles, video surveillance, augmented reality, and many
other areas [1]. However, many machine vision tasks are usually based on
decoded images that need image compression.The distortion of decoded images
deteriorate the performance of machine vision tasks.

Classical image compression algorithms, such as, JPEG [2], and
HEVC/H.265-intra [3] have achieved good Rate-distortion performance in human
perception while some learned end-to-end image compression methods [4, 5] also
achieved comparable performance. However, all of them are designed to reduce
the distortion which is perceived by human and has a gap with machine
perception. Therefore, current algorithms usually lead to suboptimal semantic
segmentation.

Some methods based on the compressed domain are proposed to solve these
problems. Pixel domain-based methods mean we need to input the images encoded
and decoded by an image compression algorithm to the semantic segmentation
network, as Fig. 1 shows. Different from that, compressed domain-based methods
mean we directly input the representations in the compressed domain into the
semantic segmentation network without the computation cost for the decoded
image. A related standard named JPEG AI [6] had been proposed. Torfason $et\
al.$ [7] had proved that the compressed domain-based method could achieve a
better performance than the pixel domain-based method, and made some
experiments on image classification and object detection. Choi $et\ al.$ [8]
further verified this by using an information theory method and proposed a
compression domain-based approach which is verified on object detection and
semantic segmentation. Bai $et\ al.$ [9] utilized a transformer to improve
image classification based on compressed domain. Wang $et\ al.$ [10] proposed
a channel selection method and an entropy estimation method to improve the
results in [7]. However, the channels selection method can only delete the
channels with the lowest entropy since the selection method is based on the
variance of the compressed representations.

Fig. 1: The pixel domain-based method.

In this paper, we propose a method based on the compressed domain to improve
segmentation tasks. The main contributions of this paper are as follows:

  * •

Based on a learned gate module, a dynamic and a static channel selection
method are proposed to select suitable channels and reduce the redundancy of
compressed representations.

  * •

We explore and analyze two different modules transforming the representation
in the compressed domain to the feature that comes from the mid-layer in the
segmentation network.

  * •

We perform experiments based on Cityscapes, and the results show that, in the
same mIoU cases, we can save up to 15.8% bitrates compared with a state-of-
the-art compressed domain-based work while saving up to about 83.6% bitrates
and 44.8% inference time compared with the pixel domain-based method.

##  2 Method

Fig. 2: The framework of our method (The knowledge distillation path is
firstly trained by original images individually. Both the weights of this path
and the pretrained weights of the compression network are frozen during the
training.).

###  2.1 Framework

The framework of our method is shown in Fig. 2. It consists of five modules:
an image compression network, a gate module, a transform module, a
segmentation network, and a knowledge distillation path. The hyperprior model
in [4] is set as the image compression network whose the pre-trained weights
are provided by CompressAI [11]. In the following steps, the weights of this
compression network are frozen so that our method will not affect the quality
(e.g, PSNR) of the decoded images.

For semantic segmentation network, we choose DeepLab v3 [12] whose first two
layers are removed to conform to the size of the compressed representation.
And then, to gain a better performance, a knowledge distillation method is
utilized. The teacher network of knowledge distillation is shown in the
knowledge distillation path that contains the first two layers of DeepLab v3,
which are pretrained by raw images, and are frozen after that. The student
network is the all networks above the knowledge distillation path.

We firstly input the raw image $x$ to the encoder of the compression network
to get a compressed representation $\hat{y}$. For the pixel domain-based
method, we need to transmit this representation $\hat{y}$ to the decoder to
get the decoded image $\hat{x}$, and input the $\hat{x}$ to the semantic
segmentation network to complete segmentation. But for our compressed domain-
based method, to get a machine vision task result, we do not need the decoder
to get $\hat{x}$, and just input $\hat{y}$ to the gate module to get a
selected representation $\hat{y}_{selected}$.

And then, the transform module is used for transforming $\hat{y}_{selected}$
to the feature $\mathcal{F}$ for the segmentation network. The feature is
further input to semantic segmentation network to get the segmentation map
$\mathcal{M}$. A cross-entropy loss $\mathcal{L}_{seg}$ of $\mathcal{M}$ and
ground truth $\mathcal{M}_{gt}$ is calculated for updating the network. We can
express it as:

$\mathcal{L}_{seg}=\mathbb{SG}(CrossEntropy(\mathcal{M},\mathcal{M}_{gt});\phi)$
(1)

where $\mathbb{SG}$ is the stop-gradient operator since the compression
network parameters $\phi$ are not updated during the training.

To further improve the performance, another knowledge distillation loss
$\mathcal{L}_{KD}$ is calculated by the knowledge distillation path. We input
$x$ into the frozen first two layers of the semantic segmentation network to
get the $\mathcal{F}_{gt}$. Then we calculate the mean square error of
$\mathcal{F}_{gt}$ and $\mathcal{F}$ as $\mathcal{L}_{KD}$, which can be
expressed as:

$\mathcal{L}_{KD}=\mathbb{SG}(MSE(\mathcal{F},\mathcal{F}_{gt});\psi)$ (2)

where $\psi$ means the weights of the knowledge distillation path, and these
weights are frozen during the training.

At last, the total loss $\mathcal{L}$ can be fomulated as:

$\mathcal{L}=\mathcal{L}_{seg}+\lambda_{1}\mathcal{L}_{KD}$ (3)

where $\lambda_{1}$ is used for controlling the weights between
$\mathcal{L}_{KD}$ and $\mathcal{L}_{seg}$, here we set it as 1 based on
auxiliary experiments.

On the other hand, by continuing to transmit the residuals between $\hat{y}$
and $\hat{y}_{selected}$, we can also get a decoded image.

###  2.2 Channel Selection

Previous works [8, 10, 13] have shown that there are some channels in the
compressed representations that are useless for machine vision tasks. Here we
propose a gate module to adaptively learn which channel is suitable for the
semantic segmentation task. On this basis, dynamic selection and static
selection methods are proposed.

Fig. 3: The framework of the gate module.

(a) (a) Module-Deconv

(b) (b) Module-Res

Fig. 4: The two different transform modules.

Fig. 5: The results of training different numbers of channels of compressed
domain based on Module-Deconv (Ours(D) means the dynamic method, while
Ours(S-128/64/32/16) means the static method using 128/64/32/16 channels
respectively, Wang-128/64/32/16 means we select 128/64/32/16 channels by using
the method in [10], ”No Gate” means the compressed domain-based method without
using any channel selection method, ”Pixel” means the pixel domain-based
method).

####  2.2.1 Gate Module

The gate is designed as Fig. 3 shows. When we get the compressed
representation $\hat{y}$, we input it to this gate module to get the selected
representation $\hat{y}_{selected}$. Because in the process of channel
selection, we need to sample channels, which may make the network non-
differentiable. Therefore, similar to [14], the gumbel-softmax layer is
introduced for making the entire gate module differentiable.

####  2.2.2 Dynamic Selection

Inputting the representation $\hat{y}$ directly into the gate module allows us
to select channels dynamically. For example, we assume the size of $\hat{y}$
is $H\times W\times C$. After going through the pooling layer and the two
convolutional layers, we have a tensor of size $1\times 1\times 2C$ where the
first $C$ number represents the probabilities that $C$ channels of compressed
representation $\hat{y}$ is sampled as 0 respectively, while the latter C
numbers mean the probabilities that the $C$ channels are sampled as 1. With
the final Gumbel-Softmax layer, we can get a tensor of size of $1\times
1\times C$, where each element has a value of 0 or 1, indicating that we do
not select or select the corresponding channel. Finally, we multiply this
tensor by $\hat{y}$ to get $\hat{y}_{selected}$. The overall flow is shown in
Fig. 3.

####  2.2.3 Static Selection

However, dynamic selection will make each image select different channels,
which may lead to the instability of training. Therefore, on the basis of
dynamic selection, we further propose a static selection method. After the
training of the whole network according to the method of dynamic selection, we
use all images in the training dataset to make inferences and count the number
of times each channel is dynamically selected. And then, we select the $N$
channels selected most frequently in this statistic. At last, we select these
channels manually instead of using the gate module to get
$\hat{y}_{selected}$.

###  2.3 Transform Module

After getting $\hat{y}_{selected}$, we input it to a transform module, because
the size of the compressed representation is different from that of the
feature in the segmentation network, and the compressed domain has a gap with
the domain in the segmentation network. To effectively bridge these gaps, and
convert the compressed representation $\hat{y}_{selected}$ into the feature
$\mathcal{F}$ for segmentation. We tested two different transform modules.

Refer to [4], we firstly design the Module-Deconv which contains two transpose
convolutional layers as Fig. 4(a) shows, that is also used in [13]. To improve
performance, according to [5], Module-Res is designed as Fig. 4(b) shows. The
residual block and PixelShuffle are used for improving performance.

##  3 Experiments

###  3.1 Experiment Setting

For the image compression network, we use the hyperprior model in [4]. For
semantic segmentation network, we use DeepLab v3 [12]. The Cityscapes dataset
[15] is used as the training set and the test set. Adam is chosen as the
optimizer. We totally train this network by 400 epochs while the initial
learning rate is set as 0.001 and drops to 1e-4 after 300 epochs. All
experiments are based on a GeForce RTX 3090 GPU with 24 GB RAM.

Table 1: The comparison between the compressed domain-based method without channel selection and the pixel domain-based method in the same bitrate cases. Bitrate(/bpp) | 0.084 | 0.111 | 0.154 | 0.216
---|---|---|---|---
| mIoU(Pixel
---
domain-based)
0.423 | 0.514 | 0.579 | 0.648
| mIoU(Compressed
---
domain-based)
| 0.645
---
(52.5%$\uparrow$)
| 0.673
---
(30.9%$\uparrow$)
| 0.693
---
(19.7%$\uparrow$)
| 0.702
---
(8.3%$\uparrow$)

###  3.2 Rate-mIou Comparison

We test dynamic and static selection methods based on Module-Deconv in this
part. In the experiments of the static selection method, we test the cases
when the number of selected channels is 128, 64, 32, and 16, respectively.

To prove that redundancy does exist in the compressed representation, we
compare our method with the case without using any channel selection method.
As the Fig. 5 shows, compared with that case, we can save up to 31.8%
bitrates.

Fig. 6: The comparison of various channels.  Table 2: The performance and MACs (Multiply-Add Cumulation) of various transform modules. Module Name |  |  Module-Deconv [13]
---
| Module-Res
---
MACs(/G) | 94 | 202
mIoU(0.067 bpp) | 0.6502 | 0.6534
mIoU(0.104 bpp) | 0.6785 | 0.6836

To verify the effectiveness of our method, we firstly compare our method with
the pixel domain-based method whose compression network is also [4]. As Table
1 shows, our compressed domain-based method is superior to the pixel domain-
based method. In addition, we compare our method with a state-of-the-art
compressed domain-based method [10]. Here, we reproduce [10] by using the same
image compression network in our method. As Fig. 5 shows, regardless of the
number of channels selected, our static method always outperforms [10]. For
example, if the number of selected channels is 32, we can save up to 15.8%
bitrates compared with [10] and save up to about 83.6% compared with the pixel
domain-based method. This shows that our method can select more appropriate
channels for segmentation tasks. The static method has a better rate-mIoU when
the selected channels are less than 128, which suggests that static method is
superior to the dynamic method.

###  3.3 The Rate-PSNR Comparsion

Since the weights of image compression network are frozen during training
process, the rate-PSNR is the same as the results of [4]. Also, we reproduce
the channel selection method [10] by using the same image compression network
as ours, which means that we have the same rate-PSNR performance.

###  3.4 The Effect of Various Channels

In Fig. 6, we test the cases of various channels. As we can see, compared with
selecting 64 or 128 channels, selecting 16 or 32 channels can get better rate-
mIou which means we reduced the redundancy in the compressed representation.

###  3.5 The Effect of Various Transform Modules

In this paper, we explore two different transform modules. As Table 2 shows,
Module-Res can get a better rate-mIoU in both bitrates while Module-Deconv
perform slightly worse but with half the complexity compared with Module-Res.

###  3.6 Inference Time

To check the efficiency of our method, we test the inference time based on
CityScapes dataset, as Table. 3 shows. ”No gate” means the compressed domain-
based method without using any channel selection method. As we can see,
compared with the pixel domain-based method, compressed domain-based methods
can save a lot of time. Moreover, with dynamic selection or static selection
we can save much more time, especially when selected channels are 16, we save
about 44.8% of the time.

Table 3: The inference time for semantic segmentation based on CityScapes dataset. Model |  | Encoding
---
time(/s)
| Decoding +
---
task time(/s)
Pixel Domain | 0.142 | 0.320
Ours(Dynamic Selection) | 0.090 | 0.217
Ours (Static Selection) | 16 channels | 0.060 | 0.195
32 channels | 0.067 | 0.198
64 channels | 0.082 | 0.208
128 channels | 0.111 | 0.226
No gate | 0.139 | 0.247

###  3.7 Scalability for Multi-tasks

Since the weights of image compression network are frozen, the scheme is
scalable for multi-tasks. We explore the potential scalability by testing
object detection with our dynamic selection method based on PASCAL VOC 07
dataset. Faster-rcnn [16] is used as task network. The results show that we
save up to about 30% bitrates to achieve similar average precision compared
with the pixel domain-based method. It’s our future work to further explore
the scalability.

##  4 Conclusions

In this paper, we propose a method based on the compressed domain to improve
segmentation tasks. We firstly proposed a gate module to learn adaptively
which channels in the compressed representations are suitable for segmentation
tasks. After that, the static selection of a specified number of channels and
dynamic selection based on the gate module are studied. These two kinds of
selection help us successfully reduce the redundancy of the compressed
representations. And then, two different transform modules are introduced for
transforming the representation in the compressed domain to the feature in the
segmentation network. At last, we do experiments based on the Cityscapes
dataset, and the results of our experiments prove that our model can achieve
better rate-mIoU and faster inference compared with the previous work.

## References

  * [1] Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos,  “Image segmentation using deep learning: A survey,”  IEEE transactions on pattern analysis and machine intelligence, 2021\.
  * [2] Gregory K Wallace,  “The jpeg still picture compression standard,”  IEEE transactions on consumer electronics, vol. 38, no. 1, pp. xviii–xxxiv, 1992.
  * [3] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand,  “Overview of the high efficiency video coding (hevc) standard,”  IEEE Transactions on circuits and systems for video technology, vol. 22, no. 12, pp. 1649–1668, 2012.
  * [4] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston,  “Variational image compression with a scale hyperprior,”  in International Conference on Learning Representations, 2018.
  * [5] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto,  “Learned image compression with discretized gaussian mixture likelihoods and attention modules,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7939–7948.
  * [6] Joao Ascenso,  “Jpeg ai use cases and requirements,”  in ISO/IEC JTC1/SC29/WG1 M90014, 2021.
  * [7] Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool,  “Towards image understanding from deep compression without decoding,”  in International Conference on Learning Representations, 2018.
  * [8] Hyomin Choi and Ivan V Bajic,  “Scalable image coding for humans and machines,”  arXiv preprint arXiv:2107.08373, 2021.
  * [9] Yuanchao Bai, Xu Yang, Xianming Liu, Junjun Jiang, Yaowei Wang, Xiangyang Ji, and Wen Gao,  “Towards end-to-end image compression and analysis with transformers,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
  * [10] Zhenzhen Wang, Minghai Qin, and Yen-Kuang Chen,  “Learning from the cnn-based compressed domain,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 3582–3590.
  * [11] Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja,  “Compressai: a pytorch library and evaluation platform for end-to-end compression research,”  arXiv preprint arXiv:2011.03029, 2020.
  * [12] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam,  “Rethinking atrous convolution for semantic image segmentation,”  arXiv preprint arXiv:1706.05587, 2017.
  * [13] Jinming Liu, Heming Sun, and Jiro Katto,  “Learning in compressed domain for faster machine vision tasks,”  in 2021 International Conference on Visual Communications and Image Processing (VCIP). IEEE, 2021, pp. 01–05.
  * [14] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren,  “Learning in the frequency domain,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1740–1749.
  * [15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele,  “The cityscapes dataset for semantic urban scene understanding,”  in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [16] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  Advances in neural information processing systems, vol. 28, pp. 91–99, 2015.