# DNN-Compressed Domain Visual Recognition with Feature Adaptation

Yingpeng Deng Lina J. Karam
Image, Video and Usability Lab, School of ECEE, Arizona State University, USA
<EMAIL_ADDRESS>

###### Abstract

Learning-based image compression was shown to achieve a competitive
performance with state-of-the-art transform-based codecs. This motivated the
development of new learning-based visual compression standards such as JPEG-
AI. Of particular interest to these emerging standards is the development of
learning-based image compression systems targeting both humans and machines.
This paper is concerned with learning-based compression schemes whose
compressed-domain representations can be utilized to perform visual processing
and computer vision tasks directly in the compressed domain. In our work, we
adopt a learning-based compressed-domain classification framework for
performing visual recognition using the compressed-domain latent
representation at varying bit-rates. We propose a novel feature adaptation
module integrating a lightweight attention model to adaptively emphasize and
enhance the key features within the extracted channel-wise information. Also,
we design an adaptation training strategy to utilize the pretrained pixel-
domain weights. For comparison, in addition to the performance results that
are obtained using our proposed latent-based compressed-domain method, we also
present performance results using compressed but fully decoded images in the
pixel domain as well as original uncompressed images. The obtained performance
results show that our proposed compressed-domain classification model can
distinctly outperform the existing compressed-domain classification models,
and that it can also yield similar accuracy results with a much higher
computational efficiency as compared to the pixel-domain models that are
trained using fully decoded images.

##  1 Introduction

Learning-based image coding algorithms have shown a competitive compression
efficiency as compared to the conventional compression methods, by using
advanced deep learning techniques. Specifically, when compared to the
JPEG/JPEG 2000 standards, learning-based codecs can produce a higher visual
quality in terms of PSNR (peak signal-to-noise ratio) and SSIM (structural
similarity) [33] / MS-SSIM (multi-scale structural similarity) [35] for some
target compression bitrates [5, 28].

Furthermore, without fully decoding the images, learning-based codecs can be
adapted to support image processing and computer vision tasks in the
compressed-domain. This latter feature was incorporated as part of the
emerging JPEG-AI standard [26]. Figure 1 shows an example for a compressed-
domain classification task. Given an original input image $\mathbf{x}$, a
learning-based codec can encode it and quantize it to form a compressed-domain
latent representation $\mathbf{\hat{y}}$ (compressed representation). This
compressed representation $\mathbf{\hat{y}}$ can be losslessly transformed
into a bitstream via entropy encoding and then recovered back to the
compressed-domain through an entropy decoding process. The decoded image
$\mathbf{\hat{x}}$ can be reconstructed by providing the compressed
representation $\mathbf{\hat{y}}$ as input to the learning-based decoder, and
can subsequently be used for training/testing a deep neural network
(DNN)-based classifier operating in the pixel domain. Alternatively, without
the decoder, the DNN-based classification can be performed directly in the
compressed domain when targeting a machine given that the compressed
representation $\mathbf{\hat{y}}$ may contain useful information to perform
image processing and computer vision tasks in the compressed domain [2].

Figure 1: Block diagram of compressed-domain and pixel-domain classifications.

One application of interest is compressed-domain classification. Recently,
Torfason et al. [29] demonstrated that two computer vision tasks, image
classification and semantic segmentation, can achieve comparable performance
using compressed-domain representations as compared to the tasks performed
using pixel-domain decompressed images. While natural image classification can
be a common benchmark case, in this work we also consider texture recognition,
which plays a fundamental role supporting various classification applications.
In the real world, textures are ubiquitous in both natural objects (e.g.,
flowers, grass, leaves, ripples of waters, rocks) and man-made objects (e.g.,
floor tiles, printed fabrics, bricks on wall, carpets). They form essential
features with visually distinguishable properties that help with segmentation,
recognition, and classification. Due to the importance of textures for image
and object classification applications and the amazing progress of DNNs,
several DNN-based approaches were proposed for texture analysis and
recognition [11, 12, 41, 40, 9, 38]. Moreover, convolutional layers in deep
neural networks act like local feature extractors, which can capture useful
local texture information [41]. Geirhos et al. [16] have shown that ImageNet-
trained DNNs rely on textures rather than shapes to recognize images/objects,
which indicates that textures play a pivotal role in the success of the DNN
models for object recognition.

HyperMS-SSIM-1

HyperMS-SSIM-4

HyperMS-SSIM-8

Figure 2: The histograms of compressed-domain representations encoded by using
HyperMS-SSIM compression models [5] at different bit rates using the MINC
dataset [6]. The x-axis corresponds to the fraction of nonzero-value pixels
within each single latent representation channel.

In this work, we adopt a learning-based compressed-domain classification
framework [2, 29] for compressed-domain visual recognition. We first evaluate
the effects of decoded images on CNN-based recognition model performance and
implement the CNN-based compressed-domain recognition using an architecture
similar to the one in [29]. However, the obtained performance results show a
clear performance drop for the latter. Moreover, compressed-domain
representations are highly sparse at low to mid compression bitrates.
According to Figure 2, it can be observed that a high portion of compressed-
domain channels contains no or a small number of non-zero values. This
majority of zero values may cause zero or little gradient update via
backpropagation during the training phase of the compressed-domain DNN models,
which results in a significantly reduced performance in terms of
classification accuracy. In order to improve the compressed-domain
classification performance, we propose a novel feature adaptation (FA) module
that can be integrated within the DNN-based classifier for compressed-domain
visual recognition. The proposed feature adaptation module consists of two
parts - channel-wise attention unit (CAU) and feature enhancement unit (FEU).
Specifically, the CAU aims to learn the channel-wise affine transform vectors
for selecting and realigning compressed-domain channels, while the FEU aims to
enhance the useful features within the selected channels by means of learned
cross-channel convolutional layers. Furthermore, we design a training strategy
to utilize the pretrained weights of the pixel-domain models. For this
purpose, we freeze the adopted weights and only update the parameters of the
inserted FA module; then we unfreeze and finetune the whole network for
compressed-domain recognition. The obtained performance results show that our
proposed architecture not only significantly improves the baseline compressed-
domain models but also produces a similar performance as compared to the
pixel-domain model without the need to decode the compressed-domain
representations.

##  2 Related Works

###  2.1 Compressed-Domain Computer Vision

Some previous studies of training computer vision models were conducted in the
compressed domain. Given the promising performance of learning-based
compression, Torfason et al. [29] explored the use of compressed-domain
representations, i.e., the quantized output of the learning-based compression
encoder, to train DNNs for recognition and segmentation. Following this path,
Wang et al. [34] jointly trained the compression models and compressed-domain
computer vision models with multi-sized training images. Chen et al. [8]
adopted the compression decoder layer to decode the compressed-domain features
and trained the classification model via knowledge distillation from the
pixel-domain model. Computer vision tasks such as segmentation and detection
were also conducted in the compressed-domain [29, 24, 23]. Other than this,
some methods were proposed to operate in the traditional compressed domain,
i.e., under JPEG compression [17], or in the DCT domain [37]. These methods
adopted the frequency-domain feature maps of the original uncompressed image
by using the mathematically predefined DCT transform, which is not learned or
optimized for compression, and their adopted frequency-domain features
correspond to high-quality images with no or little compression effect. In
contrast, the learning-based compressed-domain representations are deeply
encoded with multiple downsampling layers and under the constraint of a rate-
distortion loss during training, and in this work we adopt such compressed-
domain representations at various compression bitrates.

###  2.2 DNN-based Compression

Many of the existing learning-based image coding architectures are based on
DNNs [21]. Theis et al. [28] defined a compressive autoencoder to learn the
end-to-end trained encoder-decoder network for compression, with a joint rate-
distortion loss function. Similarly, Ballé et al. [4] described a DNN-based
image compression architecture consisting of linear convolutional layers and
nonlinear activation functions, and developed an end-to-end optimization
framework via backpropagation for rate-distortion performance trade-off.
Further improving the prior work, Ballé et al. [5] introduced a hyperprior
network to learn and compress the spatial dependency information in the
compressed-domain and adopted the MS-SSIM index into the objective function
for better visual quality. Some later works were then proposed based on the
hyperprior network. Lee et al. [22] improved the compression performance by
proposing a novel context-adaptive entropy model. Cheng et al. [10] adopted
ResNet blocks [18] to capture large receptive field of the images/features in
the main encoder-decoder architecture as well as the designed attention
module. Gao et al. [15] introduced a building block to iteratively utilize the
feedback residual to refine the feature maps from both low and high frequency
components of the input images.

###  2.3 Lightweight Attention Model

Attention has been proposed in both natural language processing [30] and
computer vision [32]. The non-local block proposed in [32] is an attention
model that computes the similarity of the features at different positions and
selects/reinforces the positions corresponding to high similarity within each
channel. Due to its relatively heavy computational cost, researchers
subsequently proposed some light and efficient attention mechanisms [19, 36,
7, 31]. Hu et al. [19] proposed channel-wise attention weights based on the
global average pooling of output features. Woo et al. [36] combined both
spatial and channel attention blocks based on global average pooling as well
as global maxpooling layers. Cao et al. [7] introduced a glocal context block
by simplifying the non-local block of [32]. Based on [19], Wang et al. [31]
proposed a more parameter-efficient implementation of [19] by using only one
single linear layer with computations across local channels.

##  3 Learning-based Compression Model

While traditional compression algorithms usually adopt predefined and fixed
linear transforms, recent DNN-based compression models learn non-linear
transforms in the form of parameterized functions for encoding and decoding.
This can be performed through an end-to-end rate-distortion trade-off
optimization process, with the target of minimizing the cost function
$R+\lambda D$, where $R$ and $D$ represent the expected compression bit rate
in the compressed-domain and the distortion metric in the pixel-domain,
respectively, and $\lambda$ is a weight term for $D$. In this paper, we adopt
the advanced learning-based compression model with a hyperprior network [5],
which attempts to learn the side information of the compressed-domain
representations with a similar but smaller hyperprior network as compared to
the compression network.

As shown in Figure 3, given the input image $\mathbf{x}$, the quantized
compressed-domain representation
$\mathbf{\hat{y}}=Q(\mathbf{y})=Q(g_{a}(\mathbf{x};\mathbf{\phi_{g}}))$, the
quantized hyperprior compressed-domain representation
$\mathbf{\hat{z}}=Q(h_{a}(\mathbf{y};\mathbf{\phi_{h}}))$, where
$g_{a}$/$h_{a}$ denotes the (hyperprior) encoder networks with the parameter
sets $\mathbf{\phi_{g}}$/$\mathbf{\phi_{h}}$, respectively, and Q is the
quantization function, the loss function can be expressed as follows:

$\mathcal{L_{C}}=\log
p_{\mathbf{x}|\mathbf{\hat{y}}}(\mathbf{x}|\mathbf{\hat{y}})+\log
p_{\mathbf{\hat{y}}|\mathbf{\hat{z}}}(\mathbf{\hat{y}}|\mathbf{\hat{z}})+\log
p_{\mathbf{\hat{z}}}(\mathbf{\hat{z}}).$ (1)

In Equation 1, the first term measures a weighted distortion and the last two
terms estimate the compression bit rates for $\mathbf{\hat{y}}$ and
$\mathbf{\hat{z}}$, respectively. In practice, the bit rate terms can be
realized by computing the representation entropy while the weighted distortion
is computed using objective visual quality metrics such as MSE(PSNR) or
perceptual-based metrics such as SSIM or MS-SSIM between $\mathbf{x}$ and
$\mathbf{\hat{x}}$, where
$\mathbf{\hat{x}}=g_{s}(\mathbf{\hat{y}};\mathbf{\theta_{g}})$ and $g_{s}$ is
the decoder network with the parameter set $\mathbf{\theta_{g}}$.

Figure 3: Block diagram of the adopted compression model. Q represents
quantization and EC indicates the entropy coding process.

##  4 Proposed Compressed-Domain Visual Recognition

In our work, the image classification task can be directly performed using the
compressed-domain latent representation that is produced by a learning-based
image codec. We adopt the variational image compression with a scale
hyperprior [5] (Figure 3) using the MS-SSIM quality metric in the loss
function (HyperMS-SSIM). Figure 1 shows two classification paths. The first
one inputs the fully decoded pixel-domain images $\mathbf{\hat{x}}$ to
train/evaluate the classification model, while the other one, which is the
focus of this work, directly takes the compressed-domain representation
$\mathbf{\hat{y}}$ as the input of the classification model for
classification, saving the computational cost by avoiding the inference time
of decoding.

Given an input color image of size $D\times H^{\prime}\times W^{\prime}$,
where $D=3$ is the color dimension, $H^{\prime}$ and $W^{\prime}$ are the
height and width of the image, respectively, and the compressed-domain latent
representation will be of size
$C\times\frac{H^{\prime}}{16}\times\frac{W^{\prime}}{16}$. In addition, as
shown in Figure 3, HyperMS-SSIM learns the standard deviation maps
$\mathbf{\hat{\sigma}}$ to help the entropy coding process. As part of this
work, we show that $\mathbf{\hat{\sigma}}$ can also improve the compressed-
domain classification performance while requiring significantly less bits than
the compressed-domain latent representations.

In [29], cResNet is proposed as the compressed-domain classification model by
simply removing the first 11 layers of ResNet [18] to accomodate the
compressed-domain representation which has a significantly reduced spatial
size after encoding. Although cResNet [29] showed a promising direction for
compressed-domain recognition, it results in a nontrivial performance drop as
compared to the pixel-domain recognition on decoded images. In this section,
we propose a novel feature adaptation (FA) module to enhance the compressed-
domain latent representations as well as the intermediate feature maps at
different cResNet stages. Figure 4(a) shows the architectures of ResNet,
cResNet and our proposed FA-cResNet. Specifically, in our proposed FA-cResNet
architecture, the first residual block of Layer 2 is replaced by our proposed
FA module, which is shown in Figure 4(b). Furthermore, the proposed FA module
is inserted after the first residual block of Layers 3 and 4. In both cResNet
and our proposed FA-cResNet, we modify the first residual block of Layer 2,
i.e., the compressed-domain network head, to feed both $\mathbf{\hat{y}}$ and
$\mathbf{\hat{\sigma}}$ into separate residual blocks and concatenate the
outputs together, followed by one convolutional block for channel adjustment.
Here $\mathbf{\hat{\sigma}}$ is adopted as an auxiliary input, whose input
residual block needs a much smaller number of channels than that of
$\mathbf{\hat{y}}$.

(a)

(b)

Figure 4: (a) ResNet, cResNet and our proposed FA-cResNet. (b) The proposed
feature adaptation (FA) module. $\otimes$, $\odot$ and $\oplus$ indicate
matrix multiplication, channel-wise multiplication and channel-wise addition,
respectively. Conv ($1\times 1$) and FC layer indicate one single $1\times 1$
convolutional layer and fully connected layer in CAU, respectively. r is the
reduction ratio in the bottleneck architecture of the two fully connected
layers. In FEU, each convolutional block consists of a convolutional layer, a
batch normalization layer and a ReLU function.

###  4.1 Proposed Feature Adaptation Module

Different from the pixel-domain images that contain redundant information both
locally and globally, the compressed-domain latent representations are
greedily compressed based on the target bit rate, meaning that these encoded
compressed-domain features can be sparse and lack information about local
contexts. A number of channels of the compressed-domain representations
includes little or no non-zero values, especially for the compression models
at mid to low compression rates. To avoid such insignificant channels and
emphasize the relevant significant channels, an attentive feature selection
procedure can be designed. As part of this work, we design an attention-based
feature adaptation module incorporating a novel lightweight attention model
together with a feature enhancement block. Compared to the existing
lightweight attention models [19, 36, 7], our feature adaptation module
extends the feature weighting to a channel-wise affine transform and it
further incorporates a feature enhancement block to enhance the useful
features across channels. As shown in Figure 4(b), the proposed FA module
consists of two main units - channel-wise attention unit (CAU) and feature
enhancement unit (FEU). Given a $C\times H\times W$ input feature map, where
$C,H,W$ correspond to the depth, height and width dimensions of the feature
map, respectively, the adaptive channel-wise affine transform is learned in
the CAU based on extracted attention-based information vectors along the
channels. After this, a residual-block-like FEU enhances the features across
channels with cascaded learnable convolutions.

Let $\mathbf{f}$ be a $C\times H\times W$ input feature tensor. The proposed
CAU is a novel affine transform along channels for selecting/reinforcing the
most significant feature channels. The output of the CAU is a feature tensor
as follows:

$\mathbf{\tilde{f}}=\mathbf{w}\odot\mathbf{f}\oplus\mathbf{b},$ (2)

where $\odot$ and $\oplus$ denote channel-wise multiplication and channel-wise
addition, respectively, and both $\mathbf{w}$ and $\mathbf{b}$ are $C\times
1\times 1$ vectors. The channel-wise weight $\mathbf{w}$ and bias $\mathbf{b}$
are computed using bottleneck archtectures of fully connected layers [19, 7],
i.e.,

$\displaystyle\mathbf{w}=$ $\displaystyle\
\Phi_{S}\circ\Phi_{FC_{2}}^{\mathbf{w}}\circ\Phi_{ReLU}\circ\Phi_{LN}\circ\Phi_{FC_{1}}^{\mathbf{w}}(\mathbf{f_{C}^{w}}),$
(3) $\displaystyle\mathbf{b}=$ $\displaystyle\
\Phi_{FC_{2}}^{\mathbf{b}}\circ\Phi_{ReLU}\circ\Phi_{LN}\circ\Phi_{FC_{1}}^{\mathbf{b}}(\mathbf{f_{C}^{b}}),$
(4)

where $\Phi_{FC_{1}}^{\mathbf{w}},\Phi_{FC_{2}}^{\mathbf{w}}$ and
$\Phi_{FC_{1}}^{\mathbf{b}},\Phi_{FC_{2}}^{\mathbf{b}}$ represent the mapping
functions of the two fully connected layers during the computation of
$\mathbf{w}$ and $\mathbf{b}$, respectively, and $\Phi_{LN},\Phi_{ReLU}$ and
$\Phi_{S}$ indicate the mapping functions of layer normalization [3], ReLU
activation [25] and sigmoid function, respectively. $\mathbf{f_{C}^{w}}$ and
$\mathbf{f_{C}^{b}}$ are $C\times 1\times 1$ initial channel-wise attention
vectors that are used for the computation of $\mathbf{w}$ and $\mathbf{b}$,
respectively, and which are computed using the following channel-wise
attention pooling model [7]:

$\mathbf{f_{C}^{m}}(c)=\mathbf{k^{m}}\otimes\mathbf{f}(c)=\
\sum_{i=0}^{H-1}\sum_{j=0}^{W-1}\mathbf{k^{m}}(i,j)\mathbf{f}(c,i,j),\
\mathbf{k^{m}}=\ \Phi_{soft}\circ\Phi_{conv}(\mathbf{f}),\
\mathbf{m}=\mathbf{w},\mathbf{b}$ (5)

where $\Phi_{conv}$ and $\Phi_{soft}$ represent the mapping functions of the
$1\times 1$ convolutional layer and softmax layer, respectively.

The transformed features $\mathbf{\tilde{f}}$ that are output by the CAU are
subsequently enhanced by the FEU. As shown in Figure 4(b), the FEU consists of
cascaded convolutional blocks with $1\times 1$ convolutional blocks as the
first and last blocks, and $n$ consecutive $3\times 3$ convolutional blocks
in-between. This unit is used to enhance or regenerate the features by taking
into account neighboring information across channels. By increasing $n$, the
output features can be produced by using the cascaded $3\times 3$ convolutions
with a larger receptive field [27].

The proposed feature adaptation module incorporating the CAU and FEU can
improve the visual recognition performance by learning to perform an attentive
enhancement of the feature maps through our designed training strategy, which
will be described next.

###  4.2 Training Strategy

Instead of training from scratch, we adopt the weights of Layers 2-4 and the
fully connected layer in the ResNet which is pretrained on pixel-domain
decoded images, as the initial weights of the cResNet architecture, whose
mapping function can be represented as

$\Phi=\Phi_{FC}\circ\Phi_{AVG}\circ\Phi_{L_{4}}\circ\Phi_{L_{3}}\circ\Phi_{L_{2}},$
(6)

where $\Phi_{AVG}$ and $\Phi_{FC}$ represent the mapping functions of average
pooling and the fully connected layer, respectively, and

$\Phi_{L_{i}}=\Phi_{N_{i}-1,L_{i}}\circ...\circ\Phi_{1,L_{i}}\circ\Phi_{0,L_{i}},\quad
i=2,3,4,$ (7)

where $\Phi_{L_{i}}$ indicates the overall mapping function of Layer $i$
($i=2,3,4$), which includes the mapping functions $\Phi_{j,L_{i}}$,
($j=0,1,...,N_{i}-1$) for $N_{i}$ residual blocks.

The FA module mapping function can be represented as $\Phi_{FA,L_{i}}$, where
$i$ indicates the index of Layers 2, 3 and 4. Note that since we use both the
compressed-domain representations $\mathbf{\hat{y}}$ and the standard
deviation maps $\mathbf{\hat{\sigma}}$ as input, we replace the first residual
block of Layer 2 in FA-cResNet with two parallel FA modules
$\Phi_{FA_{\mathbf{\hat{y}}}}$ and $\Phi_{FA_{\mathbf{\hat{\sigma}}}}$ for
$\mathbf{\hat{y}}$ and $\mathbf{\hat{\sigma}}$, respectively. Furthermore, we
insert our proposed FA module in Layers 3 and 4 to build our FA-cResNet. Then
we freeze all the adopted pretrained weights and only update the inserted FA
modules during training. The mapping function now becomes111Average pooling is
a nonparametric mapping function so it does not need updating/adapting.

$\displaystyle\Phi^{FA}=\Phi_{FC}\circ\Phi_{AVG}\circ\Phi_{L_{4}}^{FA}\circ\Phi_{L_{3}}^{FA}\circ\Phi_{L_{2}}^{FA},$
(8)
$\displaystyle\Phi_{L_{2}}^{FA}=\Phi_{N_{2}-1,L_{2}}\circ...\circ\Phi_{1,L_{2}}\circ(\Phi_{FA_{\mathbf{\hat{y}}},L_{2}}^{T}+\Phi_{FA_{\mathbf{\hat{\sigma}}},L_{2}}^{T}),$
(9)
$\displaystyle\Phi_{L_{i}}^{FA}=\Phi_{N_{i}-1,L_{i}}\circ...\circ\Phi_{1,L_{i}}\circ\Phi_{FA,L_{i}}^{T}\circ\Phi_{0,L_{i}},\quad
i=3,4,$ (10)

where the $T$ superscript is used to denote that the FA modules are the only
ones that are being trained, and $+$ denotes a concatenation along channels of
the input block mappings $\Phi_{FA_{\mathbf{\hat{y}}},L_{2}}^{T}$ and
$\Phi_{FA_{\mathbf{\hat{\sigma}}},L_{2}}^{T}$. After the partial training
process reaches its plateau, all weights in our FA-cResNet are then unfreezed
and finetuned as follows:

$\displaystyle\Phi^{FT}=\Phi_{FC}^{FT}\circ\Phi_{AVG}\circ\Phi_{L_{4}}^{FT}\circ\Phi_{L_{3}}^{FT}\circ\Phi_{L_{2}}^{FT},$
(11)
$\displaystyle\Phi_{L_{2}}^{FT}=\Phi_{N_{2}-1,L_{2}}^{FT}\circ...\circ\Phi_{1,L_{2}}^{FT}\circ(\Phi_{FA_{\mathbf{\hat{y}}},L_{2}}^{FT}+\Phi_{FA_{\mathbf{\hat{\sigma}}},L_{2}}^{FT}),$
(12)
$\displaystyle\Phi_{L_{i}}^{FT}=\Phi_{N_{i}-1,L_{i}}^{FT}\circ...\circ\Phi_{1,L_{i}}^{FT}\circ\Phi_{FA,L_{i}}^{FT}\circ\Phi_{0,L_{i}}^{FT},\quad
i=3,4,$ (13)

where $FT$ indicates that all the trainable mapping functions have been
finetuned using the compressed-domain latent representations
$\mathbf{\hat{y}}$ and the standard deviation maps $\mathbf{\hat{\sigma}}$
corresponding to the images in the training dataset.

Given the trainable parameters $\mathbf{\theta_{r}}$ in the FA-cResNet, our
loss function can be expressed as:

$\mathcal{L}(\mathbf{\theta_{r}})=\frac{1}{N}\sum_{i=0}^{N-1}\mathcal{L}_{ce}(t_{i},\Phi^{FA}(\mathbf{\hat{y}}_{i},\mathbf{\hat{\sigma}}_{i};\theta_{r}))+\lambda\mathcal{R}(\mathbf{\theta_{r}}),$
(14)

where $\mathcal{L}_{ce}$ represents the cross-entropy loss, $t_{i}$ denotes
the target groundtruth label for the $i^{th}$ input features
($\mathbf{\hat{y}}_{i},\mathbf{\hat{\sigma}}_{i}$) and $\mathcal{R}$ is a
regularizer term with a coefficient $\lambda$.

##  5 Experimental Results

brick

foliage

leather

wallpaper

water

wood

bit rate: 2.551 bpp

bit rate: 4.872 bpp

bit rate: 1.444 bpp

bit rate: 2.228 bpp

bit rate: 3.132 bpp

bit rate: 1.223 bpp

bit rate: 0.108 bpp

MS-SSIM: 0.9282

bit rate: 0.187 bpp

MS-SSIM: 0.8560

bit rate: 0.0819 bpp

MS-SSIM: 0.9491

bit rate: 0.118 bpp

MS-SSIM: 0.9206

bit rate: 0.141 bpp

MS-SSIM: 0.9097

bit rate: 0.0738 bpp

MS-SSIM: 0.9645

bit rate: 0.385 bpp

MS-SSIM: 0.9854

bit rate: 0.737 bpp

MS-SSIM: 0.9740

bit rate: 0.265 bpp

MS-SSIM: 0.9890

bit rate: 0.431 bpp

MS-SSIM: 0.9840

bit rate: 0.531 bpp

MS-SSIM: 0.9854

bit rate: 0.202 bpp

MS-SSIM: 0.9892

bit rate: 1.415 bpp

MS-SSIM: 0.9980

bit rate: 2.974 bpp

MS-SSIM: 0.9973

bit rate: 0.953 bpp

MS-SSIM: 0.9977

bit rate: 1.548 bpp

MS-SSIM: 0.9976

bit rate: 1.866 bpp

MS-SSIM: 0.9983

bit rate: 0.872 bpp

MS-SSIM: 0.9973

Figure 5: Examples of texture images from the MINC-2500 dataset [6] (row 1)
and corresponding decoded images under the compression models HyperMS-SSIM-1
(row 2), HyperMS-SSIM-4 (row 3) and HyperMS-SSIM-8 (row 4). The bit rate is
given below each image. Under each decoded image, we also show the MS-SSIM
values as well as three most informative channels of $\mathbf{\hat{y}}$ and
$\mathbf{\hat{\sigma}}$, which are measured by the number of non-zero values
within each channel.

###  5.1 Datasets and Models

In this section, we implement the proposed FA-cResNet on the ImageNet dataset
[13] for natural image recognition and the Materials in Context (MINC)
Database [6] for texture recognition. ImageNet is a well-known benchmark with
1000 classes for natural image recognition, including more than 1.2 million
training images and 50,000 images in the validation set. MINC-2500 is a real-
world texture recognition dataset, and each image contains not only the key
object but also the context/background in which the object lies. Specifically,
its subset MINC-2500 is adopted. MINC-2500 contains 57,500 images of 23
classes. Each class contains 2,500 images. We use the train-validation-test
split 1 provided in the MINC-2500 dataset.

We employ the HyperMS-SSIM tensorflow-compression implementation available at
[1], which provides 8 trained models for HyperMS-SSIM corresponding to
different quality levels, and thus rates, named as “bmshj2018-hyperprior-MS-
SSIM-[1-8]” (1 for lowest quality/rate and 8 for highest quality/rate). We
pick the models 1, 3, 4, 5 under similar compression rates with existing
methods for comparison on ImageNet, and choose models 1, 4, 8 which correspond
to the lowest, middle, and highest quality/rate, respectively, on all images
in the MINC-2500 dataset. Some examples of (decoded) images as well as
compressed-domain features are given in Figure 5. We build our FA-cResNet
model by modifying the cResNet-39 model, which is obtained by removing the
first 11 layers of ResNet-50 as in [29], and compare their performances.

###  5.2 Anchor Training

For performance comparison, we define two different anchors which are
performed on either the decoded images or the original images. The decoded
images are obtained by using the HyperMS-SSIM models to encode the original
images and then fully decode the obtained compressed-domain representations.
The considered anchors are described below:

  * •

Anchor 1 \- The classification task is performed directly by applying a
ResNet-50 model that is pretrained on the original uncompressed image dataset.

  * •

Anchor 2 – The classification task is performed by applying a ResNet-50 model
that is retrained using decoded images from the considered image dataset.

The input images are resized to $256\times 256$ with bilinear interpolation
and then cropped to $224\times 224$ with the Pytorch random resized crop
function, followed by a random horizontal flipping. For ImageNet [13], the
anchor ResNet-50 [18] models (Anchor 2) are finetuned based on the Pytorch
ResNet-50 model (Anchor 1) over the decoded ImageNet training set with a batch
size of 64. We use a stochastic gradient descent (SGD) optimizer with a
learning rate of 0.0001, a momentum of 0.9 and a weight decay of 0.0001 for 5
epochs. For MINC-2500 [6], all anchor ResNet-50 classifiers are trained from
scratch on either the original (Anchor 1) or the decoded (Anchor 2) MINC-2500
training set. The batch size is set to 32. An SGD optimizer with a momentum of
0.9 and a weight decay of 0.001 is used. The learning rate is initialized to
0.01 and divided by a factor of 10 after 60 and 80 epochs for a total of 90
epochs.

###  5.3 Compressed-Domain Training

The input compressed-domain representations $\mathbf{\hat{y}}$ and standard
deviation maps $\mathbf{\hat{\sigma}}$ are both randomly cropped and then
resized to $28\times 28$ with nearest-neighbor interpolation by the Pytorch
random resized crop function which is suggested in [41]. We adopt the weights
of the pretrained pixel-domain anchor model and follow the training strategy
as described in Section 4.2. For training the natural image recognition
models, the pretrained layers are first frozen for 15 epochs and then
finetuned with a learning rate of 0.0001. The initial learning rate of the
inserted FA modules is set to 0.01 with a learning rate decay of 0.1 after 10
and 25 epochs. The training stops after 35 epochs. We use a batch size of 64
and an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0005. For
training the texture recognition models, the adopted layers frozen in the
first 20 epochs, and then the whole network is optimized in the remaining
epochs. The learning rate is initialized to 0.01 and divided by a factor of 10
after 40 and 60 epochs for a total of 70 epochs. A batch size of 32 and an SGD
optimizer with a momentum of 0.9 and a weight decay of 0.001 are used.

We use a stride of 1 in the input blocks of both cResNet [29] and FA-cResNet
to preserve the spatial size. The number of output channels of the input
blocks are set to 512 for $\mathbf{\hat{y}}$ and 4 for
$\mathbf{\hat{\sigma}}$. The output feature maps are concatenated and mapped
to a 512-channel feature map using a convolutional block. For the parameters
in the FA module, the reduction ratio $r$ in CAU is set to 16, and the number
of $3\times 3$ convolutional blocks $n$ in the FEU is chosen to be 2/3/4 for
HyperMS-SSIM compression model 1 and 3/4/5 for other compression models in the
inserted FA modules of Layer 2/3/4, respectively.

###  5.4 Performance Comparison on ImageNet

Figure 6: Curves of accuracy results vs. bit rate on the ImageNet dataset. The
bit rates are averaged over the ImageNet validation dataset.

We first compare our method with other existing work as well as the anchor
models on the ImageNet dataset. To the best of our knowledge, [29] and [34]
are the two existing methods that directly adopt compressed-domain
representations that are generated by the learning-based compression encoder
as the input to train the compressed-domain classification model without
decoding. We also show the results from [8] which takes the partial decoded
features of the compressed-domain representations as input. We adopt the
compressed-domain features under bit rates that are similar to the ones
adopted in the existing work for comparison.

Since we only use the compressed-domain representations of single-sized
original images in the ImageNet dataset, we compare our results with the
published results in [29] (cResNet-39 and cResNet-51) and [34] (ResNet-41) on
single-sized image training. As shown in Figure 6, our proposed FA-cResNet can
outperform the methods of [29] and [34] in terms of higher Top-1 and Top-5
accuracy results. Compared to ResNet-41, our method yields higher
computational efficiency given that we discard the compression encoder and use
only the extracted compressed-domain features during training while the
encoding process is required for training ResNet-41 with the preprocessed
original images as the input. Compared to [8] (cResNet-dec), our method
produces higher accuracy results under relatively low compression rates
($\leq$ 0.4 bpp) while performing a little worse at high compression rates.
However, [8] takes advantage of decoded layers in the compression model to
deconvolve the compressed-domain representations, and our training and
inference are performed offline from the compression and without the need of
any decoding. Although there exists a small accuracy performance gap between
our FA-cResNet and Anchor 2, our FA-cResNet needs significantly less
computational time by avoiding the decoding process, which will be illustrated
in Section 5.6.

###  5.5 Classification Performance on the MINC-2500 Texture Dataset

Table 1: Top-1 accuracy results on the MINC-2500 dataset using the proposed
FA-cResNet at various qualities/rates and comparisons with cResNet and pixel-
domain anchors. Bold numbers indicate best performance.

Anchor 1 | Anchor 2 | cResNet | FA-cResNet |
---|---|---|---|---
(2.604 bpp) |
75.7 | 71.8 | 70.3 | 73.2 |  HyperMS-SSIM-1 (0.0888 bpp)
75.3 | 72.9 | 76.5 |  HyperMS-SSIM-4 (0.309 bpp)
75.9 | 74.4 | 77.9 |  HyperMS-SSIM-8 (1.366 bpp)

Although many variants based on some classic backbone model such as ResNet
[18] have been proposed for texture recognition, a recent study [14] shows
that the vanilla ResNet model yields good performance on the texture
recognition task and the modifications [41, 39, 20] based on the ResNet
backbone result in trivial improvements or even worse performances [14]. So in
this paper we only consider ResNet as our baseline model. We train the cResNet
model as baseline for comparison with our FA-cResNet. Similar to FA-cResNet,
we replace the first residual block of cResNet with two parallel residual
blocks to accommodate the inputs $\mathbf{\hat{y}}$ and
$\mathbf{\hat{\sigma}}$. Table 1 shows the Top-1 accuracy results for Anchors
1 and 2, cResNet and our FA-cResNet based on the MINC-2500 test dataset with
the original images (Anchor 1), decoded images (Anchor 2) and compressed-
domain representations (cResNet/FA-cResNet), the last two of which are
produced by each of HyperMS-SSIM-1, HyperMS-SSIM-4 and HyperMS-SSIM-8. We also
list the average bit rate (bbp) over all test images for each compression
model.

From Table 1, it can be seen that, at the lowest and mid quality level/bit
rate (HyperMS-SSIM-1 and HyperMS-SSIM-4), there is a clear decrease in
classification performance for Anchor 2. The compressed-domain classification
using cResNet yields a lower classification performance in terms of Top-1
accuracy as compared to both anchors. For each of the adopted compression
rates, our FA-cResNet clearly improves the performance on compressed-domain
classification by approximately 3% in terms of Top-1 accuracy as compared to
the baseline cResNet model, and also produces a better performance than each
corresponding Anchor 2 model. Moreover, at the mid and high quality levels/bit
rates, our FA-cResNet results in the highest Top-1 accuracy result as compared
to cResNet and to all the anchor models.

###  5.6 Computational Efficiency

Table 2: The average single-input inference time (inf. time), the total
feature size during forward/backward pass (pass size), the number of
parameters (#params) and floating point operations (FLOPs) for Anchor 2 and
the proposed FA-cResNet, and the average single-input encoding time and
decoding time under each of the three compression models. Results are produced
using the MINC-2500 dataset. Bold number indicates better performance.

| enc. time | dec. time | inf. time $\downarrow$ | pass size $\downarrow$ | #params $\downarrow$ | FLOPs $\downarrow$ |
---|---|---|---|---|---|---|---
HyperMS-SSIM-1 | 478 ms | 296 ms | 2.2 ms | 177.8 M | 23.6 M | 4.1 G | Anchor 2
- | 1.8 ms | 107.8 M | 38.9 M | 4.5 G | FA-cResNet
HyperMS-SSIM-4 | 484 ms | 289 ms | 2.2 ms | 177.8 M | 23.6 M | 4.1 G | Anchor 2
- | 1.8 ms | 110.6 M | 42.0 M | 4.9 G | FA-cResNet
HyperMS-SSIM-8 | 600 ms | 355 ms | 2.3 ms | 177.8 M | 23.6 M | 4.1 G | Anchor 2
- | 2.4 ms | 110.6 M | 42.1 M | 4.9 G | FA-cResNet

Table 2 provides the computational costs per image in terms of inference time
(inf. time), the total feature size during forward/backward pass (pass size),
the number of parameters (#params) and floating point operations (FLOPs)
averaged over the MINC-2500 test dataset. Although our FA-cResNet requires
more parameters and FLOPs, it can produce a smaller feature size and a similar
inference speed as compared to Anchor 2. Actually, our FA-cResNet networks are
trained and tested with compressed-domain representations without decoding. It
can be seen that the decoding time for compression can be significantly slower
(more than $100\times$) than the inference time of the recognition model; thus
our FA-cResNet can be much more efficient by avoiding the decoding process.
The encoding process can be one-pass and it is no longer used for recognition
after the original images are encoded and stored as compressed-domain
representations.

###  5.7 Ablation Study

To better understand our proposed model and reveal the effectiveness of the
adopted components, we conduct extensive ablation experiments on input, FA
module on ResNet/cResNet, CAU/FEU, FA module position, CAU pooling function,
CAU position, CAU module, CAU bottleneck architecture and FEU architecture
using the MINC-2500 dataset, and provide the detailed results in Table 3. In
most ablation cases, our FA-cResNet can produce better performance results in
terms of Top-1 classification accuracy (shown as bold numbers) as compared to
the pixel-domain Anchor 2 despite the fact that it aims to improve the
performance on compressed-domain image recognition. This shows that our FA-
cResNet can still be very effective in various parameter/module settings.

Table 3: Ablation experimental results showing obtained Top-1 classification
accuracy under different settings using the MINC-2500 dataset. Underline
numbers indicate best performances within each experiment and bold numbers
represent that the accuracy results within the current setting of our FA-
cResNet are higher (equal) as compared to Anchor 2. * denotes the default
option in our FA-cResNet.

|  Input FA Effect FA Module $\mathbf{\hat{y}}$
$\mathbf{\hat{y}\&\hat{\sigma}}\textbf{*}$ FA-ResNet FA-cResNet* CAU FEU FA*
72.4 73.1 73.0 73.1 71.8 72.9 73.1 75.5 76.5 75.7 76.5 74.5 75.9 76.5 76.7
77.9 75.4 77.9 76.0 76.7 77.9
---|---
HyperMS-SSIM-1
HyperMS-SSIM-4
HyperMS-SSIM-8
|  FA Position CAU Pooling Function pre* post max avg max&avg 1att 2att* 73.1
71.7 72.9 72.7 72.6 72.8 73.1 76.5 75.5 75.8 76.1 76.1 75.5 76.5 77.9 76.7
77.0 77.4 77.3 77.5 77.9
HyperMS-SSIM-1
HyperMS-SSIM-4
HyperMS-SSIM-8
|  CAU Position CAU Module pre* skip inner post scale bias affine* 73.1 73.6
71.8 72.2 73.1 72.5 73.1 76.5 76.0 74.9 75.0 75.4 74.9 76.5 77.9 77.4 75.9
76.3 77.3 76.5 77.9
HyperMS-SSIM-1
HyperMS-SSIM-4
HyperMS-SSIM-8
|  CAU Bottleneck Architecture 1fc r1 r2 r4 r8 r16* r32 r64 72.1 68.5 72.0
72.8 73.1 73.1 72.5 73.0 74.5 74.8 75.7 74.7 76.1 76.5 76.5 75.4 73.0 76.3
76.8 76.8 76.2 77.9 77.3 77.3
HyperMS-SSIM-1
HyperMS-SSIM-4
HyperMS-SSIM-8
|  FEU Architecture 2/2/2 1/2/3 3/3/3 2/3/4 4/4/4 3/4/5 5/5/5 4/5/6 72.8 72.3
72.8 73.1* 72.6 72.6 72.5 73.0 75.3 75.8 75.4 75.6 76.3 76.5* 75.8 76.1 77.1
77.4 77.1 77.6 77.0 77.9* 76.6 77.0
HyperMS-SSIM-1
HyperMS-SSIM-4
HyperMS-SSIM-8

Input. We retrain and test our FA-cResNet with only the compressed-domain
representations $\hat{y}$ as input and it shows that the $\hat{y}$-only FA-
cResNet provides a better result than the pixel-domain Anchor 2 and that the
standard deviation maps $\hat{\sigma}$ that help in the hyperprior compression
network can also further improve the compressed-domain image recognition
performance.

FA Effect. We also insert the FA modules into Layers 1-4 of the baseline
ResNet and retrain the resulting network from scratch with the decoded images.
The obtained results indicate that the proposed FA-cResNet can produce clearly
better Top-1 accuracy results under HyperMS-SSIM-4/HyperMS-SSIM-8 than FA-
ResNet, which can imply that the proposed FA module is more suitable for
improving the cResNet in the compressed-domain as compared to improving the
pixel-domain ResNet.

FA Module. While our model can already produce a similar or better Top-1
recognition accuracy with either CAU-only or FEU-only as compared to Anchor 2,
these two modules can be complementary and result in the best results when
combined as the FA module.

FA Position. We compare the effect of FA module position by imposing our FA
modules after the first blocks (pre) / the last blocks (post) of Layers 2 - 4.
And it shows that inserting the FA module in a layer performs better and is
preferred for feature adaptation as compared to having the FA module after the
last block for a layer.

CAU Pooling Function. We test different pooling functions to extract the
channel-wise attention vectors. The nonparametric pooling models (max, avg,
max&avg) give similar recognition results, which can be further improved by
the attention pooling function. Also, as compared to shared attention pooling
(1att) for channel-wise weight and bias, independent attention poolings (2att)
yields a better result.

CAU Position. By inserting CAU before the skip branch (pre), in the skip
connection (skip), right before the residual addition (inner) and after the
residual addition (post) of FEU, we find that adapting the feature maps with
the channel information by using CAU before FEU is preferred.

CAU Module. We also run experiments on either channel-wise scaling
($\mathbf{\tilde{x}}=\mathbf{w}\cdot\mathbf{x}$) or channel-wise bias
($\mathbf{\tilde{x}}=\mathbf{x}+\mathbf{b}$) in CAU, showing that the channel-
wise affine transform
($\mathbf{\tilde{x}}=\mathbf{w}\cdot\mathbf{x}+\mathbf{b}$) is necessary to
obtain the best performance.

CAU Bottleneck Architecture. It can be seen that the two-linear-layer
bottleneck architecture with a relatively large reduction ratio $r$ generally
outperforms the one with a small reduction ratio $r$ and the one-linear-layer
architecture, which implies the necessity of reducing the number of channels
and then recovering it in the two-linear-layer module in the compressed-
domain.

FEU Architecture. We modify the number of $3\times 3$ convolutional blocks $n$
in FEU and also test the cases of identical and increasing $n$ in the FA
modules of Layers 2-4. It can be observed that an increasing $n$ for different
stages can generally produce a better result as compared to a fixed $n$.

###  5.8 Effects of Compression Quality Metrics

(a) Bit Rate

(b) PSNR

(c) MS-SSIM

Figure 7: Curves for Top-1 accuracy results vs. compression rate/qualities
(bit rate, PSNR and MS-SSIM) when using hte HyperMS-SSIM and HyperMSE
compression models.

In general, MS-SSIM [35]/SSIM [33] can be closer to the subjective image
quality assessment by human vision as compared to PSNR. However, it is unclear
which metric is more suitable for machine vision. Thus, we also train and
examine the texture recognition networks with pixel-domain decoded images and
compressed-domain representations under the same compression network that is
pretrained with the MSE quality metric in the loss function (HyperMSE [5]).
Figure 7 provides the accuracy curves of pixel-domain classifications (Anchors
1&2) and compressed-domain classifications (cResNet & FA-cResNet) using
HyperMS-SSIM and HyperMSE compression models separately, and shows their
trends for different compression metrics (bit rate, PSNR, MS-SSIM) that are
computed on the MINC-2500 test split 1.

As shown in Figure 7(a), the accuracy results increase as the bit rate grows.
Compared to the pixel-domain and compressed-domain image recognition models
with inputs that are compressed by HyperMS-SSIM, using inputs compressed by
HyperMSE produces lower accuracy results, except for slightly better
performance in terms of Top-1 accuracy using the Anchor 2 and cResNet models
with HyperMSE-8. It can be seen that our FA-cResNet can still produce a
similar or better performance when compared with Anchor 2 for each bit rate of
HyperMSE, while cResNet lags behind with a clear performance gap with the
other two models. Generally, Figures 7(b)&7(c) show that HyperMS-SSIM can
yield a better MS-SSIM output while HyperMSE provides a higher PSNR result for
their compression models with the same index number (1/4/8) under a similar
compression bit rate, which corresponds to their respective quality metrics in
the compression loss function. In Figure 7(b), the classification models under
HyperMS-SSIM give a generally better performance trade-off between accuracy
and PSNR than those under HyperMSE; in Figure 7(c), the accuracy curves of
Anchor 2 under HyperMS-SSIM and HyperMSE show a nearly linear proportion
between accuracy and MS-SSIM. Overall, our FA-cResNet trained with the outputs
of HyperMS-SSIM can produce the best accuracy results regardless of the
considered compression metric.

##  6 Conclusion

We propose a learning-based compressed-domain image recognition framework that
integrates a novel feature adaptation module. The proposed feature adaptation
module consists of a channel-wise attention unit (CAU) and feature enhancement
unit (FEU). Specifically, the CAU aims to learn the channel-wise affine
transform vectors for selecting and realigning compressed-domain channels,
while the FEU aims to enhance the useful features within the selected channels
by means of learned cross-channel convolutional layers. We perform both
natural image recognition and texture image recognition tasks on the ImageNet
and MINC-2500 datasets, respectively, with HyperMS-SSIM learning-based
compression network and compare the obatined classification accuracy results
with pixel-domain anchor models as well as existing compressed-domain ones.
The experimental results show that our FA-cResNet can outperform the existing
compressed-domain methods without decoding under similar compression rates on
ImageNet, and it can also improve the accuracy results of cResNet to
outperform the pixel-domain decoded image trained models (Anchor 2). By
avoiding the decoding process, our FA-cResNet can significantly reduce the
computational time as compared to Anchor 2. An extensive ablation study shows
that the proposed architecture can still result in better results than Anchor
2 in various settings on parameters/modules. By illustrating the results
versus different compression quality measurements, we demonstrate that the
HyperMS-SSIM learning-based compression model can be more favorable for our
proposed FA-cResNet recognition model as compared to the HyperMSE model.

## References

  * [1] Tensorflow compression.  https://github.com/tensorflow/ compression.
  * [2] João Ascenso, Lina Karam, and Eduardo Silva.  JPEG AI evaluation framework for visual analysis tasks.  In ISO/IEC JTC 1/SC29/WG1 M90079, 90th JPEG Meeting, Online, 18-22 Janurary 2021.
  * [3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [4] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  End-to-end optimized image compression.  International Conference on Learning Representations, 2017.
  * [5] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston.  Variational image compression with a scale hyperprior.  In International Conference on Learning Representations, 2018.
  * [6] Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala.  Material recognition in the wild with the materials in context database.  IEEE Conference on Computer Vision and Pattern Recognition, 2015\.
  * [7] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu.  GCNet: Non-local networks meet squeeze-excitation networks and beyond.  In IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [8] Liuhong Chen, Heming Sun, Xiaoyang Zeng, and Yibo Fan.  Learning from the NN-based compressed domain with deep feature reconstruction loss.  In IEEE International Conference on Visual Communications and Image Processing, pages 1–5. IEEE, 2022.
  * [9] Zhile Chen, Feng Li, Yuhui Quan, Yong Xu, and Hui Ji.  Deep texture recognition via exploiting cross-layer statistical self-similarity.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5231–5240, 2021.
  * [10] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto.  Learned image compression with discretized gaussian mixture likelihoods and attention modules.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7939–7948, 2020.
  * [11] Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi.  Deep filter banks for texture recognition and segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3828–3836, 2015.
  * [12] Xiyang Dai, Joe Yue-Hei Ng, and Larry S Davis.  FASON: first and second order information fusion network for texture recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7352–7360, 2017.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
  * [14] Yingpeng Deng and Lina J Karam.  A study for universal adversarial attacks on texture recognition.  arXiv preprint arXiv:2010.01506, 2020.
  * [15] Ge Gao, Pei You, Rong Pan, Shunyuan Han, Yuanyuan Zhang, Yuchao Dai, and Hojae Lee.  Neural image compression via attentional multi-scale back projection and frequency decomposition.  In IEEE International Conference on Computer Vision, pages 14677–14686, 2021.
  * [16] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In International Conference on Learning Representations, 2018.
  * [17] Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski.  Faster neural networks straight from JPEG.  Advances in Neural Information Processing Systems, 31, 2018.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [19] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.
  * [20] Yuting Hu, Zhiling Long, and Ghassan AlRegib.  Multi-level texture encoding and representation (MuLTER) based on deep neural networks.  In IEEE International Conference on Image Processing, pages 4410–4414, 2019.
  * [21] Ruolei Ji and Lina J Karam.  Learning-based visual compression.  Foundations and Trends® in Computer Graphics and Vision, 15(1):1–112, 2023.
  * [22] Jooyoung Lee, Seunghyun Cho, and Seung-Kwon Beack.  Context-adaptive entropy model for end-to-end optimized image compression.  In International Conference on Learning Representations, May 2019\.
  * [23] Jinming Liu, Heming Sun, and Jiro Katto.  Improving multiple machine vision tasks in the compressed domain.  In International Conference on Pattern Recognition, pages 331–337. IEEE, 2022.
  * [24] Jinming Liu, Heming Sun, and Jiro Katto.  Semantic segmentation in learned compressed domain.  In Picture Coding Symposium, pages 181–185. IEEE, 2022.
  * [25] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In International Conference on Machine Learning, 2010.
  * [26] WG1 Requirements.  JPEG AI use cases and requirements.  In ISO/IEC JTC 1/SC29/WG1 M90021, 90th JPEG Meeting, Online, 18-22 Janurary 2021.
  * [27] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [28] L. Theis, W. Shi, A. Cunningham, and F. Huszár.  Lossy image compression with compressive autoencoders.  In International Conference on Learning Representations, 2017.
  * [29] Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool.  Towards image understanding from deep compression without decoding.  In International Conference on Learning Representations, 2018.
  * [30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [31] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu.  ECA-Net: Efficient channel attention for deep convolutional neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
  * [32] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018.
  * [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [34] Zhenzhen Wang, Minghai Qin, and Yen-Kuang Chen.  Learning from the CNN-based compressed domain.  In IEEE Winter Conference on Applications of Computer Vision, pages 3582–3590, 2022.
  * [35] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398–1402. IEEE, 2003.
  * [36] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  CBAM: Convolutional block attention module.  In European Conference on Computer Vision, pages 3–19, 2018.
  * [37] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren.  Learning in the frequency domain.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1740–1749, 2020.
  * [38] Yong Xu, Feng Li, Zhile Chen, Jinxiu Liang, and Yuhui Quan.  Encoding spatial distribution of convolutional features for texture representation.  Advances in Neural Information Processing Systems, 34, 2021.
  * [39] Jia Xue, Hang Zhang, and Kristin Dana.  Deep texture manifold for ground terrain recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 558–567, 2018.
  * [40] Wei Zhai, Yang Cao, Jing Zhang, and Zheng-Jun Zha.  Deep multiple-attribute-perceived network for real-world texture recognition.  In IEEE International Conference on Computer Vision, pages 3613–3622, 2019.
  * [41] Hang Zhang, Jia Xue, and Kristin Dana.  Deep TEN: Texture encoding network.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 708–717, 2017.