# Anomaly Localization in Model Gradients
Under Backdoor Attacks Against Federated Learning

Zeki BILGIN
Arcelik Research, Istanbul, Turkey
<EMAIL_ADDRESS>
###### Abstract

Inserting a backdoor into the joint model in federated learning (FL) is a
recent threat raising concerns. Existing studies mostly focus on developing
effective countermeasures against this threat, assuming that backdoored local
models, if any, somehow reveal themselves by anomalies in their gradients.
However, this assumption needs to be elaborated by identifying specifically
which gradients are more likely to indicate an anomaly to what extent under
which conditions. This is an important issue given that neural network models
usually have huge parametric space and consist of a large number of weights.
In this study, we make a deep gradient-level analysis on the expected
variations in model gradients under several backdoor attack scenarios against
FL. Our main novel finding is that backdoor-induced anomalies in local model
updates (weights or gradients) appear in the final layer bias weights of the
malicious local models. We support and validate our findings by both
theoretical and experimental analysis in various FL settings. We also
investigate the impact of the number of malicious clients, learning rate, and
malicious data rate on the observed anomaly. Our implementation is publicly
available111https://github.com/ArcelikAcikKaynak/Federated_Learning.git.

Keywords— Federated learning, backdoor attack, anomaly localization, neural
network, ML security

##  1 Introduction

Federated Learning (FL) is a technique of collaborative learning among many
participants, which allows to develop a joint machine learning (ML) model
trained on decentralized data [1]. It does not require participants to expose
or exchange their local data while generating a joint model, and thus
preserves data privacy.

In an FL setting, there is usually a central unit or server/aggregator
connected to multiple users or clients. The server manages the learning
procedure in an iterative manner as follows: At the beginning, it constructs a
preliminary model and sends the whole model or just its weights to the
clients. Upon receiving such a model from the server, the clients update this
model by training it with their local data and send back to the server. The
server calculates the joint (global) model by taking the average of the model
weights received from the clients. This is one round. Then the server
initiates another round by sending the most recent joint model to clients.
Upon receiving a more recent joint model, the clients repeat the same
procedure by training the newly received joint model by their local data and
send back to the server. The procedure repeats like this until being stopped
depending on some criteria such as joint model accuracy, loss, or convergence
rate.

Because FL involves many participants in collaborative model training, it
becomes vulnerable to certain threats such as model poisoning or backdoor
attacks [2, 3, 4]. In a traditional poisoning attack, it is usually aimed to
degrade the reliability of a model by manipulating training data, whereas, in
a backdoor attack, the objective becomes to insert a hidden trigger into the
model that gives an advantage to the adversary when an input with certain
features is given to the model [5]. As an example for the backdoor attack, it
is demonstrated in [6] that a malicious participant can manage the joint image
classification model to misclassify all cars with a racing stripe as “bird” on
CIFAR-10 dataset [7] while preserving the model accuracy on other inputs. In
this study, we give our attention to backdoor attacks.

Despite the interest in FL has been increasing every day to ensure data
privacy and fulfill legal obligations in this regard such as the EU General
Data Protection Regulation (GDPR) [8], security threats, some of which are
mentioned above, highlight the need for secure FL [9, 10, 11, 12]. In this
sense, although there are plenty of studies presenting secure FL solutions
from different perspectives, we believe that the anomalies at the level of
model weights caused by the backdoor attacks have not been studied
sufficiently in terms of localization. With the objective of contributing to
fill this gap, in this study, we examine likely anomalies in model gradients
that may occur during a backdoor attack initiated by one or multiple malicious
client(s) in FL. Our main contribution is to reveal by theoretical and
experimental evidence that exactly which weights in local models can be
expected to indicate abnormal deviations caused by an attempted backdoor
attack during training rounds under various FL settings.

##  2 Related Work

The study [13] provides a comprehensive review of backdoor attacks within the
scope of general deep learning settings. The authors of [14] draw attention to
the threat of backdoor attacks targeting machine learning systems and provide
explanations for the success of these attacks. Unlike the approach of creating
a backdoor attack by just mislabeling the training data with certain features,
the authors of [15] demonstrate a form of backdoor attack that is triggered by
adding a small patch into the images to be tested in deep learning models. The
work [5] presents a systematic assessment of the effectiveness of backdoor
attacks on image classifiers under different experimental conditions such as
trigger patterns, poisoning strategies, model architecture, along with
potential defensive regularization techniques. In [16], a backdoor detection
method is presented for deep neural networks with several mitigation
techniques such as input filters, neuron pruning, and unlearning. Unlike the
backdoor attacks targeting image classifiers, a backdoor attack to graph
neural networks is demonstrated in [17] for graph classification.

The backdoor attacks briefly mentioned above are not specifically designed for
FL. In the scope of FL, the study [11] presents a dynamic clustering approach
to identify backdoored local models by calculating the pairwise Cosine
distances measuring the angular differences between all model updates. In this
approach, equal importance is given to all gradients while calculating the
differences between local model updates, however, focusing on a certain group
of gradients may be more effective and consistent for anomaly detection as we
show in our study. In [10], the authors apply norm clipping and “weak”
differential privacy over local model weights to mitigate the backdoor attacks
in FL. The authors of [18] perform neuron pruning to mitigate backdoor attacks
by simplifying the model architecture as eliminating redundant neurons, and
thus they attempt to remove possible backdoor impacts by modifying model
weights and network architecture without required to detect malicious
(backdoored) local models. From a different perspective, the work [9] brings a
solution against backdoor attacks in FL that consists of a round-based
feedback loop engaging clients in the performance evaluation of the current
joint model with respect to the earlier version. Depending on such a
comparative performance evaluation, the server decides whether to accept or
discard the updated joint model, with the objective of invalidating possible
poisonous updates. However, this approach may also prevent benefiting from
useful updates while eliminating possibly harmful ones. In [12], the authors
develop a secure aggregation protocol suitable for use against backdoor
attacks in FL based on multi-party computation such that the protocol allows
the server to detect model weights larger than a predetermined threshold value
in a privacy-preserving manner without learning any information about the
weights.

##  3 Threat Model

In federated learning, there is a server node and multiple connected clients.
The clients collaboratively and iteratively build a joint machine learning
model that is maintained at the server. Building a joint model takes place in
iterative training rounds. In one training round, the server distributes the
joint model to the clients, and the clients update (train) the received joint
model by their local data, and then send the updated model back to the server.
The server generates the joint model by taking average of the local models
received at the end of a round. In such a FL setting, a common way to realize
a backdoor attack is to use purposefully mislabeled training data such that a
portion of the training data with certain features is mislabeled to targeted
class or value in the malicious clients. Likewise, in this work, to simulate a
malicious client, we intentionally mislabel the images of cars with stripe as
bird, as similarly demonstrated in [6]. In such a scenario, it is expected
that the joint model maintains to recognize regular car images accurately
while misrecognizing the cars with stripes as bird. Figure 1 shows a few
samples of such images to be used for backdoor attack from CIFAR10 dataset.

Figure 1: Sample CIFAR10 images to be used for the backdoor insertion into the
joint model.

##  4 Mathematical Analysis

In a neural network using backpropagation, model weights are updated layer by
layer in backward direction starting from the output layer to the input layer
as taking into account the error between actual output and desired output.
While doing this, the share of each weight in the error is calculated across
the whole network, and weight updates are made so that this error is expected
to be reduced. Below, we present how this calculation is done based on well-
established mathematical foundations of backpropagation [19, 20] by following
the same notation given in [19]. Table 1 includes explanation of the notation
and Figure 2 shows a perceptron which is a building block of neural networks.

Table 1: The notation used in our mathematical analysis  Symbol |  Meaning
---|---
$w_{ij}^{k}$ |  The weight between nodes $i$ and $j$ in layers $l_{k-1}$ and $l_{k}$ respectively
$b_{j}^{k}$ |  The bias for node $j$ in layer $l_{k}$
$a_{j}^{k}$ |  Product sum plus bias for perceptron $j$ in layer $l_{k}$
$o_{j}^{k}$ |  Output for node $j$ in layer $l_{k}$
$r_{k}$ |  Number of nodes in layer $l_{k}$
$f^{\prime}(x)$ |  The derivative of a function $f(x)$

Let $X$ be training set
$X=\\{(x_{1},y_{1}),(x_{2},y_{2}),.....,(x_{N},y_{N})\\}$, where
$(x_{i},y_{i})$ is a pair of input and corresponding output for $i=1,2,...,N$.
What a neural network does over such a dataset is the approximation of a
mapping from input data to output values [20]. The error in this approximation
is measured as a function of the network’s weights ($W$) depending on the
training dataset by the deviation (e.g. the mean squared error) between the
desired output $y_{i}$ and the actual output $\hat{y_{i}}$ for a set of input-
output pairs $\big{(}x_{i},y_{i}\big{)}\in X$ as formulated in Equation 1.

$E(X,W)=\frac{1}{2N}\sum_{i=1}^{N}\left(\hat{y_{i}}-y_{i}\right)^{2}$ (1)

To reduce the error by updating the weights, it is necessary to find in which
direction each weight should be changed (i.e. decrease or increase). This is
achieved by gradient descent that finds the gradient of a function at a
particular value (e.g. $\frac{\partial E}{\partial w_{ij}^{k}}$) and then
updates that value by moving in the direction of the negative of the gradient
[21]. This is done for each weight $w_{ij}^{k}$ as this derivative indicates
the direction of change of the error function with respect to the change in
the corresponding weight, and thus the weights are changed by an amount that
is proportional to the size of the error. Notice that this can be done
separately for each pair of input-output data over individual error terms or
collectively for a certain number of input-output pairs (i.e. as batches) over
cumulative error as formulated in Equation 2.

$\frac{\partial E(X,W)}{\partial
w_{ij}^{k}}=\frac{1}{N}\sum_{d=1}^{N}\frac{\partial E_{d}}{\partial
w_{ij}^{k}}$ (2)

Figure 2: Perceptron

Now, let us take a closer look at the derivative term in Equation 2 by
benefiting from the inner mechanism of a perceptron, which is the building
block of neural networks. Figure 2 shows a perceptron, where the product sum
of a node’s weights with the previous layer’s outputs is called the activation
of this node and calculated as formulated in Equation 3. By using the
activation, the derivative term in Equation 2 can be expressed as the product
of two different derivative terms thanks to the chain rule as shown in
Equation 4.

$a_{i}^{k}=\sum_{j=0}^{r_{k-1}}w_{ji}^{k}o_{j}^{k-1}$ (3)

$\frac{\partial E}{\partial w_{ij}^{k}}=\frac{\partial E}{\partial
a_{j}^{k}}\frac{\partial a_{j}^{k}}{\partial w_{ij}^{k}}$ (4)

where $a_{j}^{k}$ is the activation of node $j$ in layer $k$ before it is
passed to the nonlinear activation function ($g$) (e.g. sigmoid or relu) to
generate the output. The first term in Equation 4 is usually called the error
and denoted as given in Equation 5.

$\delta_{j}^{k}\equiv\frac{\partial E}{\partial a_{j}^{k}}$ (5)

On the other hand, when we rewrite the second term in Equation 4 by combining
with Equation 3 as given in Equation 6, we obtain the output $o_{i}^{k-1}$ of
node $i$ in layer $k-1$.

$\frac{\partial a_{j}^{k}}{\partial w_{ij}^{k}}=\frac{\partial}{\partial
w_{ij}^{k}}\left(\sum_{l=0}^{r_{k-1}}w_{lj}^{k}o_{l}^{k-1}\right)=o_{i}^{k-1}$
(6)

Thus, the partial derivative of the error function $E$ with respect to a
weight $w_{ij}^{k}$ is obtained as given in Equation 7, which is the product
of the error term $\delta_{j}^{k}$ at node $j$ in layer $k$, and the output
$o_{i}^{k-1}$ of node $i$ in layer k-1. This is reasonable because the weight
$w_{ij}^{k}$ is located between the output of node $i$ in layer $k-1$ and the
input of node $j$ in layer $k$.

$\frac{\partial E}{\partial w_{ij}^{k}}=\delta_{j}^{k}o_{i}^{k-1}$ (7)

The Output Layer

In backpropagation, computation of the error terms proceeds backward from the
output layer down to the input layer. Without loss of generality, let us
continue our mathematical analysis by assuming that the error function is the
mean squared error, the activation function is sigmoid, and there is only one
output node. Thus we can define the error function $E$ as in Equation 8.

$E=\frac{1}{2}(\hat{y}-y)^{2}=\frac{1}{2}(g(a_{1}^{m})-y)^{2}$ (8)

where $m$ is the final layer and $g(x)$ is the activation function for the
output layer.

We obtain Equation 9 from Equatıon 5 as applying the partial derivative.

$\delta_{1}^{m}=(g(a_{1}^{m})-y)g^{{}^{\prime}}(a_{1}^{m})=(\hat{y}-y)g^{{}^{\prime}}(a_{1}^{m})$
(9)

Combining all of them, the partial derivative of the error function $E$ with
respect to a weight in the final layer $w_{i1}^{m}$ becomes as given in
Equation 10.

$\frac{\partial E}{\partial
w_{i1}^{m}}=\delta_{1}^{m}o_{i}^{m-1}=(\hat{y}-y)g^{{}^{\prime}}(a_{1}^{m})o_{i}^{m-1}$
(10)

The Hidden Layers

The computation of the error terms in hidden layers is a bit different than
the output layer as it also depends on the error terms in the next forward
layer. Now consider the following equation for the error term $\delta_{j}^{k}$
in layer $1\leq k<m$:

$\delta_{j}^{k}=\frac{\partial E}{\partial
a_{j}^{k}}=\sum_{l=1}^{r^{k+1}}\frac{\partial E}{\partial
a_{l}^{k+1}}\frac{\partial a_{l}^{k+1}}{\partial a_{j}^{k}}$ (11)

where $r^{k+1}$ is the number of nodes in the next layer of ${k+1}$ and $l$
ranges from 1 to $r^{k+1}$ skipping the value 0 because the bias input
$o_{0}^{k}$ corresponding to $w_{0j}^{k+1}$ is fixed and its value is not
dependent on the outputs of previous layers. Given that first partial
derivative term in the product sum of Equation 11 is equivalent to the error
term $\delta_{l}^{k+1}$, Equation 11 can also be written as in Equation 12:

$\delta_{j}^{k}=\sum_{l=1}^{r^{k+1}}\delta_{l}^{k+1}\frac{\partial
a_{l}^{k+1}}{\partial a_{j}^{k}}$ (12)

The second term in Equation 12 can be rewritten as in Equation 13 by using the
definition of $a_{l}^{k+1}$.

$\frac{\partial a_{l}^{k+1}}{\partial
a_{j}^{k}}=w_{jl}^{k+1}g^{{}^{\prime}}(a_{j}^{k})$ (13)

where $g(x)$ is the activation function for the hidden layers.

Putting this into the above equation gives a final equation for the error term
$\delta_{j}^{k}$ in the hidden layers as in Equation 14 [19]:

$\delta_{j}^{k}=\sum_{l=1}^{r_{k+1}}\delta_{j}^{k+1}w_{jl}^{k+1}g^{{}^{\prime}}(a_{j}^{k})$
(14)

Overall

Among the above formulas, we are more concerned with the gradient calculations
because model weights are updated in proportion to the size of the associated
gradient as given in Equations 15-16.

$w_{new}=w_{old}-\alpha\frac{\partial E(X,W)}{\partial w_{old}}$ (15)

$b_{new}=b_{old}-\alpha\frac{\partial E(X,W)}{\partial b_{old}}$ (16)

where $w_{old},w_{new},b_{old}$ and $b_{new}$ are old and updated weight and
bias values respectively, and $\alpha$ is the learning rate. Since $\alpha$ is
the same for all data and local models in an iteration of gradient descent,
any abnormal deviation in weight or bias updates is expected to be due to
gradient. Equation 17 includes the gradient formulas for bias weights and
regular weights at both output layer and hidden layers. Notice that the output
of previous layer is 1 for bias weights as reflected in Equation 17.

It is seen from Equation 17 that the difference between the actual output of a
model and the desired output (i.e.$(\hat{y}-y)$) is a factor that directly
affects the value of a gradient for the output layer. Moreover, this is more
apparent for bias weights with respect to regular weights at the output layer
because the bias gradient depends on only two factors which are the output
error ($\hat{y}-y$) and the derivative of activation function
($g^{{}^{\prime}}(a_{1}^{m})$), whereas there is an additional third factor of
the output of previous layer ($o_{i}^{m-1}$) for regular weights. Therefore,
given that a typical backdoor attack is carried out by mislabeling training
data which results in higher error at the output, we can expect to see higher
gradient values especially for bias gradients at the output layer.

$\frac{\partial E(X,W)}{\partial
w_{ij}^{k}}=\begin{cases}\frac{1}{N}\sum_{d=1}^{N}[(\hat{y}-y)g^{{}^{\prime}}(a_{1}^{m})]_{d}&\text{for
output layer bias weight}\\\
\frac{1}{N}\sum_{d=1}^{N}[(\hat{y}-y)g^{{}^{\prime}}(a_{1}^{m})o_{i}^{m-1}]_{d}&\text{for
output layer regular weights}\\\
\frac{1}{N}\sum_{d=1}^{N}[g^{{}^{\prime}}(a_{j}^{k})\sum_{l=1}^{r_{k+1}}w_{jl}^{k+1}\delta_{l}^{k+1}]_{d}&\text{for
hidden layer bias weight }\\\
\frac{1}{N}\sum_{d=1}^{N}[g^{{}^{\prime}}(a_{j}^{k})o_{i}^{k-1}\sum_{l=1}^{r_{k+1}}w_{jl}^{k+1}\delta_{l}^{k+1}]_{d}&\text{for
hidden layer regular weights }\end{cases}$ (17)

##  5 Experimental Analysis

Figure 3: Neural network architecture used in our experimental work

###  5.1 Experimental Setup

We implemented a federated learning framework based on Tensorflow&Keras in
Python. In our implementation, there is a server node that keeps and
distributes the joint model, and multiple client nodes that update the joint
model using their local data. The server and clients have their own data. The
training dataset is distributed among these nodes (including the server) as
non-i.i.d. At the beginning, the server sets up a neural network model
initialized by training 2 epochs222epoch means the number of passes over the
dataset to carry out learning process with the data located at the server and
then sends this model to the clients. Upon receiving the model from the
server, clients train it 2 epochs by using their local data, and then send
their locally updated model back to the server. Thus the server collects local
models from clients and generates the updated joint model by taking the
average of collected model weights. The procedure up to this point is defined
as one round in federated learning. After calculating the updated joint model,
the server redistributes this updated joint model to clients again and thus
initiates another round. By repeating the procedure in this way, the joint
model is constantly updated in rounds until being stopped by the server
depending on some criteria such as joint model accuracy, convergence level
etc.

We used the CIFAR-10 dataset [7] for experimental analysis. The CIFAR-10
dataset consists of 60000 32x32 color images in 10 different classes, with
6000 images per class, which are split as 50000:10000 for training and test
purposes. The 10 classes in CIFAR-10 represent airplanes, cars, birds, cats,
deer, dogs, frogs, horses, ships, and trucks.

###  5.2 Malicious Client(s) and Data

We identify some of the clients as malicious and manipulate a portion of their
data to be suitable for performing a backdoor attack. Specifically, we change
the labels of the images including a car with a stripe as bird. Thus, similar
to [6], by using such a mislabeled data in the training process of a malicious
client’s model, we target to insert a backdoor into the joint model such that
it is supposed to recognize regular car images as car while recognizing the
images of cars with stripes as bird. Figure 1 shows some samples of such
backdoor data. In our experimental analysis, we increase the number of
malicious clients from 1 (i.e. 10% of all clients) to 5 (i.e. 50% of all
clients) in different set of experiments. When there are multiple malicious
clients, we distribute malicious data among them equally while keeping total
amount of malicious data the same.

###  5.3 CNN Architecture

We build a VGG-like [22] CNN architecture for experimental analysis as
illustrated in Figure 3. Because our purpose in this study is not to achieve
the best classification performance, rather to investigate abnormal variations
in model weights, we set the CNN architecture simple. In our CNN model, there
are two 2D convolution layers with 32 filters, and then a 2D max-pooling layer
followed by a fully connected dense layer including 128 neurons, and finally
another dense layer with 10 neurons as there are 10 classes in CIFAR10.

###  5.4 Results

In Section 4 we make a theoretical analysis showing that a backdoor attack
carried out by a malicious user in a federated learning setup may manifest
itself with abnormal changes in the final layer’s bias values of the local
malicious model. To validate this experimentally, we observe deviations in
local model weights with respect to the joint model for each training round.
In the following set of experiments, the learning rate is set to 0.01 unless
otherwise stated.

####  5.4.1 Varying Number of Malicious Clients

(a) 1 malicious client

(b) 2 malicious clients

(c) 5 malicious clients

Figure 4: Results for the cases including (a) 1 malicious client, (b) 2
malicious clients, and (c) 5 malicious clients out of 10 clients.

In our first experimental setup, there are 10 clients, one of which is
malicious (i.e. the client no 1), and we observe the maximum deviation in the
updated local model weights with respect to the distributed joint model
weights in all training rounds. In the malicious client, we adjust the
backdoored data rate to 1/3 by generating copies of the malicious samples. In
our investigation of the experimental results, we have not observed any
abnormal deviations in the collected local model weights except the final
layer bias values of the malicious client’s model. Figure 4(a) shows our
experimental results for the maximum deviation in the final layer bias values
of local models as well as joint model loss and joint model accuracy for both
main task and backdoor up to the 25th training round. As seen in Figure 4(a),
the maximum deviation curve of the malicious client differs clearly from other
curves at the beginning of training by showing relatively larger values, and
the magnitude of this abnormality gradually decreases until the 7th training
round, after which the malicious model acts normal like the others. Such an
abnormal behavior of the malicious client is indeed consistent with our
theoretical analysis in Section 4 and particularly validates our justification
regarding Equation 17 there. But why does the maximum deviation curve of the
malicious client gradually decrease as training proceeds? Firstly, this is
partly because of the fact that the malicious local model and the joint model
get more converged each other as training proceeds in rounds. Although this
convergence is not enough to effectively insert the backdoor into the joint
model as seen in the Loss and Accuracy plots of Figure 4(a) where backdoor
loss is relatively high and backdoor accuracy seems very low respectively, the
maximum deviation due to malicious data still decreases as the joint model is
updated by collected local models. Furthermore, since the training process
takes place in batches of data, gradients in backpropagation are calculated as
average over all samples in a batch that contains both malicious and
legitimate data in a malicious client given that the malicious data rate is
1/3. Therefore, the maximum deviation decreases as joint model accuracy on the
main task increases even if joint model accuracy on the backdoor does not
increase.

In our second experimental setup, we include 2 malicious clients out of 10
clients and distribute copies of malicious samples equally among them so that
the backdoored sample rate on each malicious client becomes 1/6 as keeping the
amount of cumulative malicious data same. In this experiment, similar to the
previous experiment, we measure the maximum deviations in local model updates
as well as joint model loss and accuracy in consecutive training rounds.
Figure 4(b) shows our experimental results in this experiment. According to
Figure 4(b), the malicious models indicate relatively larger maximum deviation
values with respect to the legitimate clients in the first training rounds,
and then their deviations gradually decrease in the following rounds and reach
the levels that cannot be differentiated from the legitimate models. This is
similar to the malicious client behavior observed in the previous experiment,
but with the difference that (i) the malicious local models and the joint
model get converged faster here at around training round 5 (w.r.t. round 7 in
the previous one-malicious client case), and (ii) two malicious clients seem
to be more successful at inserting a backdoor into the joint model as seen in
the Loss and Accuracy plots of Figure 4(b). Put it differently, this
experiment indicates that two malicious clients carry out more effective
attacks with respect to one-malicious client case even though the amount of
cumulative malicious data is identical in either case.

In our third experimental setup, we increase the number of malicious clients
to 5, which is equivalent to 50% of all clients, while keeping the amount of
cumulative malicious data the same, and repeat the earlier experiments. Figure
4(c) shows the maximum deviations in the final layer’s bias values as well as
Loss and Accuracy curves in 25 consecutive training rounds for this case. As
seen in Figure 4(c), the abnormal behavior of the malicious clients is no
longer apparent in the maximum deviation curves as all the blue and red lines
are intertwined in all rounds. This is intuitive because the malicious clients
are no longer in the minority and therefore their behavior is not perceived as
an anomaly. As seen in the Loss and Accuracy plots of Figures 4(c), joint
model loss and accuracy for the backdoor reach better values with respect to
the main task, which indicates that the number of malicious clients is an
important factor in the effectiveness of the attack.

In addition to the above interpretations, we can also infer from the
experimental results given in this part that malicious model updates are
likely to display abnormal deviations during the attempt of inserting a
backdoor, whereas they no longer indicate such anomaly during the attempts of
maintaining an already inserted backdoor, which implies that it becomes
critical to detect such anomalies in the early stages of backdoor insertion.

####  5.4.2 Impact of Learning Rate

(a) Learning rate = 0.1

(b) Learning rate = 0.01

(c) Learning rate = 0.001

Figure 5: Impact of different learning rates of (a) 0.1, (b) 0.01, and (c)
0.001 on the maximum deviation in the last layer bias values, joint model loss
and accuracy for both main task and backdoor in consecutive training rounds.

In this set of experiments, we deepen our investigation using different
learning rates while involving one malicious client with the malicious data
rate of 1/3. Specifically, we observe the maximum deviation in the final
layer’s bias values of the local models as well as Loss and Accuracy of the
joint model for both the main task and the backdoor in consecutive training
rounds. Figure 5 depicts our experimental results for three different learning
rates of 0.1, 0.01, and 0.001.

Figure 5(a) includes our experimental results for the learning rate of 0.1. As
seen in Figure 5(a), the malicious client indicates abnormal deviations with
sharp ups and downs in most training rounds, which happens most likely due to
(i) high learning rate because the amount of change in the weights during
model training is directly proportional to learning rate according to
Equations 15&16, and (ii) federated learning where the malicious client and
legitimate clients compete with each other to reflect their influence on the
joint model. More specifically, when the backdoor loss reaches a minimal level
in the joint model, the malicious client makes relatively less amount of
contribution for backdoor in the following rounds, which paves the way for
legitimate clients to dominate causing sudden changes in the direction of
increasing backdoor loss. When backdoor loss increases, malicious client’s
impact on the joint model increases again in proportional with learning rate
and loss, and the cycle continues like this until convergence occurs. This
pattern is seen in loss and accuracy plots of Figure 5(a). We also see from
the accuracy plot of Figure 5(a) that the main task and backdoor accuracy can
simultaneously reach maximum accuracy level (i.e. 100%), which means that the
backdoor is successfully inserted without degrading the model’s performance on
the main task.

Figure 5(b) includes our experimental results for the learning rate of 0.01.
This is the same experimental results given in Figure 4(a), and therefore the
interpretations made earlier for Figure 4(a) is also valid for Figure 5(b).

Figure 5(c) includes our experimental results for the learning rate of 0.001.
The malicious client continues to demonstrate abnormal deviations in this case
as seen in Figure 5(c), yet the scale of the deviation is different with
respect to other learning rates, which is intuitive due to the limiting role
of learning rate on the strength of updates as discussed above.

When we comparatively examine all the results given in Figure 5 regarding
three different learning rates, we can see that the change in the joint model
occurs at a rate proportional to the learning rate, which is intuitive.

####  5.4.3 Impact of Malicious Data Rate

(a) Malicious data rate = 1/1

(b) Malicious data rate = 1/3

(c) Malicious data rate = 1/10

Figure 6: Impact of different malicious data rates of (a) 1/1, (b) 1/10, and
(c) 1/50 on the maximum deviation in the last layer bias values, joint model
loss and accuracy for both main task and backdoor in consecutive training
rounds.

In this set of experiments, we investigate the impact of different malicious
data rates on deviations in local model weights. For this purpose, we adjust
three different malicious data rates of 1/1, 1/10, and 1/50, which represent
the ratio of malicious data to all data in malicious clients. We include one
malicious client and set the learning rate to 0.01.

Figure 6(a) shows our experimental results for the malicious data rate of 1/1
which means all data is malicious in the malicious client (i.e. no legitimate
data at all). According to Figure 6(a), the maximum deviation in the malicious
client is abnormally higher than the values seen in the legitimate clients in
all rounds. Contrary to the trend seen in other experimental results, the
maximum deviation value of the malicious client here does not decrease
gradually but remains almost constant. The reason for this lies in the fact
that gradients in backpropagation are calculated as an average value over all
samples in a batch, and therefore this gradient value remains almost constant
given all samples are malicious and backdoor accuracy stays around the same
level in all rounds (the opposite case is explained in Section 5.4.1). We can
also infer from Figure 6(a) that even having 100% malicious data is
insufficient to successfully insert a backdoor into the joint model for the
one-malicious client case.

Figure 6(b) includes our experimental results for the malicious data rate of
1/10. It is seen from Figure 6(b) that maximum deviation in the malicious
client’s model takes abnormal values in the initial training rounds, and then
gradually decreases as training proceeds and reaches the legitimate models’
level at around the 6th training round. This is due to the convergence of
local models and the joint model as well as due to legitimate samples in the
malicious client.

Figure 6(c) shows our experimental results for the malicious data rate of
1/50. Notice the scale of the maximum deviation graph, which is much smaller
than those in the other malicious data rate results, which indicates that the
magnitude of abnormal behavior of malicious client gets softened as malicious
data rate decreases.

##  6 Conclusion

In this work, we study the problem of anomaly localization in model gradients
under backdoor attacks against federated learning. To deepen our knowledge and
understanding about the possible reflections of backdoor attacks on the low-
level structure of neural networks models in various federated learning
environments, we conduct both theoretical and experimental analysis to find,
justify and validate exactly at which weights anomaly can be expected. In this
sense, we discovered that it is more likely to observe backdoor-related
anomalies in final layer bias weights of malicious local models. We also
examined the possible impact of the number of malicious clients, learning
rate, and malicious data rate on anomaly level. Our findings can be used to
optimize backdoor detecting methods such as making them focused on only the
weights where the anomaly is most expected.

## References

  * [1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [2] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos.  Attack of the tails: Yes, you really can backdoor federated learning.  arXiv preprint arXiv:2007.05084, 2020.
  * [3] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li.  Dba: Distributed backdoor attacks against federated learning.  In International Conference on Learning Representations, 2019.
  * [4] Lingjuan Lyu, Han Yu, and Qiang Yang.  Threats to federated learning: A survey.  arXiv preprint arXiv:2003.02133, 2020.
  * [5] Loc Truong, Chace Jones, Brian Hutchinson, Andrew August, Brenda Praggastis, Robert Jasper, Nicole Nichols, and Aaron Tuor.  Systematic evaluation of backdoor data poisoning attacks on image classifiers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 788–789, 2020.
  * [6] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov.  How to backdoor federated learning.  In International Conference on Artificial Intelligence and Statistics, pages 2938–2948. PMLR, 2020.
  * [7] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [8] Paul Voigt and Axel Von dem Bussche.  The eu general data protection regulation (gdpr).  A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:3152676, 2017.
  * [9] Sebastien Andreina, Giorgia Azzurra Marson, Helen Möllering, and Ghassan Karame.  Baffle: Backdoor detection via feedback-based federated learning.  arXiv preprint arXiv:2011.02167, 2020.
  * [10] Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan.  Can you really backdoor federated learning?  arXiv preprint arXiv:1911.07963, 2019.
  * [11] Thien Duc Nguyen, Phillip Rieger, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Ahmad-Reza Sadeghi, Thomas Schneider, et al.  Flguard: Secure and private federated learning.  arXiv preprint arXiv:2101.02281, 2021.
  * [12] Ferhat Karakoç, Melek Önen, and Zeki Bilgin.  Secure aggregation against malicious users.  In Proceedings of the 26th ACM Symposium on Access Control Models and Technologies, SACMAT ’21, page 115–124, New York, NY, USA, 2021\. Association for Computing Machinery.
  * [13] Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim.  Backdoor attacks and countermeasures on deep learning: a comprehensive review.  arXiv preprint arXiv:2007.10760, 2020.
  * [14] Yujie Ji, Xinyang Zhang, and Ting Wang.  Backdoor attacks against learning systems.  In 2017 IEEE Conference on Communications and Network Security (CNS), pages 1–9. IEEE, 2017.
  * [15] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash.  Hidden trigger backdoor attacks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11957–11965, 2020.
  * [16] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 707–723. IEEE, 2019.
  * [17] Zaixi Zhang, Jinyuan Jia, Binghui Wang, and Neil Zhenqiang Gong.  Backdoor attacks to graph neural networks.  arXiv preprint arXiv:2006.11165, 2020.
  * [18] Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra.  Mitigating backdoor attacks in federated learning.  arXiv preprint arXiv:2011.01767, 2020.
  * [19] Brilliant.org.  Backpropagation.  https://brilliant.org/wiki/backpropagation/.  Accessed: 19:08, July 26, 2021.
  * [20] Robert Hecht-Nielsen.  Theory of the backpropagation neural network.  In Neural networks for perception, pages 65–93. Elsevier, 1992\.
  * [21] Brilliant.org.  Backpropagation.  https://brilliant.org/wiki/artificial-neural-network/#training-the-model.  Accessed: 17:16, October 26, 2021.
  * [22] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.


# End-to-end Compression Towards Machine Vision: Network Architecture Design
and Optimization

Shurun Wang, Zhao Wang, Shiqi Wang, Yan Ye This research was supported in part
by the National Natural Science Foundation of China under 62022002, in part by
the Hong Kong Research Grants Council, Early Career Scheme (RGC ECS) under
Grant 21211018, General Research Fund (GRF) under Grant 11203220.S. Wang and
S. Wang are with the Department of Computer Science, City University of Hong
Kong, Hong Kong, China (e-mail<EMAIL_ADDRESS>shiqwang@cityu.edu.hk).Z. Wang and Y. Ye are with the Alibaba Group (e-mail:
<EMAIL_ADDRESS>yan.ye@alibaba-inc.com).

###### Abstract

The visual signal compression is a long-standing problem. Fueled by the recent
advances of deep learning, exciting progress has been made. Despite better
compression performance, existing end-to-end compression algorithms are still
designed towards better signal quality in terms of rate-distortion
optimization. In this paper, we show that the design and optimization of
network architecture could be further improved for compression towards machine
vision. We propose an inverted bottleneck structure for the encoder of the
end-to-end compression towards machine vision, which specifically accounts for
efficient representation of the semantic information. Moreover, we quest the
capability of optimization by incorporating the analytics accuracy into the
optimization process, and the optimality is further explored with generalized
rate-accuracy optimization in an iterative manner. We use object detection as
a showcase for end-to-end compression towards machine vision, and extensive
experiments show that the proposed scheme achieves significant BD-rate savings
in terms of analysis performance. Moreover, the promise of the scheme is also
demonstrated with strong generalization capability towards other machine
vision tasks, due to the enabling of signal-level reconstruction.

###### Index Terms:

Visual signal compression, machine vision, object detection, rate-distortion
optimization

##  I INTRODUCTION

Recent years have witnessed an explosive growth of visual data driven by the
unprecedented proliferation of multimedia acquisition, processing and display
devices. It has been reported that visual data accounts for the largest
proportion in the global data traffic [1]. As such, the compact representation
of visual data is highly demanded and has been extensively studied towards
human perception based on a series of human visual system (HVS)
characteristics. With the notable success of deep learning in various visual
analysis and understanding tasks, machine has become an alternative and
increasingly important terminal for ultimate consumption of visual data. This
is grounded on the widely accepted view that traditional pipelines where
manpower is relied to process and analyze the huge data volume are impractical
for real-time applications such as smart cities and intelligent security.
According to the prediction of Cisco, machine-to-machine (M2M) communications
will occupy half of the internet connections between global devices [2].

The utilization of the visual data towards machine vision primarily relies on
how the data could be efficiently represented in a compact way. Facing such
arising challenges of video compression for machine (VCM), numerous methods
have been developed beyond the traditional video coding schemes. The prominent
paradigm is referred as Analyze-then-Compress (ATC) [3], which is developed
based on the fact that features could be more compact than signals. This is in
stark contrast with the traditional Compress-then-Analyse (CTA) paradigm, as
the features are compressed in ATC whereas the visual signals are compressed
in CTA. Since machine vision relies on features for understanding and
analysis, compact feature representation can dramatically reduce the visual
data representation expense and facilitate various intelligent applications
with high throughput between front-end devices such as sensors and back-end
systems such as central servers. More specifically, the standards of Compact
Descriptors for Visual Search (CDVS) [4] and Compact Descriptors for Video
Analysis (CDVA) [5] standardized highly compact descriptors for images and
videos. The exploration of video coding for machine (VCM) has also been
launched by Moving Picture Experts Group (MPEG), in an effort to extend the
compact representation to various artificial intelligent tasks towards machine
vision [6].

Clearly, the ATC paradigm which dramatically economizes the representation
cost does not ensure the reconstruction at the signal level and significantly
affects the generalization capability for various tasks. This has been an
important aspect ignored by ATC, which lacks strong generalization capability
across different analytics tasks. More importantly, human involved monitoring,
a step that is still indispensable when the event is of sufficient interest,
cannot be supported. As such, the ATC and CTA work in their own way towards
compact visual information representation, and an advanced scheme that
fulfills both purposes is highly desirable. Herein, we propose a customized
scheme for visual signal compression scheme towards machine vision grounded on
the end-to-end deep image coding framework. The proposed scheme is designed
and optimized to fully accommodate for the characteristics of machine vision,
and moreover enjoys the advantages of both high generalization capability and
better rate-accuracy performance. Overall, the main contributions of the paper
are as follows,

  * •

We propose an inverted bottleneck structure from the perspective of channel
number distribution for the encoder of the end-to-end coding towards machine
vision. The proposed structure is motivated by the fact that the semantic
high-level features are more important in compact representations, and leads
to lower encoding complexity without degrading the rate-accuracy performance.

  * •

We propose an encoder optimization scheme to improve the rate-accuracy
performance. The potentials of end-to-end compression for machine vision are
extensively explored with the proposed generalized rate-accuracy optimization
framework.

  * •

We carry out extensive experiments based upon the object detection task to
evaluate the performance in terms of rate-accuracy, showing superior
performance of the proposed scheme. Meanwhile, the superiority of the proposed
scheme over traditional ATC and CTA approaches is also verified through this
process.

The rest of the paper is organized as follows. In Section 2, we review and
summarize the related works. In Section 3, we introduce the whole compression
and analysis pipeline based upon the end-to-end compression framework. The
proposed architecture customized for machine vision is introduced in Section
4, including the motivations, design philosophy and principles. In Section 5,
we present the optimization towards machine vision with off-line search. The
discussions regarding the connections of the proposed method with image
compression, feature compression and visual analysis are provided in Section
6. In Section 7, the experimental results are presented to show the efficiency
of proposed schemes in terms of rate-accuracy and the encoding simplification
in terms of the number of parameters and encoding time. We conclude the paper
in Section 8.

##  II Related Works

Figure 1: The pipeline of the end-to-end image compression towards machine
vision. More specifically, object detection is selected as the show case of
the machine vision task. The encoder network is specifically designed, and the
distortion of signal reconstruction $\mathcal{L}_{mse}$, the object detection
loss towards machine vision $\mathcal{L}_{task}$ and the representation
expense $\mathcal{R}$ can be acquired for joint optimization.

How the visual signals can be compactly represented towards different
objectives has been a long-standing problem. Central to such problem is the
maximization of the ultimate utility within available bit rate, where the
utility can be defined in different manners depending on the final receiver of
the visual signals. Recently, various methods have been proposed, which can be
mainly classified into visual signal compression and compact feature
representation.

###  II-A Visual Signal Compression

The traditional visual signal coding is primarily characterized based on the
throughput of the channel and perceived distortion of the reconstructed
signals. In particular, the image/video coding techniques are driven by the
development of visual data compression standards. More specifically, JPEG [7],
JPEG 2000 [8] and WebP [9] have been developed to encode the still images, and
H.264/AVC [10], H.265/HEVC [11] and the state-of-the-art standards including
Versatile Video Coding (VVC) [12] and AVS3 [13] are developed for video
compression. Central to the image and video coding is the rate-distortion
optimization (RDO), which is pursued to optimize the coding efficiency with
operational control of the encoder [14, 15, 16, 17]. In the research of video
coding, in addition to deriving the best Lagrangian multiplier [18], the
distortion measure has also been replaced towards better modeling of the
visual perception [19].

The recent advancements of deep learning have achieved substantial
breakthroughs in visual computing tasks. Benefiting from the strong
representation capability of neural networks, deep learning has also been
applied to image compression, achieving rapid progress in coding performance.
The recurrent neural network (RNN) based image compression was first proposed,
achieving comparable performance with JPEG [20]. Motivated by the discrete
cosine transform (DCT) in traditional image compression, a block based deep
learning transformation was proposed, which outperforms JPEG at low bit rates
[21]. Inspired by the local gain control, the generalized divisive
normalization (GDN) was proposed in [22], and an end-to-end image compression
based upon GDN was developed, surpassing the performance of JPEG 2000 [23].
The statistical dependency is further exploited with a variational hyper-prior
model [24]. Based on this method, a joint autoregressive and hierarchical
prior for image compression was proposed in [25], achieving performance
improvement comparing with the state-of-the-art image codecs. Although
prominent improvement has been achieved for visual signal representation, the
optimization in the compression process is still based upon HVS instead of
machine vision. As such, there are still many challenges regarding efficient
and compact representation of visual signals towards machine vision in the era
of artificial intelligence.

###  II-B Compact Visual Feature Representation

The ATC paradigm was developed grounded on the widely rooted view that visual
analytics and understanding primarily rely on features instead of textures.
Moreover, the features are much more compact than textures, enabling the
simultaneous transmission of the videos to the central cloud and
redistribution of the computational load between front-end and back-end. To
facilitate the visual retrieval in real-world applications, the standards of
Compact Descriptors for Visual Search (CDVS) [4] and Compact Descriptors for
Video Analysis (CDVA) [5] have been completed. Based on a series of deep
neural networks as the backbone for the deep learning feature extraction, such
as AlexNet [26], VGG [27], Inception Net [28] and ResNet [29], various
algorithms have been proposed. More specifically, the philosophy of hybrid
video coding has been introduced to the joint compression model for the local
and global deep learning features in [30]. An end-to-end deep learning feature
compression with multi-granularity constraint and teacher-student learning was
proposed in [31], achieving performance improvement in terms of rate-analysis
accuracy. A lossy intermediate deep learning feature compression towards
intelligent sensing was proposed in [32], which provides a prominent paradigm
for the standardization of deep learning feature compression. Moreover,
efforts have been devoted to performing image understanding tasks directly on
the compact representations for the DNN-based compression methods [33]. The
exploration of video coding for machine (VCM) [34, 35] has also been launched
by MPEG to plot a whole picture for the compact representation of visual
signals towards machine vision.

##  III The Pipeline of Compression towards Machine Vision

For machine vision, the high-level semantic information extracted from the
network plays a critical role. This is in stark contrast with traditional
compression which targets the visual signal reconstruction. The proposed
scheme, which is built upon the end-to-end compression framework, preserves
the advantage of the signal level reconstruction for high generalization
capability, and is customized for machine vision. Herein, the object detection
is adopted as the show case of the machine vision task as it plays a
fundamental role in various artificial intelligent applications such as
intelligent transportation, smart city and intelligent industry. This also
aligns with the use cases and technical requirements for VCM, since object
detection has been included into the required properties of algorithms under
various tasks, as indicated in [36]. Moreover, object detection plays
fundamental roles in the high-level understanding of visual signal, such as
event detection [37], anomaly detection [38] and tracking [39].

The whole pipeline of the end-to-end image compression towards machine vision
is shown in Fig. 1. The architecture and optimization both play indispensable
roles in compression. The architecture is composed of an end-to-end image
compression codec and a model for the machine vision task. For image
compression, the latent representation $y$ of the original image $x$ is the
output of encoder $E$, which is subsequently quantized with $Q$ as $\hat{y}$.
The decoded image $\hat{x}$ is reconstructed with decoder $D$. In order to
capture spatial dependencies in the latent representation $y$, the hyper-
latent $z$ is acquired with hyper encoder and utilized by hyper decoder after
quantization. Moreover, the bitrate $\mathcal{R}$ is estimated by the Shannon
entropy and the distribution could be modeled with the entropy and factorized
entropy model without context model, as proposed in [25]. For machine vision
task, the object detection model Faster-RCNN [40] is adopted and the results
of object detection, the predicted bounding boxes and labels
$\\{\hat{bbox},\hat{label}\\}$, could be obtained given the decoded image
$\hat{x}$. Herein, our scheme is built upon the typical end-to-end coding
framework in [25]. Within this framework, the encoder network architecture and
optimization method are specifically designed for the machine vision tasks and
introduced in the subsequent sections.

##  IV Encoding Network Architecture

In principle, the encoder is responsible for generating the latent code that
accounts for the compact representation of the original images. The
optimization objective of the end-to-end codecs has always been the quality
towards human vision perception. However, semantic information is more
important for compression towards machine vision. Such inconsistency could
result in representation redundancy and computational inefficiency for the
compression towards machine vision, especially for the front-end devices such
as cameras and mobilephones, where the encoder is deployed with limited
computation power [32]. In particular, there are four convolutional layers in
the encoder of the end-to-end image compression. As shown in Fig. 2, we
visualize the feature maps in various layers of encoder in [25] with min-max
normalization. There is an obvious delamination phenomenon among the first
three layers. Moreover, the similarity among the channels of the last layer is
also investigated with the mean absolute difference (MAD). Specifically, for
every channel $c_{i}$, the channel with the minimal MAD is selected, denoted
as $c_{j},j\neq i$. As shown in Fig. 3, it is interesting to find that around
59 channels share the same most similar channel, revealing abundant
redundancy. Motivated by this, we propose an inverted bottleneck structure for
the encoder towards machine vision to achieve a compact representation with
high efficiency and low complexity, as shown in Fig. 4. Specifically, the
layers of the encoder have been divided into two stages: feature extraction
and compact representation, denoted as S1 and S2 respectively. The channel
number increases in the feature extraction stage to extract semantic features
with high diversity and low complexity for machine vision task, and the
channel number decreases in the compression stage to achieve a compact
representation, serving as the output of encoder.

As illustrated in [41, 29], the delamination representation in deep learning
models enables the deep layers to extract the high-level semantic information
that is very abstract for facilitating the understanding. By contrast, texture
information mainly exists in the first two convolutional layers with high
redundancy, as shown in Fig. 2 (b)&(c). The features in the third
convolutional layer mainly reveal the discriminative information, such as
edges and structures, which are crucial for machine vision task, as shown in
Fig. 2 (d). Such visualization and analysis further support the design
philosophy of the inverted bottleneck structure in terms of the channel number
in the encoder, which decouples the feature representation and compact
representation to some extent, and emphasizes the semantic information for
analysis. There are several advantages of the proposed architecture. First,
the proposed inverted bottleneck structure reduces the computational
redundancy. For the first stage of feature extraction, the channel numbers of
the first two layers are reduced to economize computational cost, and the
channel number of the third layer is retained to preserve the semantic
information for machine vision. Second, the proposed inverted bottleneck
structure can also eliminate the redundancy of compact representation. The
second stage of encoder targets at the compact representation of extracted
semantic features from the first stage. As shown in Fig. 2(e), only several
feature maps contain discriminative information representation. As such, the
channel number of the last layer decreases to achieve an efficient
representation towards machine vision.

Figure 2: (a) The original image. (b)$\sim$(e) Visualizations of the feature
maps of various convolutional layers of a typical end-to-end encoder [25] by
means of min-max normalization. For each layer, the channels are randomly
selected.

Figure 3: Illustration of the channel redundancy of the $4^{th}$ convolutional
layer for a typical end-to-end encoder [25]. Specifically, the horizontal axis
indicates the channel index ranging from 1 to 192. The vertical axis
represents the number of closest channels for every channel index. In
particular, the closest channel is selected by calculating the MAD between the
current channel and every other channel, and finally the channel with the
minimal MAD is selected. The range of MAD is from 0.0010 to 0.1646.

Figure 4: The inverted bottleneck structure of the end-to-end image
compression encoder towards machine vision. “Conv” and “GDN” represent the
convolutional layers and the GDN layers respectively. The $N-K-S$ annotations
indicate the channel number $N$, the kernel size $K$ and the stride $S$ of the
convolutional layer respectively.

##  V Generalized Rate-Accuracy Optimization

Generally speaking, image compression is primarily characterized based upon
the coding bits and distortion, and the fundamental issue is to obtain the
best trade-off between them. Herein, the typical rate-distortion optimization
(RDO) problem is converted to rate-accuracy optimization (RAO), based on the
fact that the ultimate receiver is the machine vision. Such RAO problem has
been widely studied in feature compression and joint texture-feature
compression [42, 43, 44, 45]. In this work, we attempt to take a further step
to develop the generalized RAO which holds promise to improve both
generalization capability and feasibility.

###  V-A Generalized RAO Framework

The RAO framework, which serves as the foundation of the loss function, is
formulated as follows,

$\displaystyle\mathcal{L}$ $\displaystyle=\lambda_{1}\mathcal{D}+\mathcal{R},$
(1)

where $\mathcal{D}$ denotes the generalized distortion which involves both
signal fidelity and analyses accuracy, and $\mathcal{R}$ denotes the coding
bits. The Lagrange multiplier $\lambda_{1}$ controls the tradeoff between
$\mathcal{R}$ and $\mathcal{D}$, and we follow the typical end-to-end coding
for setting the Lagrange multiplier [46]. Central to this optimization problem
is the definition of $\mathcal{D}$, as it is generally acknowledged that the
final quality can only be as good as what it is optimized for. As the final
receiver is assumed to be the machine vision, the optimization goal should be
the analysis accuracy. However, such optimization requires the deterministic
network model and parameters for analysis, while there is a lack of the
generic deep learning model that could be applied to a broad range of analysis
tasks. This motivates us to incorporate the signal level distortion into $D$,
which is given by,

$\displaystyle\mathcal{D}$
$\displaystyle=\mathcal{L}_{mse}+\lambda_{2}\mathcal{L}_{task},$ (2)

where the task loss $\mathcal{L}_{task}$ indicates the accuracy in machine
vision, which is typically defined as [40] in object detection. Herein, we
introduce the mean squared error (MSE) between the original and reconstructed
images $\mathcal{L}_{mse}$, which is incorporated as the regularization term
for modeling the signal-level distortion. This indicates that the signal
quality is simultaneously preserved, which improves the generalization
capability of the proposed framework. The parameter $\lambda_{2}$ is the
weighting factor that controls the signal level distortion and accuracy.

###  V-B Generalized Distortion Modeling

In this subsection, we introduce the searching strategy to obtain the most
appropriate $\lambda_{2}$ in practice. It is worth mentioning that there are
infinite candidates of $\lambda_{2}$ values, while it is impractical to
perform the global search for the optimal one. By contrast, the empirical
selection of $\lambda_{2}$ can also limit the performance. To tackle this
problem, we propose the optimization based on the off-line searching, in an
effort to achieve enhanced performance towards machine vision.

Herein, we assume that there are $n$ quality levels in total, referred to as
$q_{i},1\leq i\leq n$. Various values of $\lambda_{1}$ lead to multiple
quality degradation levels in compression, which are denoted as the quality
level. For each quality level, it corresponds to a set of potential
$\lambda_{2}$ values, and the optimal one is selected based on the rate-
accuracy (RA) cost. To reduce the number of potential $\lambda_{2}$ values in
RA cost calculation in order to reduce the amount of tedious training and
testing, we propose the iterative approach. In particular, the candidate set
of $\lambda_{2}$ for quality $q_{i}$ at the $t^{th}$ iteration is determined
with a central point $\lambda_{2,i}$ and an interval $d$, denoted as
$\bm{\lambda_{2,i_{t}}}=\\{\lambda_{2,i_{t}}-d,~{}\lambda_{2,i_{t}},~{}\lambda_{2,i_{t}}+d\\}=\\{\lambda_{2,i_{t},1},~{}\lambda_{2,i_{t},2},~{}\lambda_{2,i_{t},3}\\}$.
The initial values of $\lambda_{2,t}$ and $d$ are empirically selected. For
$t^{th}$ iteration and quality level $q_{i}$, the end-to-end image compression
is optimized with every candidate in $\bm{\lambda_{2,i_{t}}}$ and the object
detection is performed on the decoded images for the RA cost calculation.
Herein, in RA cost calculation, we adopt the evaluation metric in MPEG VCM
[47], where the mean average precision (mAP) with the intersection of union
from 0.5 to 0.95 with interval 0.05 is selected as the performance metric of
object detection, denoted as mAP@0.5:0.95 and referred as $map$ for
convenience. As such, the accuracy variation of the object detection could be
defined as,

$\mathcal{D}_{task}=(map(x)-map(\hat{x}))/map(x).$ (3)

where $x$ denotes the original image and $\hat{x}$ denotes the decoded image.

Herein, the RA cost $J_{i_{t},j}$ is defined as

$\displaystyle J_{i_{t},j}$
$\displaystyle=\gamma_{i_{t},j}\mathcal{R}_{i_{t},j}+\mathcal{D}_{task,i_{t},j}$
(4)

where $i_{t},j$ denotes the $i^{th}$ quality level, $t^{th}$ iteration and the
$j^{th}$ $\lambda_{2}$ candidate, $j=1,2,3$. Again, $\gamma_{i_{t},j}$ denotes
the Lagrangian multiplier and is obtained with the curve fitting with the
cubic polynomial function of all RA cost points in the first iteration. For
every iteration, we select the one with the minimal RA cost.

A bidirectional search for the new candidates of $\lambda_{2}$ with interval
$d$ is performed afterwards. Specifically, in $t^{th}$ iteration, the selected
$\lambda_{2}$ with minimal RA cost for $i^{th}$ quality level is denoted as
$\lambda_{2,i_{t}}^{*}$. If $\lambda_{2,i_{t}}^{*}=\lambda_{2,i_{t-1}}^{*}$,
the interval $d$ should be shrunk with factor $w$, $d=d/w$. Otherwise, $d$
remains unchanged. The new candidates for the $t+1^{th}$ iteration at $i^{th}$
quality level could be
$\bm{\lambda_{2,i_{t+1}}}=\\{\lambda_{2,i_{t}}^{*}-d,\lambda_{2,i_{t}}^{*},\lambda_{2,i_{t}}^{*}+d\\}$.
Such strategy ensures that the proposed algorithm could achieve a
monotonically decreasing RA cost across iterations, finally terminate after
finite procedures. The pipeline of the proposed algorithm is shown in
Algorithm 1.

Algorithm 1 The algorithm pipeline of the optimization based on the off-line
searching.

0: The start points of various quality levels,
$\bm{\lambda_{2,i_{1}}}=\\{\lambda_{2,i_{1},j}\\}$, $1\leq i\leq n$ and
$j=1,2,3$. The initial interval $d$, the shrink factor $w$, the maximal
iteration number $N$ and $t=1$.

0: The optimized $\lambda_{2}$ value at various quality levels,
$\\{\lambda_{2,i}\\}$, $1\leq i\leq n$.

repeat

Optimize the compression model with the $\lambda_{2}$ candidates in $t^{th}$
iteration at $i^{th}$ quality level, $\\{\lambda_{2,i_{t},j}\\}$,$j=1,2,3$.

Evaluate the RA cost, defined in Equ. 4, with the trained network,
$\\{(\mathcal{R}_{i_{t},j},\mathcal{D}_{task,i_{t},j})\\}$.

Select the $\lambda_{2}$ candidate with minimal RA cost, denoted as
$\lambda^{*}_{2,i_{t}}$.

if $\lambda^{*}_{2,i_{t}}$=$\lambda^{*}_{2,i_{t-1}}$ then

$d=d/w$

end if

The $\lambda_{2}$ candidates in $t+1^{th}$ iteration is
$\bm{\lambda_{2,i_{t+1}}}=\\{\lambda_{2,i_{t}}^{*}-d,\lambda_{2,i_{t}}^{*},\lambda_{2,i_{t}}^{*}+d\\}$.

$t=t+1$.

until $t=N$

TABLE I: The channel distributions at two stages. “Cons”, “Down” and “Up” represent the constant, monotonically decreasing and monotonically increasing tendency at each stage respectively. The channel distributions of S2-C and S1-I are identical. | Stage1 (Conv1,2,3) | Stage2 (Conv4)
---|---|---
Tendency |  Cons |  Down |  Up |  Cons |  Down |  Up
Notation |  S1-C |  S1-D |  S1-I |  S2-C |  S2-D |  S2-I
Conv1 |  192 |  320 |  64 |  64 |  64 |  64
Conv2 |  192 |  256 |  128 |  128 |  128 |  128
Conv3 |  192 |  192 |  192 |  192 |  192 |  192
Conv4 |  192 |  192 |  192 |  192 |  128 |  256

(a)

(b)

(c)

Figure 5: The performance comparison between the three distributions for stage
1 in terms of (a) rate-mAP@0.50:0.95; (b<EMAIL_ADDRESS>(c<EMAIL_ADDRESS>
(a)

(b)

(c)

Figure 6: The performance comparison among the three distributions for stage 2
in terms of (a) rate-mAP@0.50:0.95; (b<EMAIL_ADDRESS>(c<EMAIL_ADDRESS>
##  VI Discussions

The proposed scheme has close connections to a number of image compression,
feature compression and analysis methods.

  * •

Connections to image compression. The state-of-the-art image compression
schemes remove the redundancy in terms of spatial, statistics and perception.
The proposed scheme still inherits the image compression pipeline within CTA
framework, and attempts to further remove the redundancies from the
perspective of machine vision. In this vein, it is interesting to find that
there is still large room to improve the coding performance in terms of RAO.

  * •

Connections to feature compression. Though the feature compression could lead
to more compact representation, it does not guarantee the reconstruction of
the texture and lacks the generalization capability to a certain range of
analysis tasks. The proposed scheme is optimized for machine vision, but still
preserves the capability of signal level reconstruction. This brings the
advantages of enhanced generalization capability to unseen tasks and human-
level monitoring.

  * •

Connections to analysis methods. The proposed scheme treats object detection
as the analysis task, due to the fact that object detection is the foundation
of numerous machine vision applications. It is envisioned that in the future
with the increase in the scale of deep learning models, most machine vision
tasks are expected to be unified with one architecture. This could bring more
feasibilities to the design of the coding scheme towards machine vision, in
particular from the standardization perspective.

TABLE II: The number of parameters and the encoding time for the encoder with various channel distributions. Herein, the parameter number is denoted as “$\\#$ of Param.” in the table. For every channel distribution, the encoding time is averaged with various quality levels. The proposed inverted bottleneck encoder with channel distribution S2-D achieves 48.23% and 35.35% reduction in terms of parameter number and encoding time respectively, compared with the constant channel distribution S1-C. |  S1-C |  S1-D |  S2-C(S1-I) |  S2-D |  S2-I
---|---|---|---|---|---
$\\#$ of Param. |  2,891,136 |  4,428,928 |  1,803,904 |  1,496,640 |  2,111,168
Encoding time (second) |  2.778 |  3.694 |  1.986 |  1.796 |  2.288

##  VII Experiments

(a)

(b)

(c)

Figure 7: The rate-distortion performance comparison among proposed GRAO, GRAO
(empirical), Baseline, and VVC in terms of (a) $\mathcal{D}_{task}$; (b) PSNR;
(c) MS-SSIM.

(a)

(b)

(c)

Figure 8: The performance comparison of the proposed algorithm with Baseline
and VVC in terms of (a) rate-mAP@0.50:0.95; (b<EMAIL_ADDRESS>(c) rate-
<EMAIL_ADDRESS>
To validate the efficiency of the proposed scheme, we evaluate the rate-
accuracy performance of the proposed algorithm under various compression
quality levels, compared with the state-of-the-art codecs. The experimental
setup of the proposed algorithm is first introduced. Subsequently, the
experimental results of the proposed inverted bottleneck structure are
presented, and the computational complexity is also analyzed. Moreover, the
performance of the proposed generalized RAO is investigated in terms of
machine vision and the generalization capability towards other tasks is
validated from the perspective of the signal-level reconstruction performance.
Finally, the image compression performance towards object detection with the
combination of proposed inverted bottleneck encoder and generalized rate-
accuracy optimization is presented, comparing with state-of-the-art image
coding schemes including VVC and end-to-end image compression.

###  VII-A Experimental Setup

The end-to-end image compression towards machine vision is implemented using
PyTorch [48]. For the image compression model, the learning rate of the
en/decoder and the entropy model are set to $0.0001$ and $0.001$ respectively.
The optimization method is Adaptive Moment Estimation (Adam) [49]. The
training and testing data are the training and validation part of COCO2017
dataset [50] respectively. In order to achieve stable compression performance,
the end-to-end image compression model is first trained under the conventional
rate-distortion loss function, where $\mathcal{L}_{task}$ is not involved. The
batch size and the patch size are 16 and $256\times 256$ respectively. The
model is trained with 70 epochs and the setting of $\lambda_{1}$ under various
compression levels is the same as [46]. Regarding the optimization of image
compression with the proposed joint loss function, the parameter in the codec
is initialized with the pretrained model with the same $\lambda_{1}$ value.
The parameters of Faster-RCNN are initialized and fixed with the released
model in [51]. The patch size is enlarged to $512\times 512$ to contain the
objects with multiple scales. The batch size is set to 8 with the training
epoch 1. Moreover, the initial $\lambda_{2}$ and interval $d$ are empirically
set to 40 and 32 respectively. The shrinking factor $w$ and the maximal
iteration number $N$ are 2 and 4, respectively. We evaluate the compression
performance of VVC using the open source implementation VVenC [52] with YUV444
format under slower preset.

###  VII-B Performance of Inverted Bottleneck Encoder

We divide the evaluation into two stages, where the first stage focuses on the
effectiveness of the monotonically increasing channel numbers comparing with
other solutions (monotonically decreasing and constant). Grounded on this
design, the second stage targets to investigate the advantage by decreasing
the channel number, which is further compared to the solution with increasing
and constant channel numbers. More specifically, the channel distribution
under these two stages is listed in Table I.

Figure 9: Visualization of the reconstructed images of the proposed framework,
compared with the VVC and baseline model. For every compression codec, the MAD
between the reconstructed and the original images are visualized with min-max
normalization, accompanying with the bitrate and the analysis accuracy. The
red rectangles indicate the regions that are compressed aggressively, as they
are not particularly important in the analysis task.

(a)

(b)

(c)

Figure 10: The performance comparison for instance segmentation of the
proposed algorithm with Baseline and VVC in terms of (a) rate-mAP@0.50:0.95;
(b<EMAIL_ADDRESS>(c<EMAIL_ADDRESS>
For the first stage, we evaluate the rate-analysis performance in terms of
mAP@0.5:0.95<EMAIL_ADDRESS>and<EMAIL_ADDRESS>all of which have been widely used in
object detection evaluation [40, 53]. As shown in Fig. 5, the compression
model with three distributions (S1-C, S1-D and S1-I) reveal very close
performance under various analysis accuracy measures. It is apparent that for
the model S1-I which economizes the channels could achieve the comparable
performance with lower computational complexity. These experimental results
provide useful evidence on the effectiveness of the proposed monotonically
increasing structure for feature extraction. Furthermore, based on such design
of channel distribution in stage 1, which is fixed as the monotonically
increasing distribution, we evaluate the object detection performance under
various bitrates of the three distributions in the stage 2. As shown in Fig.
6, comparable performance has also been achieved under the three distributions
and the encoder structure with S2-D distribution achieves better performance
at low bitrate. As shown in Table II, the inverted bottleneck structure with
channel distribution S2-D could reduce the computational expense with the
fewest number of parameters, which is also verified with encoding time
consumption. Consequently, by combining the stage 1 and stage 2, the proposed
inverted bottleneck structure with channel distribution S2-D could compactly
represent the images with comparable representation efficiency and lower
computational complexity.

###  VII-C Performance of Generalized Rate-accuracy Optimization

In this subsection, we evaluate the performance of proposed generalized RAO
(GRAO) framework. In particular, the performance comparison is achieved in
terms of the rate-$\mathcal{D}_{task}$ performance, and the compared methods
include the method without GRAO and the empirical selected $\lambda_{2}$ value
with GRAO, denoted as Baseline and GRAO (empirical) respectively.
Specifically, the loss function of Baseline model is
$\mathcal{L}=\lambda_{1}\mathcal{L}_{mse}+\mathcal{R}$ and the loss function
of GRAO-based models follows the formulation of Equ. 1. As shown in Fig.
7.(a), there is an obvious performance improvement of the proposed GRAO
framework and the proposed iterative GRAO can further improve the compression
performance comparing with GRAO (empirical). In particular, the performance
gain is more obvious for low bit rate scenarios due to the fact that the
optimization plays a more important role in low bit rate coding scenarios. In
contrast, the performance gain saturates at high bit rates as the degradation
of image compression for analysis tasks is not obvious in high bit rate
scenarios. The signal-level reconstruction performance of the proposed GRAO
framework is also shown in Fig. 7(b) in terms of rate-PSNR. A signal-level
representation performance with acceptable performance degradation compared
with Baseline is observed. This further demonstrates the promising
generalization capability of the proposed scheme towards other tasks due to
the signal-level information representation.

###  VII-D The Overall Performance Evaluation

We combine the proposed inverted bottleneck structure and the off-line
searching algorithm together to evaluate the performance improvement in terms
of machine vision with various measures. Herein, we directly apply the
$\lambda_{2}$ values in Section 6.3, in an effort to study the straightforward
combination of these two schemes. The performance is shown in Fig. 8 and it
confirms the proposed scheme could achieve significant performance improvement
under various evaluation metrics, especially at low bitrates. Specifically,
comparing with VVC, the proposed scheme could achieve 9.06% bit rate savings
in terms of mAP@0.50:0.95 with Bjøntegaard-Delta rate [54]. Moreover, compared
to the end-to-end compression without any modification, the proposed scheme
can also achieve significant coding bits savings, revealing the promise of the
proposed scheme in a variety of machine vision applications. The visualization
of the reconstructed images is also provided in Fig. 9. The MAD between the
original and reconstructed images of various compression codecs are shown. It
can be observed that the regions that are not particularly important in the
analysis task could be substantially compressed in the proposed scheme,
revealing the design philosophy of the proposed coding technique.

###  VII-E The Performance of Generalization Capability

The generalization capability is further investigated in this subsection. In
principle, the loss function of object detection in the optimization scheme
tends to preserve the semantic information in the compression, potentially
leading to better performance in other analysis tasks. More specifically, we
evaluate the instance segmentation performance on the reconstructed images of
the proposed scheme comparing with VVC and typical end-to-end image
compression [25] under pretrained instance segmentation Mask-RCNN [51]. As
shown in Fig. 10, there is an obvious performance improvement for instance
segmentation, although the image compression model is designed for object
detection task. This also provides more evidence regarding the generalization
capability of the proposed scheme.

##  VIII Conclusion

We propose an end-to-end compression scheme tailored for machine vision, based
upon the inverted bottleneck encoding architecture and iterative RAO scheme.
The novelty of the proposed scheme lies in the new coding network design and
the distortion modeling for RAO, which not only ensures the analysis
performance but also maintains the capability of signal reconstruction. The
benefits of the proposed scheme in terms of computational complexity, rate-
accuracy performance and generalization capability are demonstrated using
extensive experiments.

The proposed compression scheme is extensible. For example, more analysis
tasks can be combined into the analysis module, making the scheme more
sophisticated and powerful. Moreover, the proposed scheme could also be
extended to scalable representation, where the output of the end-to-end codec
could directly serve as the input module for analysis as the base layer. One
may also improve the proposed scheme by considering more variants of rate-
accuracy models for optimization. Moreover, the extension of this codec toward
a more unified compression scheme, in particular for the early feature
extraction stage, is also an interesting research direction yet to be
explored.

## References

  * [1] U. Cisco, “Cisco annual internet report (2017–2022) white paper,” 2019.
  * [2] ——, “Cisco annual internet report (2018–2023) white paper,” 2020.
  * [3] A. Redondi, L. Baroffio, M. Cesana, and M. Tagliasacchi, “Compress-then-analyze vs. analyze-then-compress: Two paradigms for image analysis in visual sensor networks,” in _IEEE International Workshop on Multimedia Signal Processing_. IEEE, 2013, pp. 278–282.
  * [4] L.-Y. Duan, V. Chandrasekhar, J. Chen, J. Lin, Z. Wang, T. Huang, B. Girod, and W. Gao, “Overview of the MPEG-CDVS standard,” _IEEE Transactions on Image Processing_ , vol. 25, no. 1, pp. 179–194, 2015.
  * [5] L.-Y. Duan, Y. Lou, Y. Bai, T. Huang, W. Gao, V. Chandrasekhar, J. Lin, S. Wang, and A. C. Kot, “Compact descriptors for video analysis: The emerging MPEG standard,” _IEEE MultiMedia_ , vol. 26, no. 2, pp. 44–54, 2018.
  * [6] S. Xia, K. Liang, W. Yang, L.-Y. Duan, and J. Liu, “An emerging coding paradigm VCM: A scalable coding approach beyond feature and signal,” _arXiv preprint arXiv:2001.03004_ , 2020.
  * [7] G. K. Wallace, “The JPEG still picture compression standard,” _IEEE Transactions on Consumer Electronics_ , vol. 38, no. 1, pp. xviii–xxxiv, 1992\.
  * [8] M. Rabbani, “JPEG2000: Image compression fundamentals, standards and practice,” _Journal of Electronic Imaging_ , vol. 11, no. 2, p. 286, 2002\.
  * [9] L. Lian and W. Shilei, “Webp: A new image compression format based on vp8 encoding,” _Microcontrollers & Embedded Systems_, vol. 3, 2012.
  * [10] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H. 264/AVC video coding standard,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 13, no. 7, pp. 560–576, 2003.
  * [11] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, no. 12, pp. 1649–1668, 2012\.
  * [12] Y.-J. Choi, D.-S. Jun, W.-S. Cheong, and B.-G. Kim, “Design of efficient perspective affine motion estimation/compensation for versatile video coding (VVC) standard,” _Electronics_ , vol. 8, no. 9, p. 993, 2019.
  * [13] J. Zhang, C. Jia, M. Lei, S. Wang, S. Ma, and W. Gao, “Recent development of AVS video coding standard: AVS3,” in _2019 Picture Coding Symposium (PCS)_. IEEE, 2019, pp. 1–5.
  * [14] K. Ramchandran and M. Vetterli, “Rate-distortion optimal fast thresholding with complete JPEG/MPEG decoder compatibility,” _IEEE Transactions on image processing_ , vol. 3, no. 5, pp. 700–704, 1994.
  * [15] G. J. Sullivan and T. Wiegand, “Rate-distortion optimization for video compression,” _IEEE signal processing magazine_ , vol. 15, no. 6, pp. 74–90, 1998.
  * [16] J. Stankowski, C. Korzeniewski, M. Domański, and T. Grajek, “Rate-distortion optimized quantization in HEVC: Performance limitations,” in _2015 Picture coding symposium (PCS)_. IEEE, 2015, pp. 85–89.
  * [17] M. Karczewicz, Y. Ye, and I. Chong, “Rate distortion optimized quantization,” _ITU-T Q_ , vol. 6, 2008.
  * [18] X. Li, N. Oertel, A. Hutter, and A. Kaup, “Laplace distribution based lagrangian rate distortion optimization for hybrid video coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 19, no. 2, pp. 193–205, 2008.
  * [19] S. Wang, A. Rehman, Z. Wang, S. Ma, and W. Gao, “SSIM-motivated rate-distortion optimization for video coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, no. 4, pp. 516–529, 2011\.
  * [20] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression with recurrent neural networks,” _ICLR_ , 2016.
  * [21] D. Liu, H. Ma, Z. Xiong, and F. Wu, “CNN-based DCT-like transform for image compression,” in _International Conference on Multimedia Modeling_. Springer, 2018, pp. 61–72.
  * [22] J. Ballé, V. Laparra, and E. P. Simoncelli, “Density modeling of images using a generalized normalization transformation,” _arXiv preprint arXiv:1511.06281_ , 2015.
  * [23] J. Ballé, V. Laparra, and E. Simoncelli, “End-to-end optimized image compression,” in _International Conference on Learning Representations_ , 2017.
  * [24] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in _International Conference on Learning Representations_ , 2018.
  * [25] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” _arXiv preprint arXiv:1809.02736_ , 2018.
  * [26] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , 2012, pp. 1097–1105.
  * [27] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [28] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [30] L. Ding, Y. Tian, H. Fan, C. Chen, and T. Huang, “Joint coding of local and global deep features in videos for visual search,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3734–3749, 2020.
  * [31] S. Wang, W. Yang, and S. Wang, “End-to-end facial deep learning feature compression with teacher-student enhancement,” _arXiv preprint arXiv:2002.03627_ , 2020.
  * [32] Z. Chen, K. Fan, S. Wang, L. Duan, W. Lin, and A. C. Kot, “Toward intelligent sensing: Intermediate deep feature compression,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 2230–2243, 2020.
  * [33] R. Torfason, F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool, “Towards image understanding from deep compression without decoding,” _arXiv preprint arXiv:1803.06131_ , 2018.
  * [34] L.-Y. Duan, J. Liu, W. Yang, T. Huang, and W. Gao, “Video coding for machines: A paradigm of collaborative compression and intelligent analytics,” _arXiv preprint arXiv:2001.03569_ , 2020.
  * [35] Y. Z. M. Rafie and S. Liu, “Draft of call for evidence for video coding for machines,” _MPEG doc. m56229 and ISO/IEC JTC 1/SC 29/WG 2_ , 2021.
  * [36] ——, “Use cases and requirements for video coding for machines,” _MPEG doc. m56227 and ISO/IEC JTC 1/SC 29/WG 2_ , 2021.
  * [37] Y. Ke, R. Sukthankar, and M. Hebert, “Event detection in crowded videos,” in _2007 IEEE 11th International Conference on Computer Vision_. IEEE, 2007, pp. 1–8.
  * [38] A. Basharat, A. Gritai, and M. Shah, “Learning object motion patterns for anomaly detection and improved object detection,” in _2008 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–8.
  * [39] S. Balaji and S. Karthikeyan, “A survey on moving object tracking using image processing,” in _2017 11th international conference on intelligent systems and control (ISCO)_. IEEE, 2017, pp. 469–474.
  * [40] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _arXiv preprint arXiv:1506.01497_ , 2015.
  * [41] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _European conference on computer vision_. Springer, 2014, pp. 818–833.
  * [42] Y. Li, C. Jia, S. Wang, X. Zhang, S. Wang, S. Ma, and W. Gao, “Joint rate-distortion optimization for simultaneous texture and deep feature compression of facial images,” in _2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM)_. IEEE, 2018, pp. 1–5.
  * [43] X. Zhang, S. Ma, S. Wang, X. Zhang, H. Sun, and W. Gao, “A joint compression scheme of video feature descriptors and visual content,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 633–647, 2016.
  * [44] S. Ma, X. Zhang, S. Wang, X. Zhang, C. Jia, and S. Wang, “Joint feature and texture coding: Toward smart video representation via front-end intelligence,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 3095–3105, 2018.
  * [45] L. Ding, Y. Tian, H. Fan, Y. Wang, and T. Huang, “Rate-performance-loss optimization for inter-frame deep feature coding from videos,” _IEEE Transactions on Image Processing_ , vol. 26, no. 12, pp. 5743–5757, 2017.
  * [46] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” _arXiv preprint arXiv:2011.03029_ , 2020.
  * [47] Y. Z. M. Rafie and S. Liu, “Draft of evaluation framework for video coding for machines,” _MPEG doc. m56228 and ISO/IEC JTC 1/SC 29/WG 2_ , 2021.
  * [48] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _arXiv preprint arXiv:1912.01703_ , 2019.
  * [49] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [50] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [51] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [52] H. Fraunhofer, “Vvenc software repository,” 2020.
  * [53] R. J. Wang, X. Li, and C. X. Ling, “Pelee: A real-time object detection system on mobile devices,” _arXiv preprint arXiv:1804.06882_ , 2018.
  * [54] L. L. Christopher Hollmann, “On the evaluation of VCM proposals,” _MPEG doc. m55854 and ISO/IEC JTC 1/SC 29/WG 2_ , 2021.