# Privacy-Preserving Autoencoder for Collaborative Object Detection

Bardia Azizian and Ivan V. Bajić The authors are with the School of
Engineering Science, Simon Fraser University, Burnaby, BC V5A 1S6 Canada
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Privacy is a crucial concern in collaborative machine vision where a part of a
Deep Neural Network (DNN) model runs on the edge, and the rest is executed on
the cloud. In such applications, the machine vision model does not need the
exact visual content to perform its task. Taking advantage of this potential,
private information could be removed from the data insofar as it does not
significantly impair the accuracy of the machine vision system. In this paper,
we present an autoencoder-style network integrated within an object detection
pipeline, which generates a latent representation of the input image that
preserves task-relevant information while removing private information. Our
approach employs an adversarial training strategy that not only removes
private information from the bottleneck of the autoencoder but also promotes
improved compression efficiency for feature channels coded by conventional
codecs like VVC-Intra. We assess the proposed system using a realistic
evaluation framework for privacy, directly measuring face and license plate
recognition accuracy. Experimental results show that our proposed method is
able to reduce the bitrate significantly at the same object detection accuracy
compared to coding the input images directly, while keeping the face and
license plate recognition accuracy on the images recovered from the bottleneck
features low, implying strong privacy protection.

###### Index Terms:

Deep neural network, coding for machines, privacy, model inversion attack,
collaborative intelligence, adversarial training, feature compression

##  I Introduction

With recent advancements in Artificial Intelligence (AI) applications such as
autonomous driving, visual surveillance, and Internet of Things (IoT), the
volume of data being transmitted between “intelligent” edge devices (e.g.,
cameras) and the cloud-based services is rapidly increasing. In these
applications, the inference is performed collaboratively between an edge
device, which often has limited computational capabilities, and a more
powerful computing entity commonly referred to as the cloud. Given the rapidly
growing number of such machine-to-machine (M2M) connections [1], there is an
urgent need for an efficient data compression methodology tailored to
automated machine-based analysis. JPEG AI [2] and MPEG-VCM [3] are two
prominent standardization groups actively working in this area to address this
pressing need for data compression for machines [4]. As part of our research
objectives in this work, we also aim to improve compression efficiency for
such collaborative M2M analysis systems.

At the same time, privacy has always been a major concern in data
communications. This concern becomes increasingly critical as the volume of
data collected and passed around among edge devices and the cloud continues to
grow. Consider a scenario where an adversary gains unauthorized access to data
collected by traffic cameras and other sensors used in traffic monitoring. If
they are able to identify individuals from such data, they could track them,
determine their lifestyle patterns, and then use such knowledge for a variety
of illegal and harmful purposes. Similar concerns exist in smart home
applications, independent living, etc. Cryptography schemes such as data
encryption [5] or secret sharing [6, 7] offer one possible solution for
protecting data, although they come with their own risks and challenges, like
the need for a trusted party.

Figure 1: An object detection model deployed in a collaborative intelligence
framework with an adversary performing a model inversion attack

Collaborative intelligence [8] (also referred to as collaborative inference)
has the potential to mitigate the risk of exposing private information, as
data passed between the edge device and the cloud are not the original input
data, but some intermediate features of a DNN model performing the intended
analysis. In this framework, the first few layers of a DNN model are executed
on the edge, and the resulting features are coded and transmitted to the
cloud, where the remaining layers are deployed. Distributing the computations
of DNNs among edge and cloud can bring latency and power efficiency [9, 10,
11]. While this approach reveals only a latent representation of the input
data, it remains vulnerable to the Model Inversion Attack (MIA) [12] through
which the input data can be recovered to some extent, as illustrated in Fig.
1. To counter this attack, one strategy would be to remove task-irrelevant
information – and, in doing so, private information as well – while retaining
only task-relevant information in the communicated data. However,
distinguishing between task-relevant, task-irrelevant, and private information
remains a challenge, especially in real-world visual data.

This paper is an extension of our previous study [13] and presents the
following novel contributions:

  1. 1.

Addressing both compression efficiency and privacy simultaneously: Data
compression and privacy have traditionally been considered two distinct
subjects in the literature. In this paper, however, we propose a practical
framework with improved privacy and compression efficiency, which could be
deployed in a collaborative intelligence setting.

  2. 2.

Deeper theoretical analysis of the proposed approach: We analyze MIA from an
information-theoretic perspective, highlighting some of the challenges and how
our approach tackles them.

  3. 3.

A novel adversarial training technique: The proposed training procedure and
the loss functions are designed in a way to encourage higher compression
efficiency and machine vision accuracy while simultaneously discouraging
precise input reconstruction from transmitted data.

  4. 4.

A new evaluation framework for privacy assessment: Most MIA-related research,
including our earlier work [13], uses perceptual quality metrics as proxy
criteria for privacy. In contrast, our proposed evaluation framework considers
personal identity as private information and examines how well can an
adversary infer personal identity from images containing faces or license
plates.111License plates can be associated with personal identity through
vehicle license records.

The remainder of the paper is organized as follows. In Section II, a review of
the related work is provided. Section III presents the preliminaries needed
for subsequent material, as well as information-theoretic analysis and the
resulting motivation for the proposed approach. Sections IV and V present our
proposed methods and evaluation framework. The experimental results are given
in Section VI, followed by the conclusions in Section VII.

##  II Related Work

For decades, image and video codecs [14, 15, 16] have evolved with a focus on
reducing the number of bits representing input data while maintaining fidelity
between the original and reconstructed input. The latter has been pursued by
improving perceptual quality metrics [17], as the end users have traditionally
been humans. However, in emerging applications such as traffic monitoring,
smart homes, etc., visual content is almost exclusively “looked at” by
machines, with only occasional, if any, human viewing. In such cases, coding
only the task-relevant features derived from the input is more advantageous
than (auto-)encoding the input itself, in terms of both compression efficiency
and privacy. Even prior to the current wave of interest in DNNs, the
significance of feature coding had been recognized through the development of
MPEG standards such as Compact Descriptors for Visual Search (CDVS) [18] and
Compact Descriptors for Visual Analysis (CDVA) [19]. More recently, feature
coding has been studied in the context of collaborative intelligence [20, 21],
as well as through standardization efforts such as MPEG-VCM [3], where a major
goal is to develop efficient compression methods for feature tensors of
intermediate DNN layers.

Another related line of research has been the use of DNNs for image coding,
with early examples found in the work of Ballé et al. [22] and subsequent work
[23, 24, 25, 26, 27]. Latest-generation DNN-based image codecs [28, 29, 30,
31, 32, 33] have surpassed state-of-the-art hand-crafted codecs such as
Versatile Video Coding (VVC)-Intra. Although DNN-based image codecs are
typically trained using quality metrics such as Peak Signal-to-Noise Ratio
(PSNR) and Structural Similarity Index Metric (SSIM) [34], which are targeted
at human viewing, they can also be trained using machine task-relevant loss
functions [35, 36]. Nonetheless, so long as the main goal is to perform a
machine vision task, encoding and decoding the entire input image seems
unnecessary. Instead, it is preferable to run a part of the machine vision DNN
model at the edge and transmit the resulting machine-targeted features to the
cloud. These features can sometimes be used for multiple tasks on the cloud
[37].

Like natural images, DNN features can be coded using either DNN-based [38] or
hand-crafted methods [39, 40, 41, 42]. The main advantage of DNN-based codecs
is their flexibility, which enables them to be trained end-to-end with a loss
function that combines various objectives. Leveraging this flexibility can
further lead to scalable coding for multiple downstream tasks [43, 44]. On the
other hand, traditional codecs may still be preferable in some cases for
reasons such as widespread availability in hardware and software, as well as
user trust developed and time-tested over the years. However, their main
drawback is that they have been optimized to encode natural images, which
could lead to suboptimal performance when encoding DNN features. A solution to
alleviate this problem would be to tailor DNN features to align with the
utilized codec through fine-tuning the DNN model. This can be achieved by
introducing appropriate differentiable proxies for the traditional codecs [45,
46, 47], thereby optimizing the entire system in an end-to-end manner.
Similarly, [48] has proposed a simpler differentiable proxy for the bitrate of
a traditional codec that takes into account its basic processing steps. We
adopted this approach in our proposed method to enhance the compressibility of
DNN features by the VVC-Intra codec.

As mentioned above, the field of image coding has not yet embraced privacy
considerations in a substantial way. However, researchers in the field of
machine/deep learning have been addressing privacy for some time [49, 50, 51].
Some focus on protecting the data used during the training phase [52], while
others seek data privacy during inference, which is also the objective of this
article. Various privacy and security attacks exist, along with corresponding
defense mechanisms, for deep learning models, based on the type of privacy
concern. In this paper, we focus on the defense against MIA [12]. In the MIA,
the adversary aims to recover the input image to the DNN model either during
the training or inference phase, by observing only the outputs or some
intermediate features of the model. For example, [53] introduced an attack in
which the image of a person used in training could be recovered through access
to the facial recognition DNN model, its confidence score output, and
knowledge of the person’s name. The authors of [12] also attempt to recover
the input image to a classification DNN model by observing the intermediate
features during inference under different conditions and the adversary’s level
of access to the model.

While the data privacy of the DNN models during the inference phase is less
explored, the privacy of the models and their training data has been on the
radar of researchers for many years. The concept of Differential Privacy (DP)
[54] was introduced to protect the private information of individuals
contributing to a dataset utilized by a machine learning or analysis model.
The main idea in DP is adding noise to the output of the deterministic
analysis model to increase uncertainty about the dataset, such that
distinguishing between two datasets differing in only one entity becomes
nearly impossible. Some research works have employed DP to protect neural
networks against input reconstruction performed by MIA [55, 56, 57, 58]. For
instance, in [56], Laplacian noise is added to the intermediate features of
the DNN model, which are going to be transmitted to the cloud. This would
randomize the deterministic operations within the DNN model and increase the
uncertainty in the input given the intercepted features. Additionally, a
nullification operation is also conducted to enhance privacy. To increase the
robustness of the cloud-side back-end to such feature perturbations, it is
retrained on the manipulated features. However, the utility of the model is
severely affected by increasing the intensity of the noise and nullification
process. The same methodology is adopted in [55], where the Privacy-Utility
trade-off is studied from the DP perspective. This trade-off can be controlled
by the privacy budget which has an inverse relationship with the Laplacian
noise distribution’s scale parameter.

The authors of [12] have extended their research in a recent paper [58], by
offering defense strategies against their proposed types of MIAs introduced in
[12]. They investigated noise obfuscation using Laplacian and Gaussian noise,
as well as the dropout technique. Among these, the dropout method, which
involves the deactivation of random neurons within a DNN layer, demonstrated
substantially better performance in terms of the Privacy-Utility trade-off
compared to noise addition.

Unlike the schemes mentioned above, [59] incorporated privacy preservation
into the feature compression process. Features with less private and more non-
private information are lightly compressed, whereas those carrying more
sensitive data undergo a more intensive compression to ensure privacy.
Distinguishing between private and non-private features is done through an
information-theoretic privacy approach called privacy fan. In this approach,
features of a DNN are ranked according to their mutual information (MI) [60]
relative to private and non-private tasks. At its core, the privacy fan is a
MI-based feature selection mechanism. However, estimating MI in a high-
dimensional feature space is extremely challenging. Our proposed approach
avoids this challenge, as it concentrates on feature modification rather than
feature selection.

##  III Preliminaries, Analysis, and Motivation

###  III-A Collaborative Intelligence Formulation

A DNN is fundamentally a parametric function with the set of parameters
$\theta$, which transforms an input vector $x\in\mathcal{X}$ to an output
vector $\bar{v}\in\mathcal{V}$, while $v\in\mathcal{V}$ represents the ground-
truth output.

$f(\cdot\,;\theta)\mathrel{\mathop{\mathchar
58\relax}}\mathcal{X}\to\mathcal{V}\\\ $ (1)

As mentioned earlier, collaborative intelligence is a framework in which the
DNN model is divided into a front-end $f_{1}(\cdot\,;\theta_{1})$ and a back-
end $f_{2}(\cdot\,;\theta_{2})$, each with its set of parameters $\theta_{1}$
and $\theta_{2}$, respectively, such that:

$y=f_{1}(x;\theta_{1});\quad\bar{v}=f_{2}(y;\theta_{2}),$ (2)

where $y$ represents the intermediate feature tensor. Throughout this paper,
lowercase letters $x$, $v$, $\bar{v}$ and $y$ denote specific outcomes of
random vectors (tensors) $\mathbf{X}$, $\mathbf{V}$, $\overline{\mathbf{V}}$,
and $\mathbfcal{Y}$, respectively.

As discussed in [8], distributing the DNN model between the edge and the cloud
can improve latency and energy efficiency. However, choosing a suitable split
point in the network is a design challenge [8, 10], which depends on many
factors such as energy considerations, computational resources of the edge and
cloud, and the communication channel. Here, our selection of the split point
is based on considerations of privacy and compression efficiency.

Consider a trained feedforward DNN with fixed parameters (weights) $\theta$.
Let $\mathbfcal{Y}_{i}$ be the feature tensor at the $i$-th layer. Then
$\mathbf{X}\to\mathbfcal{Y}_{i}\to\mathbfcal{Y}_{i+1}$ forms a Markov chain
and by the data processing inequality [60] we have that

$I(\mathbf{X};\mathbfcal{Y}_{i})\geq I(\mathbf{X};\mathbfcal{Y}_{i+1}),$ (3)

where $I(\cdot\,;\,\cdot)$ is the mutual information. Thus, deeper layers
contain less information about the input and are therefore more resistant to
MIAs. Furthermore, as proved in [43], deeper layers are more compressible at
the same level of model accuracy. As a result, deeper layers are more
preferable as split points for collaborative intelligence in terms of both
privacy and compression efficiency. On the other hand, splitting a DNN model
at a deeper layer means that the part of the model on the edge device gets
larger, increasing its computational requirements. Hence, the actual choice of
the split point is a balance between compression and privacy benefits on the
one hand, and computational cost on the other.

###  III-B Information Theoretic Formulation

Resisting against MIA is one of our main objectives in this paper. This
involves increasing uncertainty in the input when the feature tensor at the
split point is known. In the information theory terminology, this can be
equivalently expressed as maximizing the conditional entropy
$H(\mathbf{X}|\mathbfcal{Y})$. At the same time, it is crucial to maintain the
utility of the DNN model, which involves inferring the precise ground-truth
labels $\mathbf{V}$ from $\mathbfcal{Y}$, or alternatively minimizing
$H(\mathbf{V}|\mathbfcal{Y})$.

The information quantities involved in this problem setting are illustrated in
Fig. 2. Again, consider a trained feedforward DNN with fixed parameters
(weights) $\theta$. We show that the following lemma holds.

######  Lemma 1.

$H(\mathbfcal{Y}|\mathbf{V})=H(\mathbf{V}|\mathbfcal{Y})-H(\mathbf{X}|\mathbfcal{Y})+H(\mathbf{X}|\mathbf{V}).$

###### Proof.

By the chain rule for mutual information

$I(\mathbf{V},\mathbfcal{Y};\mathbf{X})=I(\mathbfcal{Y};\mathbf{X})+I(\mathbf{V};\mathbf{X}|\mathbfcal{Y})=I(\mathbf{V};\mathbf{X})+I(\mathbfcal{Y};\mathbf{X}|\mathbf{V}).$

Expressing the mutual information terms in the last equality in terms of
entropies, we have

$\begin{split}&H(\mathbf{X})-H(\mathbf{X}|\mathbfcal{Y})+H(\mathbf{V}|\mathbfcal{Y})-H(\mathbf{V}|\mathbf{X},\mathbfcal{Y})=\\\
&H(\mathbf{X})-H(\mathbf{X}|\mathbf{V})+H(\mathbfcal{Y}|\mathbf{V})-H(\mathbfcal{Y}|\mathbf{X},\mathbf{V})\end{split}$
(4)

Since the DNN has fixed weights, there is no uncertainty about $\mathbfcal{Y}$
when $\mathbf{X}$ is given. Hence, $H(\mathbfcal{Y}|\mathbf{X})=0$, and
consequently $H(\mathbfcal{Y}|\mathbf{X},\mathbf{V})=0$. Furthermore, there is
no uncertainty in the ground-truth labels when the input is known; thus
$H(\mathbf{V}|\mathbf{X})=0$, and consequently
$H(\mathbf{V}|\mathbf{X},\mathbfcal{Y})=0$. Using these in (4) and canceling
the common $H(\mathbf{X})$ term gives

$H(\mathbf{V}|\mathbfcal{Y})-H(\mathbf{X}|\mathbfcal{Y})=H(\mathbfcal{Y}|\mathbf{V})-H(\mathbf{X}|\mathbf{V})$

from which the claimed equality follows by rearrangement. ∎

Note that $H(\mathbf{X}|\mathbf{V})$, which is the uncertainty in the input
given a ground-truth label, is a function of the data(set) distribution and is
independent of the DNN model itself. Therefore, it is a constant that cannot
be changed through DNN construction and training. Then, according to Lemma 1,
maximizing $H(\mathbf{X}|\mathbfcal{Y})$ and minimizing
$H(\mathbf{V}|\mathbfcal{Y})$ implies that $H(\mathbfcal{Y}|\mathbf{V})$
becomes as small as possible. In other words, in the optimal scenario, only
the information relevant to the ground-truth should be preserved within
$\mathbfcal{Y}$, while irrelevant information (the dashed area in Fig. 2)
should be removed. According to the results in [43], the irrelevant
information indeed reduces from layer to layer, which lends further support to
the conclusion that deeper layers are more resilient against MIA.

Figure 2: The Venn diagram of the entropy for the involved random vectors

A related concept is the _information bottleneck_ [61], which is formulated as
a minimization problem

$\min_{{P_{\mathbfcal{Y}|\mathbf{X}}\mathrel{\mathop{\mathchar
58\relax}}~{}I(\mathbf{V};\mathbfcal{Y})\geq C}}I(\mathbf{X};\mathbfcal{Y}),$
(5)

where $P_{\mathbfcal{Y}|\mathbf{X}}$ is the conditional distribution of
$\mathbfcal{Y}$ given $\mathbf{X}$ and corresponds to the front-end
$f_{1}(\cdot\,;\theta_{1})$ part of the network in our setting, while $C$ is a
constant ensuring the minimum (mutual) information retained in $\mathbfcal{Y}$
about the desired output. Since
$I(\mathbf{X};\mathbfcal{Y})=H(\mathbf{X})-H(\mathbf{X}|\mathbfcal{Y})$,
$I(\mathbf{V};\mathbfcal{Y})=H(\mathbf{V})-H(\mathbf{V}|\mathbfcal{Y})$, and
$H(\mathbf{X})$ and $H(\mathbf{V})$ are (data-dependent) constants, (5) can
also be rewritten as

$\max_{{P_{\mathbfcal{Y}|\mathbf{X}}\mathrel{\mathop{\mathchar
58\relax}}~{}H(\mathbf{V}|\mathbfcal{Y})\leq
C^{\prime}}}H(\mathbf{X}|\mathbfcal{Y}),$ (6)

where $C^{\prime}$ is a new constant. Hence, information bottleneck –
maximizing $H(\mathbf{X}|\mathbfcal{Y})$ while keeping
$H(\mathbf{V}|\mathbfcal{Y})$ small – is aligned with the privacy-utility
discussion related to Lemma 1 above. In our proposed method (Section IV), we
follow this principle, but use loss terms that are more easily computable than
mutual information or conditional entropy.

###  III-C MIA Methodologies

MIA is the act of recovering the input to a DNN model from the intercepted
features. In a collaborative intelligence framework, the adversary, may be
able to access the intermediate features $y$ that are sent from the edge
device to the cloud. MIAs can be categorized into three classes based on the
adversary’s level of access to the front-end model $f_{1}(\cdot\,;\theta_{1})$
and its parameters [12]:

  1. 1.

White-box attack: The entire front-end model $f_{1}(\cdot\,;\theta_{1})$ and
its parameters $\theta_{1}$ are known to the adversary. The attack strategy
introduced in [12] involves solving an optimization problem, referred to as
regularized Maximum Likelihood Estimation (rMLE).

  2. 2.

Black-box attack: The adversary lacks information about
$f_{1}(\cdot\,;\theta_{1})$ but can query the front-end model and access an
unlimited number of arbitrary $(x,y)$ pairs. In this scenario, an Inverse
Network $f_{1}^{-1}(\cdot\,;\theta_{1}^{\prime})$ is trained by the adversary
using the ground-truth pairs of $(x,y)$, aiming to achieve
$f_{1}^{-1}(y;\theta_{1}^{\prime})\approx x$.

  3. 3.

Query-free attack: The adversary neither knows $f_{1}(\cdot\,;\theta_{1})$ nor
can query it. However, it has full access to the back-end model. In this
setting, a shadow model $g_{1}(\cdot\,;\omega_{1})$ is first constructed such
that $f_{2}(g_{1}(x;\omega_{1});\theta_{2})\approx
f_{2}(y;\theta_{2})=f_{2}(f_{1}(x;\theta_{1});\theta_{2})$. This enables
$g_{1}(\cdot\,;\omega_{1})$ to function as a substitute for
$f_{1}(\cdot\,;\theta_{1})$, which can be used to run rMLE or train the
Inverse Network.

The white-box setting is the most favorable from the adversary’s point of
view. However, the experiments in [12] indicate that the effectiveness of the
black-box attack using the Inverse Network is almost the same as that of the
white-box attack using rMLE, and even better when the Inverse Network is
trained on the same dataset as the original model. In this paper, we aim to
counter the most powerful of these attacks, which is accomplished using the
Inverse Network technique trained on the original dataset.

###  III-D Model Inversion Defense Methodologies

As previously mentioned, in the literature, DP and noise obfuscation are the
most common solutions to defend against MIA [55, 56, 57]. However, adding
independent noise to the intermediate features raises uncertainty in both
$\mathbf{X}$ and $\mathbf{V}$, as illustrated in Fig. 2. To see this, consider
the following lemma.

######  Lemma 2.

Let $\mathbfcal{N}$ be a random vector with the same dimension as
$\mathbfcal{Y}$ and independent of both $\mathbfcal{Y}$ and $\mathbf{Z}$. Then

$H(\mathbf{Z}|\mathbfcal{Y}+\mathbfcal{N})\geq H(\mathbf{Z}|\mathbfcal{Y}).$
(7)

###### Proof.

Since conditioning does not increase entropy, we have that
$H(\mathbf{Z}|\mathbfcal{Y}+\mathbfcal{N})\geq
H(\mathbf{Z}|\mathbfcal{Y}+\mathbfcal{N},\mathbfcal{N})$. Now that
$\mathbfcal{N}$ is one of the conditions, it can be subtracted from the other
condition without changing the conditional entropy:

$H(\mathbf{Z}|\mathbfcal{Y}+\mathbfcal{N},\mathbfcal{N})=H(\mathbf{Z}|(\mathbfcal{Y}+\mathbfcal{N})-\mathbfcal{N},\mathbfcal{N})=H(\mathbf{Z}|\mathbfcal{Y},\mathbfcal{N}).$

Finally, since $\mathbfcal{N}$ is independent of $\mathbf{Z}$, we have that
$H(\mathbf{Z}|\mathbfcal{Y},\mathbfcal{N})=H(\mathbf{Z}|\mathbfcal{Y})$, which
completes the proof that $H(\mathbf{Z}|\mathbfcal{Y}+\mathbfcal{N})\geq
H(\mathbf{Z}|\mathbfcal{Y})$. ∎

Replacing $\mathbf{Z}$ by either $\mathbf{X}$ or $\mathbf{V}$ in Lemma 2 shows
that noise obfuscation of features increases both
$H(\mathbf{X}|\mathbfcal{Y})$ and $H(\mathbf{V}|\mathbfcal{Y})$. That is to
say, MIA becomes more difficult (higher $H(\mathbf{X}|\mathbfcal{Y})$), but
the model becomes less accurate (higher $H(\mathbf{V}|\mathbfcal{Y})$).
Ideally, we want to maximize $H(\mathbf{X}|\mathbfcal{Y})$ while minimizing
$H(\mathbf{V}|\mathbfcal{Y})$, as discussed in Section III-B. For this reason,
our proposed method is not based on noise obfuscation, but adversarial
training with suitably chosen loss functions that promote a high value of
$H(\mathbf{X}|\mathbfcal{Y})$ and a low value of
$H(\mathbf{V}|\mathbfcal{Y})$.

##  IV Proposed Methods

Figure 3: The overall block diagram of the proposed method. $Conv(n,k,s)$ is a
2D Convolution layer followed by a Batch normalization layer and a $SiLU()$
activation, with $n$ being the number of output channels, a kernel size of
$k\times k$, and stride=$s$. $Deconv(n,k,s)$ is the same as $Conv(n,k,s)$
except with a Convolution Transpose layer. Note that the $Conv$ layers in the
autoencoder do not have batch normalization, and the last $Conv$ in AE does
not have the $SiLU()$ activation either. $Conv3\times 3(n)$ is a single
$3\times 3$ Convolution layer with stride=1 and $n$ output channels. “$\times
k$” indicates $k$ times repetition of the blocks specified by a dashed frame.

We chose object detection by YOLOv5 [62] as the machine vision task, although
the overall collaborative intelligence framework is also applicable to other
DNN models and tasks. There are different versions of YOLOv5 with varying
levels of complexity and accuracy,222https://github.com/ultralytics/yolov5 and
we opted for the YOLOv5m model as the basis for our experiments. In our
proposed framework, we follow two key objectives: resistance to MIA and
improving compression efficiency. To achieve these goals, we train an
autoencoder, inserted into the the machine vision model, together with an
auxiliary network (RecNet) aimed at reconstructing the input, in an
adversarial manner [63].

The overall block diagram of our design is shown in Fig. 3. It is worth
mentioning that the autoencoder is trained off-line prior to deployment. At
the inference stage, the YOLO front-end and “AE” are deployed on the edge
device, along with the VVC encoder, while the VVC decoder, “AD” and the YOLO
back-end are deployed in the cloud. “RecNet” is used only during training to
improve the adversarial robustness of the autoencoder, and is not part of the
deployed pipeline. These components are further elaborated on below.

###  IV-A Split Point

For collaborative edge-cloud deployment, the YOLOv5m model should be split
into two parts: the front-end, which is to be deployed on the edge device, and
the back-end, which would reside in the cloud. As discussed in Section III-A,
a deeper split point is more desirable in terms of both compression efficiency
and privacy. However, limited computing power and energy considerations of
edge devices favor choosing a shallower split point. Moreover, the YOLOv5m
model branches out at layer 5. Picking a split point deeper than this layer
would require encoding and transmitting multiple sets of feature tensors,
which would increase complexity and make compression more challenging. Thus,
we chose the deepest reasonable split point, layer 5, for the YOLOv5m model.

###  IV-B Autoencoder

At the split point, an autoencoder is integrated and trained with the goal of
reducing the dimensionality of the original latent space and generating
bottleneck features with improved compressibility and resilience to MIA. This
is a plug-and-play strategy and can be readily applied to other models and
computer vision tasks without retraining the original vision model. The
autoencoder’s architecture is shown in Fig. 3: its encoder portion is denoted
as AE, and the decoder is referred to as AD.

The role of AE is to transform the YOLO’s native latent space at the split
point (with $192$ channels) into lower-dimensional bottleneck features (here,
with $64$ channels). The spatial dimensions remain unchanged to maintain
spatial precision, which is crucial for subsequent object detection. The
resulting bottleneck feature tensor is tiled and pre-quantized to 8 bits per
element, similarly to [64], and then encoded using VVC-Intra [16]. On the
cloud side, the encoded bitstream is decoded by a VVC decoder and AD, and then
fed to the YOLOv5m back-end for object detection.

###  IV-C Loss functions

To train the autoencoder, we employ three main loss functions. Each of these
loss functions is meant to encourage a certain aspect of the bottleneck
features, as explained below.

  * •

Object detection: This is the native object detection loss function used in
YOLOv5 [62]:

$L_{obj\\_det}=w_{obj}\cdot L_{obj}+w_{box}\cdot L_{box}+w_{cls}\cdot
L_{cls},$ (8)

where $L_{obj}$, $L_{box}$, and $L_{cls}$ are objectness, bounding box
localization, and classification losses that are combined with corresponding
balancing weights.

  * •

Compressibility: $L_{cmprs}$ is a loss function borrowed from [48] and aims to
enhance the compressibility of feature tensors using traditional image codecs:

$L_{cmprs}=\frac{1}{H\cdot W\cdot C}\sum_{i=1}^{C}\|\mathbf{T}_{i}\|_{1}.$ (9)

Here, $H$, $W$, and $C$ are the height, width, and number of bottleneck
feature channels, respectively;
$\mathbf{T}_{i}=\texttt{DCT}\\{\mathbf{Z}_{i}\\}$, where DCT is the Discrete
Cosine Transform and $\mathbf{Z}_{i}$ contains the residuals of horizontal and
vertical prediction within each feature channel [48]. The purpose of such a
loss term is to mimic or approximate the processing steps performed by a
typical conventional image encoder in terms of differentiable operations and
compute a quantity ($\ell_{1}$ norm of transform coefficients) that is known
to be related to the resulting bitrate. Since we use VVC-Intra [16] to code
bottleneck features, such a loss term is appropriate for our design.

  * •

Reconstruction: The reconstruction loss function $L_{rec}$ is intended to
capture the quality of reconstruction of the input image from the bottleneck
features. $L_{rec}$ consists of two terms: the $\ell_{1}$-norm of the
reconstruction error and an edge-related loss. The latter is related to fine
details in the image, which are usually associated with private information
(e.g., facial features). $L_{rec}$ is computed as follows:

$\begin{split}L_{rec}=\frac{1}{n}\|{x}-\hat{x}\|_{1}&+\frac{\beta}{n}\|S_{h}*{x}-S_{h}*\hat{x}\|_{1}\\\
&+\frac{\beta}{n}\|S_{v}*{x}-S_{v}*\hat{x}\|_{1},\end{split}$ (10)

where $x$ and $\hat{x}$ are the original and reconstructed image,
respectively, $S_{h}$ and $S_{v}$ are the horizontal and vertical Sobel filter
[65], respectively, $*$ is the convolution operator, and $n$ is the total
number of tensor elements in the training batch. The value of $\beta$ is
empirically set to $5$.

###  IV-D Adversarial Training

Training is performed off-line. We have adopted a specific training strategy
to generate compressible bottleneck features that are resistant to MIA while
being able to support the object detection task. This is done using an
auxiliary DNN model, called the Reconstruction Network (RecNet), the purpose
of which is to mimic an adversary performing MIA and try to reconstruct the
input image from the bottleneck features. As illustrated in Fig. 3, RecNet’s
architecture roughly mirrors that of the YOLOv5m front-end and AE. This is in
line with a white-box attack, where the adversary knows the architecture of
the model under attack.

The autoencoder and RecNet are adversarially trained together. Throughout the
training process, RecNet tries to recover the input image from the bottleneck
features as best it can, while the AE attempts to disrupt RecNet’s performance
by manipulating the generated bottleneck features. Meanwhile, both AE and AD
work to maintain high object detection accuracy. Furthermore, bottleneck
features are progressively optimized for compression efficiency via the
compressibility loss function during training. Note that the pre-trained
YOLOv5m model remains intact, and its weights are frozen throughout the entire
training process.

The training process for each data batch is summarized in the following steps:

  1. 1.

The input images pass through the front-end, AE, and RecNet, and the
corresponding reconstruction loss $L_{rec}$ is calculated according to
equation (10).

  2. 2.

The gradient of $L_{rec}$ backpropagates only through the RecNet and updates
its weights. The autoencoder’s weights are frozen during this step.

  3. 3.

The same batch of images goes through the whole network, and the total loss is
computed as follows:

$L_{tot}=L_{obj\\_det}+w_{cmprs}\cdot L_{cmprs}-w_{rec}\cdot L_{rec},$ (11)

where $L_{obj\\_det}$, $L_{cmprs}$, and $L_{rec}$ are the loss terms defined
in Section IV-C. The balancing weights for combining these loss terms are
denoted $w_{cmprs}$ and $w_{rec}$. Experiments are performed for different
combinations of these weights in Section VI.

  4. 4.

The gradient of $L_{tot}$ backpropagates through the autoencoder and updates
its weights. RecNet’s weights remain frozen during this step. Note that the
negative sign of $L_{rec}$ in equation (11) leads AE to make reconstruction
more difficult for RecNet. Meanwhile, the positive sign of $L_{obj}$ and
$L_{cmprs}$ drives AE and AD to improve object detection accuracy and create
compressible features.

##  V Proposed Evaluation Framework

Measuring the privacy of a coding system has always been a challenge. Many
researchers rely on perceptual metrics to quantify how much private data in an
image has been preserved. For example, a low value of Peak Signal-to-Noise
Ratio (PSNR) or Structural Similarity Index Metric (SSIM) [34] between the
target and the original image is often considered a sign of high privacy
protection. However, private data is not uniformly distributed across the
entire image, whereas perceptual metrics naturally consider the whole image.
To illustrate why such metrics might not be appropriate proxies for privacy
protection, Fig. 4(a) shows an image that exposes some private information
(face and license plate). In Fig. 4(b), noise is added to the entire image
except the face and license plate regions, creating low PSNR, but with private
information still fully exposed. In Fig. 4(c), noise is added only to the face
and license plate region, leading to an image with much higher PSNR but with
private information now obscured by noise. Therefore, in this example, PSNR is
a poor proxy for privacy protection.

(a)

(b)

(c)

Figure 4: Image quality may not be a good proxy for privacy: (a) original
image; (b) low-quality image (PSNR = 28.0 dB) but with private information
exposed; (c) higher-quality image (PSNR = 41.4 dB) with private information
protected

For this reason, we propose an evaluation framework in which privacy is
measured more directly, instead of relying on proxies such as PSNR and SSIM.
Specifically, our evaluation is based on the face and license plate
recognition accuracy obtained on the images recovered by running a MIA on the
bottleneck features. These evaluation metrics are elaborated in Sections V-A
and V-B, while Section V-C presents the benchmark evaluation models.

###  V-A Face Recognition Metric

Faces contain critical information through which the identity of a person can
be divulged. In order to quantify the privacy level of our proposed method, we
measure how accurately a face recognition model can identify faces within the
images recovered from bottleneck features by MIA. Various face recognition
models, especially those based on DNNs, are available. We employed the
implementation of MagFace [66], along with its pre-trained model accessible on
GitHub.333https://github.com/IrvingMeng/MagFace MagFace is a well-known DNN-
based face recognition model with high accuracy. As reported in [66], MagFace
has achieved $99.83\%$ verification accuracy on the LFW dataset [67].

Face recognition models are often evaluated on the verification task, where
the goal is to determine whether two distinct images show the same individual.
However, privacy primarily concerns the precision with which an individual can
be recognized from an image. Hence, we believe identification task, as
outlined in [68, 69], is more closely related to the notion of privacy. In the
identification task, two sets of images are involved: the gallery set and the
probe set. The gallery set comprises sample images of individuals, while the
probe set contains images of people of interest. The face recognition model
should match the images within the probe set with the corresponding images in
the gallery showing the same individuals.

The LFW dataset consists of $13,233$ images belonging to $5,749$ individuals,
$1,680$ of which have at least two images available. From these $1,680$
individuals, one of their photos is assigned to the gallery set, and the rest
are assigned to the probe set. Next, we collect the output feature vectors
generated by MagFace for all the images and compute the distance between all
pairs of members from the gallery and probe sets. The image in the gallery
with the lowest feature distance to the currently-investigated probe image is
selected as the match. In the end, the identification accuracy is calculated
based on the number of correctly detected matches.

###  V-B License Plate Recognition Metric

License plate data is another form of sensitive information that can be
exploited to track specific vehicles for potentially malicious purposes using
citywide traffic monitoring cameras. Hence, license plate recognition accuracy
can be considered another direct measure of privacy. For this purpose, we used
LPRNet [70], a DNN-based license plate recognition model, whose implementation
is available on
GitHub.444https://github.com/xuexingyu24/License_Plate_Detection_Pytorch

The common approach to reporting the accuracy of license plate recognition
methods is to consider the output of the model correct only when all license
plate characters are recognized correctly. However, from the privacy
perspective, even a partially successful recognition could be considered a
privacy leakage. Therefore, we adopted character-by-character recognition
accuracy as our criterion. In our experiments, this accuracy is calculated on
a set of $5,000$ images from the CCPD dataset [71].

###  V-C Benchmarks

We compare the performance of our proposed method with the benchmark pipelines
shown in Fig. 5. In “Benchmark-input,” the input image is encoded using the
VVC codec, and object detection is performed on the decoded image ($\hat{x}$).
The accuracy of face and license plate recognition is also measured on the
decoded image. “Benchmark-latent” deals with the feature channels at YOLOv5’s
fifth layer, where 192 channels are clipped, quantized to 8 bits, tiled into a
gray-scale matrix, and then encoded using the VVC codec. The decoded features
are then fed to the YOLOv5 back-end for object detection. To measure privacy
protection, an approximation to the input image ($\hat{x}$) is created by an
inverse network (InvNet), whose architecture mirrors the layers that produce
the features, similar to a white-box attack. The accuracy of face and license
plate recognition is measured on the recovered image $\hat{x}$.

The pipelines for “Benchmark-bottleneck” and our proposed method are shown in
Fig. 5(c). In both cases, the 64 bottleneck channels of the autoencoder are
tiled into grayscale matrix after clipping and quantization, and encoded using
the VVC codec. The only difference between the two is that our method’s
autoencoder is trained adversarially with non-zero weights $w_{cmprs}$ and
$w_{rec}$, while the autoencoder in Benchmark-bottleneck is trained only using
YOLO’s native object detection loss (i.e., $w_{cmprs}=w_{rec}=0$ in equation
(11)). The InvNet for the proposed method and “Benchmark-bottleneck” mirrors
both the YOLOv5m front-end and the encoding part (AE) of the autoencoder.
Again, the accuracy of face and license plate recognition is measured on the
recovered image $\hat{x}$.

(a) Benchmark-input

(b) Benchmark-latent

(c) Benchmark-bottleneck / Proposed

Figure 5: The evaluation pipelines

##  VI Experiments

The training was performed on the COCO-2017 object detection dataset [72]
using an NVIDIA Tesla V100-SXM2 GPU with 32GB of memory. For the proposed
method, in the first step, only the autoencoder was trained with
$L_{obj\\_det}$ for 50 epochs. Next, we trained the RecNet with $L_{rec}$ for
20 epochs, while keeping the autoencoder’s weights frozen to those obtained in
the first step. Finally, with the autoencoder and RecNet initialized to the
weights obtained in the first two steps, the adversarial training was
conducted with various values of $w_{cmprs}$ and $w_{rec}$, as detailed in
Section IV-D, for 40 epochs. In all stages, we used the Stochastic Gradient
Descent (SGD) optimizer with an initial learning rate of 0.01, which changed
with a cosine learning rate decay [73] over the training epochs. VVenC [74]
was employed as the VVC-Intra codec.

As noted in [42], feature compression performance can be impacted by the
selected clipping range. We empirically selected $\pm 6\sigma$ as the clipping
range, where $\sigma$ is the standard deviation of the to-be-coded feature
tensor values on the COCO validation set.

As mentioned earlier, RecNet is an auxiliary DNN model exploited in the
adversarial training stage and is not part of the final system for
collaborative object detection. In a real-world situation, an adversary would
train their own Inverse Network (InvNet) using input-feature pairs generated
by querying the front-end model deployed on the edge device. Hence, to mimic
this scenario, we also train a new, randomly initialized InvNet on the
bottleneck features generated by our autoencoder, in order to recover the
input. This InvNet and the benchmarks’ InvNets are trained with an
$\ell_{1}$-norm loss of the error (the first term in equation (10)) on the
COCO dataset, the same dataset used for training the autoencoder and YOLOv5
model. The correspondence of datasets for training the main model and InvNets
could result in better efficacy of MIA according to [12]. Therefore, the
resistance of the models would be examined against a stronger MIA.

###  VI-A Resistance to MIA

Experiments in this section focus on the resistance of the models against MIA,
without involving feature compression by the VVC codec. In other words, the
gray-toned blocks within the evaluation pipelines shown in Fig. 5 are all
bypassed. Fig. 6 shows images recovered from bottleneck features by running a
MIA. These images correspond to the benchmark pipelines and our proposed
autoencoder, which was adversarially trained with different values of
$w_{rec}$ while setting $w_{cmprs}$ to $0$.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: Examples of images recovered through MIA: (a) original image (b)
Benchmark-latent (c) Benchmark-bottleneck (d) $(w_{rec},w_{cmprs})=(0.5,0)$
(e) $(w_{rec},w_{cmprs})=(1.0,0)$ (d) $(w_{rec},w_{cmprs})=(2.0,0)$

As seen in Fig. 6, the adversary’s Inverse Networks are able to recover the
input images from the Benchmark-latent and Benchmark-bottleneck to a good
extent. In particular, facial and license plate details are still visible in
these images. This suggests that even with a reduction in the number of
feature channels from 192 at the YOLOv5m model’s split point to 64 at the
autoencoder’s bottleneck, there remains sufficient information in the
bottleneck for license plate recognition and person identification. On the
other hand, adversarially trained autoencoders, particularly those trained
with higher values of $w_{rec}$, are capable of effectively removing this
information. This is because higher values of $w_{rec}$ emphasize $L_{rec}$ in
the total loss (see equation (11)), which encourages AE to cause a drop in the
quality of the recovered images. This degradation in quality is more
concentrated near the edges and textured areas, due to the use of Sobel
filters in the reconstruction loss as presented in equation (10).

Next, we present a comparison between the resistance of our proposed approach
to MIA and well-known defense strategies, such as adding Gaussian or Laplacian
noise [55, 56] and nullification (Dropout) [56, 58]. These manipulations are
applied with different intensities to either the input or latent features of
YOLOv5m at layer 5 to obtain the corresponding Privacy-Utility curves. Here,
utility is measured by the accuracy of object detection on the COCO-2017
validation set (mean Average Precision, mAP, at IoU threshold of 0.5, shown on
the vertical axis), and privacy is quantified in three different ways along
the horizontal axis: PSNR of the recovered images on the COCO-2017 validation
set in Fig. 7(a), face recognition accuracy on the LFW dataset in Fig. 7(b),
and license plate recognition accuracy on the CCPD dataset in Fig. 7(c). Our
autoencoder was trained with $w_{cmprs}=0$ and a set of values for
$w_{rec}\in\\{0.0,0.1,0.5,0.8,1.2,2.0\\}$ to plot the orange Privacy-Utility
curves shown in Fig. 7.

According to Fig. 7, the three leftmost points on our Privacy-Utility curves,
corresponding to $w_{rec}\in\\{0.8,1.2,2.0\\}$, provide excellent privacy,
with face and license plate recognition accuracies below $20\%$ and $10\%$,
respectively. This level of privacy is achieved while sacrificing less than
$1\%$ precision in the object detection task. On the contrary, the other
methods experience a significant drop in object detection accuracy to reach
this level of privacy, despite starting out with a slightly higher mAP value.

(a)

(b)

(c)

Figure 7: Privacy-Utility curves illustrating resistance to MIA

###  VI-B Compression Efficiency

In this section, we examine compression efficiency in addition to object
detection and privacy. For this purpose, the VVC codec is included in all
evaluation pipelines in Fig. 5. For our approach, the key hyperparameters for
training the autoencoder are $w_{rec}$ and $w_{cmprs}$ through which the
amount of existing private information in the bottleneck and the bitrate of
the coded bitstream can be controlled. We trained autoencoders with different
values of $w_{rec}\in\\{1.0,1.5,2.0\\}$ and $w_{cmprs}\in\\{1,2,3,4\\}$. Note
that, according to Fig. 7, $1\leq w_{rec}\leq 2$ can offer suitable privacy
protection. The bottleneck feature channels of each obtained autoencoder were
compressed using the VVC codec with various Quantization Parameter (QP)
values. Therefore, each triplet $(w_{rec},w_{cmprs},QP)$ corresponds to a
specific point in the Rate-Utility plane. The Pareto front of these points
represents the performance of our proposed method alongside the benchmarks in
Fig. 8.

For the benchmark methods, the curves are obtained by varying the QP of the
VVC codec. Fig. 8(a) shows the Rate-Utility curves on the COCO-2017 validation
set, where the rate is shown as the average number of bits per pixel (bpp),
and the utility is mAP at the IoU threshold of 0.5, as before. Compressing
YOLOv5m features directly (Benchmark-latent) shows the worst performance,
because there are many channels (192) and the VVC codec is not tailored to
coding features directly. Reducing the number of channels via autoencoder
(Benchmark-bottleneck) improves the performance, but the features are still
not VVC-friendly. Therefore, coding the input directly (Benchmark-input) shows
better performance. However, the proposed approach creates VVC-friendly
features at the autoencoder’s bottleneck via the compressibility loss
$L_{cmprs}$ in (9), thereby surpassing the performance of Benchmark-input.

(a)

(b)

(c)

(d)

Figure 8: (a) Rate-Utility curves; (b), (c), (d) Privacy-Utility curves

Figs. 8(b)-8(d) show the Privacy-Utility curves like those shown in Fig. 7,
but this time different points on the curves are obtained by varying the
compression rate (via QP) rather than changing the parameters of the defense
mechanism. The results collectively show that our proposed method outperforms
the benchmarks in various Privacy-Utility scenarios. One interesting point to
note here is the performance of Benchmark-input, which is based on VVC coding
of the input image directly. As shown in Fig. 8(a), this approach was the
second best in terms of Rate-Utility, behind our method, which was not
surprising because VVC-Intra is optimized for coding images. However, in terms
of Privacy-Utility in Figs. 8(b)-8(d), it is now significantly worse than
other benchmarks. The likely reason is that compression artifacts introduced
by VVC hurt object detection more than they affect face recognition or license
plate recognition. By operating on YOLOv5m features rather than input itself,
the other two benchmarks and the proposed method manage to reduce the impact
of compression on object detection performance.

We use Bjøntegaard-Delta (BD) values [75] to summarize the difference between
various curves in Fig. 8. Table I shows BD-Rate and BD-mAP555Calculated based
on the MPEG-VCM reporting template [76] for the curves in Fig. 8(a), as well
as the BD-PSNR for the curves shown in Fig. 8(b). In the table, the symbol
$\uparrow$ means the higher the better, while $\downarrow$ means the lower the
better. The BD-PSNR values indicate that the adversarially trained autoencoder
can effectively reduce the PSNR of recovered images by $15.97$ dB and $7.07$
dB on average compared to Benchmark-input and Benchmark-latent, respectively.
Moreover, a $19.4\%$ bit saving is achieved relative to Benchmark-input.

TABLE I: Bjøntegaard-Delta (BD) values with respect to “Benchmark-input” |  BD-Rate $\downarrow$ |  BD-mAP $\uparrow$ |  BD-PSNR $\downarrow$
---|---|---|---
Benchmark-input | $0.0\%$ | $0.0$ |  $0.00$ dB
Benchmark-latent | $110.5\%$ | $-6.6$ |  $-8.90$ dB
Benchmark-bottleneck | $43.7\%$ | $-3.1$ |  $-9.17$ dB
Proposed | $-19.4\%$ | $1.9$ |  $-15.97$ dB

###  VI-C Execution Time

As demonstrated in [8] and mentioned earlier in the text, collaborative
inference has the potential to reduce overall latency compared to “cloud-only”
solutions where input is directly compressed and transmitted to the cloud. We
find that this is also true for the proposed system. Table II shows the
average execution time of different components of our proposed system, in the
rows labeled “Proposed.” The bottom of the table, in the rows labeled “Input,”
shows the execution time of the “Benchmark-input” method where input is
compressed directly. For all components except the VCC codec,666VVenC [74],
which was only able to run on CPU. we show execution times on both the CPU and
GPU. The CPU used for this experiment was AMD Ryzen 9 5900HS (3.30 GHz) and
the GPU was NVIDIA GP102 TITAN X.

TABLE II: Average execution time (in milliseconds per input image) for different components of the system on the COCO-2017 validation set System component | Execution time (ms)
---|---
CPU | GPU
Proposed | YOLOv5m front-end | $120.1$ | $10.5$
Autoencoder-AE | $29.3$ | $2.6$
VVC bottleneck encoding | $1481.1$
VVC bottleneck decoding | $797.7$
Autoencoder-AD | $27.7$ | $1.6$
YOLOv5m back-end | $184.8$ | $26.7$
Total | $\mathbf{2640.7}$ | $\mathbf{2320.2}$
Input | VVC input encoding | $1833.4$
VVC input decoding | $843.3$
YOLOv5m | $293.8$ | $48.7$
Total | $\mathbf{2970.5}$ | $\mathbf{2725.4}$

As seen in the table, most of the processing time is spent on VVC encoding,
whereas the proposed autoencoder consumes the smallest portion, demonstrating
its low complexity. The proposed autoencoder generates a feature tensor that
is tiled into a grayscale-like image of the same resolution as the input
image. However, because the input image has 3 color channels, its encoding
using VVC takes longer than the encoding of the tiled feature tensor. And the
time difference ($1833.4-1481.1=352.3$ ms) is enough to compensate for running
the YOLOv5m front-end and the autoencoder, even on a CPU ($120.1+29.3=149.4$
ms).

Simpler edge devices usually lack a GPU and therefore have to perform
computations on a CPU. Hence, one practical realization of the system would be
to run edge-side computations on a CPU and cloud-side computations on a GPU.
In this case, the overall latency of our proposed method becomes $2456.5$ ms,
which is around $10\%$ less than the time required when the input is
compressed directly, which is $2725.4$ ms.

The above analysis was focused on the execution time of various components.
But, the actual end-to-end inference latency also depends on the communication
time, which is the time it takes for the encoded bitstream to be transmitted
from the edge device to the cloud. As shown in Table I, at the same object
detection accuracy, the proposed method provides BD-Rate savings of $19.4\%$,
meaning that it produces, on average, a $19.4\%$ smaller bitstream. For a
given transmission rate (in bits/second), such a bitstream would take $19\%$
less time to be transmitted compared to the Benchmark-input bitstream. Hence,
the actual time savings in practice could be even higher than the $10\%$
achieved on execution time only.

###  VI-D Ablation Study

In this section, we analyze the impact of $w_{rec}$ and $w_{cmprs}$ on our
system’s performance. Fig. 9 shows the Rate-Utility and Privacy-Utility curves
for some of these values, with their corresponding BD values with respect to
Benchmark-latent given in Table III. According to these results, Benchmark-
bottleneck reduces the bitrate by $32\%$ compared to coding the original
latent space of the YOLOv5m model. Thus, part of the bitrate reduction in the
proposed system is attributed to the dimensionality reduction of the latent
space. However, compressibility loss also plays a crucial role in minimizing
the bitrate, which is seen when changing $w_{cmprs}$.

Another interesting point is that the average PSNR difference of the images
recovered by MIA from Benchmark-bottleneck and Benchmark-latent is negligible
($0.27$ dB). Hence, feature dimensionality alone is not sufficient to disrupt
MIA. On the other hand, adversarially trained autoencoders with $w_{rec}>0$
provide a more significant reduction in quality of MIA-generated images,
especially when $w_{rec}$ increases.

It is also interesting to note that $w_{rec}$ and $w_{cmprs}$ provide somewhat
independent “knobs” to control the quality of MIA-generated images and
bitrate, respectively. For instance, increasing $w_{rec}$ from $0.5$ to $1.0$
or $2.0$ (with $w_{cmprs}=0$) minimally alters the average bitrate, yet leads
to an average PSNR drop of about $2.3$ dB in the quality of MIA-generated
images. Similarly, increasing $w_{cmprs}$ from $0.0$ to $3.0$ (with
$w_{rec}=0.5$) has a relatively minor impact on the quality of MIA-generated
images compared to the effect of $w_{rec}$, yet it leads to an additional
$30\%$ bitrate reduction relative to Benchmark-latent.

(a)

(b)

Figure 9: (a) Rate-Utility curves (b) Privacy-Utility curves

##  VII Conclusion

In this paper, we presented a novel feature coding scheme for collaborative
object detection with strong resistance to model inversion attack (MIA). This
was achieved via an autoencoder that transforms the features of the YOLOv5m
object detection model into a lower-dimensional space while maintaining object
detection accuracy, and improving compressibility as well as resistance to
MIA. Moreover, the proposed approach enables lower end-to-end inference
latency compared to encoding the input image directly. The proposed approach
is plug-and-play in the sense that it utilizes a standard widely available
codec (VVC) and does not require retraining the object detector. In addition
to the conventional evaluation of MIA resistance in terms of the quality of
MIA-generated images, the proposed system was also evaluated on measures that
are more directly related to privacy, such as face and license plate
recognition accuracy, showing superior performance compared to benchmarks in
each case.

TABLE III: Bjøntegaard-Delta (BD) values with respect to “Benchmark-latent” |  BD-Rate $\downarrow$ |  BD-mAP $\uparrow$ |  BD-PSNR $\downarrow$
---|---|---|---
Benchmark-latent | $0.0\%$ | $0.0$ |  $0.00$ dB
Benchmark-bottleneck | $-32.0\%$ | $5.0$ |  $-0.27$ dB
$w_{rec}=0.5,w_{cmprs}=0.0$ | $-34.1\%$ | $5.2$ |  $-4.47$ dB
${w_{rec}=1.0,w_{cmprs}=0.0}$ | $-36.7\%$ | $5.5$ |  $-5.95$ dB
${w_{rec}=2.0,w_{cmprs}=0.0}$ | $-32.7\%$ | $5.0$ |  $-6.79$ dB
${w_{rec}=0.5,w_{cmprs}=1.0}$ | $-57.2\%$ | $7.1$ |  $-5.05$ dB
${w_{rec}=0.5,w_{cmprs}=3.0}$ | $-63.3\%$ | $7.2$ |  $-5.91$ dB

## References

  * [1] Cisco, “Cisco annual Internet report (2018–2023),” Mar. 2020.
  * [2] ISO/IEC JTC 1/SC29/WG1, “Final call for proposals for JPEG AI,” Jan. 2022.  N100095.
  * [3] W. Gao, S. Liu, X. Xu, M. Rafie, Y. Zhang, and I. Curcio, “Recent standard development activities on video coding for machines,” arXiv:2105.12653, 2021.
  * [4] L. Duan, J. Liu, W. Yang, T. Huang, and W. Gao, “Video coding for machines: A paradigm of collaborative compression and intelligent analytics,” IEEE Transactions on Image Processing, vol. 29, pp. 8680–8695, 2020.
  * [5] K.-A. Shim, “A survey of public-key cryptographic primitives in wireless sensor networks,” IEEE Communications Surveys & Tutorials, vol. 18, no. 1, pp. 577–601, 2016.
  * [6] X. Liu, X. Zhao, Z. Xia, Q. Feng, P. Yu, and J. Weng, “Secure outsourced sift: Accurate and efficient privacy-preserving image sift feature extraction,” IEEE Transactions on Image Processing, vol. 32, pp. 4635–4648, 2023.
  * [7] S. Hu, Q. Wang, J. Wang, Z. Qin, and K. Ren, “Securing sift: Privacy-preserving outsourcing computation of feature extractions over encrypted image data,” IEEE Transactions on Image Processing, vol. 25, no. 7, pp. 3411–3425, 2016.
  * [8] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” ACM SIGARCH Computer Architecture News, vol. 45, no. 1, pp. 615–629, 2017.
  * [9] A. E. Eshratifar and M. Pedram, “Energy and performance efficient computation offloading for deep neural networks in a mobile cloud computing environment,” in Proc. ACM GLSVLSI, p. 111–116, 2018.
  * [10] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “JointDNN: An efficient training and inference engine for intelligent mobile cloud computing services,” IEEE Trans. Mobile Computing, vol. 20, no. 2, pp. 565–576, 2021.
  * [11] A. Banitalebi-Dehkordi, N. Vedula, J. Pei, F. Xia, L. Wang, and Y. Zhang, “Auto-split: A general framework of collaborative edge-cloud ai,” in Proc. ACM KDD, p. 2543–2553, 2021.
  * [12] Z. He, T. Zhang, and R. B. Lee, “Model inversion attacks against collaborative inference,” in Proc. 35th Annual Computer Security Applications Conference, p. 148–162, 2019.
  * [13] B. Azizian and I. V. Bajić, “Privacy-preserving feature coding for machines,” in Proc. PCS, pp. 205–209, 2022.
  * [14] M. Rabbani and R. Joshi, “An overview of the JPEG 2000 still image compression standard,” Signal Processing: Image Communication, vol. 17, no. 1, pp. 3–48, 2002.
  * [15] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 12, pp. 1649–1668, 2012.
  * [16] B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (VVC) standard and its applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 10, pp. 3736–3764, 2021.
  * [17] W. Lin and C.-C. Jay Kuo, “Perceptual visual quality metrics: A survey,” J. Visual Commun. Image Represent., vol. 22, no. 4, pp. 297–312, 2011.
  * [18] MPEG-CDVS, “Compact descriptors for visual search,” 2015.  ISO/IEC JTC 1 15938-13.
  * [19] MPEG-CDVA, “Compact descriptors for video analysis,” 2019.  ISO/IEC JTC 1 15938-15.
  * [20] I. V. Bajić, W. Lin, and Y. Tian, “Collaborative intelligence: Challenges and opportunities,” in Proc. IEEE ICASSP, pp. 8493–8497, 2021.
  * [21] N. Shlezinger and I. V. Bajić, “Collaborative inference for AI-empowered IoT devices,” IEEE Internet of Things Magazine, vol. 5, no. 4, pp. 92–98, 2022.
  * [22] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” arXiv:1611.01704, 2016.
  * [23] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in Proc. ICLR, 2018.
  * [24] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in NeurIPS, 2018.
  * [25] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proc. IEEE/CVF CVPR, June 2020.
  * [26] Y.-H. Ho, C.-C. Chan, W.-H. Peng, H.-M. Hang, and M. Domański, “ANFIC: Image compression using augmented normalizing flows,” IEEE Open J. Circuits Syst., vol. 2, pp. 613–626, 2021.
  * [27] S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, and S. Wang, “Image and video compression with neural networks: A review,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 6, pp. 1683–1698, 2020.
  * [28] D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang, “ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding,” in Proc. IEEE/CVF CVPR, 2022.
  * [29] W. Jiang, J. Yang, Y. Zhai, P. Ning, F. Gao, and R. Wang, “MLIC: Multi-reference entropy model for learned image compression,” arXiv:2211.07273, 2023.
  * [30] W. Jiang and R. Wang, “MLIC++: Linear complexity multi-reference entropy modeling for learned image compression,” arXiv:2307.15421, 2023.
  * [31] W. Duan, Z. Chang, C. Jia, S. Wang, S. Ma, L. Song, and W. Gao, “Learned image compression using cross-component attention mechanism,” IEEE Transactions on Image Processing, vol. 32, pp. 5478–5493, 2023.
  * [32] H. Fu, F. Liang, J. Lin, B. Li, M. Akbari, J. Liang, G. Zhang, D. Liu, C. Tu, and J. Han, “Learned image compression with gaussian-laplacian-logistic mixture model and concatenated residual modules,” IEEE Transactions on Image Processing, vol. 32, pp. 2063–2076, 2023.
  * [33] T. Chen, H. Liu, Z. Ma, Q. Shen, X. Cao, and Y. Wang, “End-to-end learnt image compression via non-local attention optimization and improved context modeling,” IEEE Transactions on Image Processing, vol. 30, pp. 3179–3191, 2021.
  * [34] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [35] N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari, and E. Rahtu, “Image coding for machines: an end-to-end learned approach,” in Proc. IEEE ICASSP, pp. 1590–1594, 2021.
  * [36] N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari, H. R. Tavakoli, and E. Rahtu, “Learned image coding for machines: A content-adaptive approach,” in Proc. IEEE ICME, pp. 1–6, 2021.
  * [37] S. R. Alvar and I. V. Bajić, “Pareto-optimal bit allocation for collaborative intelligence,” IEEE Transactions on Image Processing, vol. 30, pp. 3348–3361, 2021.
  * [38] Z. Yuan, S. Rawlekar, S. Garg, E. Erkip, and Y. Wang, “Feature compression for rate constrained object detection on the edge,” in Proc. IEEE MIPR, pp. 1–6, 2022.
  * [39] H. Choi and I. V. Bajić, “Deep feature compression for collaborative object detection,” in Proc. IEEE ICIP, pp. 3743–3747, 2018.
  * [40] H. Choi and I. V. Bajić, “Near-lossless deep feature compression for collaborative intelligence,” in Proc. IEEE MMSP, pp. 1–6, 2018.
  * [41] A. E. Eshratifar, A. Esmaili, and M. Pedram, “Bottlenet: A deep learning architecture for intelligent mobile cloud computing services,” in Proc. IEEE/ACM ISLPED, pp. 1–6, 2019.
  * [42] R. A. Cohen, H. Choi, and I. V. Bajić, “Lightweight compression of neural network feature tensors for collaborative intelligence,” in Proc. IEEE ICME, pp. 1–6, 2020.
  * [43] H. Choi and I. V. Bajić, “Scalable image coding for humans and machines,” IEEE Trans. Image Process., vol. 31, pp. 2739–2754, 2022.
  * [44] N. Yan, C. Gao, D. Liu, H. Li, L. Li, and F. Wu, “Sssic: Semantics-to-signal scalable image coding with learned structural representations,” IEEE Transactions on Image Processing, vol. 30, pp. 8939–8954, 2021.
  * [45] O. G. Guleryuz, P. A. Chou, H. Hoppe, D. Tang, R. Du, P. Davidson, and S. Fanello, “Sandwiched image compression: Wrapping neural networks around a standard codec,” in Proc. IEEE ICIP, pp. 3757–3761, 2021.
  * [46] A. Said, M. K. Singh, and R. Pourreza, “Differentiable bit-rate estimation for neural-based video codec enhancement,” in Proc. PCS, 2022.
  * [47] O. G. Guleryuz, P. A. Chou, H. Hoppe, D. Tang, R. Du, P. Davidson, and S. Fanello, “Sandwiched image compression: Increasing the resolution and dynamic range of standard codecs,” in Proc. PCS, 2022.
  * [48] S. R. Alvar and I. V. Bajić, “Multi-task learning with compressible features for collaborative intelligence,” in Proc. IEEE ICIP, pp. 1705–1709, 2019.
  * [49] F. Mireshghallah, M. Taram, P. Vepakomma, A. Singh, R. Raskar, and H. Esmaeilzadeh, “Privacy in deep learning: A survey,” arXiv:2004.12254, 2020.
  * [50] X. Liu, L. Xie, Y. Wang, J. Zou, J. Xiong, Z. Ying, and A. V. Vasilakos, “Privacy and security issues in deep learning: A survey,” IEEE Access, vol. 9, pp. 4566–4593, 2021.
  * [51] A. Boulemtafes, A. Derhab, and Y. Challal, “A review of privacy-preserving techniques for deep learning,” Neurocomputing, vol. 384, pp. 21–45, 2020.
  * [52] S. Ge, B. Liu, P. Wang, Y. Li, and D. Zeng, “Learning privacy-preserving student networks via discriminative-generative distillation,” IEEE Transactions on Image Processing, vol. 32, pp. 116–127, 2023.
  * [53] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proc. ACM CCS, p. 1322–1333, 2015.
  * [54] C. Dwork, “Differential privacy: A survey of results,” in Theory and Applications of Models of Computation (M. Agrawal, D. Du, Z. Duan, and A. Li, eds.), pp. 1–19, Springer, 2008.
  * [55] J. Ryu, Y. Zheng, Y. Gao, S. Abuadbba, J. Kim, D. Won, S. Nepal, H. Kim, and C. Wang, “Can differential privacy practically protect collaborative deep learning inference for the internet of things?,” arXiv:2104.03813, 2021.
  * [56] J. Wang, J. Zhang, W. Bao, X. Zhu, B. Cao, and P. S. Yu, “Not just privacy: Improving performance of private deep learning in mobile cloud,” in Proc. ACM KDD, p. 2407–2416, 2018.
  * [57] I. Jarin and B. Eshete, “Pricure: Privacy-preserving collaborative inference in a multi-party setting,” in Proc. ACM IWSPA, p. 25–35, 2021.
  * [58] Z. He, T. Zhang, and R. B. Lee, “Attacking and protecting data privacy in edge–cloud collaborative inference systems,” IEEE Internet of Things Journal, vol. 8, no. 12, pp. 9706–9716, 2021.
  * [59] S. R. Alvar and I. V. Bajić, “Scalable privacy in multi-task image compression,” in Proc. IEEE VCIP, pp. 1–5, 2021.
  * [60] T. M. Cover and J. A. Thomas, Elements of Information Theory.  Wiley, 2nd ed., 2006.
  * [61] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” in Proc. 37th annual Allerton Conference on Communication, Control, and Computing, 1999.
  * [62] G. Jocher et al., “ultralytics/yolov5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference,” Feb. 2022.
  * [63] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NeurIPS, 2014.
  * [64] H. Choi and I. V. Bajić, “Deep feature compression for collaborative object detection,” in 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3743–3747, 2018.
  * [65] R. C. Gonzalez and R. E. Woods, Digital Image Processing.  2018\.
  * [66] Q. Meng, S. Zhao, Z. Huang, and F. Zhou, “MagFace: A universal representation for face recognition and quality assessment,” in Proc. IEEE/CVF CVPR, pp. 14225–14234, 2021.
  * [67] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” Tech. Rep. 07-49, University of Massachusetts, Amherst, October 2007.
  * [68] L. Best-Rowden, H. Han, C. Otto, B. F. Klare, and A. K. Jain, “Unconstrained face recognition: Identifying a person of interest from a media collection,” IEEE Trans. Inform. Forensics and Security, vol. 9, no. 12, pp. 2144–2157, 2014.
  * [69] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Web-scale training for face identification,” in Proc. IEEE/CVF CVPR, 2015.
  * [70] S. Zherzdev and A. Gruzdev, “LPRNet: License Plate Recognition via Deep Neural Networks,” arXiv:1806.10447, 2018.
  * [71] Z. Xu, W. Yang, A. Meng, N. Lu, and H. Huang, “Towards end-to-end license plate detection and recognition: A large dataset and baseline,” in Proc. ECCV, pp. 255–271, 2018.
  * [72] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. ECCV, Sept. 2014.
  * [73] T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, June 2019.
  * [74] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “VVenC: an open and optimized VVC encoder implementation,” in Proc. IEEE ICME Workshops, pp. 1–2, 2021.
  * [75] G. Bjøntegaard, “Calculation of average psnr differences between rd-curves,” in VCEG Meeting (ITU-T SG16 Q.6), 2001.  VCEG-M33.
  * [76] C. Hollmann, S. Liu, W. Gao, and X. Xu, “[VCM] on VCM reporting template.” ISO/IEC JTC 1/SC 29/WG2 M56185, Jan. 2021.

  *[DNN]: Deep Neural Network
  *[AI]: Artificial Intelligence
  *[IoT]: Internet of Things
  *[M2M]: machine-to-machine
  *[DNNs]: Deep Neural Network
  *[MIA]: Model Inversion Attack
  *[VVC]: Versatile Video Coding
  *[DP]: Differential Privacy
  *[MIAs]: Model Inversion Attack
  *[MI]: mutual information
  *[QP]: Quantization Parameter