# FrankenSplit: Saliency Guided Neural Feature Compression with Shallow
Variational Bottleneck Injection

Alireza Furutanpey<EMAIL_ADDRESS>Distributed Systems Group
TU ViennaViennaAustria ,  Philipp Raith<EMAIL_ADDRESS>Distributed
Systems Group
TU ViennaViennaAustria and  Schahram Dustdar<EMAIL_ADDRESS>Distributed Systems Group
TU ViennaViennaAustria

###### Abstract.

The rise of mobile AI accelerators allows latency-sensitive applications to
execute lightweight Deep Neural Networks (DNNs) on the client side. However,
critical applications require powerful models that edge devices cannot host
and must therefore offload requests, where the high-dimensional data will
compete for limited bandwidth. Split Computing (SC) alleviates resource
inefficiency by partitioning DNN layers across devices, but current methods
are overly specific and only marginally reduce bandwidth consumption. This
work proposes shifting away from focusing on executing shallow layers of
partitioned DNNs. Instead, it advocates concentrating the local resources on
variational compression optimized for machine interpretability. We introduce a
novel framework for resource-conscious compression models and extensively
evaluate our method in an environment reflecting the asymmetric resource
distribution between edge devices and servers. Our method achieves 60% lower
bitrate than a state-of-the-art SC method without decreasing accuracy and is
up to 16x faster than offloading with existing codec standards.

Split Computing, Edge Computing, Edge Intelligence, Learned Image Compression,
Data Compression, Feature Compression, Knowledge Distillation

††copyright: none††conference: ; , 2023; ††ccs: Computer systems organization
Embedded and cyber-physical systems††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Distributed artificial
intelligence††ccs: Computing methodologies Computer vision tasks

##  1\. Introduction

Deep Learning (DL) has demonstrated that it can solve real-world problems in
challenging areas ranging from Computer Vision (CV) (Voulodimos et al., 2018)
to Natural language Processing (NLP) (Otter et al., 2020). Complementary with
the advancements in mobile edge computing (MEC) (Feng et al., 2022) and
energy-efficient AI accelerators, visions of intelligent city-scale platforms
for critical applications, such as mobile augmented reality (MAR) (Rausch et
al., 2021), disaster warning (Arinta and Andi W.R., 2019), or facilities
management (Xin et al., 2022), seem progressively feasible. Nevertheless, the
accelerating pervasiveness of mobile clients gave unprecedented growth in
Machine-to-Machine (M2M) communication (Cisco, 2020), leading to an
insurmountable amount of network traffic. A root cause is the intrinsic
limitation of mobile devices that allows them to realistically host a single
lightweight Deep Neural Network (DNN) in memory at a time. Hence, clients must
frequently offload inference requests since the local resources alone cannot
meet the complex and demanding requirements relying on multiple highly
accurate DNNs (Romero et al., 2021).

The downside to traditional offloading approaches is that by constantly
streaming high-dimensional visual data, the limited bandwidth will inevitably
lead to network congestion resulting in erratic response delays, and it leaves
valuable client-side resources idle. Recently, Split Computing (SC) emerged as
an alternative to alleviate inefficient resource utilization and to facilitate
low-latency and performance-critical mobile inference. The basic idea is to
partition a DNN to process the shallow layers with the client and send a
reduced representation to the remaining deeper layers deployed on a server. SC
has the potential to draw resources from the entire edge-cloud compute
continuum. However, for performance and latency-sensitive applications,
current methods have limited applicability and only conditionally utilize
onboard resources for two reasons. First, split computation is only sensible
if the time for executing shallow layers locally, transferring the features,
and remotely executing the remaining layers is less than the time of directly
offloading the task. Due to fluctuating network conditions and the inherent
resource asymmetry between the client and the server, finding the right
balance is challenging. Consequently, most SC methods rely on carefully
calibrated runtimes that periodically measure internal and external
conditions. Second, a particular set of shallow layers corresponds to a model
for only one task, i.e., SC does not replace offloading for numerous types of
applications (e.g., MAR) where inference from multiple DNNs within a small
time frame is necessary.

We argue that flawed underlying assumptions incur the limitations of SC.
Specifically, the rationale of SC is that analogously to how lightweight
embedded DNNs can perform fast local inference utilizing specialized
accelerators, the same principles apply when repurposing the resources for
executing the shallow layers of a larger DNN. Additionally, work on SC
increasingly draws from embedded AI for some form of data reduction, such as
feature tensor quantization (Almeida et al., 2022) and channel pruning (Lee et
al., 2021). Notably, the number of layers a constrained client can
realistically onload is a negligible amount of the total computation time,
i.e., server-grade hardware can execute the same layers by orders of magnitude
faster than a constrained client. In other words, the potential of SC is
inhibited by primarily focusing on shifting parts of the model execution from
the server to the client, and the viability of SC methods is not determined by
how well they can partially compute a split network but by how well they can
reduce the input size. Therefore, we pose the following question: Is it more
efficient to focus the local resources on compressing the data rather than
executing shallow layers of a network that would constitute a negligible
amount of the total computation cost on the server?

Although some work report outstanding results by modifying the layers deployed
on the client to reduce the transfer size, they target a specific class of DNN
architectures or evaluate their method on small trivial datasets (Matsubara et
al., 2022a). Here, we advocate dedicating the client resources solely to
finding a minimal compressed representation that is interpretable by arbitrary
DNN architectures and multiple downstream tasks.

To this end, we draw from recent advancements in lossy learned image
compression (LIC). Despite outperforming handcrafted codecs (Yang et al.,
2022), such as WebP, or PNG, LIC is unsuitable for real-time inference in MEC
since they consist of large autoencoders and other complex mechanisms that are
demanding even for server-grade hardware. Moreover, research in compression
primarily focuses on reconstruction for human perception containing
information superfluous for M2M communication, i.e., it is possible to
considerably reduce bitrate by compressing features exclusively for machine
interpretability.

Feature compression has its roots in the information bottleneck (IB) principle
(Tishby et al., 2000) and provides the underlying objective to fit existing
LIC training objectives. Nevertheless, analogous to the deep variational IB
(DVIB) (Alemi et al., 2016), current neural compression methods typically
place the bottleneck at the penultimate layer, i.e., a distributed setting
assumes the client hosts almost the entire network. Contrastingly, the
opposite is needed in MEC.

This work coins the term Shallow Variational Information Bottleneck Injection
(SVBI) to address the increasing network traffic incurred by M2M communication
and introduces FrankenSplit. FrankenSplit is a novel training and design
heuristic for embeddable feature compression models. It generalizes SC to
arbitrary DNN architectures and multiple downstream tasks with a single
encoder. It maximizes resource efficiency by consistently outperforming
offloading in various environments without relying on a complex runtime system
that continuously probes conditions. Importantly, even without drawing from
work on efficient neural network components, our encoder has only 0.14M. Yet,
it can provide clients in highly constrained environments with state-of-the-
art image classification accuracy in 0.15 seconds. For reference, MobileNets
(Sandler et al., 2018; Howard et al., 2019) see widespread praise in MEC for
their lightweight architecture with comparably mediocre accuracy despite
having 25x-39x more parameters than our encoder.

For transparency and to encourage advancing this research directive, we open-
source our repository111https://github.com/rezafuru/FrankenSplit including
configurations and several pre-trained weights. We summarize the contributions
of this work as

  * •

Thoroughly exploring how shallow and deep bottleneck injection differ for
feature compression.

  * •

Introducing a novel saliency-guided training method to overcome the challenges
of SVBI to train a lightweight encoder with limited capacity.

  * •

Introducing an autoencoder design heuristic that permits attaching various
decoders from arbitrary DNN architectures to one universal encoder.

Section 2 discusses relevant work on SC and LIC. Section 3 describes the
problem domain and progressively introduces the solution approach. Section 4
extensively justifies relevant performance indicators and evaluates several
implementations of FrankenSplit against various baselines to assess our
method’s efficacy. Lastly, Section 5 summarizes this work and highlights
limitations to motivate follow-up work.

##  2\. Related Work

###  Neural Data Compression

####  2.1.1. Learned Image Compression

The goal of (lossy) image compression is minimizing bitrates while preserving
information critical for human perception. Transform coding is a basic
framework of lossy compression, which divides the compression task into
decorrelation and quantization (Goyal, 2001). Decorrelation reduces the
statistical dependencies of the pixels, allowing for more effective entropy
coding, while quantization represents the values as a finite set of integers.
The core difference between handcrafted and learned methods is that the former
relies on linear transformations based on expert knowledge. Contrarily, the
latter is data-driven with non-linear transformations learned by neural
networks (Ballé et al., 2020).

Ballé et al. introduced the Factorized Prior (FP) entropy model and formulated
the neural compression problem by finding a representation with minimal
entropy (Ballé et al., 2017). An encoder network transforms the original input
to a latent variable, capturing the input’s statistical dependencies. In
follow-up work, Ballé et al. (Ballé et al., 2018) and Minnen et al. (Minnen et
al., 2018) extend the FP entropy model by including a hyperprior as side
information for the prior. Minnen et al. (Minnen et al., 2018) introduce the
Joint Autoregressive and Hierarchical Priors (JAHP) entropy model, which adds
a context model to the existing scale hyperprior latent variable models.
Typically, context models are lightweight, i.e., they add a negligible number
of parameters, but their sequential processing increases the end-to-end
latency by orders of magnitude.

####  2.1.2. Feature Compression

Singh et al. demonstrate a practical method for the Information Bottleneck
principle in a compression framework by introducing the bottleneck in the
penultimate layer and replacing the distortion loss with the cross-entropy for
image classification (Singh et al., 2020). Dubois et al. generalized the VIB
for multiple downstream tasks and were the first to describe the feature
compression task formally (Dubois et al., 2021). However, their encoder-only
CLIP compressor has over 87 million parameters. Both Dubois and Singh et al.
consider feature compression for mass storage, i.e., they assume the data is
already present at the target server. In contrast, we consider how resource-
constrained clients must first compress the high-dimensional visual data
before sending it over a network.

Closest to our work is the Entropic Student (ES) proposed by Matsubara et al.
(Matsubara et al., 2022b, c), as we follow the same objective of real-time
inference with feature compression. Nevertheless, they simply apply the
learning objective, and a scaled-down version of autoencoder from (Ballé et
al., 2017, 2018). Moreover, they do not analyze the intrinsic differences
between feature and image compression nor explain their solution approach.
Contrastingly, we carefully examine the problem domain of resource-conscious
feature compression to identify underlying issues with current methods,
allowing us to conceive novel solutions with significantly better rate-
distortion performance.

###  Split Computing

We distinguish between two orthogonal approaches to SC.

####  2.2.1. Split Runtimes

Split runtime systems are characterized by performing no or minimal
modifications on off-the-shelf DNNs. The objective is to dynamically determine
split points according to the available resources, network conditions, and
intrinsic model properties. Hence, split runtimes primarily focus on profilers
and adaptive schedulers. Kang et al. performed extensive compute cost and
feature size analysis on the layer-level characterizations of DNNs and
introduced the first split runtime system (Kang et al., 2017). Their study has
shown that split runtimes are only sensible for DNNs with an early natural
bottleneck, i.e., models performing aggressive dimensionality reduction within
the shallow layers. However, most modern DNNs increase feature dimensions
until the last layers for better representation. Consequently, follow-up work
focuses on tensor manipulation (Li et al., 2018; Laskaridis et al., 2020;
Almeida et al., 2022). We argue against split runtimes since they introduce
considerable complexity. Worse, the system must be tuned toward external
conditions, with extensive profiling and careful calibration. Additionally,
runtimes raise overhead and another point of failure by hosting a network-
spanning system. Notably, even the most sophisticated methods still rely on a
natural bottleneck, evidenced by how state-of-the-art split runtimes still
report results on old superseded DNNs with an early bottleneck (Liu et al.,
2022).

####  2.2.2. Artificial Bottleneck Injection

By shifting the effort towards modifying and re-training an existing base
model (backbone) to replace the shallow layers with an artificial bottleneck,
bottleneck Injection retains the simplicity of offloading. Eshratifar et al.
replace the shallow layers of ResNet-50 with a deterministic autoencoder
network (Eshratifar et al., 2019). A follow-up work by Jiawei Shao and Jun
Zhang further considers noisy communication channels (Shao and Zhang, 2020).
Matsubara et al. (Matsubara et al., 2019), and Sbai et al. (Sbai et al., 2021)
propose a more general network agnostic knowledge distillation (KD) method for
embedding autoencoders, where the output of the split point from the
unmodified backbone serves as a teacher. Lastly, we consider the work in
(Matsubara et al., 2022b) as the state-of-the-art for bottleneck injection.

Although bottleneck injection is promising, there are two problems with
current methods. They rely on autoencoder for crude data compression or are
intended for a specific class of neural network architecture.

This work addresses both limitations of naive bottleneck injection methods.

##  3\. Problem Formulation & Proposed Method

The aim is for a constrained client to request real-time predictions from a
large DNN while maximizing resource efficiency and minimizing bandwidth
consumption. Figure 1 illustrates inference strategies we will consider during
evaluation.

Figure 1. Edge Inference Strategies

Strategy a) corresponds to offloading strategies with CPU-bound handcrafted
codecs applied to the input and is undesirable since lossy codecs achieve low
bitrates at a high predictive loss. Strategy b) utilizes the onboard
accelerator and applies a LIC method. Although they achieve low bitrates at a
considerably less predictive loss, they incur an excessive overhead. Strategy
c) is our proposed method with an embeddable variational compression model. To
overcome the limitations of a) and b),  (i) we require a resource-conscious
encoder that is maximally compressive about a useful representation without
increasing the predictive loss. Additionally,  (ii) the decoder should exploit
the available server-side resources but still incur minimal overhead on the
backbone. Lastly,  (iii) a compression model should fit for different
downstream tasks and architectural families (e.g., CNNs or Vision
Transformers) . Our solution approach focuses on two distinct but intertwined
aspects. First is an appropriate training objective for feature compression
with limited model capacity. The second concerns a practical implementation by
introducing an architectural design heuristic for edge-oriented variational
autoencoders.

###  Rate-Distortion Theory and the Information Bottleneck

By Shannon’s rate-distortion theory (Shannon, 1959), we seek a mapping from a
random variable (r.v.) $X$ to an r.v. $Z$, minimizing the bitrate the outcomes
of $X$ and can be reconstructed according to a distortion constraint $D_{c}$.
More formally, given a distortion measure $\mathcal{D}$ and a distortion
constraint $D_{c}$, the minimal bitrate is

(1)
$\underset{P_{Z|X}}{\mathrm{min}}\;I(X;Z)\;\text{s.t.}\;\mathcal{D}(X,Z)\leq
D_{c}$

where $I(X;Z)$ is the mutual information between the input and latent vector
and is defined as

(2) $\mathrm{I}(X;Z)=\int\int
p(x,z)\log\left(\frac{p(x,z)}{p(x)p(z)}\right)dxdz$

Notably, optimizing for perceptual quality results in $Z$ to retain
information redundant for downstream tasks.

Consider the Data Processing Inequality (DPI). For any 3 r.v.s $X,Y,Z$ that
form a Markov chain $X\leftrightarrow Y\leftrightarrow Z$ the following holds:

(3) $I(X;Y)\geq I(X;Z)$

Additionally, as argued by Tishby and Zaslavsky (Tishby and Zaslavsky, 2015),
we can view layered neural networks as a Markov chain of successive
representations and describe the information flow in an n-layered sequential
DNN, layer with the information path:

(4) $I(X;Y)\geq I(R_{1};Y)\geq I(R_{2};Y)\geq\dots I(R_{n};Y)\geq
I(\tilde{Y};Y)$

In other words, the final representation before a prediction $R_{n}$ for a
discriminative task cannot have more mutual information with the target than
the input $X$ and typically has less. Empirically, this is established by how
slight perturbations and certain transformations do not affect predictions.
Hence, when the task is to predict the ground-truth labels Y from a joint
distribution $P_{X,Y}$, the rate-distortion objective is essentially given by
the information bottleneck principle (Tishby et al., 2000). By relaxing the
(1) with a lagrangian multiplier, the objective is to maximize:

(5) $I(Z;Y)-\beta I(Z;X)$

Specifically, an encoding $Z$ should be a minimal sufficient statistic of $X$
respective $Y$, i.e., we want $Z$ to contain relevant information regarding
$Y$ while discarding irrelevant information from $X$. Practical
implementations differ by the target task and how they approximate (5). For
example, an approximation of $I(Z;Y)$ for an arbitrary classification task is
the conditional cross entropy (CE) (Yang et al., 2022):

(6) $\mathcal{D}=H(P_{Y},P_{\tilde{Y}|Z})$

Unsurprisingly, using (6) for estimating the distortion measure in (5) for
end-to-end optimization of a neural compression model is not a novel idea.
However, a common assumption in such work is that the latent variable is the
final representation $R_{n}$ before performing the prediction, i.e., the
encoder must consist of an entire backbone model which we refer to as Deep
Variational Information bottleneck Injection (DVBI). Conversely, we work with
resource-constrained clients, i.e., to conceive lightweight encoders, we must
shift the bottleneck to the shallow layers, which we refer to as shallow
variational bottleneck injection (SVBI). Intuitively, the existing methods for
DVBI should generalize to SVBI should, e.g., approximating the distortion term
with (6) as in (Singh et al., 2020). Although by shifting the bottleneck to
the shallow layers, the encoder has less capacity for finding a minimal
sufficient representation, the objective is still an approximation of (1).

Head Distillation (HD) (Matsubara et al., 2019; Sbai et al., 2021) is an
alternative method that naturally relates to shallow bottleneck. In HD, the
removed head network teaches the artificial bottleneck. However, we will show
that HD is a suboptimal approximation of (1) for variational compression. In
the following, we bridge the gap between deep and shallow bottleneck injection
by formulating the VIB objective for HD.

###  Head Distilled Deep Variational IB

Ideally, the bottleneck is embeddable in an existing predictor
$P_{\mathcal{T}}$ without decreasing the performance. Therefore, it is not the
hard labels $Y$ that define the task but the soft labels $Y_{\mathcal{T}}$.
For simplicity, we handle the case for one task and defer how SVBI naturally
generalizes to multiple downstream tasks and DNNs to Section 4.3.6.

To perform SVBI, take a copy of $P_{\mathcal{T}}$. Then, mark the location of
the bottleneck by separating the copy into a head $\mathcal{P}_{h}$ and a tail
$\mathcal{P}_{t}$. Importantly, both parts are deterministic, i.e., for every
realization of r.v. $X$ there is a representation $\mathcal{P}_{h}(x)=h$ such
that $\mathcal{P}_{\mathcal{T}}(x)=\mathcal{P}_{h}(\mathcal{P}_{t}(x))$.
Lastly, replace the head with an autoencoder and a parametric entropy model.
The encoder is deployed at the sender, the decoder at the receiver, and the
entropy model is shared.

We distinguish between two optimization strategies to train the compression
model. First, is direct optimization corresponding to the DVIB objective in
(5), except we replace the CE with the standard KD loss (Hinton et al., 2015)
to approximate $I(Z;Y)$. Second is indirect optimization and describes HD with
the objective:

(7) $I(Z;H)-\beta\;I(Z;X)$

Unlike the former, the latter does not directly correspond to (1) for a
representation $Z$ that is a minimal sufficient statistic of $X$ respective
$Y_{\mathcal{T}}$. Instead, it replaces $Y$ with a proxy task for the
compression model to replicate the output of the replaced head, i.e., training
methods approximating (7) optimize for a $Z$ that is a minimal sufficient
statistic of $X$ respective $H$. Figure 2 illustrates the difference between
estimating the objectives (5) and (7). Intuitively, with faithful replication
of $H$, the partially modified DNN has an information path equivalent to its
unmodified version. A sufficient statistic retains the information necessary
to replicate the input for a deterministic tail, i.e., the final prediction
does not change. The problem of (7) is that it is a suboptimal approximation
of (1). Although sufficiency holds, it does not optimize $Z$ respective
$Y_{\mathcal{T}}$. The marginal distribution of $Y_{\mathcal{T}}$ now arises
from the r.v. $X$ and the parameters of $\mathcal{P}_{\mathcal{T}}$. Moreover,
since maximizing $I(Z;H)$ is only a proxy objective for
$I(Z;Y_{\mathcal{T}})$, it corresponds to the Markov chain
$Y_{\mathcal{T}}\leftrightarrow H\leftrightarrow X\leftrightarrow Z$. Notice
how $X$ is now conditional independent of $Y_{\mathcal{T}}$, breaking the
assumption from the previous subsection. Consequently, when training a
compression model with (7), we skew the rate-distortion optimization towards a
higher rate than necessary by setting the lower bound for the distortion
estimation too high. Nevertheless, while direct optimization works well for
DVBI (Singh et al., 2020), we show how it results in poor r-d performance for
SVBI in Section 4. Interestingly, HD performs considerably better despite its
inherent suboptimality, which is why it is the focus of this work.

Figure 2. Left: direct optimization. Right: HD

###  Loss Function for End-to-end Optimization

Unlike most work on artificial bottleneck injection, our bottleneck is not a
deterministic autoencoder. Instead, we embed a stochastic compression model.
Categorically, we follow nonlinear transform coding (Ballé et al., 2020) (NTC)
to implement a neural compression algorithm. For an image vector $x$, we have
a parametric analysis transform $g_{a}(x;\phi_{g})$ maps $x$ to a latent
vector $z$. Then, a quantizer $Q$ discretizes $z$ to $\bar{z}$, such that an
entropy coder can losslessly compress $\bar{z}$ to a sequence of bits. Here,
we start diverging from NTC for image compression. Rather than an approximate
inverse of the analysis, the parametric synthesis transform
$g_{s}(\bar{z};\theta_{g})$ maps $\bar{z}$ to a representation $\tilde{h}$
that is suitable for various tail predictors and downstream tasks. Our
objective resembles variational autoencoder-based image compression
optimization, as introduced in (Ballé et al., 2017, 2018). Analogous to
variational inference, we approximate the intractable posterior
$p(\tilde{z}|x)$ with a parametric variational density $q(\tilde{z}|x)$ as
follows (excluding constants):

(8) $\mathbb{E}_{\boldsymbol{x}\sim
p_{\boldsymbol{x}}}D_{\mathrm{KL}}\left[q\|p_{\tilde{\boldsymbol{z}}\mid\boldsymbol{x}}\right]=\mathbb{E}_{\boldsymbol{x}\sim
p_{\boldsymbol{x}}}\mathbb{E}_{\tilde{\boldsymbol{z}}\sim
q}[\underbrace{-\mathrm{log}\;p(x|\tilde{z})}_{\text{
distortion}}-\overbrace{\mathrm{log}\;p(\tilde{z})}^{\text{weighted rate}}]$

By assuming a gaussian distribution such that the likelihood of the distortion
term is given by

(9)
$P_{x|\tilde{z}}(x\;|\;\tilde{z},\theta_{g})=\mathcal{N}(x\;|\;g_{s}(\tilde{z};\theta_{g}),\;\mathbf{1})$

we can use the square sum of differences between $h$ and $\tilde{h}$ as our
distortion loss.

The rate term describes the cost of compressing $\tilde{z}$. Analogous to the
LIC methods discussed in Section 2.1, we apply uniform quantization
$\bar{z}=\lfloor\tilde{z}\rceil$. Since discretization leads to problems with
the gradient flow, we apply a continuous relaxation by adding uniform noise
$\eta\;\thicksim\;\mathcal{U}(-\frac{1}{2},\frac{1}{2})$. Combining the rate
and distortion term, we derive the loss function for approximating objective
(7) as

(10) $\mathcal{L}_{\mathrm{}}=\|\mathcal{P}_{h}(x)\text{ -
}(g_{s}(g_{a}(x;\phi_{g})+\eta;\theta_{g})\|^{2}_{2}+\beta\;\mathrm{log}(g_{a}(x;\theta_{g})+\eta)$

Note, in the relaxed lagrangian of (1), the weight is on the distortion while
we weight the rate term to align closer to the variational information
bottleneck objective. Moreover, omitting the second term $(\beta=0)$ results
in the head distillation objective for naive bottleneck injection.

Since the loss in (10) is an approximation of the objective (7), the model
weights will not converge towards an optimum for objective (1). Nevertheless,
notice how the suboptimality stems from treating every pixel of $H$ equally
crucial to the remaining layer. In the previous subsection, we established
that as the remaining layers further process a representation $h$, its
information content decreases monotonously. Intuitively, the shallow layers
extract high-level features, while deeper layers are more focused. The
implication here is that the MSE in (10) overly strictly penalizes pixels at
spatial locations which contain redundant information that later layers can
safely discard. Contrarily, the loss may not penalize the salient pixels
enough when $\tilde{h}$ is numerically close to $h$. This insight explains why
the objective in (5) does not yield good results when the bottleneck is at a
shallow layer, as the mutual information $I(Y;\tilde{Y})$ is not sufficiently
high. Since the representation of the last hidden layer and the representation
of the shallow layer are so far apart in the information path, there is
insufficient information to minimize $\mathcal{D}(H;\tilde{H})$.

Finally, assume we have access to a vector $\mathcal{S}$. Each
$s_{i}\in\mathcal{S}$ is a weight term for a spatial location, which is
salient about the conditional probability distributions of the remaining tail
layers. Then, we should be able to improve the rate-distortion performance by
replacing the distortion term in (10) with

(11)
$\mathcal{L}_{\mathrm{distortion}}=s_{i}\cdot\frac{1}{N}\sum_{i}(h_{i}-\tilde{h}_{i})^{2}$

###  Saliency Maps

Class Activation Mapping (CAM) is a method to increase the explainability of
trained models (Zhou et al., 2016). Nevertheless, it effectively measures the
importance of each spatial location. As illustrated in Figure 3, CAM allows us
to extract attention maps from a pre-trained teacher model at any layer. The
original literature extensively covers computing the saliency maps, so we do
not cover the details.

Figure 3. Extracted Saliency Maps using Grad-CAM

To derive vector $\mathcal{S}$ for the loss in (11), we use one of the Grad-
CAM variants (Selvaraju et al., 2019) to calculate and persist a saliency map
for each sample as a preliminary step. Since Grad-CAM allows us to derive
pixel importance at any layer, each saliency map is the unweighted average of
the maps created by the top-level layers.

###  Network Architecture

The beginning of this section broke down our aim into three problems. We
addressed the first with SVBI and proposed a novel training method for low-
capacity models. To not inflate the significance of our contribution, we
refrain from including efficient neural network components based on existing
work in efficient neural network design. A generalizable resource-asymmetry-
aware autoencoder design remains; components should be reusable and attachable
to arbitrary network architectures once trained.

Figure 4. Single encoder for three architectural families

####  3.5.1. Inter- and Intra-architecture module re-attachment

When performing predictions with the latent, it may seem redundant to have a
synthesis transform. Rather, we could directly feed the encoder output to the
tail of a backbone model. However, this locks the encoder to only a single
backbone. The challenge is to conceive a design heuristic that can exploit the
available server resources to aid the lightweight encoder with minimal
overhead on the prediction task. We assume that, after training an embedded
compression model with one particular backbone as described in Section 3.3, it
is possible to reuse the encoder or decoder for different backbones with
little effort based on the following two observations. First is how most
modern DNNs consist of an initial embedding followed by a few top-level
layers. The top-level layers consist of coarse-grained layers, while the
coarse-grained ones recursively consist of finer-grained ones. The terminology
differs for each work, but here we will refer to top-level layers as stages
and the coarse-grained layers as blocks. Within architectural families, the
individual components are identical. The difference between variants is
primarily the embed dimensions or the block ratio of the deepest stages. For
example, the block ratio of ResNet-50 is 3:4:6:3, while the block ratio of
ResNet-101 is 3:4:23:3. Importantly, stages define the shallow layers (head
model) as all the layers before the deepest stage. Additionally, there is a
natural interface for the decoder to attach to multiple backbones with minimal
effort, which we refer to as intra-architecture module re-attachment. This
second observation is how network families introduce different inductive
biases (e.g., convolutions versus attention modules with sliding windows),
leading to diverging representations among non-related architectures.
Therefore, it is naive to disregard architecture-induced bias by directly
repurposing neural compression models for SC. For example, Matsubara et al.
(Matsubara et al., 2022c) use a scaled-down version of Ballé et al.’s (Ballé
et al., 2017) convolutional neural compression model, demonstrating strong
rate-distortion performance for bottlenecks reconstructing a convolutional
layer. However, we will show that this does not generalize to other
architectural families, such as hierarchical vision transformers (Liu et al.,
2021). One potential solution is to leverage identical components from a
target network. While this approach may be inconsequential for decoders, given
the homogeneity of server hardware, it is inadequate for encoders due to the
heterogeneity of edge devices. Vendors have varying support for the basic
building blocks of a DNN, and particular operations may be prohibitively
expensive for the client. Consequently, regardless of the encoder or decoder,
we account for the heterogeneity with a universal encoder composed of three
downsampling residual blocks of two stacked $3\times 3$ convolutions, totaling
around 140k parameters. Since the mutual information between the original
input and shallow layers is still high, it should be possible to add a that
maps the representation of the encoder to one suitable for different
architectures based on invariance towards invertible transformations of mutual
information. Specifically, for invertible functions $\phi,\psi$, it holds
that:

(12) $I(X;Y)=I(\phi(X);\psi(Y))$

We refer to re-mapping a representation from one family to another as inter-
architecture module re-attachment.

####  3.5.2. Resource-Asymmetry Aware Network Design

The number of autoencoder parameters is typically comparable between the
encoder and decoder. Increasing the decoder parameters does not change the
inability to recover lost signals (Section 3.1), i.e., it does not offset the
limited encoder capacity for accurately distinguishing between valuable and
redundant information. Nevertheless, when optimizing the entire compression
pipeline simultaneously, the training algorithm should consider the ability
for restoration by the decoder, which in turn should help the encoder
parameters to converge towards finding a minimal sufficient statistic
respective to the task. Hence, Inspired by (Liang et al., 2021), we partially
treat decoding as an image restoration problem and add restoration blocks with
optional upsampling between backbone-specific transformation blocks. The
incurred latency is minimal and is offset by skipping the shallow layers.

Figure 5. High-level deployment sketches

To summarize, our network architecture is a general design heuristic, i.e.,
the individual components are interchangeable with the current state-of-the-
art building blocks for vision models. Figure 4 illustrates the purpose of the
decoder in restoring and transforming the latent. The number in parentheses
represents stage depth, but to space constraints, the figure does not
represent the exact architectures of the models, and we cannot describe each
decoder architecture in detail. Instead, the accompanying repository contains
descriptive configurations. Complementary, Figure 5 illustrates with a
deployment sketch how a universal encoder overcomes the limitation of SC and
local inference only to permit inference from one DNN at a time.

##  4\. Evaluation

###  Experiment Setting

The experiments reflect the deployment strategies illustrated in Figure 1.
Ultimately, we aim to demonstrate that FrankenSplit enables latency-sensitive
and high-performance applications. Regardless of the particular task, a mobile
edge client requires access to a DNN with high predictive strength on a
server. Therefore, we must show whether FrankenSplit adequately solves two
problems associated with offloading high-dimensional image data for real-time
inference tasks. First, whether it considerably reduces the bandwidth
consumption compared to existing methods without sacrificing predictive
strength. Second, whether it improves inference times over various
communication channels, i.e., it must remain competitive even when stronger
connections are available.

Lastly, FrankenSplit should still be applicable as newer DNN architectures
emerge, i.e., the evaluation should assess whether our method generalizes to
arbitrary backbones. However, since it is infeasible to perform exhaustive
experiments on all existing visual models, we focus on three well-known
representatives and a subset of their variants instead. Namely, (i) ResNet (He
et al., 2015) for classic residual CNNs. (ii) Swin Transformer (Liu et al.,
2021) for hierarchical vision transformers, which are receiving increasing
adaptation for a wide variety of vision tasks. (iii) ConvNeXt for modernized
state-of-the-art CNNs. Table 1 summarizes the relevant characteristics of the
unmodified backbones subject to our experiments.

Table 1. Overview of backbones executed on the server

Backbone | Ratios | Params |  | Inference (ms)
---
| Top-1 Acc. (%)
---
Swin-T | 2:2:6:2 | 28.33M | 4.77 | 81.93
Swin-S | 2:2:18:2 | 49.74M | 8.95 | 83.46
Swin-B | 2:2:30:2 | 71.13M | 13.14 | 83.88
ConvNeXt-T | 3:3:9:3 | 28.59M | 5.12 | 82.70
ConvNeXt-S | 3:3:27:3 | 50.22M | 5.65 | 83.71
ConvNeXt-B | 3:3:27:3 | 88.59M | 6.09 | 84.43
ResNet-50 | 3:4:6:3 | 25.56M | 5.17 | 80.10
ResNet-101 | 3:4:23:3 | 44.55M | 10.17 | 81.91
ResNet-152 | 3:8:36:3 | 60.19M | 15.18 | 82.54

####  4.1.1. Baselines

Since our work aligns closest to learned image compression, we extensively
compare FrankenSplit with learned and handcrafted codecs applied to the input
images, i.e., the input to the backbone is the distorted output. Comparing
task-specific methods to general-purpose image compression methods may seem
unfair. However, FrankenSplit’s universal encoder has up to 260x less
trainable parameters and further reduces overhead by not including side
information or a sequential context model.

The naming convention for the learned baselines is the first author’s name,
followed by the entropy model. Specifically, we choose the work by Balle et
al. (Ballé et al., 2017, 2018) and Minnen et al. (Minnen et al., 2018) for LIC
methods since they represent foundational milestones. Complementary, we
include the work by Cheng et al. (Cheng et al., 2020) to demonstrate
improvements with architectural enhancement.

As the representative for disregarding autoencoder size to achieve state-of-
the-art r-d performance in LIC, we chose the work by Chen et al. (Chen et al.,
2021) Their method differs from other LIC baselines by using a partially
parallelizable context model, which trades off compression rate with execution
time according to the configurable block size. However, due to the large
autoencoder, we found evaluating the inference time on constrained devices
impractical when the context model is purely sequential and set the block size
to 64x64. Additionally, we include the work by Lu et al. (Lu et al., 2022) as
a milestone of the recent effort on efficient LIC with reduced autoencoders
but only for latency-related experiments since we do not have access to the
trained weights. Nevertheless, according to their results, we can assume a
near-identical r-d performance as the work by Cheng et al.

As a baseline for the state-of-the-art SC, we include the Entropic Student
(ES) (Matsubara et al., 2022b, c). Crucially, the ES demonstrates the
performance of naively applying a minimally adjusted LIC method for feature
compression without considering the intrinsic properties of the problem domain
we have derived in Section 3.1. One caveat is that we intend to show how
FrankenSplit generalizes beyond CNN backbones, despite the encoder’s
simplistic CNN architecture. Although Matsubara et al. evaluate the ES on a
wide range of backbones, most have no lossless configurations. However,
comparing bottleneck injection methods using different backbones is fair, as
we found that the choice does not significantly impact the r-d performance.
Therefore, for an intuitive comparison, we choose ES with ResNet-50 using the
same factorized prior entropy model as FrankenSplit.

We separate the experiments into two categories to assess whether our proposed
method addresses the abovementioned problems.

####  4.1.2. Criteria rate-distortion performance

We primarily measure the bitrate in bits per pixel (bpp) which is sensible
because it permits directly comparing models with different input sizes.
Choosing a distortion measure to draw meaningful and honest comparisons is
challenging for feature compression. Unlike evaluating reconstruction fidelity
for image compression, PSNR or MS-SSIM does not provide intuitive results
regarding predictive strength. Similarly, adhering to the customs of SC by
reporting absolute values, such as top-1 accuracy, gives an unfair advantage
to experiments conducted on higher capacity backbones and veils the efficacy
of a proposed method. Consequently, we evaluate the distortion with the
relative measure predictive loss. To ensure a fair comparison, we give the LIC
and handcrafted baselines a grace threshold of 1.0% top-1 accuracy,
considering it is possible to mitigate some predictive loss incurred by codec
artifacts (Luo et al., 2020). For example, if a model has a predictive loss of
1.5% top-1 accuracy, we report it as 0.5% below our definition of lossless
prediction. However, for FrankenSplit, we set the grace threshold at 0.4%,
reflecting the configuration with the lowest predictive loss of the ES. Note
that the ES improves r-d performance by finetuning the backbone weights by
applying a secondary training stage. In contrast, we put our work at a
disadvantage by training FrankenSplit exclusively with the methods introduced
in this work.

####  4.1.3. Measuring latency and overhead

The second category concerns with execution times of prediction pipelines with
various wireless connections. To account for the resource asymmetry in MEC, we
use NVIDIA Jetson boards222https://www.nvidia.com/en-gb/autonomous-
machines/embedded-systems/ to represent the capable but resource-constrained
mobile client, and the server hosts a powerful GPU. Table 2 summarizes the
hardware we use in our experiments.

Table 2. Clients and server hardware configuration

Device | Arch | CPU | GPU
---|---|---|---
Server | x86 | 16x Ryzen @ 3.4 GHz | RTX 3090
Client (TX2) | arm64x8 | 4x Cortex @ 2 GHz | Vol. 48 TC
Client (NX) | arm64x8 | 4x Cortex @ 2 GHz | Pas. 256 CC

###  Training & Implementation Details

We optimize our compression models initially on the 1.28 million ImageNet
(Russakovsky et al., 2015) training samples for 15 epochs, as described in
section 3.3 and section 3.4, with some slight practical modifications for
stable training. Importantly, FrankenSplit’s HD training method is different
from the baseline ES. The former exclusively uses the MSE between the
synthesis transform and the pruned backbone head, while the latter
additionally includes the MSE between the remaining tail stages. We found
including the MSE between the deeper layers leads to unstable and considerably
prolonged training.

We use Adam optimization (Kingma and Ba, 2014) with a batch size of $16$ and
start with an initial learning rate of $1\cdot 10^{-3}$, then gradually lower
it to $1\times 10^{-6}$ with an exponential scheduler.

The hyperparameter $\beta$ controls the rate term during training, i.e.,
higher $\beta$ trades lower bitrates with higher distortion. We aim to
minimize bitrate without sacrificing predictive strength. Hence, we first seek
the lowest $\beta$ resulting in lossless prediction. For the lossless
configuration, we ensure that it consistently results in lossless prediction
by training models five times and reporting the worst result.

To implement our method, we use PyTorch (Pytorch, 2018), CompressAI (Bégaint
et al., 2020) for entropy estimation and entropy coding, and pre-trained
backbones from PyTorch Image Models (Wightman, 2019). All baseline
implementations and weights were either taken from CompressAI or the official
repository of a baseline. To compute the saliency maps, we use a modified
XGradCAM method from the library in (Gildenblat and contributors, 2021) and
include necessary patches in our repository. Lastly, to ensure
reproducibility, we use torchdistill (Matsubara, 2021).

###  Rate-Distortion Performance

We measure the predictive loss by the drop in top-1 accuracy using the
ImageNet validation set for the standard classification task with 1000
categories. Analogously, we measure filesizes of the entropy-coded binaries to
calculate the average bpp. Figure 6 shows rate-distortion curves with the
Swin-B backbone. The architecture of FrankenSplit-FP (FS-FP) and FrankenSplit-
SGFP (FS-SGFP) are identical. We train both models with the loss functions
derived in Section 3.3. The difference is that FS-SGFP is saliency guided,
i.e., FS-FP represents the suboptimal HD training method and is an ablation to
our proposed solution.

####  4.3.1. Effect of Saliency Guidance

Although FS-FP performs better than almost all other models, it is trained
with the suboptimal objective discussed in Section 3.2. Empirically, this is
demonstrated by FS-SGFP outperforming FS-FP on the r-d curve. By simply
guiding the distortion loss with saliency maps, we achieve a 25% lower bitrate
at no additional cots.

Figure 6. Rate-distortion curve for ImageNet

####  4.3.2. Comparison to the ES

Even without saliency guidance, FS-FP consistently outperforms the ES by a
large margin. Specifically, FS-FP and FS-SGFP achieve 32% and 63% lower
bitrates for the lossless configuration.

For a direct comparison to the ES, we ensured that our bottleneck injection
incurs comparable overhead. Moreover, the ES has an advantage due to fine-
tuning tail parameters in an auxiliary training stage. Therefore, we attribute
the performance gain to the more sophisticated architectural design decisions
described in Section 3.5.

####  4.3.3. Comparison to Image Codecs

For almost all lossy codec baselines, Figure 6 illustrates that FS-(SG)FP has
a significantly better r-d performance. Comparing FS-FP to Ballé-FP
demonstrates the r-d gain of task-specific compression over general-purpose
image compression. Although the encoder of FrankenSplit has 25x fewer
parameters, both codecs use an FP entropy model with encoders consisting of
convolutional layers. Yet, the average file size of FS-FP with a predictive
loss of around 5% is 7x less than the average file size of Ballé-FP with
comparable predictive loss.

FrankenSplit also beats modern general-purpose LIC without including any of
their complex or heavy-weight mechanisms. The only baseline, FrankenSplit does
not convincingly outperform is Chen-BJHAP. Nevertheless, incurred overhead is
as vital for real-time inference, which we will evaluate Section 4.4.

####  4.3.4. Naivety of Direct Optimization for SVBI

In Section 3.3 we mentioned that direct optimization does not work for SVBI as
it does for DVBI, where the bottleneck is at the penultimate layer.
Interestingly, direct optimization performs incomparably worse than HD,
despite the latter’s inherent suboptimality. Contrasting the r-d performance
on the simple CIFAR-10 (Krizhevsky et al., 2009) dataset summarized in the
left with the r-d performance on the ImageNet dataset on the right in Figure 7
provides empirical evidence. Other than training direct optimization methods
for more epochs to account for slower convergence, all models are identical
and optimized with the setup described in Section 4.2. On the trivial task,
SVBI-CE and SVBI-KD yield moderate performance gain over JPEG. Contrarily, the
same training method entirely falters on the ImageNet dataset and does not get
close to lossless prediction even at high bitrates. When the joint
distribution $P_{X,Y}$ arises from a trivial dataset, $I(Y;\tilde{Y})$ has
just enough information to propagate down for the compression model to come
close to a sufficient statistic of $Z$ respective $Y$. However, when $P_{X,Y}$
arises from a non-trivial dataset, where the information content of
$I(Y;\tilde{Y})$ is not enough to approximate a sufficient statistic, it
performs significantly worse.

Figure 7. Contrasting the r-d performance

####  4.3.5. Generalization to arbitrary backbones

In the previous, we intentionally demonstrated the r-d performance of
FrankenSplit with a Swin backbone since it backs up our claim from Section 3.5
that the synthesis transform can map the CNN encoder output to the input of a
hierarchical vision transformer. Nevertheless, we initially set out to plot
the r-d curves for all the models summarized in Table 1 but found that the
choice of backbone has a negligible impact on the rate-distortion performance.
Additionally, due to the relative distortion measure, different backbones will
not lead to significantly different r-d curves for the baseline codecs. The
insignificance of teacher choice for r-d performance is consistent with all
our claims and findings. With the objective in (7), we learn the
representation of a shallow layer, which generalizes well since such
representations typically have high mutual information with the original
input. Moreover, the difference between the smaller and larger model
variations from the same architectural family is primarily due to the deepest
stage or the embed size. Likewise, plotting results for intra-architecture
module re-attachment is redundant, as it will result in nearly identical r-d
curves after finetuning the decoder for a few epochs. What remains is to
demonstrate the necessity of our heuristic regarding inductive bias.

Table 3. Effect of blueprints on a backbone

Blueprint-Backbone | File Size (kB) | Predictive Loss (%)
---|---|---
Swin-Swin | 8.70 | 0.00
5.08 | 0.40
3.19 | 0.77
ConvNext-Swin | 19.05 | 2.49
15.07 | 3.00
10.05 | 3.35
ResNet-Swin | 22.54 | 0.82
18.19 | 0.99
8.27 | 1.34

As detailed in Section 3.5, the decoder consists of interchangeable
restoration and transformation blocks. Irrespective of the backbone,
restoration blocks are residual sub-pixel or transpose convolutions.
Contrarily, the transformation block depends on the target backbone, and we
argued that the r-d performance improves when the transformation block induces
the same bias as the target backbone. Specifically, we create a synthesis
transform blueprint for each of the three architectural families (Swin,
ResNet, and ConvNeXt) that results in comparable rate-distortion performance
from Figure 6 for their intended variations. For example, for any ConvNeXt
backbone variation, we train the bottleneck with the ConvNeXt synthesis
blueprint. However, as summarized in Table 3, once we attempt to use the
ResNet or ConvNeXt restoration blocks, the rate-distortion performance for the
Swin-B is significantly worse. Most notably, the lossless configuration for
Swin-Swin has less than half the average file size of ConvNeXt-Swin with 2.49%
predictive loss.

####  4.3.6. Generalization to multiple Downstream Tasks

Dubois et al. (Dubois et al., 2021) generalize a bottleneck to multiple tasks
by seeking a maximal invariant through extensive augmentation and contrastive
SSL (Oord et al., 2018). Here we argue that our SVBI naturally generalizes
despite its simpler training method involving no augmentation or contrastive
SSL for two reasons. First, the head distillation method optimizes our encoder
to learn high-level representations, i.e., the mutual information between the
bottleneck output and the original input is still high. Additionally,
hierarchical vision models are primarily feature extractors. Since the
bottleneck learns to approximate the representation from a challenging
distribution, it should be possible to embed the same compression module on
backbones trained for other tasks. We provide empirical evidence by finetuning
a predictor prepared on ImageNet for different classification tasks.

Table 4. Execution times with various backbones

Backbone |  | Overhead Prams
---
(%)
| Inf. Server+NX
---
(m/s)
| Inf. Server+TX2
---
(m/s)
Swin-T | 2.51 | 7.83 | 9.75
Swin-S | 1.41 | 11.99 | 13.91
Swin-B | 1.00 | 16.12 | 18.04
ConvNeXt-T | 3.46 | 6.83 | 8.75
ConvNeXt-S | 1.97 | 8.50 | 10.41
ConvNeXt-B | 0.90 | 9.70 | 11.62
ResNet-50 | 3.50 | 13.16 | 10.05
ResNet-101 | 2.01 | 8.13 | 15.08
ResNet-152 | 1.48 | 18.86 | 20.78

For FrankenSplit, we applied none or only rudimentary augmentation to evaluate
how our method handles a type of noise it did not encounter during training.
Hence, we include the Food-101 (Bossard et al., 2014) dataset since it
contains noise in high pixel intensities. Additionally, we include CIFAR-100
(Krizhevsky et al., 2009). Lastly, to contrast more challenging tasks, we
include Flower-102 (Nilsback and Zisserman, 2008) datasets.

We freeze the entire compression model trained on the ImageNet dataset and
finetune the tails separately for five epochs with no augmentation, a learning
rate of $5\cdot 10^{-5}$ using Adam optimization. The teacher backbones
achieve an $87.73\%$, $88.01\%$, and $89.00\%$ top-1 accuracy, respectively.
Figure 8 summarizes the r-d performance for each task. Note that the focus of
this study is not transfer learning but to show how the bottleneck retains
information necessary to accommodate multiple predictors.

Figure 8. Rate-distortion curve for multiple downstream tasks

Our method still demonstrates clear r-d performance gains over the baselines.
More importantly, notice how FS-SGFP outperforms FS-FP on the r-d curve for
the Food-101 dataset, with a comparable margin to the ImageNet dataset.
Contrarily, on the Flower-102 datasets, there is noticeably less performance
difference between them. Presumably, on trivial datasets, the suboptimality of
HD is less significant. Considering how easier tasks require less model
capacity, the diminishing efficacy of our saliency-guided training method is
consistent with our claims and derivations in Section 3. The information of
the shallow layer may suffice, i.e., the less necessary the activations of the
deeper layers are, the better a minimal sufficient statistic respective $H$
approximates a minimal sufficient statistic respective $Y$.

###  Prediction Latency and Overhead

We exclude entropy coding from our measurement, since not all baselines use
the same entropy coder. For brevity, the results implicitly assume the Swin-B
backbone for the remainder of this section. Inference times with other
backbones for FrankenSplit can be derived from Table 4.

Table 5. Inference pipeline execution times

Model |  | Prams
---
Enc./Dec.
| Enc. NX/TX2
---
(ms)
| Dec.
---
(ms)
| Full NX/TX2
---
(ms)
FrankenSplit |  | 0.14M/
---
2.06M
| 2.92/
---
4.87
2.00 |  | 16.34/
---
18.29
Ballé-FP |  | 3.51M/
---
351M
| 27.27/
---
48.93
1.30 |  | 41.71/
---
63.37
Minnen-MSHP |  | 14.04M/
---
11.65M
| 29.51/
---
52.39
1.52 |  | 44.17/
---
67.05
Minnen-JHAP |  | 21.99M/
---
19.59M
| 4128.17/
---
4789.89
275.18 |  | 4416.7/
---
5078.2
Cheng-JHAP |  | 16.35M/
---
22.27M
| 2167.34/
---
4153.95
277.26 |  | 2457.7/
---
4444.3
Lu-JHAP |  | 5.28M/
---
4.37M
| 2090.88/
---
5011.56
352.85 |  | 2456.8/
---
5377.8
Chen-BJHAP |  | 36.73M/
---
28.08M
| 3111.01/
---
5837.38
43.16 |  | 3167.3/
---
5893.6

Analogously, the inference times of applying LIC models for different
unmodified backbones can be derived using Table 1. Notably, the relative
overhead decreases the larger the tail is, which is favorable since we target
inference from more accurate predictors.

####  4.4.1. Computational Overhead

We first disregard network conditions to get an overview of the computational
overhead of applying compression models. Table 5 summarizes the execution
times of the prediction pipeline’s components. Enc. NX/TX2 refers to the
encoding time on the respective client device. Analogously, dec. refers to the
decoding time at the server. Lastly, Full NX/TX2 is the total execution time
of encoding at the respective client plus decoding and the prediction task at
the server. Lu-JHAP demonstrates how LIC models without a sequential context
component are noticeably faster, but are still 9.3x-9.6x slower than
FrankenSplit despite a considerably worse r-d performance. Comparing Ballé-FP
to Minnen-MSHP reveals that including side information only incurs minimal
overhead, even on the constrained client. On the server, FrankenSplit’s is
slightly slower than some baselines due to the attention mechanism of the
decoder blueprint for the swin backbone. However, unlike all other baselines,
the computational load of FrankenSplit is near evenly distributed between the
client and the server, i.e., FrankenSplit’s design heuristic successfully
considers resource asymmetry. The significance of considering resource
asymmetry is emphasized by how the partially parallelized context model of
Chen-BHJAP leads to faster decoding on the server. Nevertheless, it is slower
than other JHAP baselines due to the overhead of the increased encoder size
outweighing the performance gain of the blocked context model on constrained
hardware.

####  4.4.2. Competing against Offloading

Unlike SC methods, we do not reduce the latency on the server, i.e.,
performance gains of FrankenSplit solely stem from reduced transfer times. The
average compressed filesize gives the transfer size from the ImageNet
validation set.

Table 6. Total Latency with various wireless standards

| Standard/
---
Data Rate
Mbps
Model |  | Transfer
---
(ms)
| Total [TX2]
---
(ms)
| Total [NX]
---
(ms)
BLE/ 0.27 | FS-SGFP (0.23) | 142.59 | 160.48 | 158.53
FS-SGFP (LL) | 209.89 | 227.78 | 225.83
PNG | 2532.58 | 2545.72
4G/ 12.0 | FS-SGFP (0.23) | 3.21 | 21.09 | 19.15
FS-SGFP (LL) | 4.72 | 22.61 | 20.66
PNG | 56.98 | 70.13
Wi-Fi/ 54.0 | FS-SGFP (0.23) | 0.71 | 18.6 | 16.65
FS-SGFP (LL) | 1.05 | 18.93 | 16.99
PNG | 12.66 | 25.81
5G/ 66.9 | FS-SGFP (0.23) | 0.58 | 18.46 | 16.51
FS-SGFP (LL) | 0.85 | 18.73 | 16.78
PNG | 10.22 | 23.36

Using the transfer size, we evaluate transfer time on a broad range of
standards. Since we did not include the execution time of entropy coding for
learned methods, the encoding and decoding time for the handcrafted codecs is
set to 0. The setting favors the baselines because both rely on entropy coding
and sequential CPU-bound transforms. Table 6 summarizes how our method
performs in various standards. Due to space constraints, we cannot include
comparisons to LIC models. Still, with Table 5 and the r-d results from the
previous subsection, we can infer that the LIC baselines have considerably
higher latency than FrankenSplit. Regardless, our method should always be
preferable since the reduced transfer size is significantly smaller. In highly
constrained networks, such as BLE, FrankenSplit is up to 16x faster than PNG.
For stronger connections, such as 4G LTE, it is still up to 3.3x faster.
Generally, the less constrained the bandwidth, the more favorable it is to
offload. Yet, FrankenSplit is significantly better than offloading even for
5G.

##  5\. Conclusion

In this work, we conceived a novel lightweight compression framework to
facilitate critical MEC applications relying on large DNNs. We demonstrated
that a minimalistic implementation of our design heuristic is sufficient to
outperform numerous baselines. However, our method still has several
limitations. Notably, it assumes that the client has an onboard accelerator.
Moreover, the simplistic Factorized Prior entropy model is not input adaptive,
i.e., it does not adequately discriminate between varying inputs. Although
adding side information with hypernetworks taken from LIC trivially improves
r-d performance, our results show that it is naive to repurpose image
compression methods directly. Hence, conceiving an efficient way to include
task-dependent side information is a promising research directive.

## References

  * (1)
  * Alemi et al. (2016) Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2016.  Deep Variational Information Bottleneck.  _CoRR_ abs/1612.00410 (2016).  arXiv:1612.00410 http://arxiv.org/abs/1612.00410
  * Almeida et al. (2022) Mario Almeida, Stefanos Laskaridis, Stylianos I Venieris, Ilias Leontiadis, and Nicholas D Lane. 2022\.  Dyno: Dynamic onloading of deep neural networks from cloud to device.  _ACM Transactions on Embedded Computing Systems_ 21, 6 (2022), 1–24.
  * Arinta and Andi W.R. (2019) Rania Rizki Arinta and Emanuel Andi W.R. 2019.  Natural Disaster Application on Big Data and Machine Learning: A Review. In _2019 4th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE)_. 249–254.  https://doi.org/10.1109/ICITISEE48480.2019.9003984
  * Ballé et al. (2020) Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, and George Toderici. 2020.  Nonlinear Transform Coding.  _CoRR_ abs/2007.03034 (2020).  arXiv:2007.03034 https://arxiv.org/abs/2007.03034
  * Ballé et al. (2017) Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. 2017.  End-to-end Optimized Image Compression. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=rJxdQ3jeg
  * Ballé et al. (2018) Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018.  Variational image compression with a scale hyperprior.  https://doi.org/10.48550/ARXIV.1802.01436
  * Bégaint et al. (2020) Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. 2020.  CompressAI: a PyTorch library and evaluation platform for end-to-end compression research.  _arXiv preprint arXiv:2011.03029_ (2020).
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014\.  Food-101–mining discriminative components with random forests. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_. Springer, 446–461.
  * Chen et al. (2021) Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang. 2021.  End-to-End Learnt Image Compression via Non-Local Attention Optimization and Improved Context Modeling.  _IEEE Transactions on Image Processing_ 30 (2021), 3179–3191.  https://doi.org/10.1109/TIP.2021.3058615
  * Cheng et al. (2020) Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. 2020\.  Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules.  https://doi.org/10.48550/ARXIV.2001.01568
  * Cisco (2020) U Cisco. 2020.  Cisco annual internet report (2018–2023) white paper.  _Cisco: San Jose, CA, USA_ 10, 1 (2020), 1–35.
  * Dubois et al. (2021) Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison. 2021.  Lossy compression for lossless prediction.  _Advances in Neural Information Processing Systems_ 34 (2021), 14014–14028.
  * Eshratifar et al. (2019) Amir Erfan Eshratifar, Amirhossein Esmaili, and Massoud Pedram. 2019.  BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services. In _2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)_. 1–6.  https://doi.org/10.1109/ISLPED.2019.8824955
  * Feng et al. (2022) Chuan Feng, Pengchao Han, Xu Zhang, Bowen Yang, Yejun Liu, and Lei Guo. 2022\.  Computation offloading in mobile edge computing networks: A survey.  _Journal of Network and Computer Applications_ (2022), 103366.
  * Gildenblat and contributors (2021) Jacob Gildenblat and contributors. 2021.  PyTorch library for CAM methods.  https://github.com/jacobgil/pytorch-grad-cam.
  * Goyal (2001) V.K. Goyal. 2001\.  Theoretical foundations of transform coding.  _IEEE Signal Processing Magazine_ 18, 5 (2001), 9–21.  https://doi.org/10.1109/79.952802
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Deep Residual Learning for Image Recognition.  _CoRR_ abs/1512.03385 (2015).  arXiv:1512.03385 http://arxiv.org/abs/1512.03385
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the Knowledge in a Neural Network.  https://doi.org/10.48550/ARXIV.1503.02531
  * Howard et al. (2019) Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019\.  Searching for mobilenetv3. In _Proceedings of the IEEE/CVF international conference on computer vision_. 1314–1324.
  * Kang et al. (2017) Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017.  Neurosurgeon: Collaborative intelligence between the cloud and mobile edge.  _ACM SIGARCH Computer Architecture News_ 45, 1 (2017), 615–629.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Laskaridis et al. (2020) Stefanos Laskaridis, Stylianos I Venieris, Mario Almeida, Ilias Leontiadis, and Nicholas D Lane. 2020\.  SPINN: synergistic progressive inference of neural networks over device and cloud. In _Proceedings of the 26th annual international conference on mobile computing and networking_. 1–15.
  * Lee et al. (2021) Joo Chan Lee, Yongwoo Kim, SungTae Moon, and Jong Hwan Ko. 2021\.  A splittable dnn-based object detector for edge-cloud collaborative real-time video inference. In _2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 1–8.
  * Li et al. (2018) Hongshan Li, Chenghao Hu, Jingyan Jiang, Zhi Wang, Yonggang Wen, and Wenwu Zhu. 2018\.  Jalad: Joint accuracy-and latency-aware deep structure decoupling for edge-cloud execution. In _2018 IEEE 24th international conference on parallel and distributed systems (ICPADS)_. IEEE, 671–678.
  * Liang et al. (2021) Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021\.  Swinir: Image restoration using swin transformer. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1833–1844.
  * Liu et al. (2022) Hongzhou Liu, Wenli Zheng, Li Li, and Minyi Guo. 2022\.  LoADPart: Load-Aware Dynamic Partition of Deep Neural Networks for Edge Offloading. In _2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)_. 481–491.  https://doi.org/10.1109/ICDCS54860.2022.00053
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10012–10022.
  * Lu et al. (2022) Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. 2022.  Transformer-based Image Compression. In _2022 Data Compression Conference (DCC)_. 469–469.  https://doi.org/10.1109/DCC52660.2022.00080
  * Luo et al. (2020) Xiyang Luo, Hossein Talebi, Feng Yang, Michael Elad, and Peyman Milanfar. 2020.  The rate-distortion-accuracy tradeoff: Jpeg case study.  _arXiv preprint arXiv:2008.00605_ (2020).
  * Matsubara (2021) Yoshitomo Matsubara. 2021\.  torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation. In _International Workshop on Reproducible Research in Pattern Recognition_. Springer, 24–44.
  * Matsubara et al. (2019) Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, and Sameer Singh. 2019\.  Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems. In _Proceedings of the 2019 Workshop on Hot Topics in Video Analytics and Intelligent Edges_ (Los Cabos, Mexico) _(HotEdgeVideo’19)_. Association for Computing Machinery, New York, NY, USA, 21–26.  https://doi.org/10.1145/3349614.3356022
  * Matsubara et al. (2022a) Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. 2022a.  Split computing and early exiting for deep learning applications: Survey and research challenges.  _Comput. Surveys_ 55, 5 (2022), 1–30.
  * Matsubara et al. (2022b) Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, and Stephan Mandt. 2022b.  SC2: Supervised Compression for Split Computing.  https://doi.org/10.48550/ARXIV.2203.08875
  * Matsubara et al. (2022c) Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, and Stephan Mandt. 2022c.  Supervised compression for resource-constrained edge computing systems. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2685–2695.
  * Minnen et al. (2018) David Minnen, Johannes Ballé, and George Toderici. 2018\.  Joint Autoregressive and Hierarchical Priors for Learned Image Compression.  https://doi.org/10.48550/ARXIV.1809.02736
  * Nilsback and Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman. 2008.  Automated Flower Classification over a Large Number of Classes. In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_. 722–729.  https://doi.org/10.1109/ICVGIP.2008.47
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Otter et al. (2020) Daniel W Otter, Julian R Medina, and Jugal K Kalita. 2020\.  A survey of the usages of deep learning for natural language processing.  _IEEE transactions on neural networks and learning systems_ 32, 2 (2020), 604–624.
  * Pytorch (2018) Automatic Differentiation In Pytorch. 2018\.  Pytorch.
  * Rausch et al. (2021) Thomas Rausch, Waldemar Hummer, Christian Stippel, Silvio Vasiljevic, Carmine Elvezio, Schahram Dustdar, and Katharina Krösl. 2021\.  Towards a Platform for Smart City-Scale Cognitive Assistance Applications. In _2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_. 330–335.  https://doi.org/10.1109/VRW52623.2021.00066
  * Romero et al. (2021) Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021.  INFaaS: Automated Model-less Inference Serving. In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_. USENIX Association, 397–411.  https://www.usenix.org/conference/atc21/presentation/romero
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115, 3 (2015), 211–252.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4510–4520.
  * Sbai et al. (2021) Marion Sbai, Muhamad Risqi U. Saputra, Niki Trigoni, and Andrew Markham. 2021.  Cut, Distil and Encode (CDE): Split Cloud-Edge Deep Inference. In _2021 18th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)_. 1–9.  https://doi.org/10.1109/SECON52354.2021.9491600
  * Selvaraju et al. (2019) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  _International Journal of Computer Vision_ 128, 2 (oct 2019), 336–359.  https://doi.org/10.1007/s11263-019-01228-7
  * Shannon (1959) Claude E Shannon. 1959\.  Coding theorems for a discrete source with a fidelity criterion. In _IRE National Convention Record, 1959_ , Vol. 4. 142–163.
  * Shao and Zhang (2020) Jiawei Shao and Jun Zhang. 2020.  BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems. In _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. 1–6.  https://doi.org/10.1109/ICCWorkshops49005.2020.9145068
  * Singh et al. (2020) Saurabh Singh, Sami Abu-El-Haija, Nick Johnston, Johannes Ballé, Abhinav Shrivastava, and George Toderici. 2020.  End-to-end Learning of Compressible Features.  _CoRR_ abs/2007.11797 (2020).  arXiv:2007.11797 https://arxiv.org/abs/2007.11797
  * Tishby et al. (2000) Naftali Tishby, Fernando C. Pereira, and William Bialek. 2000.  The information bottleneck method.  https://doi.org/10.48550/ARXIV.PHYSICS/0004057
  * Tishby and Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky. 2015.  Deep learning and the information bottleneck principle. In _2015 ieee information theory workshop (itw)_. IEEE, 1–5.
  * Voulodimos et al. (2018) Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, Eftychios Protopapadakis, et al. 2018\.  Deep learning for computer vision: A brief review.  _Computational intelligence and neuroscience_ 2018 (2018).
  * Wightman (2019) Ross Wightman. 2019\.  PyTorch Image Models.  https://github.com/rwightman/pytorch-image-models.  https://doi.org/10.5281/zenodo.4414861
  * Xin et al. (2022) Qin Xin, Mamoun Alazab, Vicente García Díaz, Carlos Enrique Montenegro-Marin, and Rubén González Crespo. 2022\.  A deep learning architecture for power management in smart cities.  _Energy Reports_ 8 (2022), 1568–1577.
  * Yang et al. (2022) Yibo Yang, Stephan Mandt, and Lucas Theis. 2022.  An Introduction to Neural Data Compression.  https://doi.org/10.48550/ARXIV.2202.06533
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016.  Learning deep features for discriminative localization. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2921–2929.