#  Enhancing Split Computing and Early Exit Applications through Predefined
Sparsity ††thanks:  This study was carried out within the PNRR research
activities of the consortium iNEST (Interconnected North-Est Innovation
Ecosystem) funded by the European Union Next-GenerationEU (Piano Nazionale di
Ripresa e Resilienza (PNRR) – Missione 4 Componente 2, Investimento 1.5 – D.D.
1058 23/06/2022, ECS_00000043), and by the European Union’s Horizon Europe
research and innovation programme under the Marie Sklodowska-Curie grant
agreement No. 101109243. This manuscript reflects only the Authors’ views and
opinions. Neither the European Union nor the European Commission can be
considered responsible for them. This work was also partially supported by the
US NSF grant 2038960.

Luigi Capogrosso1, Enrico Fraccaroli12, Giulio Petrozziello3, Francesco
Setti1,
Samarjit Chakraborty2, Franco Fummi1, Marco Cristani1  1 _Dept. of Engineering
for Innovation Medicine_ , University of Verona, Italy<EMAIL_ADDRESS>3 _Dept. of Computer Science_ , University of Verona, Italy,
<EMAIL_ADDRESS>2 _Dept. of Computer Science_ , University of
North Carolina at Chapel Hill, USA<EMAIL_ADDRESS>
###### Abstract

In the past decade, Deep Neural Networks (DNNs) achieved state-of-the-art
performance in a broad range of problems, spanning from object classification
and action recognition to smart building and healthcare. The flexibility that
makes DNNs such a pervasive technology comes at a price: the computational
requirements preclude their deployment on most of the resource-constrained
edge devices available today to solve real-time and real-world tasks. This
paper introduces a novel approach to address this challenge by combining the
concept of predefined sparsity with Split Computing (SC) and Early Exit (EE).
In particular, SC aims at splitting a DNN with a part of it deployed on an
edge device and the rest on a remote server. Instead, EE allows the system to
stop using the remote server and rely solely on the edge device’s computation
if the answer is already good enough. Specifically, how to apply such a
predefined sparsity to a SC and EE paradigm has never been studied. This paper
studies this problem and shows how predefined sparsity significantly reduces
the computational, storage, and energy burdens during the training and
inference phases, regardless of the hardware platform. This makes it a
valuable approach for enhancing the performance of SC and EE applications.
Experimental results showcase reductions exceeding $4\times{}$ in storage and
computational complexity without compromising performance. The source code is
available at https://github.com/intelligolabs/sparsity_sc_ee.

###### Index Terms:

Split Computing, Early Exit, Deep Neural Networks, Predefined Sparsity, Edge
Devices.

##  I Introduction

Remarkable improvements in areas like computer vision, speech recognition, and
autonomous systems are significantly supported thanks to the rise of Deep
Neural Networks (DNNs) [1, 2]. At the same time, over the last decade, DNNs
have grown significantly in size and complexity; some now have millions or
even billions of trainable parameters [3]. However, these powerful models have
a significant drawback: they demand substantial computational resources and
storage, making deploying them directly on edge devices impractical.

Nowadays, the most commonly used approach is to train huge DNNs in server
computing settings using high-performance hardware accelerators like Graphical
Processing Units (GPUs) [4], and Tensor Processing Units (TPUs) [5]. Then,
once trained, the DNN model is used for inference, a less computationally
expensive activity that, in the case of real-time and real-world problems, is
often performed on edge devices, and more in general on embedded systems [6].

#### Motivations for this paper

As a result, a significant amount of research effort has been directed toward
improving embedded technologies. This focus has enabled the development of
real-time solutions for a wide range of real-world complex applications. In
this regard, hardware-specific (e.g., edge TPUs) and Micro-Controller Unit
(MCU)-based embedded systems have earned a lot of attention, primarily due to
their low power requirements and high performance, and secondarily for their
maintainability, adaptability, and reliability [7].

DNNState-of-the-Art approachPruning andQuantizationEdgeDevice _Training_ and
_deploy_ ApplysparsityApplySplitInsertEarly ExitEdgeDeviceRemoteDevice
_Training_ and _deploy_

Figure 1: Difference between our approach of predefined sparsity applied to SC
and EE, against the state-of-the-art pruning and quantization.

In particular, the importance of model reduction techniques for accelerating
DNNs is widely acknowledged nowadays, aiming to optimize their performance
[8]. These approaches focus on reducing the number of trainable parameters
within a DNN, resulting in substantial advantages across various aspects:
computational resources needed, storage requirements, and energy consumption.
Since efficiency is paramount, improvements in this area are crucial if we aim
to deploy DNNs on edge devices for solving real-world applications [9].

At the same time, current state-of-the-art approaches for efficient edge
computing rely on advanced Machine Learning (ML) paradigms, such as SC and EE
[10, 11, 12, 13, 14]. In particular, SC, where a DNN is intelligently split
with a part of it deployed on an edge device, and the rest on a remote server,
and EE, where the model is built with multiple “exits” across the layers and
each exit can produce the model output, represents the state-of-the-art
framework for structuring distributed deep learning applications [15]. These
approaches allow DNNs to be leveraged for latency-sensitive applications in
scenarios where deploying the entire DNN remotely is impractical due to
limited local computation bandwidth, and also locally since DNN would require
higher memory requirements than those available on the edge device. They
combine local and remote computing advantages, leading to lower latency and,
more importantly, drastically reducing the required transmission bandwidth.

#### Innovations in this paper

Model $\mathcal{M}(\cdot{})$ that needs tosolve real-world taskComputationally
intensiveHigh storage requirementsHigh energy consumption Model
$\mathcal{M}(\cdot{})$ after applyingthe predefined sparsityNot so
computationally intensiveLess storage requirementsLess energy consumption
Model $\mathcal{M}(\cdot{})$ after applying SplitComputing and Early
ExitState-of-the-art real-world taskNot so computationally intensiveLess
storage requirementsLess energy consumption _Network_ $\mathcal{M}(\cdot{})$
headon EdgeEarlyexit$\mathcal{M}(\cdot{})$ tailon Remote Figure 2: Starting
from a DNN $\mathcal{M}(\cdot{})$, we first apply the predefined sparsity, and
then we train the network. After the training stage, we split the network
following the SC and EE paradigm. As a result, the final architecture is not
so computationally intensive, doesn’t require huge storage spaces, and has
less energy consumption, all without compromising the overall performance.

Figure 1 exemplifies the proposal of this paper, where we explore the
application of predefined sparsity as a model reduction method within the
context of SC and EE. We aim to showcase how, in the context of SC and EE,
predefined sparsity can significantly reduce computational demands, storage
requirements, and energy consumption compared to state-of-the-art approaches.
Furthermore, regardless of the underlying implementation platform, our
approach yields advantages for both the training and inference stages.

In particular, our pipeline is exemplified by the flow presented in Figure 2.
Our strategy for producing sparsity involves defining a preset set of sparse
neuron connections before the training process, i.e., we eliminate a specified
set of connections inside the neural network, and we keep this configuration
constant throughout both the training and inference phases. Specifically, we
apply this strategy within an SC and EE scenario, illustrating how predefined
sparsity can enhance performance in applications operating within this
context.

As a killer application where this proposal can bring real benefits, consider
a common smart manufacturing setting, such as a real-time quality control
system on a production line. The edge devices need to assess whether a product
has a defect on time and catch all the defective ones. It can base its
decision on the inference provided by the early exit strategy, which is fairly
accurate but not as accurate as the inference produced by the full DNN. As
such, it will move the defective product to a buffer on the side of the line
while the remote device runs the remaining layers of the DNN. Once the remote
device completes the inference, a worker or a conveyor belt system can
reintroduce the falsely marked non-defective products inside the production
line.

In summary, the main contributions of this paper are:

  * •

A strategy for implementing predefined sparsity, where a predefined set of
sparse neuron connections is established before training and remains constant
throughout training and inference.

  * •

Our approach involves the removal of specific connections within the neural
network, reducing computational and storage complexity during inference and
throughout the training process.

  * •

We apply this methodology in an SC and EE scenario, demonstrating its
potential to further enhance application performance within this context.

The rest of the paper is organized as follows. Section II presents background
information. Our proposal’s details are outlined in Section III, followed by
experiments in Section IV. Finally, conclusions are drawn in Section V.

##  II Related Work

This section provides an overview of i) DNNs sparsity methods, and ii)
distributed deep learning applications with a specific focus on SC and EE.

###  II-A Deep Neural Networks (DNNs) sparsity

While numerous methods concerning sparse DNNs can be found in the literature
[16, 17, 18], most of these do not reduce the computation and storage
complexity associated with training, but only with inference.

An example is the dropout. This technique is used to prevent overfitting. It
randomly drops some neurons out (i.e., ignoring) during each training
iteration. This helps the network learn more robust and generalized features,
as it can’t rely too much on any one neuron [19]. Due to lack of space, and
since it is only a related topic, refer to [20] for more details.

There are other approaches, such as pruning and trimming methods, that process
the trained DNN to generate a sparse DNN for inference. For example, in [21],
the authors observe that a large fraction of the computations performed by
DNNs are intrinsically ineffectual as they involve a multiplication where one
of the inputs is zero. This observation motivates “Cnvlutin”, a value-based
approach that eliminates most of the ineffectual operations. Furthermore, in
[22], the authors present an algorithm that prunes (specifies) a trained
network layer-wise, removing connections at each layer by solving a convex
optimization program. This program seeks a sparse set of weights at each
layer, keeping the layer inputs and outputs consistent with the originally
trained model.

As mentioned in Section I, other methods aim to reduce the complexity of
performing inference on a trained DNN. These methods encompass techniques like
quantization [16] and compression [23]. It’s important to note that all these
methods primarily target reducing complexity in the inference models rather
than significantly simplifying the training process.

One approach that strives to minimize complexity in both training and
inference involves using neural networks with structured weight matrices that
are not necessarily sparse [24].

Lastly, it’s worth mentioning that several authors have recently introduced
the concept of predefined sparse neural networks [25]. This surge in research
activity is motivated by the recognition that specialized hardware is
typically required to run large and complex neural networks effectively.

In this article, we leverage the previously mentioned _predefined sparsity_.
Specifically, our approach offers two key benefits. Firstly, it is hardware
agnostic. This is in contrast to quantization, which relies on specific
hardware capabilities. Secondly, it allows us to streamline our model before
the training begins. This differs from pruning, which tackles complexity
reduction after training. As a result, our model has a smaller footprint on
the GPU right from the start of training.

Our work goes a step further. While default sparsity has been explored
previously, we are the first to investigate its advantages within the context
of SC and EE. This exploration aims to unlock even greater efficiency for
applications that utilize this technique.

###  II-B Distributed deep learning

We focus on architectures operating through a DNN model
$\mathcal{M}(\cdot{})$, whose task is to produce the inference output $y$ from
an input $x$. We can identify three major types of architectures used for
distributed deep learning applications in the literature: Local-only Computing
(LoC), Remote-only Computing (RoC), and SC.

#### Local-only Computing (LoC)

Under this policy, the entire computation is performed on the sensing devices.
Therefore, the edge device entirely executes the function $\mathcal{M}(x)$.
Its advantage lies in offering low latency due to the proximity of the
computing element to the sensor [7]. However, it may not be compatible with
DNN-based architectures that demand robust hardware capabilities. Usually,
simpler DNN models $\bar{\mathcal{M}}(x)$ that use specific architectures
(e.g., depth-wise separable convolutions) are used to build lightweight
networks, such as MobileNetV3 [26].

Besides designing lightweight neural models, in the last few years, great
progress has been made in the area of DNN compression. Compression techniques,
such as network pruning and quantization [27], or knowledge distillation [28]
achieve a more efficient representation of one or more layers of the neural
network, but with a possible quality degradation.

#### Remote-only Computing (RoC)

The input $x$ is transferred through the communication network and then is
processed at the remote system through the function $\mathcal{M}(x)$. This
architecture preserves full accuracy considering the higher power budget of
the remote system, but it leads to high latency and bandwidth consumption due
to the input transfer.

#### Split Computing (SC)

A typical SC scenario is discussed in [10], where the authors show that
neither LoC nor RoC approaches are optimal, and a split configuration is an
ideal solution. The SC paradigm divides the DNN model into a head, executed by
the local sensing device, and a tail, executed by the remote system. It
combines the advantages of both LoC and RoC thanks to the lower latency and,
more importantly, drastically reduces the required transmission bandwidth by
compressing the input to be sent $x$ through the use of an autoencoder [29].
We define the encoder and decoder models as $z_{l}=\mathcal{F}(x)$ and
$\bar{x}=\mathcal{G}(z_{l})$, which are executed at the edge, and remotely,
respectively. The distance $d(x,\bar{x})$ defines the performance of the
encoding-decoding process.

One of the earliest works on SC is the study by Kang  _et al_. [30], in which
the authors show that the initial layers of a DNN are the most suitable
candidates for partitioning, as they optimize both latency and energy
consumption. Additionally, latency reduction is usually achieved through
quantization, as explored in [31], and the utilization of lossy compression
techniques prior to data transmission, as investigated in [32]. In addition to
lossy compression techniques, in [33], the authors also explore lossless
techniques to encode intermediate results without modifying the ML model.
Instead, the concept of employing autoencoders to compress the data further to
be transferred is discussed in various studies, such as [34].

The prevalent methods for identifying potential splitting points have evolved
from architecture-based techniques to more refined neuron-based methods.
Within the domain of architecture-based approaches, in [12], the authors state
that candidate split locations are where the size of the DNN layers decreases:
the rationale is that compressing information by autoencoders, where
compression would still occur due to the shrinking of the architecture,
certainly seems reasonable. In [13] and [14], the authors show that not only
the architecture of the layers but also the saliency of individual layers is a
crucial factor when deciding where to split. A neuron’s saliency is determined
by its gradient in relation to the accurate decision. Consequently, optimal
splitting points should be strategically positioned following layers housing a
concentration of impactful neurons to preserve the information flowing until
then.

At the same time, current state-of-the-art approaches in different ML
applications rely on advanced learning procedures, such as the _Multi-Task
Learning (MTL)_ [35]. In particular, MTL is a paradigm in which multiple
related tasks are jointly learned to improve the generalizability of a model
by using shared knowledge across different aspects of the input. As a result,
in [36], the authors propose, for the first time ever, how to partition multi-
tasking DNN to be deployed within a SC framework. With this design, the
authors can handle multiple tasks concurrently instead of the current focus on
Single-Task Learning (STL) in SC, and through MTL, they increase task
performance, overcoming the challenge of preserving only the performance of
the main task.

#### Early Exit (EE)

This scenario adds an early exiting branch to a standard SC architecture.
Formally, we can define $B_{i},i=1\dots N$ (with $N=L$, and $L$ is the number
of layers of the DNN) as the branch model that takes as input $z_{l}$ and
produces an estimate of the desired output $y$. In practice, the EE
architecture is a modification of an existing neural network, adding one or
more classification branches where, before the computation of all network’s
layers, the confidence of the intermediate result is checked to be enough to
be considered the final result [37].

EE architecture can be exploited in a distributed deep learning application
where the intermediate result can be directly transmitted, as in local
computing, or refined at the remote side, as in SC. In this scenario, the
level of transmission traffic depends on the input, thus varying
stochastically. Therefore, the interdependencies between computation and
communication cannot be analytically modeled, and real experiments are needed
to validate a given implementation.

In this paper, we’re not looking to develop a new method for finding the best
layers in a DNN to use SC or the most efficient way to put EE into action. We
aim to explore and show how the predefined sparsity can improve these existing
frameworks. Thus, instead of comparing our work to SC and EE methods, we see
it as a way to make them even better, investigating how sparsity can add
benefits on top of what SC and EE already offer.

##  III Method

To understand the core concepts of our research, let us delve into the
mathematical background. These concepts were initially presented in [25].

Let us take a $(L+1)$-layer MultiLayer Perceptron (MLP), described
collectively by the following neuronal configuration:

$N_{net}=(N_{0},\dots,N_{i-1},N_{i},\dots,N_{L})\hbox to0.0pt{\;,\hss}$

where $N_{i}$ represents the number of nodes in the $i$-th layer. We use the
convention that layer $i$ is to the right of layer $i-1$. Given $L$ junctions
between layers, with junction $i$ connecting the $N_{i-1}$ nodes of its left
layer ${i-1}$ with the $N_{i}$ nodes of its right layer $i$.

We can define predefined sparsity as simply not having all $N_{i-1}\cdot
N_{i}$ edges (or weights) present in junction $i$. Furthermore, we can define
structured predefined sparsity so that for a given junction $i$, each node in
its left layer has fixed out-degree, i.e., $d^{out}_{i}\in\mathbb{N}$
connections to its right layer, and each node in its right layer has fixed in-
degree, i.e., $d^{in}_{i}\in\mathbb{N}$ connections from its left layer.

In particular, a MLP have $d^{out}_{i}=N_{i}$ and $d^{in}_{i}=N_{i-1}$ with
$N_{i-1}\cdot N_{i}$ edges present in the $i^{th}$ junction. While a sparse
MLP has at least one junction with less than this number of edges. The number
of edges in junction $i$ is given by the formula:

$|W_{i}|=N_{i-1}\cdot d^{out}_{i}=N_{i}\cdot d^{in}_{i}\hbox to0.0pt{\;.\hss}$
(1)

The density of junction $i$ is measured relative to MLP, and is denoted by the
function:

$\rho_{i}=\frac{|W_{i}|}{N_{i-1}\cdot N_{i}}\hbox to0.0pt{\;.\hss}$ (2)

In our structured predefined sparse network, the density of the $i$-th
junction $\rho_{i}$ cannot be arbitrary. By replacing Equation 1 in Equation
2, we can define:

$\rho_{i}=\frac{d^{in}_{i}}{N_{i}}=\frac{N_{i}}{d^{out}_{i}}\hbox
to0.0pt{\;,\hss}$ (3)

where $d^{out}_{i}$ and $d^{in}_{i}$ are natural numbers. The number of
possible $\rho_{i}$ values is the same as the number of ($d^{out}_{i}$,
$d^{in}_{i}$) values satisfying the structured predefined sparsity
constraints:

$d^{out}_{i}=\frac{N_{i}\cdot d^{in}_{i}}{N_{i-1}},\qquad
d^{in}_{i}\leq{}N_{i-1}\;\hbox to0.0pt{\;.\hss}$ (4)

Now, the smallest value of $d^{in}_{i}$ which satisfies the assignment to
$d^{out}_{i}$ in Equation 4, and $d^{out}_{i}\in\mathbb{N}$, is the following:

$\frac{N_{i-1}}{\gcd(N_{i-1},N_{i})}\hbox to0.0pt{\;,\hss}$

and other values are integer multiples. Since $d^{in}_{i}$ is upper bounded by
$N_{i-1}$, the total number of possible ($d^{out}_{i},d^{in}_{i}$) is
$\gcd(N_{i-1},N_{i})$. We can now define the set of possible densities
$\rho_{i}$, as follows:

$\Biggl{\\{}\rho_{i}=\frac{k}{\gcd(N_{i-1},N_{i})},\quad\rho_{i}\in{}(0,1],\quad
k\in{}\mathbb{N}\Biggr{\\}}\;.$ (5)

Specifying $N_{net}$ and the out-degree configuration
$d^{out}_{net}=(d^{out}_{1},\dots,d^{out}_{L})$ determines the density of each
junction and the overall density, defined as:

$\rho_{net}=\frac{\sum_{i=1}^{L}{|W_{i}|}}{\sum_{i=1}^{L}{N_{i-1}\cdot
N_{i}}}\;.$ (6)

It’s worth noting that, for an MLP using structured predefined sparsity, only
the weights corresponding to connected edges are stored in memory and used in
the computation. Specifically, the parameters are updated based on the
gradient of a loss function with respect to the parameters. Let’s denote the
parameters of the network as $\theta{}$ and the loss function as
$\mathcal{L}(\theta{})$. The gradient of the loss function for the parameters
is denoted as $\nabla_{\theta{}}\mathcal{L}$. Then, the parameters are updated
using a gradient descent step:

$\theta_{\text{new}}=\theta_{\text{old}}-\eta\cdot\nabla_{\theta}\mathcal{L}\hbox
to0.0pt{\;,\hss}$ (7)

where $\eta$ is the learning rate, controlling the step size in the direction
opposite to the gradient.

##  IV Experiments

This section describes the experimental trials performed to validate our
proposal, along with their implementation details and results.

$-0.75$$-0.5$$-0.25$$0$$0.25$$0.5$$0.75$$10^{0}$$10^{1}$$10^{2}$$10^{3}$$10^{4}$$10^{5}$Weight
valuesFrequency (a) Weights distribution in layer 1 [800, 180].

$-0.75$$-0.5$$-0.25$$0$$0.25$$0.5$$0.75$$10^{0}$$10^{1}$$10^{2}$$10^{3}$$10^{4}$$10^{5}$Weight
valuesFrequency (b) Weights distribution in layer 2 [180, 180].

$-0.75$$-0.5$$-0.25$$0$$0.25$$0.5$$0.75$$10^{0}$$10^{1}$$10^{2}$$10^{3}$$10^{4}$$10^{5}$Weight
valuesFrequency (c) Weights distribution in layer 3 [800, 10].

Figure 3: Histograms of weights in each junction resulting from training a
deep MLP on the MNIST dataset. The network configuration
$H=[H_{0},\dots,H_{n}]$ used is [800, 180, 180, 10].

#### Models details

We use three types of MLPs in our experiments: shallow, deep, and sparse. We
can characterize them as follows:

  * •

Shallow: This is the simplest type. It contains only one hidden layer besides
the input and output layers and represents the base case. The hidden layer
neurons receive information from the input layer and process it before
transmitting it to the final output layer;

  * •

Deep: Each neuron within a hidden layer is fully connected to all neurons in
the subsequent layer;

  * •

Sparse: The _predefined sparsity_ pattern is applied. This sparsity pattern
essentially removes certain connections between neurons in adjacent layers.
Specifically, the sparse MLP aims to balance the simplicity of the shallow MLP
and the learning capacity of the deep MLP.

Each network configuration is defined by using two lists. The first list
identifies the number of neurons in the hidden layers:

$H=[H_{0},\dots,H_{n}]\hbox to0.0pt{\;,\hss}$

where $n$ is the number of hidden layers, and intuitively, $H_{0}$ and $H_{n}$
represent the number of neurons in the input and output layers, respectively.
The second list identifies the out-degree of each neuron for each layer:

$G=[G_{0},\dots,G_{L-1}]\hbox to0.0pt{\;.\hss}$

Specifically, $G_{0}$ is the out-degree of each input layer node, and the
output layer $G_{L}$ is not reported because it is zero.

Then, the shallow, deep, and sparse MLP has been split. Our research primarily
focuses on using existing sparsity patterns in split computing applications
for these networks. Inspired by the approach presented in [30], we opt to
split the network at the midpoint, ensuring uniformity in our approach. While
more recent studies, such as [12, 13], have proposed advanced techniques for
selecting optimal splitting points, exploring these methods will be part of
future investigations.

Finally, the inserted EEs are composed of linear layers followed by Rectified
Linear Unit (ReLU) activation functions, where the dimension of the last layer
matches the desired output size of the DNN.

Even with the rise of specialized architectures like Convolutional Neural
Networks (CNNs), MLPs remains a valuable tool for researchers. Their
fundamental structure allows for in-depth exploration of core deep learning
concepts without the added complexity of specialized architectures. This focus
on MLPs aligns with the experimental nature of this research, where we aim to
isolate the effects of the proposed technique and gain a deeper understanding
of its fundamental workings. Furthermore, recent advancements in research in
MLP have demonstrated their continued effectiveness in various real-world
applications, highlighting their ongoing relevance in the field [38, 39].

#### Datasets

In this research, we focus on the image classification task. We utilize the
MNIST dataset [40], a well-known collection of handwritten digits. This
contains 60,000 training and 10,000 testing images for the multi-class image
classification task. The images were centered in a $28\times{}28$ image by
computing the center of mass of the pixels and translating the image to
position this point at the center of the $28\times{}28$ field.

MNIST has to be considered as a placeholder for bigger datasets (e.g.,
ImageNet [41]); nonetheless, the focus here is to show the potentialities of
the predefined sparsity applied in an SC and EE and not beating the state-of-
the-art in a specific computer vision challenge.

While this article focuses on a computer vision task, the concepts explored
here remain valid also for broader applications. For example, they can be
effectively used for other tasks, such as time series forecasting.

#### Training details

All the source code is implemented in TensorFlow [42]. We train our models for
50 epochs, with a learning rate of $1\times{}10^{-5}$, using Adam [43] as an
optimizer, on an NVIDIA RTX 3060 Ti.

###  IV-A Why predefined sparsity in SC and EE?

The motivation behind predefined sparsity can be exemplified by examining the
weights histograms of a trained MLP shown in Figure 3. Specifically, the three
histograms show the weight distributions for each layer in a 3-layer MLP model
trained on the MNIST dataset with hidden layers having the following number of
neurons $H=[800,180,180,10]$.

As we can see from Figures 3(a) and 3(b), the first layers of the network have
a significant concentration of weights around zero, suggesting that these
weights might not be crucial for the network’s performance. While Figure 3(c)
highlights how the weights in the last layer assume a broader spectrum of
values.

This finding suggests that the benefits of _predefined sparsity_ can be
especially pronounced in the earlier layers of the network. As a result, in
resource-constrained environments, like those encountered in SC and EE
applications, predefined sparsity offers significant advantages because by
reducing the number of connections, we can decrease the size and complexity of
the network portion deployed on the edge device. This translates to lower
memory requirements and faster processing times during training and inference.

Furthermore, the storage footprint is directly proportional to the number of
edges. Operating at a sparsity level of, for example, 50% results in a two-
fold reduction in complexity. This translates to significant efficiency gains,
making deploying more complex models on devices with limited resources
possible. These findings led us to want to study them in the context of SC and
EE.

###  IV-B Results

Table I: Results regarding the accuracy and the number of parameters with different configurations of head MLPs. Type |  | Neurons
---
per Layer
| Out-degree
---
per Node
| Accuracy
---
(%)
| Number of
---
Parameters
Deep | [800 | 3 3 | 10] | [3 3 10] | 76.58 | 2443
Shallow | [800 | 3 | 10] | [3 10] | 80.89 | 2455
Sparse | [800 | 40 40 | 10] | [2 9 10] | 93.93 | 2450
Deep | [800 | 5 5 | 10] | [5 5 10] | 90.69 | 4095
Shallow | [800 | 5 | 10] | [5 10] | 89.84 | 4065
Sparse | [800 | 40 40 | 10] | [4 10 10] | 95.30 | 4090
Deep | [800 | 7 7 | 10] | [7 7 10] | 92.85 | 5743
Shallow | [800 | 7 | 10] | [7 10] | 92.22 | 5687
Sparse | [800 | 40 40 | 10] | [6 11 10] | 95.89 | 5730
Deep | [800 | 14 14 | 10] | [14 14 10] | 95.35 | 11574
Shallow | [800 | 14 | 10] | [14 10] | 95.44 | 11364
Sparse | [800 | 40 40 | 10] | [13 17 10] | 96.54 | 11570
Deep | [800 | 20 20 | 10] | [20 20 10] | 96.03 | 16650
Shallow | [800 | 21 | 10] | [21 10] | 96.03 | 17041
Sparse | [800 | 40 40 | 10] | [19 24 10] | 96.98 | 16650
Table II: Results regarding the accuracy and the number of parameters with different configurations of tail MLPs. Type |  | Neurons
---
per Layer
| Out-degree
---
per Node
| Accuracy
---
(%)
| Number of
---
Parameters
Deep | [800 | 6 6 | 10] | [6 6 10] | 90.68 | 4918
Shallow | [800 | 6 | 10] | [6 10] | 91.03 | 4876
Sparse | [800 | 40 40 | 10] | [5 11 10] | 95.65 | 4930
Deep | [800 | 10 10 | 10] | [10 10 10] | 94.59 | 8230
Shallow | [800 | 10 | 10] | [10 10] | 94.00 | 8120
Sparse | [800 | 40 40 | 10] | [9 14 10] | 96.24 | 8250
Deep | [800 | 14 14 | 10] | [14 14 10] | 95.43 | 11574
Shallow | [800 | 14 | 10] | [14 10] | 94.88 | 11364
Sparse | [800 | 40 40 | 10] | [13 17 10] | 96.72 | 11570
Deep | [800 | 28 28 | 10] | [28 28 10] | 96.79 | 23530
Shallow | [800 | 29 | 10] | [29 10] | 96.79 | 23529
Sparse | [800 | 80 80 | 10] | [26 22 10] | 97.33 | 23530
Deep | [800 | 39 39 | 10] | [39 39 10] | 97.35 | 33199
Shallow | [800 | 41 | 10] | [41 10] | 97.34 | 33261
Sparse | [800 | 80 80 | 10] | [37 33 10] | 97.75 | 33210
$3000$$6000$$9000$$12000$$15000$$18000$$75$$80$$85$$90$$95$$100$Number of
parametersAccuracy (%)SparseShallowDeep Figure 4: Accuracy tests by the number
of parameters of deep, shallow, and sparse head MLPs.
$5000$$10000$$15000$$20000$$25000$$30000$$35000$$75$$80$$85$$90$$95$$100$Number
of parametersAccuracy (%)SparseShallowDeep Figure 5: Accuracy tests by number
of parameters of deep, shallow, and sparse tail MLPs.

Tables I and II provide a comparative overview of the three neural network
configurations, detailing their architectural details and performances.
Specifically, each table reports the number of neurons per layer, focusing on
the number of neurons in hidden layers (in bold), the out-degree for each
node, their accuracy, and the number of parameters.

Figures 4 and 5 show the accuracy plots against the number of parameters for
the three neural network configurations. These two plots reveal the advantage
of sparse split models: they present remarkable stability in accuracy even
with significant reductions in trainable parameters. This characteristic
results in two key advantages, particularly desired and pursued in resource-
constrained settings like SC and EE.

Unlike traditional deep and shallow DNN, where accuracy improvements often
depend on a significant increase in the number of parameters, sparse split
models achieve optimal inference performance without requiring massive
parameter expansions. As a result, they are much more efficient when using the
limited processing power and storage space available on edge devices, and so
very suitable for distributed deep leasing scenarios through SC and EE.

Furthermore, the predefined sparsity of these networks leads to decreased
memory usage, both during training and inference. Regarding the training, this
is an advantage because researchers with limited resources can also engage in
cutting-edge research in deep learning without the need for expensive, high-
end computational resources. Instead, the reduced complexity translates into
faster processing times, enabling real-time operation on edge devices with
limited processing power.

###  IV-C Discussion

#### Early Exiting and reduced communication

In SC and EE applications, where data security and bandwidth limitations are
paramount, sparse split models represent an ideal solution.

The early exiting strategy applied to the head provides an acceptable level of
accuracy with a smaller subset of parameters, as highlighted by the achieved
accuracy shown in Figure 4. This allows the network to terminate computations
early, significantly reducing the amount of data transmitted between the edge
device and the server. This not only minimizes the risk of data breaches but
also conserves precious network bandwidth, enabling efficient communication
even in low-connectivity environments.

#### Balancing training time and efficiency

While the benefits of sparse split models are evident, it’s important to
acknowledge the trade-off with training time. When we introduce predefined
sparsity into a MLPs, we aim to train the model to learn a function using a
significantly reduced number of connections compared to a non-sparse network.
This approach creates a more lightweight structure for the model but also
leads to a more challenging optimization problem. Consequently, the gradient
descent algorithm may require more iterations to converge due to the increased
complexity introduced by the sparsity constraints. In particular, sparse split
models in our experiments exhibited a $2\times{}$ slowdown in training time
compared to their dense counterparts.

However, this drawback needs to be considered in the context of the specific
application and its resource constraints. In many cases, the significant gains
in memory usage, communication efficiency, and inference speed during
deployment far outweigh the potential increase in training time.

#### Potential further exploration

While this work demonstrates the effectiveness of predefined sparsity with SC
and EE, further exploration is possible. First, future research could look at
applying this technique to different DNN architectures on top of existing SC
and EE methods.

It would also be interesting to see how different hardware edge devices can
benefit from this. For example, we know that devices like GPUs and Field
Programmable Gate Arrays (FPGAs) handle sparse data well. So, using this
technique with these devices on the edge (like GPU-based or FPGA-based
hardware) could be a promising direction.

Finally, further exploration could involve applying these techniques to real-
world applications. This would allow us to assess the effectiveness of the
approach in practical scenarios and identify any challenges that may arise.

##  V Conclusion

In this paper, we presented the effect of predefined sparsity within the SC
and EE paradigm. This approach, demonstrably effective for the first time in
an SC and EE scenario, significantly reduces the computational, storage, and
energy demands during training and inference, regardless of the hardware
platform. The experimental results showcase impressive reductions exceeding
$4\times{}$ in both storage and computational complexity while maintaining
comparable accuracy. This paves the way for deploying complex DNN models on
edge devices for real-world SC and EE applications.

## References

  * [1] S. Pouyanfar, S. Sadiq, Y. Yan _et al._ , “A survey on deep learning: Algorithms, techniques, and applications,” _ACM Computing Surveys (CSUR)_ , 2018.
  * [2] S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” _Computer Science Review_ , 2021.
  * [3] S. Smith, M. Patwary, B. Norick _et al._ , “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” _arXiv preprint arXiv:2201.11990_ , 2022.
  * [4] K. Y. Chan, B. Abu-Salih, R. Qaddoura _et al._ , “Deep neural networks in the cloud: Review, applications, challenges and research directions,” _Neurocomputing_ , 2023.
  * [5] N. P. Jouppi, C. Young, N. Patil _et al._ , “In-datacenter performance analysis of a tensor processing unit,” in _Proceedings of the 44th annual international symposium on computer architecture_ , 2017.
  * [6] R. Singh and S. S. Gill, “Edge ai: a survey,” _Internet of Things and Cyber-Physical Systems_ , 2023.
  * [7] L. Capogrosso, F. Cunico, D. S. Cheng _et al._ , “A machine learning-oriented survey on tiny machine learning,” _IEEE Access_ , 2024.
  * [8] G. Menghani, “Efficient deep learning: A survey on making deep learning models smaller, faster, and better,” _ACM Computing Surveys_ , 2023.
  * [9] E. Fragkou, M. Koultouki, and D. Katsaros, “Model reduction of feed forward neural networks for resource-constrained devices,” _Applied Intelligence_ , 2023.
  * [10] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “Jointdnn: An efficient training and inference engine for intelligent mobile cloud computing services,” _IEEE Transactions on Mobile Computing_ , 2019.
  * [11] M. Jankowski, D. Gündüz, and K. Mikolajczyk, “Joint device-edge inference over wireless links with pruning,” in _2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2020.
  * [12] M. Sbai, M. R. U. Saputra, N. Trigoni _et al._ , “Cut, distil and encode (cde): Split cloud-edge deep inference,” in _2021 18th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)_. IEEE, 2021.
  * [13] F. Cunico, L. Capogrosso, F. Setti _et al._ , “I-split: Deep network interpretability for split computing,” in _2022 26th International Conference on Pattern Recognition (ICPR)_. IEEE, 2022.
  * [14] L. Capogrosso, F. Cunico, M. Lora _et al._ , “Split-et-impera: A framework for the design of distributed deep learning applications,” in _2023 26th International Symposium on Design and Diagnostics of Electronic Circuits and Systems (DDECS)_. IEEE, 2023.
  * [15] Y. Matsubara, M. Levorato, and F. Restuccia, “Split computing and early exiting for deep learning applications: Survey and research challenges,” _ACM Computing Surveys_ , 2022.
  * [16] Y. Gong, L. Liu, M. Yang _et al._ , “Compressing deep convolutional networks using vector quantization,” _arXiv preprint arXiv:1412.6115_ , 2014.
  * [17] W. Chen, J. Wilson, S. Tyree _et al._ , “Compressing neural networks with the hashing trick,” in _International conference on machine learning_. PMLR, 2015.
  * [18] W. Wen, C. Wu, Y. Wang _et al._ , “Learning structured sparsity in deep neural networks,” _Advances in neural information processing systems_ , 2016.
  * [19] N. Srivastava, G. Hinton, A. Krizhevsky _et al._ , “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , 2014.
  * [20] A. Labach, H. Salehinejad, and S. Valaee, “Survey of dropout methods for deep neural networks,” _arXiv preprint arXiv:1904.13310_ , 2019.
  * [21] J. Albericio, P. Judd, T. Hetherington _et al._ , “Cnvlutin: Ineffectual-neuron-free deep neural network computing,” _ACM SIGARCH Computer Architecture News_ , 2016.
  * [22] A. Aghasi, A. Abdi, N. Nguyen _et al._ , “Net-trim: Convex pruning of deep neural networks with performance guarantee,” _Advances in neural information processing systems_ , 2017.
  * [23] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [24] V. Sindhwani, T. Sainath, and S. Kumar, “Structured transforms for small-footprint deep learning,” _Advances in Neural Information Processing Systems_ , 2015.
  * [25] S. Dey, K.-W. Huang, P. A. Beerel _et al._ , “Pre-defined sparse neural networks with hardware acceleration,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , 2019.
  * [26] A. Howard, M. Sandler, G. Chu _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019.
  * [27] T. Liang, J. Glossner, L. Wang _et al._ , “Pruning and quantization for deep neural network acceleration: A survey,” _Neurocomputing_ , 2021.
  * [28] J. Gou, B. Yu, S. J. Maybank _et al._ , “Knowledge distillation: A survey,” _International Journal of Computer Vision_ , 2021.
  * [29] Y. Matsubara, S. Baidya, D. Callegaro _et al._ , “Distilled split deep neural networks for edge-assisted real-time systems,” in _Proceedings of the 2019 Workshop on Hot Topics in Video Analytics and Intelligent Edges_ , 2019.
  * [30] Y. Kang, J. Hauswald, C. Gao _et al._ , “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” _ACM SIGARCH Computer Architecture News_ , 2017.
  * [31] G. Li, L. Liu, X. Wang _et al._ , “Auto-tuning neural network quantization framework for collaborative inference between the cloud and edge,” in _Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I 27_. Springer, 2018.
  * [32] H. Choi and I. V. Bajić, “Deep feature compression for collaborative object detection,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018.
  * [33] D. Carra and G. Neglia, “Dnn split computing: Quantization and run-length coding are enough,” in _GLOBECOM 2023-2023 IEEE Global Communications Conference_. IEEE, 2023.
  * [34] A. E. Eshratifar, A. Esmaili, and M. Pedram, “Bottlenet: A deep learning architecture for intelligent mobile cloud computing services,” in _2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)_. IEEE, 2019.
  * [35] R. Caruana, “Multitask learning,” _Machine learning_ , 1997.
  * [36] L. Capogrosso, E. Fraccaroli, S. Chakraborty _et al._ , “Mtl-split: Multi-task learning for edge devices using split computing,” _arXiv preprint arXiv:2407.05982_ , 2024.
  * [37] C. Lo, Y.-Y. Su, C.-Y. Lee _et al._ , “A dynamic deep neural network design for efficient workload allocation in edge computing,” in _2017 IEEE International Conference on Computer Design (ICCD)_. IEEE, 2017.
  * [38] C. Turetta, G. Skenderi, L. Capogrosso _et al._ , “Towards deep learning-based occupancy detection via wifi sensing in unconstrained environments,” in _2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2023.
  * [39] S.-A. Chen, C.-L. Li, N. Yoder _et al._ , “Tsmixer: An all-mlp architecture for time series forecasting,” _arXiv preprint arXiv:2303.06053_ , 2023.
  * [40] L. Deng, “The mnist database of handwritten digit images for machine learning research,” _IEEE Signal Processing Magazine_ , 2012.
  * [41] J. Deng, W. Dong, R. Socher _et al._ , “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. IEEE, 2009.
  * [42] M. Abadi, A. Agarwal, P. Barham _et al._ , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [43] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


J134100.78+412314.0 | $0.3488$ | $-60.0$ | $60.0$ | $1.3\pm 0.9$ | $83.7\pm 1.2$ | $14.50_{-0.02}^{+0.02}$ | $0$ | $14.5$ | $1$
J134100.78+412314.0 | $0.6208$ | $-100.0$ | $50.0$ | $-0.5\pm 3.9$ | $105.6\pm 5.6$ | $14.35_{-0.05}^{+0.04}$ | $0$ | $9.5$ | $1$
J134100.78+412314.0 | $0.6214$ | $-50.0$ | $70.0$ | $-8.9\pm 2.2$ | $104.7\pm 3.1$ |  $\cdots$ | $0$ | $8.1$ | $1$
J134100.78+412314.0 | $0.6861$ | $-40.0$ | $120.0$ | $49.3\pm 1.4$ | $91.8\pm 1.9$ | $14.71_{-0.02}^{+0.02}$ | $0$ | $10.6$ | $1$
J134447.55+554656.8 | $0.4042$ | $-75.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.67$ | $-1$ | $4.6$ | $0$
J135726.26+043541.3 | $0.6102$ | $-40.0$ | $75.0$ | $6.5\pm 2.4$ | $79.9\pm 3.4$ | $14.41_{-0.05}^{+0.04}$ | $0$ | $8.6$ | $1$
J140732.25+550725.4 | $0.2433$ | $-75.0$ | $75.0$ |  $\cdots$ |  $\cdots$ | $<13.36$ | $-1$ | $11.8$ | $1$
J140732.25+550725.4 | $0.2469$ | $-50.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.29$ | $-1$ | $12.3$ | $1$
J140923.90+261820.9 | $0.5749$ | $-50.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<13.05$ | $-1$ | $21.5$ | $1$
J140923.90+261820.9 | $0.5996$ | $-50.0$ | $40.0$ | $-8.7\pm 1.2$ | $80.2\pm 1.6$ | $13.99_{-0.02}^{+0.02}$ | $0$ | $21.2$ | $1$
J140923.90+261820.9 | $0.6827$ | $-70.0$ | $80.0$ | $3.5\pm 1.8$ | $96.2\pm 2.5$ | $14.07_{-0.02}^{+0.02}$ | $0$ | $20.9$ | $1$
J141038.39+230447.1 | $0.3498$ | $-30.0$ | $40.0$ | $9.0\pm 2.5$ | $54.0\pm 3.5$ | $13.92_{-0.06}^{+0.06}$ | $0$ | $9.6$ | $1$
J141038.39+230447.1 | $0.3510$ | $-50.0$ | $55.0$ |  $\cdots$ |  $\cdots$ | $<13.40$ | $-1$ | $9.8$ | $1$
J141038.39+230447.1 | $0.5351$ | $-20.0$ | $75.0$ | $23.8\pm 4.5$ | $76.7\pm 6.4$ | $13.73_{-0.09}^{+0.07}$ | $0$ | $12.4$ | $1$
J141910.20+420746.9 | $0.2890$ | $-70.0$ | $70.0$ | $6.8\pm 3.0$ | $91.8\pm 4.3$ | $14.49_{-0.05}^{+0.05}$ | $0$ | $7.6$ | $0$
J141910.20+420746.9 | $0.4256$ | $-70.0$ | $10.0$ | $-27.9\pm 5.3$ | $60.4\pm 7.5$ | $13.73_{-0.13}^{+0.10}$ | $0$ | $8.8$ | $1$
J141910.20+420746.9 | $0.5221$ | $-100.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<13.76$ | $-1$ | $5.9$ | $0$
J141910.20+420746.9 | $0.5346$ | $-40.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.79$ | $-1$ | $4.5$ | $0$
J141910.20+420746.9 | $0.6083$ | $-30.0$ | $60.0$ |  $\cdots$ |  $\cdots$ | $<13.75$ | $-1$ | $4.7$ | $0$
J142735.59+263214.6 | $0.3661$ | $0.0$ | $105.0$ | $55.0\pm 5.4$ | $88.6\pm 7.6$ | $14.38_{-0.10}^{+0.08}$ | $0$ | $6.1$ | $0$
J142859.03+322506.8 | $0.3823$ | $-20.0$ | $60.0$ | $23.4\pm 3.6$ | $56.7\pm 5.1$ | $13.78_{-0.08}^{+0.07}$ | $0$ | $10.0$ | $1$
J143511.53+360437.2 | $0.3730$ | $-60.0$ | $60.0$ |  $\cdots$ |  $\cdots$ | $<13.46$ | $-1$ | $9.2$ | $1$
J143511.53+360437.2 | $0.3876$ | $-100.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<14.12$ | $-1$ | $5.5$ | $0$
J143748.28-014710.7 | $0.2989$ | $-90.0$ | $55.0$ |  $\cdots$ |  $\cdots$ | $<12.89$ | $-1$ | $28.6$ | $1$
J143748.28-014710.7 | $0.6127$ | $-90.0$ | $60.0$ | $-16.3\pm 2.7$ | $113.6\pm 3.8$ | $13.88_{-0.03}^{+0.03}$ | $0$ | $21.9$ | $1$
J143748.28-014710.7 | $0.6812$ | $-100.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<13.41$ | $-1$ | $20.4$ | $1$
J150030.64+551708.8 | $0.3473$ | $-75.0$ | $75.0$ |  $\cdots$ |  $\cdots$ | $<13.21$ | $-1$ | $13.2$ | $1$
J150030.64+551708.8 | $0.3480$ | $-60.0$ | $80.0$ | $0.6\pm 3.6$ | $95.9\pm 5.0$ | $13.99_{-0.04}^{+0.04}$ | $0$ | $12.6$ | $1$
J152424.58+095829.7 | $0.5185$ | $-70.0$ | $70.0$ |  $\cdots$ |  $\cdots$ | $<13.06$ | $-1$ | $18.4$ | $1$
J152424.58+095829.7 | $0.5718$ | $-55.0$ | $30.0$ |  $\cdots$ |  $\cdots$ | $<13.20$ | $-1$ | $11.1$ | $1$
J152424.58+095829.7 | $0.6754$ | $-40.0$ | $95.0$ | $24.1\pm 2.3$ | $110.8\pm 3.2$ | $14.09_{-0.03}^{+0.03}$ | $0$ | $15.7$ | $1$
J152424.58+095829.7 | $0.7289$ | $-30.0$ | $50.0$ | $3.3\pm 4.5$ | $55.7\pm 6.3$ | $13.69_{-0.10}^{+0.08}$ | $0$ | $10.2$ | $1$
J154121.48+281706.2 | $0.2825$ | $-120.0$ | $120.0$ | $-9.1\pm 16.2$ | $169.9\pm 22.9$ | $14.18_{-0.13}^{+0.10}$ | $0$ | $6.8$ | $0$
J155048.29+400144.9 | $0.3126$ | $-100.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<13.59$ | $-1$ | $8.4$ | $1$
J155048.29+400144.9 | $0.4273$ | $-80.0$ | $110.0$ | $24.3\pm 4.2$ | $151.8\pm 6.0$ | $14.35_{-0.04}^{+0.04}$ | $0$ | $10.7$ | $1$
J155048.29+400144.9 | $0.4920$ | $-90.0$ | $90.0$ |  $\cdots$ |  $\cdots$ | $<13.46$ | $-1$ | $11.1$ | $1$
J155048.29+400144.9 | $0.4926$ | $-50.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.33$ | $-1$ | $11.3$ | $1$
J155232.54+570516.5 | $0.3660$ | $-40.0$ | $50.0$ | $12.0\pm 3.0$ | $78.4\pm 4.2$ | $14.03_{-0.05}^{+0.05}$ | $0$ | $9.3$ | $1$
J155232.54+570516.5 | $0.3665$ | $-40.0$ | $60.0$ | $0.5\pm 2.3$ | $90.2\pm 3.3$ | $14.22_{-0.04}^{+0.04}$ | $0$ | $9.1$ | $1$
J155304.92+354828.6 | $0.2179$ | $-50.0$ | $40.0$ | $-7.8\pm 3.0$ | $59.6\pm 4.2$ | $13.98_{-0.06}^{+0.06}$ | $0$ | $9.6$ | $1$
J155304.92+354828.6 | $0.4755$ | $-75.0$ | $75.0$ |  $\cdots$ |  $\cdots$ | $<13.54$ | $-1$ | $9.1$ | $1$
J155504.39+362848.0 | $0.5760$ | $-75.0$ | $75.0$ |  $\cdots$ |  $\cdots$ | $<13.81$ | $-1$ | $4.4$ | $0$
J161916.54+334238.4 | $0.2694$ | $-50.0$ | $40.0$ | $-7.2\pm 5.7$ | $47.9\pm 8.1$ | $13.85_{-0.13}^{+0.10}$ | $0$ | $12.2$ | $1$
J161916.54+334238.4 | $0.4423$ | $-75.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<13.13$ | $-1$ | $19.1$ | $1$
J163201.12+373749.9 | $0.2740$ | $-60.0$ | $25.0$ | $-0.6\pm 2.2$ | $54.3\pm 3.1$ | $13.96_{-0.04}^{+0.03}$ | $0$ | $28.9$ | $1$
J163201.12+373749.9 | $0.4177$ | $-30.0$ | $45.0$ | $8.6\pm 1.5$ | $58.9\pm 2.1$ | $13.56_{-0.03}^{+0.03}$ | $0$ | $23.2$ | $1$
J215647.46+224249.8 | $0.5928$ | $-30.0$ | $40.0$ | $3.8\pm 4.7$ | $67.3\pm 6.6$ | $13.89_{-0.14}^{+0.11}$ | $0$ | $6.7$ | $0$
J215647.46+224249.8 | $0.6200$ | $-50.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.60$ | $-1$ | $7.0$ | $0$
J215647.46+224249.8 | $0.6210$ | $-50.0$ | $43.0$ |  $\cdots$ |  $\cdots$ | $<13.57$ | $-1$ | $7.1$ | $0$
J215647.46+224249.8 | $0.6214$ | $-27.0$ | $50.0$ |  $\cdots$ |  $\cdots$ | $<13.53$ | $-1$ | $7.0$ | $0$
J225357.75+160853.1 | $0.3206$ | $-40.0$ | $40.0$ | $-1.7\pm 3.8$ | $59.3\pm 5.4$ | $13.96_{-0.08}^{+0.08}$ | $0$ | $7.8$ | $0$
J225738.20+134045.4 | $0.4989$ | $-40.0$ | $75.0$ | $22.7\pm 6.6$ | $92.4\pm 9.4$ | $14.27_{-0.11}^{+0.09}$ | $0$ | $8.0$ | $1$
J234500.43-005936.0 | $0.5481$ | $-75.0$ | $100.0$ |  $\cdots$ |  $\cdots$ | $<14.24$ | $-1$ | $4.8$ | $0$

Note. — (1) J–Name of the background quasar; (2) Redshift of the absorber; (3)
Start velocity of the integration range; (4) End velocity of the integration
range; (5) Adopted absolute velocity centroid offset of O vi w.r.t H i; (6)
Adopted velocity width of O vi absorption; (7) Adopted measured column density
of O vi absorption; (8) Detection Flag for O vi – 0–detection, $-1$–upper
limit, $-2$–lower limit, $-3$–contaminated, $-4$–not covered by the spectrum;
(9) Average $S/N$ of the spectrum per pixel when both O vi transitions are
covered and used, otherwise the $S/N$ of the spectrum of the adopted
transition; (10) Robustness Flag for the spectrum – 1-Included in the robust
sample, $0$ not included in the robust sample.

This is needed for the table to fully resolve.