# AccEPT: An Acceleration Scheme for Speeding Up Edge Pipeline-parallel
Training

Yuhao Chen, Yuxuan Yan, Qianqian Yang, Yuanchao Shu, Shibo He, Zhiguo Shi, and
Jiming Chen This paper was presented in part at the IEEE International
Conference on Ubiquitous Communication (Ucom), Xi’an, China, July 2023[1]. Y.
Chen, Y. Yan, Q. Yang, S. He, Z. Shi, J. Chen are with the State Key
Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou
310027, China. (E-mail: {csechenyh, yxyan44, qianqianyang20, ycshu, s18he,
shizg<EMAIL_ADDRESS>
###### Abstract

It is usually infeasible to fit and train an entire large deep neural network
(DNN) model using a single edge device due to the limited resources. To
facilitate intelligent applications across edge devices, researchers have
proposed partitioning a large model into several sub-models, and deploying
each of them to a different edge device to collaboratively train a DNN model.
However, the communication overhead caused by the large amount of data
transmitted from one device to another during training, as well as the sub-
optimal partition point due to the inaccurate latency prediction of
computation at each edge device can significantly slow down training. In this
paper, we propose AccEPT, an acceleration scheme for accelerating the edge
collaborative pipeline-parallel training. In particular, we propose a light-
weight adaptive latency predictor to accurately estimate the computation
latency of each layer at different devices, which also adapts to unseen
devices through continuous learning. Therefore, the proposed latency predictor
leads to better model partitioning which balances the computation loads across
participating devices. Moreover, we propose a bit-level computation-efficient
data compression scheme to compress the data to be transmitted between devices
during training. Our numerical results demonstrate that our proposed
acceleration approach is able to significantly speed up edge pipeline parallel
training up to 3 times faster in the considered experimental settings.

###### Index Terms:

Data compression scheme, distributed parallel edge training, adaptive
quantization, bitwise compression

##  1 Introduction

Edge computing has made it possible to run various deep learning (DL)
applications such as computer vision[2, 3], natural language processing[4] and
human activity recognition[5] on edge devices. These edge devices, like mobile
phones, cameras, and watches, run the DL model locally with the raw data
samples directly collected from the on-device sensors, providing real-time
response as well as data privacy when compared with uploading the raw data to
the cloud to run the model. Moreover, the raw data sampled by the edge device
may be highly personalized. For example, for the human activity recognition
task, the data distribution may vary for different individuals like heights
and weights, or be greatly affected by environmental noise like weather
changes, resulting in performance degradation of a pre-trained model. Hence,
there are emerging demands of on-device training to improve the performance of
on-device inference and provide personalization. For example, Federated
Learning(FL)[6], a framework proposed by Google, is currently widely used for
on-device training. However, due to the limited computation and storage
capabilities, mobile devices are usually not able to train an entire DL model
efficiently or even fail to run [7].

To tackle this issue, one of the solutions is the model partitioning
technique, which partitions a DL model into several sub-models, and deploys
them across a number of edge devices that execute the whole model
collaboratively[8, 9, 10, 11, 12]. With this technique, the data privacy is
preserved since the raw data is only processed on the local device. The real-
time response is also guaranteed by leveraging the computing resources of
multiple edge servers. Moreover, compared with other techniques where the DL
model is compressed to fit into a single edge device[13, 14, 15], the model
partitioning method also enjoys the advantage of running the original DL model
such that preserves the model performance.

Most of the existing works on model partitioning consider on-device inference
by partitioning a model into two parts, with one part on the edge device and
the other on the edge server[16, 17, 18, 19, 20]. The edge device executes the
first part of the model, and transmits the intermediate results to the server,
which computes the remaining part, and returns the final output back to the
device. However, few studies investigate model partitioning based on-device
training. Some recent works proposed a learning framework called split
learning, which also splits a model into two parts and deploys one to the edge
device and the other to the server [12, 21, 9]. In this approach, however, the
training process on the edge device is stalled until the backward gradients
are received, which remarkably slows down the training process.

To address this issue, in our previous work, we proposed an edge pipeline-
parallel training approach called FTPipeHD[10], which partitions a DL model
into several sub-models and adopts the idea of the pipeline parallelism for
GPU training acceleration to the on-device training. The pipeline mechanism
avoids the stalling of devices by keeping feeding the data batches into the
model without waiting for the newest weights to arrive, which achieves a
significant speedup with a cost of slight performance degradation. Moreover,
since edge devices have time-varying and heterogeneous computing resources,
FTPipeHD also periodically estimates the computing capacity of each device
during training. Based on this estimation, the DL model is dynamically
partitioned as the training proceeds in order to accelerate training. Fig. 1
demonstrates the basic idea of the edge pipeline-parallel training, where a DL
model is split into three sub-models and deployed across three edge devices.

Figure 1: An example of the distributed parallel edge training based on model
partitioning.

Although FTPipeHD solves the stalling problem and provides a certain level of
acceleration, there are still some factors that slow down training. Firstly,
the latent features in the forward pass and the gradients in the backward
pass, which need to be transmitted, can cause non-negligible communication
latency during training. Unlike pipeline parallel training with multiple GPUs,
where data is transmitted via high-speed cables, edge devices are generally
interconnected by wireless links like WiFi and cellular networks, which
usually have lower bandwidth compared with their wired counterparts.
Furthermore, for transmission through cellular networks, e.g., Long Term
Evolution (LTE), 4G, and 5G, a large amount of data transmission may incur
prohibitive costs charged by network operators, which may cause
dissatisfaction among users of the edge devices. Secondly, the inaccurate
estimation of the computing capacities of heterogeneous devices makes the
model partitioning sub-optimal, thus slowing down training. As we will
demonstrate in this paper, the execution time by each device estimated by
FTPipeHD is highly inaccurate compared with the true execution time.

To tackle the aforementioned issues, we introduce AccEPT in this paper, an
acceleration scheme for Accelerating Edge Pipeline-parallel Training. Instead
of predicting the layer-level execution time as in FTPipeHD, the scheme uses a
light-weight latency predictor to accurately predict the sub-model’s execution
time by each device. To reduce the transmission latency, we propose a novel
bit-level fast data compression scheme to effectively reduce the amount of
transmission data. We implement AccEPT on edge devices and conduct
comprehensive experiments. We summarize the main contributions of this paper
as follows:

  * •

We propose an acceleration scheme called AccEPT that speeds up the edge
pipeline-parallel training, which consists of a light-weight latency predictor
and a bit-level computation-efficient data compression scheme. The latency
predictor provides an accurate sub-model’s execution time prediction to better
partition the model for training, while the compression scheme reduces the
communication overhead.

  * •

We propose a light-weight latency predictor to dynamically estimate the sub-
model’s execution time by each device, which is pre-trained on data collected
from various devices. During training, the proposed latency predictor performs
continuous learning to adapt to unseen devices.

  * •

We propose a novel bit-level computation efficient data compression scheme
that consists of an adaptive quantizer and a bit-wise encoder. The quantizer
first quantizes the 32-bit floating-point data of the features and the
gradients into lower-bit integer representation. Then the encoder is applied
to the quantized data to reduce the redundant bits. A corresponding
decompression scheme is also proposed.

  * •

We implement AccEPT on a set of commonly used edge devices and perform
comprehensive evaluations to demonstrate the acceleration performance of
AccEPT. Evaluation results show that AccEPT can speed up the edge pipeline
training up to 3 times faster in the considered experimental settings.

The rest of this paper is organized as follows. Section 2 gives the background
and motivation of our work. In section 3, we introduce the system overview and
design details of AccEPT, followed by section 4 where we present the numerical
results to validate the performance of the proposed scheme. Finally, we
conclude our work in section 5.

##  2 Background and Motivation

###  2.1 Pipeline-parallel Training

We follow the idea of pipeline parallel training introduced in my previous
work[10], referred to as FTPipeHD, to enable distributed on-device training.
By FTPipeHD, one device owns the raw data and manages the training, referred
to as the _central node_ , while the other devices participating in the
training are referred to as the _worker nodes_. We present in Fig. 2 an
example of this pipeline parallel training scheme with three devices, where
device 1 is the central node, and $i^{ver}_{\mathrm{f}}$ and
$i^{ver}_{\mathrm{b}}$ denote the forwarding and backwarding of the $i$-th
training batch with the weights of the version number $ver$. We note that each
device performs the forward and backward run of the local sub-model
alternatively, which is referred to as _one-forward-one-backward (1F1B)_ rule.
As illustrated in Fig. 2, the consecutive forward and backward run by a device
may be based on different versions of weights and different batches.
Therefore, to guarantee the model convergence, the weights used to perform the
forward run of a data batch are stashed for the later corresponding backward
propagation of the same data batch.

Figure 2: The illustration of the pipeline parallel training across three
devices.

For instance, consider the training of the data batch 1, which is available at
the central node. In traditional training without pipelining, the training of
batch 1 starts only when device 1 receives the backward gradients of batch 0
and updates the weights accordingly, while by the pipelining scheme, the
training of the batch 1 starts once the forward run of batch 0 is completed
based on the initial weights of version 0 instead of the updated weights of
version 1. We can observe from Fig. 2 that on device 1, although the weights
have been updated to version 1 by the backward run of batch 0, batch 1 is
still backwarded with the weights of version 0 to ensure that the same version
of weights is used in the forward and backward pass of the same batch. We also
note that for a certain batch, different versions of weights may be used on
different devices, e.g., $5_{\mathrm{f}}^{3}$ on device 1,
$5_{\mathrm{f}}^{4}$ on device 2, and $5_{\mathrm{f}}^{5}$ on device 3. This
pipeline parallel training scheme aims to minimize the idle time of devices
during the training to accelerate the training.

###  2.2 The Large Amount of Transmission Data

Unlike model inference where the batch size of the input data is generally
$1$, training a model usually requires a large batch size to provide
computation efficiency and ensure convergence. In edge pipeline-parallel
training, suppose that the model is partitioned at layer $l$ between devices
$d_{0}$ and $d_{1}$, then the features and the gradients both of size $n\times
w\times h\times c$, should be transmitted between two nodes, with a batch size
of $n$, width of $w$, height of $h$ and a channel number of $c$. Each element
is a 32-bit (4 bytes) floating-point value. Fig. 3 demonstrates the size of
output of each layer $l$ in MobileNetV2 when batch size is 128 and the size of
the input image is $3\times 32\times 32$. It can be observed that the largest
output is about 4MB. If this is transmitted over the 4G cellular network whose
bandwidth is 50Mbps on average, it still takes about 655ms to transmit it.
Moreover, if the model is trained for 200 epochs and there are 391 batches in
each epoch, about $4*200*391\approx 312,800\mathrm{MB}\approx 305\mathrm{GB}$
should be transmitted throughout the training, which may cause large data
costs charged by network operators. To compress the transmitted data,
researchers have proposed several data compression methods for the distributed
inference based on model partitioning, where DNN-based compressor[20] can
compress the data as much as possible while preserving the inference
performance. However, for the training problem considered in this paper,
compressors with large computation overhead can slow down the training of the
whole model, and hence, are undesirable. Therefore, a computation-efficient
compression scheme is needed to reduce the amount of data transmitted between
nodes during the edge pipeline parallel training.

Figure 3: Output size of each layer in MobileNetV2 when batch size is 128.

###  2.3 The Inaccurate Estimation of the Execution Time

To minimize the training time, the layer-level execution time on each edge
device should be accurately estimated so that the model can be optimally
partitioned. In FTPipeHD, before the edge pipeline-parallel training, the
central node $d_{0}$ profiles the total execution time of a forward pass and a
backward pass for each layer, denoted as $T_{\mathrm{e},j}^{0}$ for the $j$-th
layer. The execution time of any sub-model on $d_{0}$ can be calculated by
summing up the execution time of corresponding layers. During training,
$d_{0}$ periodically collects the execution time of sub-models from each
worker node. A ratio between the sub-model’s execution time on the worker node
and on the central node is calculated to represent the computing capacity of
the worker node. Then for the $j$-th layer, its execution time
$T_{\mathrm{e},j}^{i}$ on the worker node $d_{i}$ is estimated by multiplying
this ratio with $T_{\mathrm{e},j}^{0}$. However, such an estimation is not
accurate. Fig. 4 demonstrates the estimation error between the estimated and
true layer-level latency on $d_{1}$ for a 20-layer MobileNetV2[3], where the
MobileNetV2 is partitioned by assuming all devices have the same computing
capacity and $d_{1}$ is responsible for training the layers from 9 to 15. It
can be seen from Fig. 4 that most of the estimation errors exceed $20\%$,
where the estimation error of layer 20 is $187.51\%$. We also note that even
though the overall execution time from layer 9 to 15 on $d_{1}$ is known by
$d_{0}$, the estimation errors of layer 9 to 15 are still notable. The
inaccurate estimation of execution time at each device leads to suboptimal
model partitioning. Hence, it calls for a better prediction scheme in order to
speed up the whole training process.

Figure 4: Estimation error of the layer-level latency on $d_{1}$ for the
20-layer MobileNetV2. Figure 5: The system overview of our proposed scheme.

##  3 System Design

###  3.1 System Overview

Motivated by observations in section 2, the proposed AccEPT scheme accelerates
the edge pipeline-parallel training by accurately estimating the execution
time for a better model partition point and reducing the amount of
transmission data. To achieve this, AccEPT first pre-trains an adaptive light-
weight latency predictor, which is a regression model containing only four
fully-connected layers, with the data collected from various devices. During
the training, the latency predictor estimates the execution time of each
worker node and collects the exact execution time from unseen devices to
update itself, which obtains a more accurate execution time estimation and
thus a better partition point. Then, AccEPT uses a bit-level fast compression
scheme to compress the data at each node before transmitting it to the next
node. Specifically, it first quantizes the data to low-bit integers. Then it
employs a bit-wise encoding to combine multiple low-bit integers into an 8-bit
integer. At the receiver side, the transmitted data is decompressed in an
inverse operation. On acquiring the accurate execution time and calculating
the transmission time based on the reduced amount of transmitted data, AccEPT
solves a dynamic programming problem to obtain the optimal model partition
point, which aims to minimize the overall training time. Fig. 5 illustrates
the overview of the proposed scheme.

###  3.2 Adaptive Light-weight Latency Predictor

Instead of estimating the layer-level execution time on each device, the
proposed adaptive light-weight latency predictor provides an accurate sub-
model’s execution time of each worker and fast adaptation to unseen devices.
It consists of three steps, i.e., the pre-training, dynamic prediction and
adaptive update. Note that the latency predictor is specific to a certain
model. All sub-models described below come from the same model.

Pre-training. Before the edge pipeline parallel training, we pre-train the
latency predictor on the representative data collected from various devices.
The latency predictor is a regression model with four fully-connected layers,
which is defined as $T_{\mathrm{e},s}^{i}=f(s,h_{i})$. The input of the
latency predictor, namely $s$ and $h_{i}$, is the vector encoding of a sub-
model and the vector characterizing the hardware performance of the worker
node $d_{i}$, respectively. The output $T_{\mathrm{e},s}^{i}$ is the estimated
execution time of the sub-model represented by $s$ on the worker node $d_{i}$.

In particular, we use the binary encoding to encode a sub-model. Suppose that
the model being partitioned has $l$ layers, then $s$ is a vector of length
$l$, with the $i$-th element representing the $i$-th layer. For each element
in the encoding vector $s$, 1 means the presence of the corresponding layer in
the sub-model, while 0 means the absence of the corresponding layer. For
example, for a model with 10 layers, a sub-model which consists of its 3rd
layer to 7th layer, the encoding of the sub-model $s$ is
$[0,0,1,1,1,1,1,0,0,0]$. Note that the layer of a model can be a block
consisting of several operators like the residual block in the MobileNetV2.

To characterize the hardware performance of each worker node, i.e., to obtain
$h_{i}$, we use the execution time of ten representative sub-models on the
worker node. Firstly, for a model with $l$ layers, there are totally
$M=l(l+1)/2$ sub-models. We sort all these sub-models according to their FLOPs
and choose ten sub-models whose indices satisfy $k*(M-1)/9,k=0,1,...,9$, as
the representative sub-models. Then for an edge device, we put the execution
time of the ten sub-models into a vector of length 10, which is $h_{i}$. Our
experiments show that the process described above requires only a negligible
amount of time.

To pre-train the regression model $f(s,h_{i})$, we construct the training
dataset $D=\\{(s,h_{i},T_{\mathrm{e},s}^{i})\\}$ by collecting data from
various edge devices including edge servers, laptops and desktop personal
computers (PC), whose specifications will be elaborated in the experiments.
Note that the elements in the encoding vector are discrete, while those in the
hardware performance vector are continuous. Hence, we use a dual-stream
regression model inspired by Maple[22]. Specifically, the first stream
processes the sub-model encoding vector through two fully-connected layers,
the output of which will be concatenated with the hardware performance vector
in the second stream and processed through additional layers. Finally, the
output of the second stream is the predicted execution time of the sub-model.

Dynamic Prediction. During the edge pipeline parallel training, the latency
predictor dynamically adjusts its predicted execution time based on the
feedback of the exact execution time of each worker node, which will be
introduced in the following. At the beginning of the edge pipeline parallel
training, the central node $d_{0}$ requires all worker nodes to send $s$ and
$h$ to $d_{0}$. After receiving $s_{j}$ and $h_{j}$ from $d_{j}$, the latency
predictor calculates $T_{\mathrm{e},s}^{j}$. Similar to FTPipeHD[10], each
$d_{j}$ periodically sends its exact execution time of the local sub-model $s$
back to $d_{0}$ during the training. Different from FTPipeHD where this
execution time is used to estimate the computing capacity of $d_{i}$, the
latency predictor computes the difference between the received exact execution
time and the estimated one. If the difference is larger than a pre-defined
threshold $\epsilon$, it means that the computing resources of $d_{j}$ have
changed substantially and $h_{j}$ should be recalculated. Based on the updated
$h_{j}$, the execution time of the sub-model on $d_{j}$ is predicted again and
thus the optimal model partition point is recomputed.

Adaptive Update. To adapt to unseen devices encountered during the training,
the proposed latency predictor will adaptively update itself. Specifically,
the latency predictor collects the exact execution time
$\hat{T}_{\mathrm{e},s}^{i}$ periodically sent by each worker node to
construct a new dataset $\hat{D}=\\{(s,h_{i},\hat{T}_{\mathrm{e},s}^{i})\\}$,
where $s$ and $h_{i}$ is sent to $d_{0}$ at the beginning of the edge pipeline
parallel training. Then it performs the continuous learning to update itself
using $\hat{D}$. As our experiment results show, even with eight samples, the
proposed predictor can adapt well to unseen devices.

###  3.3 Bit-level Fast Compression Scheme

To reduce the transmission latency between nodes, the proposed bit-level fast
compression scheme compresses the data before transmission. The proposed
scheme consists of an adaptive quantizer and a bit-wise encoder.

The adaptive quantizer aims to reduce the transmitted data size by
transforming the floating-point data into a low $k$-bit integer
representation. However, in the modern computer system, the minimum number of
bits to store an integer is eight. Even the boolean type data is typically
represented with eight bits in mainstream programming languages like C++ and
Python due to the memory alignment requirements. Therefore, if $k$ is smaller
than 8, each value of the transmitted data will contain $8-k$ redundant bits.
One straightforward method is to concatenate several quantized values into one
8-bit integer. However, if 8 cannot be divided by $k$, there would still be
some redundant bits. To solve this problem, we adopt multi-bit quantization
(MBQ)[23] for the latent features, where the value is quantized into a linear
combination of several one-bit values, making the quantized values easier to
concatenate. For the gradients to be transmitted, we use the stochastic
uniform quantization to balance the computation costs between the forward and
backward pass as the calculation of gradients is usually more computation-
heavy[24]. Additionally, to reduce the transmitted data size as well as
guarantee model convergence, the bit width $k$ is adaptively adjusted as the
training proceeds.

After applying the adaptive quantizer to the transmitted data, the bit-wise
encoder concatenates the quantized data along the batch size dimension due to
the fact that batch size is typically a power of 2. After encoding the
quantized data, there would be no redundant bits. We introduce the forward
quantizer, forward encoder, backward quantizer and backward encoder separately
in the following.

Forward Quantizer. Similar to section 2.2, assume that the latent features
$\boldsymbol{x}\in\mathbb{R}^{n\times w\times h\times c}$ is a four-
dimensional tensor. We first obtain a vectorized form of $\boldsymbol{x}$,
denoted by $\boldsymbol{x_{\mathrm{v}}}\in\mathbb{R}^{N\times 1}$, where
$N=n\times w\times h\times c$. We then exploit MBQ[23] to quantize each
element into a $k$-bit integer. More specifically, the quantized feature
$\boldsymbol{x_{\mathrm{v}}}$ is approximated by the linear combination of $k$
binary bases, $\boldsymbol{\hat{x}_{\mathrm{v}}}$, given as,

$\boldsymbol{x_{\mathrm{v}}}\approx\boldsymbol{\hat{x}_{\mathrm{v}}}=\boldsymbol{B_{\mathrm{v}}\alpha},$
(1)

where $\boldsymbol{B_{\mathrm{v}}}\in\\{-1,+1\\}^{N\times k}$ is a binary
matrix, and $\boldsymbol{\alpha}\in\mathbb{R}^{k\times 1}$ is the coefficient
vector. Since every element of $\boldsymbol{B_{\mathrm{v}}}$ is only 1 bit,
eight of them can be combined into an 8-bit integer, which allows us to do
bit-wise encoding easily. The optimal $\boldsymbol{B^{*}}$ and
$\boldsymbol{\alpha^{*}}$ are obtained by minimizing the quantization error:

$\boldsymbol{B_{\mathrm{v}}^{*}},\boldsymbol{\alpha^{*}}=\mathop{\arg\min}\limits_{\boldsymbol{B_{\mathrm{v}}},\boldsymbol{\alpha}}\|\boldsymbol{x_{\mathrm{v}}}-\boldsymbol{B_{\mathrm{v}}\alpha}\|^{2}.$
(2)

However, we note that the optimization problem given in (2) is NP-hard[25].
Hence, we calculate the $\boldsymbol{B_{\mathrm{v}}^{*}}$ and
$\boldsymbol{\alpha^{*}}$ alternatively at each epoch, following the idea in
[26]. When training batch 0 of every training epoch, we initialize the
$\boldsymbol{B^{0}_{\mathrm{v}}}$ and $\boldsymbol{\alpha^{0}}$ by the
residue-based method proposed in [27]. In this residue-based method, we
iteratively calculate them by minimizing the error between the residue and the
linear combination $\boldsymbol{B^{0}_{\mathrm{v}}}\boldsymbol{\alpha^{0}}$,
where the residue is defined as the error between the original data and the
$\boldsymbol{B^{0}_{\mathrm{v}}}\boldsymbol{\alpha^{0}}$ of the previous
iteration. Then, we denote the vectorized form of the intermediate feature as
$\boldsymbol{x_{\mathrm{v}}^{t}}$, and the set of all the possible linear
combinations of a given $\boldsymbol{\alpha}^{t}$ and binary bases as
$\mathbb{P}^{t}$, where $t$ refers to the batch $t$. For example, if $k=2$,
$\boldsymbol{\alpha}^{t}=(\alpha_{1},\alpha_{2})$, we have
$\mathbb{P}^{t}=\\{-\alpha_{1}-\alpha_{2},\alpha_{1}+\alpha_{2},-\alpha_{1}+\alpha_{2},\alpha_{1}+\alpha_{2}\\}$.
When training batch $t$, we first determine the
$\boldsymbol{B_{\mathrm{v}}^{t}}$ by finding the value in $\mathbb{P}^{t-1}$
that is closest to each value in $\boldsymbol{x_{\mathrm{v}}^{t}}$, which can
be done by sorting $\mathbb{P}^{t-1}$ and performing element-wise binary
search on each value in $\boldsymbol{x_{\mathrm{v}}^{t}}$. This can be further
speeded up by tensor-level operation that treats
$\boldsymbol{x_{\mathrm{v}}^{t}}$ as a tensor in practical implementation
enabled by coding libraries like Pytorch or Tensorflow. Experiments show that
the tensor-level search is 10 times faster than the element-wise binary
search. With $\boldsymbol{B^{t}_{\mathrm{v}}}$ determined, we then update
$\boldsymbol{\alpha}^{t}$ as follows, which is based on the closed-form
solution to (2):

$\boldsymbol{\alpha}^{\mathrm{cur}}=(\boldsymbol{{B^{t}_{\mathrm{v}}}}^{\mathrm{T}}\boldsymbol{B_{\mathrm{v}}^{t}})^{-1}\boldsymbol{B^{t}_{\mathrm{v}}}^{\mathrm{T}}\boldsymbol{x_{\mathrm{v}}^{t}},$
(3)

$\boldsymbol{\alpha}^{t}=\beta\boldsymbol{\alpha}^{t-1}+(1-\beta)\boldsymbol{\alpha}^{\mathrm{cur}},$
(4)

where $\beta=\mathop{\min}(0.9,(1+t)/(10+t))$.

As shown in [28], quantization may introduce the biased error to the data,
which distorts the distribution of the original data, and may hurt the
convergence of the training. Hence, inspired by [28], we further introduce a
bias correction to the quantized data. Note that the quantization can be
viewed as a noise added to the original data, i.e.,
$\boldsymbol{\hat{x}}=\boldsymbol{x}+n_{\mathrm{q}}$, we have

$\displaystyle\mathbb{E}[\boldsymbol{\hat{x}}]$
$\displaystyle=\mathbb{E}[\boldsymbol{x}+n_{\mathrm{q}}]$ (5)
$\displaystyle=\mathbb{E}[\boldsymbol{x}]+\mathbb{E}[n_{\mathrm{q}}],$

where we observe that we can subtract the mean of the quantization error
$n_{\mathrm{q}}$, i.e., $\mathbb{E}[n_{\mathrm{q}}]$, from the quantized data
to eliminate the biased error. Note that in our proposed scheme, the
dequantization is performed on the next device. Therefore, the forward
compression scheme calculates the $\mathbb{E}[n_{\mathrm{q}}]$ and transmits
it to the next device, which is then subtracted from the dequantized data.

The bit width $k$ is adaptively adjusted as the training proceeds. At the
beginning of the training, we use a small $k$, which is 2, to reduce the
communication latency, such that accelerating the convergence of the DNN
model. We increase the $k$ by 1 every time the learning rate of the model is
altered at a certain epoch according to the optimizer strategy, up to an upper
limit of 4 bits. We note a larger bit width reduces the quantization error,
and hence helps the convergence of the model in the later stage of training.
Experimental results demonstrate that the model accuracy of the proposed
adaptive bit width outperforms that of the fixed 2, 3 and 4 bit width and even
the one without quantization.

Note that in DNN model training, all the operations in the forward pass should
be differentiable so that the gradients can be calculated in the backward
pass. The quantization operation, however, is not differentiable. To tackle
this issue, we adopt the generally-used straight through estimator (STE)[29]
to approximate the gradients of the quantization operation for simplicity.
Specifically, suppose that
$\boldsymbol{\hat{x}_{\mathrm{v}}}=q(\boldsymbol{x_{\mathrm{v}}})$ and
$I(\cdot)$ is the identity function, then in the backward pass, we have

$\frac{\partial\boldsymbol{\hat{x}_{\mathrm{v}}}}{\partial\boldsymbol{x_{\mathrm{v}}}}=\frac{\partial
q(\boldsymbol{x_{\mathrm{v}}})}{\partial\boldsymbol{x_{\mathrm{v}}}}\approx\frac{\partial
I(\boldsymbol{x_{\mathrm{v}}})}{\partial\boldsymbol{x_{\mathrm{v}}}}=1.$ (6)

Forward Encoder. After quantizing the latent features, the forward encoder
concatenates the binary bases along the first dimension. Firstly, since -1
requires at least 2 bits to represent, we replace all the -1s in binary bases
with 0 before encoding. We then concatenate eight elements of $\boldsymbol{B}$
into one 8-bit integer. More specifically, the binary vector
$\boldsymbol{B_{\mathrm{v}}}$ is first reshaped back to $\boldsymbol{B}$ of
size $n\times w\times h\times c\times k$. We then combine every consecutive
eight binary elements along the first dimension into an 8-bit integer, which
results in a tensor of size $\frac{n}{8}\times w\times h\times c\times k$,
denoted by $\boldsymbol{B_{\mathrm{c}}}$. Fig. 6 gives a simple example of
encoding an $8\times 1\times 1\times 1\times 4$ binary tensor into an $1\times
1\times 1\times 1\times 4$ one. In this example, we first replace all the -1s
with 0s. Then in the red-boxed part, eight binary numbers are encoded into one
8-bit integer, that is, $(00110110)_{2}=54$. Moreover, the proposed encoding
method can be viewed as an encryption process, which enhances data privacy to
a certain extent.

Figure 6: An example of the forward encoder.

Backward Quantizer. For the gradients to be transmitted, we aim to quantize
the floating-point gradients $\boldsymbol{g}\in\mathbb{R}^{n\times w\times
h\times c}$ to $\boldsymbol{g_{\mathrm{q}}}\in\mathbb{Z}^{n\times w\times
h\times c}$ in $k$-bit integer representation. More specifically, we first
scale the $\boldsymbol{g}$ into the range $[-2^{k-1},2^{k-1}-1]$ by dividing
it with the scaling factor $s=\frac{c}{2^{k-1}-1}$ with
$c=\mathbb{\max}|\boldsymbol{g}|$. Next, we employ stochastic rounding instead
of the nearest rounding, which is a more common rounding technique. This is
due to the fact that with the nearest rounding, small gradients are always
rounded to zero, which may cause gradient vanishing. Instead, the stochastic
approach rounds the data upwards or downwards with a certain probability,
i.e.,

$\displaystyle\boldsymbol{g_{\mathrm{q}}}=$
$\displaystyle\lfloor\boldsymbol{g}/s\rfloor,$ w.p. $1-pr$ (7a)
$\displaystyle\boldsymbol{g_{\mathrm{q}}}=$
$\displaystyle\lfloor\boldsymbol{g}/s\rfloor+1,$ w.p. $pr$, (7b)

where $pr=\boldsymbol{g}/s-\lfloor\boldsymbol{g}/s\rfloor$. Numerical
analysis[30] has also revealed that stochastic rounding is an unbiased
rounding scheme.

The bit width is also adaptively adjusted in the backward quantizer as
follows. Initially, we set bit width $k$ to 8, which then drops to 4 when the
learning rate changes. Note that the quantization bit width of the gradients
reduces as the training proceeds in contrast to the case in the forward
compression. This is due to the fact that at the beginning of training,
gradients change more significantly so that more accurate gradients are needed
for the model to learn. As the training proceeds, the gradients get smaller
when the model converges. Hence, a small bit width is sufficient.

Backward Encoder. Similar to the forward encoder, to make full use of the 8
bits of an integer, we encode the quantized gradients
$\boldsymbol{g_{\mathrm{q}}}$ into $\boldsymbol{g_{\mathrm{c}}}$. If the bit
width $k$ equals 8, the $\boldsymbol{g_{\mathrm{q}}}$ will be directly
transmitted without encoding, i.e.,
$\boldsymbol{g_{\mathrm{c}}}=\boldsymbol{g_{\mathrm{q}}}$. For $k=4$, we add 8
to each value of $\boldsymbol{g_{\mathrm{q}}}$, which moves the data range
from $[-2^{4-1},2^{4-1}-1]$ to $[0,15]$. Then similarly, we combine two
tensors along the first dimension into one tensor. In this case, the quantized
gradients $\boldsymbol{g_{\mathrm{q}}}$ reduces to
$\boldsymbol{g_{\mathrm{c}}}$ of size $\frac{n}{2}\times w\times h\times c$.

Transmission Data Comparison. Now we compare the size of transmission data
before and after the compression scheme. For the forward run, before
compressing the features, the latent features
$\boldsymbol{x}\in\mathbb{R}^{n\times w\times h\times c}$ to be transmitted
are in 32-bit floating-point representation, with a total size of $(n\times
w\times h\times c\times 32)$ bits. After compression, the transmission data is
comprised of the coefficient vector $\boldsymbol{\alpha}\in\mathbb{R}^{k\times
1}$, the encoded binary bases
$\boldsymbol{B_{\mathrm{c}}}\in\mathbb{Z}^{\lceil\frac{n}{8}\rceil\times
w\times h\times c\times k}$ and the mean of the quantization error
$m_{\mathrm{q}}\in\mathbb{R}$, where the total size equals
$(32k+\lceil\frac{n}{8}\rceil\times w\times h\times c\times k\times 8+32)$.
The compress ratio $c_{\mathrm{f}}$ is given by

$\displaystyle c_{\mathrm{f}}=\frac{n\times w\times h\times c\times
32}{\lceil\frac{n}{8}\rceil\times w\times h\times c\times k\times 8+32(k+1)}.$
(8)

Note that if the size of $\boldsymbol{x}$ is large enough and $n$ is usually a
multiple of 8, the compression ratio is approximately $\frac{32}{k}$.
Similarly, for the gradients, the compressed transmission data contains the
encoded gradients $\boldsymbol{g_{\mathrm{c}}}$ and a 32-bit scaling factor
$s\in\mathbb{R}$, with the total size being $\frac{n}{8/k}\times w\times
h\times c\times 8+32$, where $k$ equals 4 or 8. Thus the compress ratio
$c_{\mathrm{b}}$ is also approximately $\frac{32}{k}$.

###  3.4 Bit-level Fast Decompression Scheme

After encoding the quantized data, the data is transmitted to the next edge
device, where the received data is then decompressed by the bit-level fast
decompression scheme that performs the reverse operation to the compression.
The decompression consists of a decoder and a dequantizer. Similarly, we
introduce the forward decoder, forward dequantizer, backward decoder and
backward dequantizer separately.

Forward Decoder. To decode the received $\boldsymbol{B_{\mathrm{c}}}$, we
extract eight 1-bit values from each 8-bit integer of
$\boldsymbol{B_{\mathrm{c}}}$. We then perform the minus recovery, which
replaces all the 0s to -1 to obtain the original binary bases
$\boldsymbol{B}$.

Forward Dequantizer. After decoding, we have the coefficient vector
$\boldsymbol{\alpha}\in\mathbb{R}^{k\times 1}$, the binary bases
$\boldsymbol{B}\in\\{-1,+1\\}^{n\times w\times h\times c\times k}$ and the
mean of the quantization error $m_{\mathrm{q}}\in\mathbb{R}$. The dequantized
data $\boldsymbol{\hat{x}}$ can be simply obtained by matrix multiplication
$\boldsymbol{B_{\mathrm{v}}\alpha}$, where $\boldsymbol{B_{\mathrm{v}}}$ is
the vectorized form of $\boldsymbol{B}$. We further subtract
$\boldsymbol{\hat{x}}$ by $m_{\mathrm{q}}$ to correct bias as mentioned in
section 3.3, which is then fed into the local sub-model to perform the forward
run at the current worker node.

Backward Decoder. If the current bit width is 8, the decoding phase is
skipped. For $k=4$, the gradient decoder decodes the
$\boldsymbol{g_{\mathrm{c}}}$ simply by dividing each 8-bit integer into two
4-bit integers and the minus values in $\boldsymbol{g_{\mathrm{q}}}$ are then
recovered by subtracting each element by 8.

Backward Dequantizer. The $\boldsymbol{g_{\mathrm{q}}}$ is dequantized to
$\boldsymbol{\hat{g}}$ by multiplying it with the scaling factor $s$, i.e.,
$\boldsymbol{\hat{g}}=\boldsymbol{g_{\mathrm{q}}}*s$. The decompressed
gradients $\boldsymbol{\hat{g}}$ is an approximated version of the original
gradients $\boldsymbol{g}$, both of which are in floating-point
representation. The $\boldsymbol{\hat{g}}$ is then used for back propagation
on the current device.

###  3.5 Optimal Model Partitioning

After accurately predicting the execution time on each node and reducing the
transmitted data size between nodes, we accelerate the edge pipeline-parallel
training by finding the optimal model partition point. Following FTPipeHD, we
calculate the optimal model partition point by solving a dynamic programming
(DP) problem. Let $A(L,N)$ denote the optimal time used by the slowest node
among $N$ nodes to collaboratively train a $L$-layer model. We start with
$N=1$, i.e., only $d_{0}$ participate in the training, that is,

$A(l,1)=T_{\mathrm{e}}^{0}(1,l),\forall l\in[1:L],$ (9)

where $T_{\mathrm{e}}^{0}(a,b)=\sum\limits_{j=a}^{b}T_{\mathrm{e},j}^{0}$.
Then we find $A(L,N)$ by iteratively calculating the following at each step:

$\displaystyle A(l,n)=\min\limits_{1\leq p<l}\max\begin{cases}A(p,n-1)\\\
2\times T_{\mathrm{c},p}^{n-2}\\\
T_{\mathrm{e}}^{n-1}(p+1,l)\end{cases}\forall l\in[n:L],$ (10)

where $n$ is increased by one at each iteration until $n=N$, and
$T_{\mathrm{c},j}^{i}$ denotes the communication time of sending the output of
the $j$-th layer from $d_{i}$ to $d_{i+1}$. After calculating $A(L,N)$, the
optimal partition point is then constructed by a backtracking method, which
can be referred to FTPipeHD for more details. When calculating $A(l,n)$, we
note that there are two main differences compared with the approach used in
FTPipeHD. Firstly, the $T_{\mathrm{e}}^{n-1}(p+1,l)$ in AccEPT is calculated
by the latency predictor and more accurate, while it is calculated by the
estimated computing capacity in FTPipeHD. Secondly, we reduce the transmission
data with the proposed data compression scheme, leading to a smaller
$T_{\mathrm{c},j}^{i}$ used in this work. As our experiments show later, the
proposed acceleration scheme can acquire a better model partition point and
further reduce the total training time.

##  4 Evaluation

###  4.1 AccEPT Implementation

We implement the proposed AccEPT on off-the-shelf devices with Pytorch of
version 1.4.0. Note that any version higher than 1.4.0 may not be feasible due
to the lack of support for reloading the weights during the training, which is
necessary for the pipeline parallel training. We employ the commonly used
lightweight Flask of version 1.1.2 to transmit data between devices. The
implementation of the asynchronous pipeline-parallel training uses a condition
lock to strictly follow the 1F1B rule.

###  4.2 Numerical Evaluation of the Adaptive Latency Predictor

To construct the dataset $D$ used for pre-training the latency predictor, we
gather data from diverse devices, including CPUs such as Intel(R) Xeon(R) CPU
E5-2640, AMD EPYC 7402, and Huawei Kunpeng 920. To improve the generalization
performance of the predictor, we simulate various levels of computing
capacities by the taskset tool in Linux, which controls the number of
processor cores executing a given process on the device. We measure the
latency of all possible sub-models on each device, yielding around 5000
samples over 24 different resource configurations.

(a)  (b)  (c)  (d)

Figure 7: Performance comparisons between different latency prediction methods
with different numbers of CPU cores used.

####  4.2.1 Prediction Accuracy Comparison

We first evaluate the prediction accuracy of our latency predictor, where we
use MNIST to train MobilenetV2[3] as the edge pipeline-parallel training task.
We use the widely used FLOPs-based method[31, 32] and loop up table(LUT)-based
method[33, 34] as the baselines. The FLOPs-based method assumes that a model’s
FLOPs and its execution time are linearly correlated. Thus by utilizing this
linear correlation, the execution latency of a model that has not been
executed yet can be predicted. On the other hand, the LUT-based method
involves pre-computing and storing the execution latency of each operator in a
model into a table. It then looks up the execution latency of each operator
from the table and sums up the overall execution latency of a model. For
comparison, We take the Intel(R) Xeon(R) Gold 6133 as the unseen device that
is not used in the training, and conduct experiments with its number of CPU
cores limited to 1, 2, 3, and 4, respectively. We then employ FLOPs-based,
LUT-based, and our own latency predictor method to predict the execution
latency of every sub-model of MobileNetV2 in this evaluation. We also record
actual execution latency to calculate the accuracy of the predictions. The
accuracy is calculated based on the ±5% and ±10% error-bound of the latency,
both of which are widely used in the literature[35]. Moreover, to ensure the
correctness of the collected latency data, we measure the execution time of
each sub-model 100 times and take the average. The experimental results are
shown in Fig. 7, with each sub-figure representing a different number of CPU
cores. As it can be seen in Fig. 7, the latency predicted by the FLOPs is
significantly inaccurate under all resource conditions. The LUT-based
prediction method shows a performance decline as the number of CPU cores
increases, which can be attributed to that computation optimizations at the
hardware or runtime level occur when more cores are used. Our latency
predictor consistently outperforms the other two methods and demonstrates
greater stability across all cases. It achieves an accuracy of approximately
88% with a ±5% error-bound and over 95% with a ±10% error-bound on average.

Next, we evaluate the adaptive ability of the predictor, where we measure the
prediction accuracy of the predictor after updating it by a new dataset
$\hat{D}$. We adopt a cross-validation approach to avoid impact from different
devices. For each iteration, we select one device as the unseen device and
construct a new dataset $\hat{D}$ with it, while the other devices are used to
generate the training dataset that pre-trains the latency predictor. We vary
the total number of samples in the $\hat{D}$ from 2 to 14. These samples are
selected from the hardware performance samples mentioned in 3.2. If there are
no sufficient samples, we randomly sample the execution time of all sub-models
and add the results to $\hat{D}$. The execution time of the sub-models that
are not in $\hat{D}$ are then used as the test dataset. Then we combine the
$\hat{D}$ with the pre-training dataset $D$ to continuously train the pre-
trained latency predictor, and measure the prediction performance with the
test dataset. We finally aggregate the results from all the iterations and
calculate the average prediction accuracy, which is demonstrated in Fig. 8.

From the results shown in Fig. 8, we can see that the performance of the
latency predictor continuously improves as the number of samples in the new
dataset $\hat{D}$ increases. The predictor eventually stabilizes around 90%
accuracy under a ±5% error-bound and around 96% accuracy under a ±10% error-
bound. This demonstrates that our latency predictor can be adapted to unseen
devices with relatively few new training samples. Specifically, the results
indicate that the performance of the latency predictor convergences after
retraining the predictor with only eight new samples. However, in a relatively
long period, the resources of each device, such as the device’s battery status
and the computation workloads, keep varying, which can affect the execution
time of each sub-model. Hence, we periodically collect the sub-model’s
execution time from each device and use these newly collected data to update
the latency predictor, which helps the predictor adapt to the time-varying
resources on the devices.

Figure 8: Evaluations of the prediction performance by the adaptively updated
latency predictor with respect to different numbers of adaptive samples.

####  4.2.2 Acceleration by the Latency Predictor

Next, we compare the edge pipeline parallel training time leveraging our
latency predictor with two benchmarks, i.e., FTPipeHD and the edge pipeline
parallel training with average partitioning. To enhance the reliability of the
results, we consider three different scenarios for experiments where different
devices are used as the central nodes and the work nodes, which are shown in
Table I.

TABLE I: Three scenarios with different device as the central node Scenario No. | Central Node(Num of cores) | Worker Node(Num of cores)
---|---|---
1 | Xeon E5-2640(2) |  | Xeon Gold 6133(1)
---
Xeon Gold 6133(1)
2 | Xeon E5-2640(3) |  | Kunpeng 920(1)
---
Xeon Silver 4210R(1)
3 | AMD EPYC 7402(4) |  | Xeon E5-2640(3)
---
Kunpeng 920(2)
Xeon Gold 6133(1)
TABLE II: The model partition points for different methods Method | Partition Point
---|---
| Scenario 1 | Scenario 2 | Scenario 3
Average Partition | (7,14) | (7,14) | (5,10,15)
FTPipeHD | (8,14) | (10,13) | (8,12,16)
Our Method | (7,13) | (11,15) | (8,13,16)
Method | Training Time(min)
---|---
| Scenario 1 | Scenario 2 | Scenario 3
Average Partition | 44.44 | 41.72 | 27.07
FTPipeHD | 42.26 | 38.46 | 26.22
Our Method | 38.69 | 35.42 | 26.19

We list the model partition points calculated by the three methods in Table.
II and show the training time for the first ten epochs in Table. 4.2.2. As can
be observed from Table. II and Table. 4.2.2, our latency predictor achieves a
significant acceleration on training in Scenario 1 and Scenario 2, and a
slight improvement in Scenario 3. This is because the overall computing
capacities of the devices in Scenario 3 are much more than the other two
scenarios, and there is still a surplus of computing resources after each
device has allocated resources for training their local sub-models. Therefore,
each device is able to execute its local sub-model in a relatively short
period regardless of partitioning points. However, in scenarios where the
overall computing capacities of devices are low or there is a large difference
in computing capacity between devices, such as Scenario 1 and Scenario 2, it
is important to balance the computing load in order to accelerate the overall
training.

###  4.3 Numerical Evaluation of the Bit-level Fast Compression Scheme

Next, we evaluate the performance of the proposed bit-level fast compression
scheme in accelerating pipeline training. We choose two representative DL
applications, i.e., the image classification task and the human activity
recognition task. For the image classification task, we use the CIFAR10
dataset and train a MobileNetV2 model. For the human activity recognition
task, we use the UCI-HAR dataset[36] collected from smartphone’s inertial
sensors and train a simple five-layer Convolutional Neural Network (CNN) model
proposed in [37].

####  4.3.1 Communication Overhead Comparison

We first compare the size of the transmitted data before and after applying
our proposed compression scheme, where the corresponding compression ratio
(CR) is also calculated. The data compression results are shown in Table III.
It can be observed from Table III that even for the layer in MobileNetV2 that
outputs the minimum amount of data, it still needs to transmit 80.47KB if no
compression is applied. The experimental results show that the proposed
compression scheme achieves nearly a compression ratio of $\frac{32}{k}$ with
$k$-bit quantization (i.e., the _$k$ -bit comp._) for most cases. The gap
between this theoretical compression ratio and the actual one is caused by
transmitting the additional data, e.g. the coefficients $\boldsymbol{\alpha}$,
and information needed for the distributed parallel training including the
weight version and the batch id. This gap becomes larger if the size of the
features is smaller since the size of the additional data and the distributed
parallel training information are getting not negligible. Note that with
quantization, we can only achieve a compression ratio of at most 4 even if we
quantize a 32-bit floating point into an integer with less than 8 bits, since
the minimum number of bits to represent a value in mainstream programming
languages is 8. Moreover, if we use the MBQ without encoding, as shown in the
_$k$ -bit w/o enc._, the compression ratio is usually lower than 4. This is
because each value is decomposed into $k$ binary bases and each binary value
is stored and transmitted as an 8-bit integer. These results validate the
significance of the proposed encoder in reducing the amount of transmitted
data, i.e., communication overhead, together with the proposed quantizer.

TABLE III: The transmission data size and the compression ratio comparison

| Largest Latent Output | Smallest Latent Output
---|---|---
| MobileNetV2 | Simple CNN | MobileNetV2 | Simple CNN
|  | Size
---
(KB)
CR |  | Size
---
(KB)
CR |  | Size
---
(KB)
CR |  | Size
---
(KB)
CR
Original | 4096 | 1.00 | 2016 | 1.00 | 80.47 | 1.00 | 25.45 | 1.00
2-bit comp.1 | 257.1 | 15.93 | 127.1 | 15.86 | 6.09 | 13.20 | 2.65 | 9.62
2-bit w/o enc.2 | 2049 | 2.00 | 1009 | 2.00 | 41.08 | 1.96 | 13.57 | 1.88
3-bit comp. | 385.1 | 10.63 | 190.1 | 10.61 | 8.60 | 9.35 | 3.43 | 7.41
3-bit w/o enc. | 3073 | 1.33 | 1513 | 1.33 | 61.08 | 1.32 | 19.82 | 1.28
4-bit comp. | 513.1 | 7.98 | 253.1 | 7.97 | 11.1 | 7.25 | 4.22 | 6.03
4-bit w/o enc. | 4097 | 1.00 | 2017 | 1.00 | 81.09 | 0.99 | 26.08 | 0.98
uniform comp.3 | 512.5 | 7.99 | 252.5 | 7.98 | 10.51 | 7.66 | 3.62 | 7.02
uniform w/o enc. | 1024 | 4.00 | 504.5 | 4.00 | 20.49 | 3.93 | 6.73 | 3.78

  * 1

The $k$-bit comp. refers to the $k$-bit forward compression.

  * 2

The $k$-bit w/o enc. refers to the $k$-bit forward quantizer without the
encoder.

  * 3

The uniform comp. refers to the 4-bit uniform backward compression.

Next, we measure the actual network latency of transmitting the latent
features of MobileNetV2. We conduct experiments by transmitting the latent
features from a MacBook Pro to a desktop PC using a real WiFi connection. The
network latency is measured as the time interval between the start of data
transmission from the transmitting device and the successful reception of data
by the receiving device. The WiFi network used is under a commonly used
network bandwidth ranging from 1Mbps to 50Mbps, which is controlled by using
the Network Link Conditioner tool in macOS. We conduct 50 measurements at each
bandwidth and calculate the average network latency. Then we compare the
latency between the data without compression and the data with the fixed
$k$-bit compression, which is shown in Fig. 9 where _Max_ and _Min_ means
sending the intermediate results with the maximal and minimum amount of latent
features, respectively. As the figure shows, it still takes around 1300ms to
transmit the largest latent features without compression even when the
bandwidth is 50Mbps, while the proposed compression significantly reduces the
latency to around 100ms. For the smallest latent features, when the bandwidth
is 50Mbps, the transmission latency reduces from 55ms to about 25ms when 2-bit
compression is used. We can also observe a significant reduction in latency
across different bandwidths. We can note that the proposed compression
significantly reduces the transmission network latency up to 1/13. We also
note that the latencies by compression with different bit widths are getting
closer with the increase of the bandwidth. Hence, when the available bandwidth
is large, we can use 4-bit compression to improve the model accuracy. On the
other hand, when the bandwidth is limited, we can use 2-bit compression to
reduce the transmission latency.

(a)  (b)

Figure 9: Comparison of the transmission network latency between the $k$-bit
compression scheme and the no compression scheme on the MobileNetV2.(a)
Network latency for the largest latent features. (b) Network latency for the
smallest latent features.

####  4.3.2 Model Performance Evaluation

Then, we evaluate the model performance under compression schemes with
different bit widths and compare them with the scheme without compression in
terms of model performance. We first compare our proposed MBQ scheme with
other quantization approaches. Then we compare the performance of the proposed
adaptive scheme with the fixed bit-width schemes. Note here we use three
independent processes on the CPU to simulate three edge devices in the
distributed parallel manner.

We first compare the proposed MBQ scheme with two quantization methods and the
case with no quantization used. One baseline quantization method is the
uniform quantization with the scaling factor $s$ initially set to 0.25 and
updated by its gradients with respect to the training loss. The other is
AMSGrad[25], which prunes the number of $\alpha$ by computing the increment of
the training loss incurred by each $\alpha$. For the proposed scheme and both
baselines, the forward bit width $k$ is set to either 2, 3 or 4, while the
backward bit width is set to 8 bits. The model accuracy on the test dataset is
listed in Table IV, where _No compression_ refers to FTPipeHD[10] without any
data compression.

TABLE IV: The model accuracy comparison under different quantization methods

| 2 Bits | 3 Bits | 4 Bits
---|---|---|---
| CIFAR10 | HAR | CIFAR10 | HAR | CIFAR10 | HAR
No compression | 85.72% | 97.47% | 85.72% | 97.47% | 85.72% | 97.47%
Proposed MBQ | 85.23% | 94.93% | 85.09% | 96.04% | 85.57% | 96.79%
Uniform Quant. | 84.66% | 80.85% | 84.38% | 78.67% | 83.77% | 94.30%
AMSGrad MBQ | 81.67% | 92.84% | 82.01% | 87.02% | 77.93% | 93.90%

As we can observe from Table IV, the proposed MBQ has better testing accuracy
than other quantization methods in both CIFAR10 and HAR. We have also found
that the AMSGrad MBQ sometimes diverges the model during training, resulting
in the degradation of model accuracy. When compared with FTPipeHD, the
proposed MBQ method has a comparable testing accuracy on CIFAR10, while it
suffers a slight accuracy drop on HAR due to the compression. It also can be
observed that as the bit width decreases, the testing accuracy drops on HAR.
However, the testing accuracies under different bit widths on CIFAR10 remain
approximately the same. This implies that the quantization may be more likely
to degrade the model when the model is small.

Next, we evaluate the performance of the adaptive compression scheme, which is
compared to the fixed bit-width and the scheme without compression. We
consider three fixed bit-width compression cases with the bit width in the
forward compression to be 2, 3 and 4, respectively, while the bit width in
backward compression is 8 in all three cases. For the adaptive scheme, we
increase the forward bit width at epoch 130 and 175 in the MobileNetV2 and
epoch 60 and 160 in the simple CNN model from 2 to 4, respectively. We train
both the MobileNetV2 and the simple CNN for a total 200 epochs. Hence we have
the average forward bit width for the training of the two models to be 2.48
and 2.9, respectively. We also decrease the backward bit width at epoch 160
for the MobileNetV2 model and at epoch 120 for the simple CNN model from 8 to
4. Similarly, the average backward bit width is 7.2 and 6.4.

Figure 10: Comparison between the adaptive compression scheme, fixed $k$-bit
scheme and the no compression scheme in terms of model accuracy on the HAR
dataset. Figure 11: Comparison between the adaptive compression scheme, fixed
$k$-bit scheme and the no compression scheme in terms of model accuracy on
CIFAR10 dataset.

Fig. 10 and Fig. 11 illustrate the model accuracy on HAR and CIFAR10
respectively. It can be observed that all the accuracy curves fluctuate
greatly during the training due to the parallel training manner and the
quantization, however, becomes stable as the model converges. For the HAR
dataset, all the schemes with compression suffer from performance degradation
in terms of accuracy compared to the scheme without compression. As for the
CIFAR10 dataset, however, the adaptive compression scheme even outperforms the
one without compression. This may be due to the fact that 2-bit compression
introduces more noise to the data at the beginning of the training, which can
be beneficial for the training. Moreover, as the learning rate decreases, a
higher bit width can help the convergence of the model.

###  4.4 Numerical Evaluation of the AccEPT Scheme

Finally, we evaluate the performance of the AccEPT scheme on accelerating the
edge pipeline parallel training, and compare to the cases with only latency
predictor (LP) applied or only bit-level compression (BLC) applied, and
FTPipeHD. We use Scenario 1 and 2 from Table I as the experimental settings
and train MobileNetV2 on the MNIST dataset as the training task, with the
epochs set to 100. We also measure the training time under bandwidths ranging
from 0.4Mbps to 4Mbps between devices. The evaluation results are demonstrated
in Table V.

From the Table. V, we can see that the AccEPT scheme outperforms other methods
in terms of the training time, with AccEPT achieving acceleration ratios of at
most 3.01 compared to FTPipeHD. Under low-bandwidth conditions where the
communication latency between devices dominates the total training time, our
proposed compression greatly reduces the amount of information to be
transmitted and significantly accelerates the training, with BLC achieving
acceleration ratios of 2.84 and 2.07 compared to FTPipeHD in Scenarios 1 and
2, respectively. As the bandwidth increases, the communication latency
decreases, and computation time becomes the main part in slowing down
training. At this point, the effect of the latency predictor becomes obvious,
with LP achieving acceleration ratios of 1.15 and 1.10 in Scenarios 1 and 2 at
4Mbps compared to FTPipeHD, respectively.

TABLE V: Comparison between AccEPT and other Methods on Training Time (Min)

Scenario# | Method | Bandwidth(Mbps)
---|---|---
0.25 | 0.5 | 1 | 2 | 4
1 | FTPipeHD | 3439.17 | 1755.93 | 894.41 | 620.03 | 488.85
BLC1 | 1212.64 | 952.16 | 546.26 | 577.76 | 466.341
LP2 | 2326.17 | 1280.96 | 628.88 | 455.57 | 423.07
AccEPT | 1139.95 | 711.25 | 437.19 | 423.59 | 403.75
2 | FTPipeHD | 2295.11 | 1235.48 | 621.758 | 484.66 | 397.4
BLC | 1110.78 | 721.68 | 451.04 | 442.07 | 390.12
LP | 2069.48 | 1178.71 | 584.86 | 400.83 | 360.69
AccEPT | 967.86 | 639.66 | 420.15 | 364.24 | 351.41

  * 1

Only the bit-level compression(BLC) is implemented.

  * 2

Only the latency predictor(LP) is implemented.

(a)  (b)

Figure 12: Training time evaluation on edge devices: (a) The Raspberry Pi used
for evaluation; (b) Training Time Comparison between different schemes.

We also deploy our proposed AccEPT to train the simple CNN model on the HAR
dataset with three Raspberry Pi Model 4B, which are widely used edge devices.
We note that due to the memory limitation of the Raspberry Pi, the training of
the MobileNetV2 is not considered here. To simulate the heterogeneous devices
scenario, we use taskset tool to limit the number of CPU cores used in two
devices to 2, where the rest one to 4. The three edge devices communicate with
each other through a wireless local area network(LAN) provided by a mobile
phone, where we conduct the experiments under different bandwidths ranging
from 0.25Mbps to 4Mbps, which is controlled by the tc tool. The actual
deployment is shown in Fig. 12a. Similarly, we compare the training time
between the AccEPT, the cases with only the bit-level compression or only the
latency predictor and the FTPipeHD, which is shown in Fig. 12b.

It can be observed from Fig. 12b that although the size of transmission data
is relatively small and the possible partition points are limited in the
simple CNN compared with the one in the MobileNetV2, AccEPT still provides a
significant training acceleration. Similar to the results in Table. V, when
the bandwidth is low, the time for transmitting the data between devices
dominates the training time, making the data compression contribute more to
the acceleration compared with the latency predictor. As the bandwidth
increases, the training time of the case with only the latency predictor
applied becomes smaller than that with only the bit-level compression,
demonstrating that the latency predictor becomes more important when the
transmission time decreases. We also note that due to the relatively small
amount of transmitted data in the Simple CNN, the benefits of using the data
compression are getting less significant with the increase of bandwidth. The
evaluation results on both the edge servers and the Raspberry Pi devices
validate the effectiveness of the proposed AccEPT scheme in accelerating
pipeline training under different conditions.

##  5 Conclusion

In this paper, we proposed an acceleration scheme to speed up the distributed
pipeline parallel training, called AccEPT. We first introduced an adaptive
light-weight latency predictor to accurately predict the sub-model’s execution
time on each participating device, which helps to better partition the model
across devices. Then we proposed a bit-level compression scheme to reduce the
data size of the latent features transmitted during training, which decreases
the communication overhead. In these two ways, the pipeline parallel training
can be significantly accelerated compared to the existing methods, as
demonstrated by the experimental results on both the edge servers and the
Raspberry Pi devices.

## References

  * [1] Y. Yan, Y. Chen, Q. Yang, and Z. Shi, “Edge collaborative learning acceleration based on latency prediction,” in _2023 International Conference on Ubiquitous Communication (Ucom)_. IEEE, 2023, pp. 254–258.
  * [2] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [3] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [4] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _arXiv preprint arXiv:2107.13586_ , 2021.
  * [5] F. Gu, M.-H. Chung, M. Chignell, S. Valaee, B. Zhou, and X. Liu, “A survey on deep learning for human activity recognition,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 8, pp. 1–34, 2021.
  * [6] J. Konečnỳ, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device intelligence,” _arXiv preprint arXiv:1610.02527_ , 2016.
  * [7] F. Liu, P. Shu, H. Jin, L. Ding, J. Yu, D. Niu, and B. Li, “Gearing resource-poor mobile devices with powerful clouds: architectures, challenges, and applications,” _IEEE Wireless communications_ , vol. 20, no. 3, pp. 14–22, 2013.
  * [8] A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “Pipedream: Fast and efficient pipeline parallel dnn training,” _arXiv preprint arXiv:1806.03377_ , 2018.
  * [9] M. G. Poirot, P. Vepakomma, K. Chang, J. Kalpathy-Cramer, R. Gupta, and R. Raskar, “Split learning for collaborative deep learning in healthcare,” _arXiv preprint arXiv:1912.12115_ , 2019.
  * [10] Y. Chen, Q. Yang, S. He, Z. Shi, J. Chen, and M. Guizani, “Ftpipehd: A fault-tolerant pipeline-parallel distributed training approach for heterogeneous edge devices,” _IEEE Transactions on Mobile Computing_ , 2023\.
  * [11] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu _et al._ , “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” _Advances in neural information processing systems_ , vol. 32, pp. 103–112, 2019.
  * [12] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split learning for health: Distributed deep learning without sharing raw patient data,” _arXiv preprint arXiv:1812.00564_ , 2018.
  * [13] A. Gordon, E. Eban, O. Nachum, B. Chen, H. Wu, T.-J. Yang, and E. Choi, “Morphnet: Fast & simple resource-constrained structure learning of deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1586–1595.
  * [14] X. Zeng, K. Cao, and M. Zhang, “Mobiledeeppill: A small-footprint mobile deep learning system for recognizing unconstrained pill images,” in _Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services_ , 2017, pp. 56–67.
  * [15] G. Zhou, Y. Fan, R. Cui, W. Bian, X. Zhu, and K. Gai, “Rocket launching: A universal and efficient framework for training well-performing light net,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [16] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” _ACM SIGARCH Computer Architecture News_ , vol. 45, no. 1, pp. 615–629, 2017.
  * [17] J. H. Ko, T. Na, M. F. Amir, and S. Mukhopadhyay, “Edge-host partitioning of deep neural networks with feature space encoding for resource-constrained internet-of-things platforms,” in _2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2018, pp. 1–6.
  * [18] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “Jointdnn: An efficient training and inference engine for intelligent mobile cloud computing services,” _IEEE Transactions on Mobile Computing_ , 2019.
  * [19] S. Laskaridis, S. I. Venieris, M. Almeida, I. Leontiadis, and N. D. Lane, “Spinn: synergistic progressive inference of neural networks over device and cloud,” in _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking_ , 2020, pp. 1–15.
  * [20] S. Yao, J. Li, D. Liu, T. Wang, S. Liu, H. Shao, and T. Abdelzaher, “Deep compressive offloading: Speeding up neural network inference by trading edge computation for network latency,” in _Proceedings of the 18th Conference on Embedded Networked Sensor Systems_ , 2020, pp. 476–488.
  * [21] D. Liu, X. Chen, Z. Zhou, and Q. Ling, “Hiertrain: Fast hierarchical edge ai learning with hybrid parallelism in mobile-edge-cloud computing,” _IEEE Open Journal of the Communications Society_ , vol. 1, pp. 634–645, 2020.
  * [22] S. Abbasi, A. Wong, and M. J. Shafiee, “Maple: Microprocessor a priori for latency estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2747–2756.
  * [23] X. Lin, C. Zhao, and W. Pan, “Towards accurate binary convolutional neural network,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [24] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania _et al._ , “Pytorch distributed: Experiences on accelerating data parallel training,” _arXiv preprint arXiv:2006.15704_ , 2020.
  * [25] Z. Qu, Z. Zhou, Y. Cheng, and L. Thiele, “Adaptive loss-aware quantization for multi-bit networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7988–7997.
  * [26] D. Zhang, J. Yang, D. Ye, and G. Hua, “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 365–382.
  * [27] Y. Guo, A. Yao, H. Zhao, and Y. Chen, “Network sketching: Exploiting binary structure in deep cnns,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5955–5963.
  * [28] M. Nagel, M. v. Baalen, T. Blankevoort, and M. Welling, “Data-free quantization through weight equalization and bias correction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1325–1334.
  * [29] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” _arXiv preprint arXiv:1308.3432_ , 2013.
  * [30] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in _International conference on machine learning_. PMLR, 2015, pp. 1737–1746.
  * [31] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl for model compression and acceleration on mobile devices,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 784–800.
  * [32] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” _arXiv preprint arXiv:1608.08710_ , 2016.
  * [33] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 734–10 742.
  * [34] X. Dai, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, Y. Jia _et al._ , “Chamnet: Towards efficient network design through platform-aware model adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 398–11 407.
  * [35] L. Dudziak, T. Chau, M. Abdelfattah, R. Lee, H. Kim, and N. Lane, “Brp-nas: Prediction-based nas using gcns,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 10 480–10 490, 2020.
  * [36] D. Anguita, A. Ghio, L. Oneto, X. Parra Perez, and J. L. Reyes Ortiz, “A public domain dataset for human activity recognition using smartphones,” in _Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning_ , 2013, pp. 437–442.
  * [37] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao, “Fedhealth: A federated transfer learning framework for wearable healthcare,” _IEEE Intelligent Systems_ , vol. 35, no. 4, pp. 83–93, 2020.

|  Yuhao Chen received the Bachelor of Engineering degree from the College of
Information Science and Electronic Engineering, Zhejiang University, in 2019.
He is currently a Ph.D. candidate in cyberspace security at the College of
Control Science and Engineering, Zhejiang University. His research interests
include edge computing systems in IoT and semantic communication.
---|---
|  Yuxuan Yan obtained his Bachelor of Engineering degree from the College of
Information Science and Electronic Engineering at Zhejiang University in 2023.
He is currently pursuing a Master’s degree in the College of Information
Science and Electronic Engineering at Zhejiang University, specializing in the
field of edge computing and semantic communication.
---|---
|  Qianqian Yang (Member, IEEE) received the B.Sc. degree in automation from
Chongqing University, Chongqing, China, in 2011, the M.S. degree in control
engineering from Zhejiang University, Hangzhou, China, in 2014, and the Ph.D.
degree in electrical and electronic engineering from Imperial College London,
U.K. She has held visiting positions at CentraleSupelec in 2016 and the New
York University Tandon School of Engineering from 2017 to 2018. After her
Ph.D., she served as a Post-Doctoral Research Associate for Imperial College
London, and as a Machine Learning Researcher for Sensyne Health Plc. She is
currently a Tenure-Tracked Professor with the Department of Information
Science and Electronic Engineering, Zhejiang University, China. Her main
research interests include wireless communications, information theory,
machine learning and medical imaging. She serves as a Reviewer for IEEE
TRANSACTIONS ON INFORMATION THEORY, IEEE TRANSACTIONS ON COMMUNICATIONS, IEEE
TRANSACTIONS ON WIRELESS COMMUNICATIONS, etc.
---|---
|  Yuanchao Shu is currently a Qiushi Professor with the College of Control
Science and Engineering at Zhejiang University, China. Prior to joining
academia, he was a Principal Researcher with the Office of the CTO, Microsoft
Azure for Operators, and the Mobility and Networking Research Group at
Microsoft Research Redmond. His research interests lie broadly in mobile,
sensing and networked systems. His previous research results have led to over
60 publications at top-tier peer-reviewed conferences and journals including
ACM MobiCom, MobiSys, SenSys, UbiComp, USENIX Security, NSDI, JSAC, TMC, TWC,
TPDS, TKDE etc. Dr. Shu currently serves on the editorial board of IEEE
Transactions of Wireless Communications, ACM Transactions on Sensor Networks,
and was a member of the organizing committee and TPC of conferences including
MobiCom, MobiSys, SenSys, SEC, Globecom, ICC etc. He won five Best Paper/Demo
(Runner-Up) Awards, and was the recipient of ACM China Doctoral Dissertation
Award (2/yr) and IBM PhD Fellowship. Dr. Shu received his Ph.D. from Zhejiang
University, and was also a joint Ph.D. student in the EECS Department at the
University of Michigan, Ann Arbor. Dr. Shu is a senior member of ACM and IEEE.
---|---
|  Shibo He (M’13-SM’19) received the Ph.D. degree in control science and
engineering from Zhejiang University, Hangzhou, China, in 2012. He is
currently a Professor with Zhejiang University. He was an Associate Research
Scientist from March 2014 to May 2014, and a Postdoctoral Scholar from May
2012 to February 2014, with Arizona State University, Tempe, AZ, USA. From
November 2010 to November 2011, he was a Visiting Scholar with the University
of Waterloo, Waterloo, ON, Canada. His research interests include Internet of
Things, crowdsensing, big data analysis, etc. Prof. He serves on the editorial
board for the IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, Springer Peer-to-Peer
Networking and Application and KSII Transactions on Internet and Information
Systems, and is a Guest Editor for Elsevier Computer Communications and
Hindawi International Journal of Distributed Sensor Networks. He was a
Symposium Co-Chair for the IEEE GlobeCom 2020 and the IEEE ICC 2017, TPC Co-
Chair for i-Span 2018, a Finance and Registration chair for ACM MobiHoc 2015,
a TPC Co-Chair for the IEEE ScalCom 2014, a TPC Vice Co-Chair for ANT
2013´lC2014, a Track Co-Chair for the Pervasive Algorithms, Protocols, and
Networks of EUSPN 2013, a Web Co-Chair for the IEEE MASS 2013, and a Publicity
Co-Chair of IEEE WiSARN 2010, and FCN 2014.
---|---
|  Zhiguo Shi (M’10-SM’15) received the B.S. and Ph.D. degrees in electronic
engineering from Zhejiang University, Hangzhou, China, in 2001 and 2006,
respectively. Since 2006, he has been a Faculty Member with the Department of
Information and Electronic Engineering, Zhejiang University, where he is
currently a Full Professor. From 2011 to 2013, he visited the Broadband
Communications Research Group, University of Waterloo, Waterloo, ON, Canada.
His current research interests include signal and data processing. Prof. Shi
serves as an Editor for the IEEE Network, IEEE Transactions on Vehicular
Technology, IET Communications, Journal of The Franklin Institute. He was a
recipient of the Best Paper Award from the ISAP 2020, IEEE GLOBECOM 2019, IEEE
iThings 2019, EIA MILCOM 2018, IEEE WCNC 2017, CSPS 2017, IEEE/CIC ICCC 2013,
IEEE WCNC 2013, IEEE WCSP 2012. He was also a recipient of first prize for Wu
Wenjun AI Science and Technology Award in 2020, first prize for Science and
technology Development of the Ministry of Education in 2015 and second prize
for the Scientific and Technological Award of Zhejiang Province in 2012.
---|---
|  Jiming Chen (Fellow, IEEE) received the PhD degree in control science and
engineering from Zhejiang University, Hangzhou, China, in 2005. He is
currently a professor with the Department of Control Science and Engineering,
the vice dean of the Faculty of Information Technology, Zhejiang University.
His research interests include IoT, networked control, wireless networks. He
serves on the editorial boards of multiple IEEE Transactions, and the general
co-chairs for IEEE RTCSA’19, IEEE Datacom’19 and IEEE PST’20. He was a
recipient of the 7th IEEE ComSoc Asia/Pacific Outstanding Paper Award, the
JSPS Invitation Fellowship , and the IEEE ComSoc AP Outstanding Young
Researcher Award. He is an IEEE VTS distinguished lecturer. He is a fellow of
the CAA.
---|---