captionUnknown document class

# EdgeConvEns: Convolutional Ensemble Learning
for Edge Intelligence

Ilkay Sikdokur, İnci M. Baytaş, and Arda Yurdakul I. Sikdokur is with the
Department of Computational Science and Engineering, Bogazici University,
Istanbul, Turkey (e-mail: ilkay.sikdokur@boun.edu.tr).İ. M. Baytaş and A.
Yurdakul are with the Department of Computer Engineering, Bogazici University,
Istanbul, Turkey (e-mail<EMAIL_ADDRESS>yurdakul@boun.edu.tr).

###### Abstract

Deep edge intelligence aims to deploy deep learning models that demand
computationally expensive training in the edge network with limited
computational power. Moreover, many deep edge intelligence applications
require handling distributed data that cannot be transferred to a central
server due to privacy concerns. Decentralized learning methods, such as
federated learning, offer solutions where models are learned collectively by
exchanging learned weights. However, they often require complex models that
edge devices may not handle and multiple rounds of network communication to
achieve state-of-the-art performances. This study proposes a convolutional
ensemble learning approach, coined EdgeConvEns, that facilitates training
heterogeneous weak models on edge and learning to ensemble them where data on
edge are heterogeneously distributed. Edge models are implemented and trained
independently on Field-Programmable Gate Array (FPGA) devices with various
computational capacities. Learned data representations are transferred to a
central server where the ensemble model is trained with the learned features
received from the edge devices to boost the overall prediction performance.
Extensive experiments demonstrate that the EdgeConvEns can outperform the
state-of-the-art performance with fewer communications and less data in
various training scenarios.

##  I Introduction

Edge computing is vital for next-generation solutions in various industries
such as automotive [1], energy [2], and agriculture [3]. Edge intelligence
offers innovative frameworks for Internet of Things (IoT) applications.
Sensors are used in data collection for safe driving [1], anomaly detection
[2], real-time and predictive analytics [3]. The data-driven problems usually
require computation-intensive solutions, e.g., Deep Neural Network (DNN)
training. Deep edge intelligence, where DNNs are employed on edge, has become
imperative to address complex edge intelligence applications such as image
classification [4], speech recognition [5], and natural language processing
[6]. However, centralized training with millions of parameters is essential to
attain state-of-the-art performances. In contrast, edge intelligence face
problems with distributed data and limited computational resources.
Consequently, two main challenges emerge in deep edge intelligence
applications; data cannot be centralized due to privacy concerns and limited
bandwidth, and edge devices may not supply sufficient memory and computational
requirements to train complex DNNs [7].

Figure 1: Deep edge intelligence setting covered in this study. Cameras
represent edge devices: processing capacity is proportional to the size. Five
different colored shapes represent different classes: the bigger the size is,
the more frequently it is observed by each edge device.

One prominent approach to the distributed and heterogeneous data challenge is
federated learning, a popular approach in IoT solutions [8]. However,
centralized and decentralized federated learning requires multiple
communication rounds to update model parameters increasing the data transfer
rate. Furthermore, federated learning frameworks, particularly with
centralized orchestrator setup, require the clients to have similar
architectures with a sufficient number of parameters to solve complex tasks.
Therefore, the federated learning paradigm does not directly address the lack
of computational resources at the edge of networks. Tiny Machine Learning
(TinyML) is one of the recent techniques to address the low-resource device
challenge. A typical TinyML framework comprises training a machine learning
model on a high-performance computer, compressing the learned model to reduce
memory requirements, and deploying it on an embedded system for inference [7].
Although TinyML solutions can mitigate the memory requirements, data privacy,
and on-device model training challenges remain.

This study aims to improve overall prediction performance under the
distributed data setup by learning to fuse the knowledge acquired from on-
device training on edge with low computational resources. The proposed
solution, EdgeConvEns in Figure 1, combines weak learning at the edge devices
with convolutional ensemble learning at the server. Unlike traditional
federated learning, edge devices can train different models in the EdgeConvEns
framework due to device limitations. Only the learned feature representations
are transferred without the requirement for global updates. The data need not
be equally available at the edge devices. Some classes can be observed by one
device more frequently than others. For example, a camera viewing the right
lane of a highway observes heavy vehicles more frequently than a camera on the
left lane. Unlike TinyML solutions, the proposed framework trains edge models
on edge devices with their local data instead of creating a compressed global
model and sending it back to the edge. The contributions of this study are
outlined as follows:

  * •

Limited training of configurable weak models on heterogeneous edge devices is
achieved on Field-Programmable Gate Array (FPGA) fabric. Autonomously
generating a partially parallelized task-level streaming architecture
constrained with available resources on various edge devices enables
acceleration of neural network operations.

  * •

Variational Auto-Encoder (VAE) models are trained on the server for each edge
model to generate statistically similar features to the ones learned on edge
for replacing the missing features due to disconnection problems in edge
intelligence.

  * •

A convolutional ensemble learning scheme that fuses the underlying information
in the representations learned on edge is proposed to boost overall prediction
performance.

EdgeConvEns can attain satisfactory performances compared to the state-of-the-
art with a much smaller number of parameters. Communication and memory
utilization of the proposed framework are investigated under various data
distribution and training scenarios with benchmark datasets in image
classification and regression.

This work is organized as follows. Selected previous works in edge
intelligence, FPGA acceleration in neural networks, federated learning and
ensemble learning are analyzed in the next section. Section III presents
details of the proposed design and its components. Section IV contains the
implementation details, experimental results and a comparative analysis with
earlier works. The last section concludes this work with a review of the
achievements of the study.

##  II Related Work

The edge intelligence literature showcases various ways of deploying model
training and inference. In some applications, training and inference are on a
cloud server, whereas some applications divide training and inference into
cloud server and edge, respectively. Zhou et al. proposes a data offloading
scale of seven levels based on the design choice of edge intelligence
applications given in Figure 2 [9]. Volume and path length of data unloading
decrease as the level increases. The proposed EdgeConvEns can be placed
between Levels 6 and 7 since the models on edge are independently trained, and
their inference is also independently done on individual edge devices.
However, the features of the inference data obtained from the edge devices are
required on the server device for making the final prediction. Thus, the
proposed approach results in less transmission latency of data offloading,
higher data privacy, and lower communication bandwidth. EdgeConvEns can be
categorized as Device-to-Edge (D2E), except that edge models do not receive
any information from the server for training and inference. When the edge
model training is completed on the edge device, data offload between the
device and server can be less compared to other D2E approaches [10].

Figure 2: Edge intelligence levels by data offloading [9].

As edge intelligence has grown more prevalent for applications like IoT,
decentralized learning techniques on edge have become imperative. One of these
techniques is Federated Learning (FL), a mainstream decentralized learning
method commonly used in IoT. The FL methods encounter several challenges, such
as the high number of communication rounds and weight transfer due to the
iterative updates of local and global models [8]. In addition, all clients and
the central server have the same model architecture, which dictates a minimum
resource requirement for training the models on each edge device [8, 11, 12,
13, 14, 15, 16, 17]. On the other hand, the EdgeConvEns is based on ensemble
learning discarding the exchange of model parameters that causes a
communication bottleneck. In the proposed approach, there is only a one-way
transfer from edge devices to the server in which the overall prediction is
boosted. In recent FL studies [18, 19, 20], training is also done with either
a one-way model parameter or embedding vector transfer, but EdgeConvEns yields
a better prediction accuracy with a lower number of parameters according to
our comparative experimental analysis.

Figure 3: Illustration of the system for the image classification task. Red-
dashed lines denote the successful transfer of edge features. Missing features
are filled by VAE outputs (blue-dashed lines). Green-dashed lines represent
the complete feature set to train the CNN-based ensemble model. The system is
constructed the same for regression problems.

Ensemble methods are broadly used in deep learning. Some recent studies
propose IoT frameworks based on ensemble models [21], where traditional
sampling strategies, such as bagging and boosting, are used. Knowledge
distillation is also utilized in ensemble frameworks for edge intelligence.
Such methods employ an ensemble of logit outputs and gradients of larger
models to train smaller models [22, 23]. EdgeConvEns proposes convolution-
based ensemble learning that fuses feature vectors transferred from edge for
an improved classification performance compared to the traditional and
knowledge distillation-based methods.

Field-Programmable Gate Arrays (FPGA) have recently been employed in edge
intelligence applications to accelerate computationally expensive operations
[24]. The training time of Convolutional Neural Networks (CNN) can be reduced
by accelerating matrix multiplication. These studies focus on the
parallelization of the calculations by certain levels only, such as fully
connected layer calculations [25], convolution layer calculations [26], and
parallelization on minibatch level [27]. The proposed EdgeConvEns partially
parallelizes the computations at each layer by chosen array dimensions,
including minibatch level, channel, and filter level for convolutional
operations, and chosen partition level for fully connected layer operations.
It also allows for the adjustment of resource utilization by changing the
factor of parallelization. Some works also accelerate the training phase by
binarizing [28], quantizing and using smaller data types [29] for the weights.
Even though the same optimizations can also be preferred in EdgeConvEns, the
acceleration of the training is achieved without using any quantization method
or smaller data types in this work.

Figure 4: Two-dimensional t-distributed stochastic neighbour embedding (t-SNE)
plots of the average feature vectors extracted from 20 edge models. The
feature vectors are obtained from the training set of the CIFAR-10 dataset for
two different classes such as (a) automobile and (b) bird. Each symbol
represents a different model.

##  III Proposed System

EdgeConvEns, illustrated in Figure 3, consists of a server and edge devices
with varying hardware capacities that presumably have medium computational
power. The edge models in this study are shallow neural networks with poor
evaluation performances trained for image recognition and regression. The
server ensemble model aims to boost the overall prediction performance by
learning to fuse the edge model embeddings. The server has no computational
hardware restrictions for training a complex ensemble model. The learned
ensemble model makes the final predictions without broadcasting any learned
model back to the edge devices.

First, we independently train the edge models with their training datasets,
which are subsets acquired by random sampling with replacement from a more
extensive training set of independent and identically distributed (i.i.d.)
samples. Thus, edge datasets might have overlapping and distinct samples.
After the training of edge models, learned embeddings of each edge’s training
set are transferred to the server. On the server, we first train a VAE for
each edge model with respective embeddings to capture their distributions. The
trained VAEs are utilized for imputing the missing embeddings by generating
one from the corresponding edge’s VAE. Finally, the ensemble model is trained
with the training embeddings, either the original embeddings from the edge or
generated from the respective edge’s VAE.

###  III-A Edge Models

In the proposed framework, the edge models do not need to share the same model
architecture and are trained independently. However, the dimensionality of the
learned embeddings that are transferred to the server, denoted by $L_{com}$ in
Figure 3, should be the same since the ensemble model on the server applies
convolution on the matrix of edge embeddings. The edge model denoted by
$EM_{i}$ is trained with a subset of the training data denoted by
$D_{train}^{EM_{i}}$, where $D_{train}$ is the main training data. After
training, embeddings of $D_{train}^{EM_{i}}$ denoted by
$F_{train}^{EM_{i}}\in\mathbb{R}^{L_{com}}$ is obtained. The embeddings,
$F_{train}^{EM_{i}}$, are then transferred to the server as illustrated with
red-dashed lines in Figure 3.

The ensemble model expects to receive an embedding from each edge model to
learn how to fuse them to improve the prediction performance. However, there
may be several problems in acquiring the embedding from an edge model in real-
world applications, such as connection problems between edge and server and
data availability on edge. Hence, every edge model may not produce an
embedding vector to transfer to the server for the same input data.

###  III-B Server Models

####  III-B1 Variational Auto-Encoder (VAE) Models

This study proposes to use a generative approach to fill the necessary feature
vectors not received from an edge model ${EM_{i}}$ to obtain the final
prediction of the ensemble model. In Figure 4, the dimensions of the feature
vectors obtained from the layers of 20 edge models for CIFAR-10 are reduced to
two by using the t-distributed Stochastic Neighbour Embedding (t-SNE) method
[30] and their average positions by edge models are plotted. It can be seen
that the edge outputs to be used in the ensemble model follow different
distributions. EdgeConvEns leverages VAE models for imputing the missing
embeddings. VAE models are trained and deployed on the server for each edge
model to generate embeddings sampled from a similar distribution as the edge
embeddings’. In Figure 3, the red-dashed line from our edge device to the VAE
model shows the transfer of the edge embeddings $F_{train}^{EM_{i}}$. Each VAE
model, $VAE_{i}$, is independently trained with respective edge embeddings,
$F_{train}^{EM_{i}}$, to capture the edge models’ embedding distributions. For
this purpose, embeddings received from the edge models are encoded into a
distribution over latent space. When a random sample from the encoder’s
distribution is decoded to reduce the reconstruction error, the VAE’s decoder
starts exhibiting a generative nature. Thus, the trained VAE’s decoder can be
used to generate embeddings, $VAE_{i}^{dec}(z)$, from a random variable in the
latent space, $z\in\mathbb{R}^{d}$.

Figure 5: Illustration of the convolutional ensemble operation for a single
image. $F^{EM_{i}}$: the feature vectors transferred from the $i$-th edge
model. $VAE^{dec}_{i}(z)$: the feature vectors generated by $i$-th VAE model
for the same image.

####  III-B2 Ensemble Model

Since the weak models on edge devices cannot yield state-of-the-art
performances, a convolutional ensemble approach shown by the green model in
Figure 3 is proposed for learning a fusion of embeddings to improve the final
predictive performance. Let $K_{train}$ contain the indices of the samples for
the full training dataset $D_{train}$ and $K_{train}^{EM_{i}}$ contain the
indices for the edge model’s ($EM_{i}$) training dataset,
$D_{train}^{EM_{i}}$. Since $D_{train}^{EM_{i}}\subseteq D_{train}$, it can be
deduced that $K_{train}^{EM_{i}}\subseteq K_{train}$. The training dataset for
the ensemble model, $D^{Ens}_{train}$, is constructed as follows.

${D^{Ens}_{train}[k,i]=\left\\{\begin{matrix}F_{train}^{EM_{i}}&if\ k\in
K_{train}^{EM_{i}}\\\ VAE_{i}^{dec}(z)&if\ k\notin
K_{train}^{EM_{i}}\end{matrix}\right.}.$ (1)

Thus, the ensemble model’s training data, $D^{Ens}_{train}$, of size
$n(D_{train})\times N\times L_{com}$ is constructed by stacking the
embeddings. Figure 3 shows the ensemble training data construction with blue
and green-dashed lines.

Figure 5 illustrates the matrix formed by stacking the representations
obtained from either edge models or VAE for a single sample. The red two-
dimensional frame shows the convolutional kernel to extract a unified
representation of each sample. The motivation behind the convolutional
ensemble is to learn a fusion that complements each edge model’s information
by investigating the inter-edge interactions in the embeddings. Hence, the
kernel size directly affects the prediction performance. In this study, the
architecture of the ensemble model comprises a convolutional layer with a
kernel of size $(\frac{N}{2},\frac{L_{com}}{2})$ and a hidden fully-connected
layer.

##  IV Experimental Results

###  IV-A Implementation

####  IV-A1 Partially Parallelized Streaming in FPGA-based Edge Devices

Inference and training of DNNs heavily require matrix multiplications.
Acceleration of matrix multiplication is performed by the proposed task-level
parallelism, which allows the designer to choose the level of acceleration to
ensure targeted hardware utilization. The proposed method transfers data using
data stream method [31] where the data flow from the source to the destination
module via First-In-First-Out (FIFO) buffers. FIFO buffers allow the consumer
module to use the data when the source module sends the data providing
accelerated computations. The data interface is chosen as AXI4-Stream [32] for
all modules. The task-level streaming method has constraints, such as reading
and writing an array element only once, in addition to the in-order access to
the buffers.

In EdgeConvEns, all of the arrays used in training are manually tiled in
preprocessing phase by the factors decided by the designer. Then, the arrays
are partitioned into tiles during synthesis by the factorized part of the
chosen dimension. The factors need to be decided to fit the edge model into
the chosen edge device by the designer. Assume that the matrices $M_{1}[X][Y]$
and $M_{2}[Y][Z]$ are aimed to be multiplied into the result matrix
$M_{3}[X][Z]$. It can be seen that the $Y$ is the common dimension of
multiplication. Assume that $Y=f*y$ where $f$ and $y$ are positive integers.
Then, the matrix multiplication can be written as

$M_{3_{i,j}}=\sum_{k=1}^{f}\sum_{l=1}^{\frac{Y}{f}}M_{1}[i][k][l]*M_{2}[k][l][j].$
(2)

In this form, the matrices $M_{1}$ and $M_{2}$ are manually tiled into the $f$
factor on the $Y$ dimension. After this modification, the follow-up arrays can
be automatically partitioned by the new factor dimension $f$. Then, the
partitioned independent memory parts are streamed through the system in FIFO
buffers.

In Figure 6, the factorization operation used in EdgeConvEns is illustrated.
In the first step, the tensors $I$ and $K$, are factorized by manually
selected factors, namely $BS_{f}$ and $F_{f}$. Calculation operations require
multiple uses of the same data which violates data streaming; therefore,
required parts are duplicated in the second step. In the third step, the
duplicated tensors are streamed for parallel computation. In the fourth step,
the aimed calculations are done in parallel. Finally, the acquired partial
results are aggregated into the result tensor in the last step.

In this study, training of edge models is implemented on Xilinx FPGAs using
high-level synthesis (HLS) [33]. Vitis HLS offers several directives which
enable array partitioning, streaming, and parallel computation. In the
algorithm, completely partitioning the manual tiles is executed with the
”$\\#$pragma HLS ARRAY$\\_$PARTITION complete” command where the ”variable=”
parameter specifies the array and the ”dim=” parameter is used for the
dimensions to partition. The arrays are also streamed inside the module, and
pipelining is applied with the ”$\\#$pragma HLS DATAFLOW” command. In
addition, the task-level parallelism on computations is applied with the
”$\\#$pragma HLS UNROLL” command. The same method applies to fully-connected
layer computations with appropriate inputs and outputs.

Figure 6: Illustration of duplication, operation, and aggregation of arrays in
dataflow parallelization.

The mini-batch size and the input channel size, convolution filters, and
length of the fully connected layers need to be manually factorized into tiles
for CNN calculations. The other factorizable dimension of the array is
automatically factorized based on the manual factorization of the weight of
the previous layer. The proposed Algorithm 1 automatically sets the tiling
factor of the layer on the corresponding dimension based on the manually tiled
dimension of the previous layer. In the algorithm, dimensions of the $i$-th
convolution kernel are denoted by
$[F_{f}^{i},C_{f}^{i}][F_{p}^{i},C_{p}^{i}][KW^{i}][KH^{i}]$ for
$F^{i}=F_{f}^{i}\times F_{p}^{i}$ number of filters and for
$C^{i}=C_{f}^{i}\times C_{p}^{i}$ number of channels. Dimensions for $i$-th
fully connected layer weights are denoted by
$[L1_{f}^{i},L2_{f}^{i}][L1_{p}^{i},L2_{p}^{i}]$ as $L1^{i}=L1_{f}^{i}\times
L1_{p}^{i}$ and $L2^{i}=L2_{f}^{i}\times L2_{p}^{i}$. The $i$-th convolutional
layer output dimensions are
$[BS_{f},C_{f}^{i}][BS_{p},C_{p}^{i}][W^{i}][H^{i}]$, where $BS=BS_{f}\times
BS_{p}$ is the minibatch size. The $i$-th fully connected layer outputs
dimensions are $[BS_{f},L2_{f}^{i}][BS_{p},L2_{p}^{i}]$.

Batch of images: $I[BS_{f},C_{f}][BS_{p},C_{p}][W][H]$,

Edge model: $EM$

Manual factorizations of layers: $f$

for $i<=n(l)$ do $\triangleright$ Forward Pass

if $l^{i}=$ Convolution then

if $i=1$ then

$dim(w^{i})=[f^{i},C_{f}][\frac{F^{i}}{f^{i}},\frac{C^{i}}{C_{f}}][KW^{i}][KH^{i}]$

else if $i>1$ then

$dim(w^{i})=[f^{i},f^{i-1}][\frac{F^{i}}{f^{i}},\frac{F^{i-1}}{f^{i-1}}][KW^{i}][KH^{i}]$

$dim(o^{i})=[BS_{f},f^{i}][BS_{p},\frac{F^{i}}{f^{i}}][W^{i}][H^{i}]$

else if $l^{i}=$ Fully Connected Layer then

if $i=1$ then

$dim(w^{i})=[C_{f},f^{i}][\frac{L1^{i}}{C_{f}},\frac{L2^{i}}{f^{i}}]$

else if $i>1$ then

$dim(w^{i})=[f^{i-1},f^{i}][\frac{L1^{i}}{f^{i-1}},\frac{L2^{i}}{f^{i}}]$

$dim(o^{i})=[BS_{f},f^{i}][BS_{p},\frac{L2^{i}}{f^{i}}]$

for $i<=n(l)$ do $\triangleright$ Backward Pass

if $l^{i}=$ Fully Connected Layer then

$dim(\partial\mathcal{L}^{i})=[BS_{f},f^{i}][BS_{p},\frac{L2^{i}}{f^{i}}]$

$dim(gr^{i})=[f^{i-1},f^{i}][\frac{L1^{i}}{f^{i-1}},\frac{L2^{i}}{f^{i}}]$

else if $l^{i}=$ Convolution then

$dim(\partial\mathcal{L}^{i})=[BS_{f},f^{i}][BS_{p},\frac{L2^{i}}{f^{i}}]$

$dim(gr^{i})=[f^{i},f^{i-1}][\frac{F^{i}}{f^{i}},\frac{F^{i-1}}{f^{i-1}}][KW^{i}][KH^{i}]$

Algorithm 1 Automation process of the tiling.

In Algorithm 1, $l$ denotes the layers, $w$ and $o$ represent the weights and
the outputs of the corresponding layers. $\partial\mathcal{L}$ stands for the
loss sent back from the next layer, and $gr$ shows the corresponding gradients
of the layers. Here, $dim()$ outputs the dimensions of the corresponding
array, and $f$ is the manually selected tiling factorizations.

####  IV-A2 Experimental Setup

As mentioned in Section I, the data is assumed to be distributed as some edge
devices observe some classes more frequently than others. Two metrics, namely
$\alpha$ and $\delta$, are defined to represent this behavior. The $\alpha$
parameter denotes the minimum percentage of the whole training data of classes
used in the training edge models. For example, when $\alpha=0.25$, then at
least 25% of the training data of each class is randomly sampled for each edge
device. The second metric $\delta$ denotes the maximum discrepancy rate
between the percentage of sampled training data and sampled test data for a
class. For example, if 10% of the whole training data of class 1 is used for
training an edge model and $\delta=0.5$, then the percentage of test data of
class 1 that the edge model accesses is between 5% and 15%. The experiments
are conducted by taking random subsets of the whole train and test data.
$K^{EM_{i}}_{train}$ is chosen as $\bigcup_{i=1}^{c}\\{k_{i}\subset
K_{train},\frac{\left|k_{i}\right|}{\left|K_{train_{i}}\right|}=X\sim\mathcal{U}_{[\alpha,1]}$
and $k_{i}$ is chosen randomly$\\}$. In addition, $K^{EM_{i}}_{test}$ is
chosen as $\bigcup_{i=1}^{c}\\{k_{i}\subset
K_{test},\frac{\left|k_{i}\right|}{\left|K_{test_{i}}\right|}=max\\{min\\{\left|K_{train_{i}}\right|*(1+X),1\\},0\\}$
and $X\sim\mathcal{U}_{[-\delta,\delta]}\\}$. Different values of $\alpha$ and
$\delta$ remarkably change the class data percentage for train and test data
of edge devices, which affects the overall prediction accuracy with timing and
memory requirements.

The classification experiments are conducted on four different image
classification benchmark datasets, CIFAR-10 [34], CIFAR-100 [34], MNIST [35],
and Fashion MNIST [36]. The regression performance of the method is evaluated
on Boston Housing [37], California Housing [38], and Pecan Street [39]
datasets.

Experiments on classification utilize train and test data randomly generated
for each edge device based on $\alpha$ and $\delta$ values. The continuous
values of the targets in the regression datasets are divided into ten groups
in training data, where all groups have the same density of data to mimic the
classification behavior. Test data are also grouped based on the training
grouping. After creating classes for regression datasets, $\alpha$ is set to
0.05 in the regression experiments.

####  IV-A3 Edge and Server Models

In the experiments, each weak edge model is arbitrarily designed. For the
image classification task, the networks are built with structures such as one
convolutional layer with $5\times 5$ kernel size and $f_{e}$ filters, one
$2\times 2$ max pooling layer, one convolutional layer with $5\times 5$ kernel
size and $f_{e}$ filters, one or two Fully Connected Layers (FCL) with
64-dimensional output, and an output FCL with c-dimensional output where
$f_{e}\in\left\\{1,2,4\right\\}$ and $c$ is the number of classes. For the
regression task, the networks are built with structures such as one or two
Fully Connected Layers (FCL) with 64-dimensional output and an output FCL with
1-dimensional output.

Also, the training data and training epochs are randomly sampled for each edge
model. The minimum, mean and maximum statistics of the training epoch, model
accuracy, and the total number of model parameters are given in Table I for
each classification dataset. It can be seen that the accuracy values of the
edge models vary quite a lot because edge models differ in structure, and some
edge models are trained for a few epochs. Due to their relatively small and
shallow structures, the edge models do not produce a good accuracy.

The edge models are trained independently from each other and the models that
are on the server. The cross-entropy, given as
$-\sum_{i=1}^{c}y_{i}log(\hat{y}_{i})$, and mean squared error (MSE), given as
$\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$, are used as the loss
functions during edge model training for image classification and regression
problems, respectively. In the loss functions, $y_{i}$ and $\hat{y}_{i}$
denote the ground truth label and the prediction for the $i$-th class out of
$c$ classes for cross-entropy loss, where they denote $i$-th prediction and
observation in the $n$-sized sample for MSE loss, respectively. The loss
function used in the training of VAE models is given as
$\left\|F_{train}^{EM_{i}}-VAE_{i}^{dec}(z)\right\|^{2}+KL[\mathcal{N}(\mu_{enc},\sigma_{enc}),\mathcal{N}(0,1)]$,
where $z$ denotes a random sample vector from the latent space with the
distribution $\mathcal{N}(\mu_{enc},\sigma_{enc})$, $\mu_{enc}$ and
$\sigma_{enc}$ denote mean and standard deviation terms of the encoding, $KL$
denotes the Kullback-Leibler (KL) divergence and $\mathcal{N}$ denotes normal
distribution. The ensemble model is trained with the same loss functions as
the edge models. It should be highlighted that the edge and ensemble models
are implemented using float (32-bit) datatype. ADAM optimizer is used to train
VAE and ensemble models with a learning rate of $10^{-4}$. For edge models,
Stochastic Gradient Descent (SGD) optimizer is used with $10^{-4}$ learning
rate.

TABLE I: Edge model properties.

Dataset |  | Number of
---
Training Epochs
| Number of
---
Parameters
(thousand)
| Accuracy
---
(%)
Min | Mean | Max | Min | Mean | Max | Min | Mean | Max
CIFAR-10 | 11 | 32 | 49 | 7 | 25 | 44 | 10.36 | 37.46 | 51.37
CIFAR-100 | 10 | 25 | 46 | 13 | 25 | 50 | 4.53 | 8.43 | 17.23
MNIST | 12 | 24 | 45 | 4 | 15 | 21 | 93.57 | 96.74 | 97.94
| Fashion
---
MNIST
10 | 32 | 47 | 4 | 18 | 21 | 71.76 | 79.91 | 90.16

###  IV-B Training Scenarios

The training of the proposed system can be done in three different scenarios
based on the feature vector transfer scheme from the edge to the server. These
three scenarios bring advantages and disadvantages to the training process and
stand on a trade-off between training time, memory requirement, and overall
accuracy. The appropriate scenario can be chosen according to the device
choice and specifications of the transfer medium. In all scenarios, $Ep^{EM}$,
$Ep^{VAE}$, and $Ep^{Ens}$ denote the training epochs of edge models, VAE
models, and the ensemble model, respectively.

####  IV-B1 Abundant Memory on the Server

Scenario 1 is considered when all feature vectors obtained from the inference
of training data are transferred from the edge models to the server at once.
Therefore, VAE training and ensemble learning can start once the feature
vector transfer is completed. In this scenario, the memory requirements on the
server side are the highest among the three scenarios since all the feature
vectors must be stored on the server device. It also achieves the highest
accuracy among all the scenarios discussed in this study.

Firstly, edge models are trained independently using their training data for
$Ep^{EM}$ epochs. After their training, feature vectors, $F_{train}^{EM_{i}}$,
with $L_{com}$ length are obtained by inference. After the inference, the
feature vectors are transferred to the server in only one transfer. When the
transfer is done, the VAE model of each edge model is trained for $Ep^{VAE}$
epochs. Then, missing feature vectors are replaced via these trained VAE
models for each edge model. Ultimately, the final ensemble model is trained
using all feature vectors received and replaced.

####  IV-B2 Limited Memory on the Server

In Scenario 2, only a mini-batch of ensemble training data consisting of
feature vectors is sent to the server. This scenario remarkably reduces the
memory requirement on the server side. The total ensemble training epoch
$Ep^{Ens}$ is divided by a factor $Ep^{Ens}_{d}$. The stored mini-batch is
repeatedly used in training for $\frac{Ep^{Ens}}{Ep^{Ens}_{d}}$ epoch. Every
mini-batch is transferred to the server for $Ep^{Ens}_{d}$ times. This
scenario requires less data load per transfer but more communication.

Training the ensemble learner using the same mini-batch repeatedly causes bias
in training hence, degrading the accuracy. However, this degradation can be
alleviated to a certain level by decreasing the repeated use of a mini-batch
in training. For example, increasing the number of $Ep^{Ens}_{d}$ alleviates
such bias in training at the cost of increasing the number of transfers. The
training of VAE models is also deteriorated by low $Ep^{Ens}_{d}$. It yields
lower accuracy in the feature generation than in Scenario 1.

A special case of Scenario 2, where $Ep^{Ens}=Ep^{Ens}_{d}$, is named Scenario
3. In this case, a mini-batch is used in training for one epoch at one
communication. On the server, extra storage memory is not required for keeping
the mini-batch for consecutive use in training. Since each mini-batch is used
in training for one epoch, the accuracy acquired is close to the first
scenario. The disadvantage of this scenario is the increased number of one-way
communication.

###  IV-C Quantitative Analysis

For the experiments, $L_{com}$ is set to 64 and edge models are trained for 30
epochs. The ensemble model is trained for 100 epochs. The transfer rate
between the edge and server is assumed to be 450 Mbps. Memory and
communication evaluations are done for the CIFAR-10 dataset. Moreover, 60%,
70%, and 87% of the full training set are used when $\alpha$ values are set to
0.05, 0.3, and 0.7 for memory and communication evaluations. The VAE and the
ensemble model architecture are the same for all datasets and training
scenarios. VAE model consists of an FCL with 64 neurons in encoder and decoder
parts and the latent space is taken as $\mathbb{R}^{32}$. Total number of
parameters used for the encoder and decoder parts of each $VAE_{i}$ is 260K.
VAE models are trained for 50 epochs. Ensemble model consists of 64
$\frac{N}{2}\times\frac{L_{com}}{2}$ kernels and an FCL with 64 neurons. The
ensemble model has 182K parameters for all datasets and training scenarios.
The accuracy values are given for $\alpha=0.05$ and $\delta=0$ unless
mentioned otherwise.

The FPGA experiments are conducted on Xilinx Artix-7 AC701 Evaluation Platform
using CIFAR-10 dataset. The synthesis and implementation of the system are
done using Vitis HLS 2022.2 and Vivado 2022.2 platforms. The VAE and ensemble
model experiments are conducted on GeForce RTX 2080 Ti device.

Firstly, the effectiveness of the proposed partial task-level streaming
implemented on FPGA devices is presented. Different model layers are
parallelized in the experiments with the given parallelization factors. The
training is done with mini-batches of size two. The results can be seen in
Table II. The structure of the edge model used in this experiment is one
convolutional layer with $5\times 5$ kernel size and two filters, one $2\times
2$ max pooling layer, one convolutional layer with $5\times 5$ kernel size and
two filters, one Fully Connected Layers (FCL) with 64-dimensional output, and
an output FCL with 10-dimensional output. In the table, ”Without
Optimizations” is the training of the model without any task-level parallelism
applied. $BS_{f}$ and $F_{f}$ denote the parallelization factors of the mini-
batch dimension and the filter dimension of the data for edge training,
respectively. Therefore, $BS_{f=x}$-$F_{f=y,z}$ points to that mini-batch are
parallelized by factor $x$, the first convolutional operation is parallelized
by factor $y$, and the second convolutional operation is parallelized by
factor $z$. The utilization values are taken from the post-implementation
phase. Latency values are taken from the C/RTL Cosimulation results.
Experimental results show that task-level streaming yields remarkable
acceleration even without factorizing different layer calculations. Also,
parallelizing different layers with different factors shows changing effects
on acceleration and resource utilization. These results show that the proposed
method can be configured for devices with various hardware capacities.

TABLE II: Latency and resource utilization of edge models.

Configuration |  | Latency
---
(ms)
| BRAM
---
(%)
| DSP
---
(%)
| FF
---
(%)
| LUT
---
(%)
| LUTRAM
---
(%)
| IO
---
(%)
| Without
---
Optimizations
194 | 20 | 6 | 3 | 6 | 1 | 39
| EdgeConvEns
---
$BS_{f=1}$-$F_{f=1,1}$
90 | 64 | 13 | 6 | 11 | 2 | 61
| EdgeConvEns
---
$BS_{f=1}$-$F_{f=1,2}$
72 | 69 | 17 | 7 | 14 | 3 | 61
| EdgeConvEns
---
$BS_{f=1}$-$F_{f=2,1}$
51 | 75 | 17 | 7 | 14 | 3 | 61
| EdgeConvEns
---
$BS_{f=2}$-$F_{f=1,1}$
46 | 76 | 22 | 9 | 18 | 4 | 86

The importance of replacing the missing feature vectors with appropriate
values can be seen in Table III. The dataset used in this experiment is
CIFAR-10. The average edge model accuracy is 40%, and 20 edge models are used.
The results show that replacing the missing feature vectors with trivial
values such as zero, the mean, and the maximum element of the successfully
transferred feature vectors detriments the overall accuracy of the ensemble
model. The feature vectors generated with VAE produce remarkably more accurate
results in terms of overall ensemble accuracy.

TABLE III: Accuracy with different filling methods. Replacement Method | Accuracy (%)
---|---
| $\bm{\alpha=0.3}$
---
$\bm{\delta=0.2}$
| $\bm{\alpha=0.5}$
---
$\bm{\delta=0.5}$
| $\bm{\alpha=0.7}$
---
$\bm{\delta=0}$
With Zero | 62 | 58 | 58
With Mean | 52 | 54 | 57
With Max | 35 | 42 | 52
With VAE | 82 | 73 | 63

Examining the loss and accuracy plots of the training of the proposed ensemble
model is essential. Figure 7 shows that the ensemble model converges to a
solution as the training continues. The plots are given for $\alpha=0.05$ and
$\delta=0.5$ for CIFAR-10, and Pecan Street datasets on Scenario 1.

Figure 7: The loss plots of the ensemble model trained for (a) CIFAR-10
dataset for image classification, and (b) Pecan Street dataset for regression
tasks.

Figure 8: Performance metrics of the ensemble model on different classes where
$N=20$. (a) CIFAR-10, and (b) MNIST.

Figure 8 shows the box plot of the performance metrics of accuracy, the area
under the ROC curve, precision, and recall values calculated for each class in
test data for CIFAR-10 and MNIST datasets. The values are separately
calculated based on one-hot encodings for the classes. Even though the edge
models are not fully trained for all classes, the ensemble model performs for
all classes at a similar level in terms of prediction.

Figure 9: Ensemble accuracy for different training scenarios and $\alpha$,
$\delta$ values for CIFAR-10 dataset. Green, orange, and blue colors represent
different $\alpha$ values, $0.05$, $0.3$, and $0.7$ respectively. Dark colors
represent Scenario 1 and 3, and the light colors represent Scenario 2.

In Figure 9, the accuracy of the ensemble model for different choices of
$\alpha$ and $\delta$ values and training scenarios are given for CIFAR-10.
The figure shows that the ensemble accuracy increases as the $\alpha$ and
$\delta$ values decrease. As expected, training with Scenario 1 and Scenario 3
yield better accuracy than Scenario 2. That means the model learns better as
the heterogeneity of training data observed by edge devices increases, and the
prediction performance is preserved as the heterogeneity in test data stays
similar. It is also shown that the accuracy of the ensemble model is limited
to the accuracy of the edge model. For CIFAR-10, the prediction accuracy of
91.14%, 93.47%, and 94.23% is obtained using 20 edge models with a mean
accuracy of 31.60%, 37.46%, and 48.41%, respectively.

The effect of changing the number of edge models and the transferred feature
vector size $L_{com}$ is also investigated in Table IV for CIFAR-10 and
CIFAR-100 datasets. For changing the number of edge models, $L_{com}$ is taken
as 64. For changing $L_{com}$, the number of edge models is taken as 20. It
can be seen that increasing the number of edge models also increases the final
ensemble accuracy. The same effect for changing $L_{com}$ can be observed, but
the increase ceases to be material after a certain point of $L_{com}$ value,
which is after 128 for this example.

TABLE IV: Test accuracy with different $N$ and $L_{com}$.

Dataset | $N$ | $\bm{L_{com}}$
---|---|---
10 | 20 | 30 | 50 | 32 | 64 | 128 | 256
CIFAR-10 | 80.80 | 93.47 | 96.24 | 98.79 | 90.89 | 93.47 | 93.33 | 93.47
CIFAR-100 | 41.60 | 55.48 | 64.10 | 75.77 | 51.77 | 55.48 | 67.37 | 67.40

Comparison with recent edge intelligence studies regarding final accuracy for
image classification and regression problems are presented in Table V and
Table VI. The best-reported values are taken from the compared studies and our
study. In the image classification comparison, accuracy results for CIFAR-10
and CIFAR-100 of EdgConvEns are taken from Table IV. MNIST and Fashion MNIST
results of EdgConvEns are given for $N=20$ and $L_{com}=64$. In regression
comparison, $N=50$ is taken for our results. Table VI shows that EdgeConvEns
produces state-of-the-art accuracy for image classification and regression
problems though it uses less accurate edge models than the ones used in the
compared studies. For example, the edge models used in [20] have 37.67% and
78.94% accuracy on CIFAR-100 and CIFAR-10, respectively. Table I shows that
the average edge model accuracy of EdgeConvEns is 8.43% and 37.46%,
respectively.

TABLE V: Accuracy comparison for image classification.

Method | CIFAR-10 | CIFAR-100 | MNIST |  | Fashion
---
MNIST
FedAvg [8] | 81.72% | - | 95.04% | 94.50%
FedProx [11] | 83.25% | - | 96.26% | 94.53%
FedDyn [12] | 85.19% | 53.27% | - | -
SCAFFOLD [13] | 85.99% | 53.32% | - | -
FedGen [14] | 83.91% | 50.38% | - | -
FedProto [17] | 84.49% | - | 97.13% | 97.10%
FedFTG [15] | 87.34% | 56.94% | - | -
FedDC [16] | 86.18% | 55.52% | 98.45% | -
| Bayesian
---
Nonparametric
[18] 45.80% | - | 97.80% | -
AE-KD [22] | 93.01% | 72.36% | - | -
EKD [23] | 92.33% | 67.78% | - | -
FedGKT [20] | 92.97% | 69.57% | - | -
| Majority
---
Voting
42.30% | 14.06% | 93.12% | 82.74%
| Average
---
Voting
43.26% | 17.62% | 93.46% | 82.98%
EdgeConvEns | 98.79% | 75.77% | 99.32% | 97.40%

TABLE VI: Accuracy comparison for regression.

Work | Dataset | Metric |  | Reported
---
Value
EdgeConvEns
SAFA [40] |  | Boston
---
Housing
| Accuracy [40]
---
(higher is better)
0.643 | 0.728
FedAG [41] |  | Boston
---
Housing
| RMSE
---
(lower is better)
4.070 | 3.479
PrivFL [42] |  | Boston
---
Housing
| MAE
---
(lower is better)
3.266 | 2.788
Tiresias [43] |  | California
---
Housing
| R2 Score
---
(higher is better)
0.717 | 0.726
Sherpa [44] |  | California
---
Housing
| R2 Score
---
(higher is better)
0.502 | 0.868
DER Forecast [45] | Pecan Street |  | RMSE
---
(lower is better)
1.98 | 0.63

###  IV-D Memory and Communication Efficiency

The average time required for the edge model training is 345, 402, and 500
minutes for $\alpha$ values of 0.05, 0.3, and 0.7, respectively. It can be
seen that the edge models are trained in a shorter time when they are trained
with less amount of data. In the whole training process, the training of the
edge models takes the most time due to the hardware constraints of the edge
devices.

In Figure 10, the time requirements for feature vector transfer
($T^{EM}_{transfer}$), VAE training ($T^{VAE_{EM}}_{train}$), and ensemble
model training ($T^{Ens}_{train}$) is shown. The results are invariant with
the dataset used for training due to the same $L_{com}$ in all edge models.
The total transfer time for an edge model is the most for Scenario 3 as a
mini-batch is repeatedly transferred to the server. The number of
communication is calculated as 39063 for Scenario 3. It is less in Scenario 2
due to the repeated use of the same mini-batch to complete the ensemble
training epochs. The number of communication is calculated as 7813 for
Scenario 2. It is the least for Scenario 1 because the feature vectors are
sent to the server once. The ensemble learning time slightly increases from
Scenario 1 to Scenario 3 due to repeated VAE decoding operations as the same
mini-batch is received by the server multiple times. It can also be seen that
the latency for feature vector transfer and VAE training visibly decreases for
decreasing $\alpha$ values as expected. The cumulative amount of data
($S^{cum}_{transfer}$) of feature vectors transferred to the server is shown
in Figure 10. The largest amount is in Scenario 3 due to repeated transfers,
while the least amount of feature vectors data transfer is in Scenario 1 due
to one-time transfer. End-to-end latency is approximately 0.05 seconds for
inference in all training scenarios.

Figure 10: (a) Overall latency for feature vector transfer, VAE training, and
ensemble training, (b) Total cumulative transferred data amount from the edge.

Table VII shows the size of the transferred feature vectors in a one-way
communication from one edge device to the server ($M^{EM}_{transfer}$),
required storage memory on the server ($M^{server}_{memory}$), and required
storage memory on an edge device ($M^{EM}_{memory}$) by training scenarios and
$\alpha$ values. It shows that the largest memory is needed for Scenario 1
because the whole feature vector data of each edge model are transferred to
the server once and stored there. However, the required memory remarkably
reduces when the system is trained with Scenario 2 and Scenario 3. Scenario 3
requires no memory space allocated for the transferred feature vector data,
whereas Scenario 2 requires it for only a mini-batch.

TABLE VII: Memory requirements on edge devices and server. Scenario | $\bm{\alpha}$ |  | $\bm{M^{EM}_{transfer}}$
---
(MB)
| $\bm{M^{server}_{memory}}$
---
(MB)
| $\bm{M^{EM}_{memory}}$
---
(MB)
Scenario 1 | 0.05 | 7.7 | 256.0 | 7.7
0.3 | 9.0 | 9.0
0.7 | 11.1 | 11.1
Scenario 2 | All | 0.03 | 0.7 | 0.03
Scenario 3 | 0

##  V Conclusion

This study proposes EdgeConvEns, a convolutional ensemble learning framework
for deep edge intelligence. The EdgeConvEns framework aims to boost the
system’s overall classification and regression performance by learning a
unified representation from the collective information extracted by weak edge
models accessing heterogeneously distributed data. Thus, we target to reduce
the computational requirements on edge while offering performances comparable
to state-of-the-art. The proposed framework also tackles missing information
due to failures in network communication by a VAE-based feature imputation
approach. Moreover, EdgeConvEns provides a customizable acceleration for
training DNNs on Xilinx FPGA devices. Thus, the training can be accelerated
using different parallelization factors such that the hardware limits are met
for different target devices. EdgeConvEns also makes the training of the
system available for devices with differing hardware capacities, thanks to
different learning scenarios. The experiments conducted with benchmark
datasets and various training scenarios demonstrate that the proposed
EdgeConvEns outperforms conventional ensemble learning and standard federated
learning techniques.

## References

  * [1] X. Krasniqi and E. Hajrizi, “Use of IoT Technology to Drive the Automotive Industry From Connected to Full Autonomous Vehicles,” _IFAC-PapersOnLine_ , vol. 49, no. 29, pp. 269–274, 2016.
  * [2] W. Z. Khan, M. Y. Aalsalem, M. K. Khan, M. S. Hossain, and M. Atiquzzaman, “A Reliable Internet of Things Based Architecture for Oil and Gas Industry,” in _2017 19th International Conference on Advanced Communication Technology_. IEEE, 2017, pp. 705–710.
  * [3] N. Misra, Y. Dixit, A. Al-Mallahi, M. S. Bhullar, R. Upadhyay, and A. Martynenko, “IoT, Big Data and Artificial Intelligence in Agriculture and Food Industry,” _IEEE Internet of Things Journal_ , vol. 9, no. 9, pp. 6305–6324, 2020.
  * [4] C.-F. R. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-Attention Multi-Scale Vision Transformer for Image Classification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 357–366.
  * [5] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, and T. N. Sainath, “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 82–97, 2012.
  * [6] T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent Trends in Deep Learning Based Natural Language Processing,” _IEEE Computational Intelligence Magazine_ , vol. 13, no. 3, pp. 55–75, 2018.
  * [7] S. A. R. Zaidi, A. M. Hayajneh, M. Hafeez, and Q. Ahmed, “Unlocking Edge Intelligence Through Tiny Machine Learning (TinyML),” _IEEE Access_ , vol. 10, pp. 100 867–100 877, 2022.
  * [8] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-Efficient Learning of Deep Networks From Decentralized Data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.
  * [9] Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing,” _Proceedings of the IEEE_ , vol. 107, no. 8, pp. 1738–1762, 2019.
  * [10] E. Li, Z. Zhou, and X. Chen, “Edge Intelligence: On-Demand Deep Learning Model Co-Inference With Device-Edge Synergy,” in _Proceedings of the 2018 Workshop on Mobile Edge Communications_ , 2018, pp. 31–36.
  * [11] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated Optimization in Heterogeneous Networks,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 429–450, 2020.
  * [12] D. A. E. Acar, Y. Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama, “Federated Learning Based on Dynamic Regularization,” arXiv:2111.04263, 2021.
  * [13] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic Controlled Averaging for Federated Learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5132–5143.
  * [14] Z. Zhu, J. Hong, and J. Zhou, “Data-Free Knowledge Distillation for Heterogeneous Federated Learning,” arXiv:2105.10056, 2021.
  * [15] L. Zhang, L. Shen, L. Ding, D. Tao, and L.-Y. Duan, “Fine-Tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 174–10 183.
  * [16] L. Gao, H. Fu, L. Li, Y. Chen, M. Xu, and C.-Z. Xu, “FedDC: Federated Learning With Non-IID Data via Local Drift Decoupling and Correction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 112–10 121.
  * [17] Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated Prototype Learning Across Heterogeneous Clients,” in _AAAI Conference on Artificial Intelligence_ , vol. 1, 2022, p. 3.
  * [18] M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian Nonparametric Federated Learning of Neural Networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 7252–7261.
  * [19] X. Fang and M. Ye, “Robust Federated Learning With Noisy and Heterogeneous Clients,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 072–10 081.
  * [20] C. He, M. Annavaram, and S. Avestimehr, “Group Knowledge Transfer: Federated Learning of Large CNNS at the Edge,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 14 068–14 080, 2020.
  * [21] C.-C. Lin, D.-J. Deng, C.-H. Kuo, and L. Chen, “Concept Drift Detection and Adaption in Big Imbalance Industrial IoT Data Using an Ensemble Learning Method of Offline Classifiers,” _IEEE Access_ , vol. 7, pp. 56 198–56 207, 2019.
  * [22] S. Du, S. You, X. Li, J. Wu, F. Wang, C. Qian, and C. Zhang, “Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 345–12 355, 2020.
  * [23] U. Asif, J. Tang, and S. Harrer, “Ensemble Knowledge Distillation for Learning Improved and Efficient Networks,” arXiv:1909.08097, 2019.
  * [24] S. Biookaghazadeh, M. Zhao, and F. Ren, “Are FPGAs Suitable for Edge Computing?” arXiv:1804.06404, 2018.
  * [25] I. Sikdokur, I. Baytas, and A. Yurdakul, “Image Classification on Accelerated Neural Networks,” arXiv:2203.11081, 2022.
  * [26] A. Ahmad and M. A. Pasha, “Optimizing Hardware Accelerated General Matrix-Matrix Multiplication for CNNs on FPGAs,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 67, no. 11, pp. 2692–2696, 2020\.
  * [27] C. Luo, M.-K. Sit, H. Fan, S. Liu, W. Luk, and C. Guo, “Towards Efficient Deep Neural Network Training by FPGA-based Batch-Level Parallelism,” _Journal of Semiconductors_ , vol. 41, no. 2, p. 022403, 2020.
  * [28] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized Neural Networks: Training Deep Neural Networks With Weights and Activations Constrained To +1 or -1,” arXiv:1602.02830, 2016.
  * [29] F. Zhu, R. Gong, F. Yu, X. Liu, Y. Wang, Z. Li, X. Yang, and J. Yan, “Towards Unified Int8 Training for Convolutional Neural Network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1969–1979.
  * [30] L. Van der Maaten and G. Hinton, “Visualizing Data Using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, no. 11, pp. 2579–2605, 2008\.
  * [31] M. Gokhale, J. Stone, J. Arnold, and M. Kalinowski, “Stream-Oriented FPGA Computing in the Streams-C High Level Language,” in _Proceedings 2000 IEEE Symposium on Field-Programmable Custom Computing Machines_. IEEE, 2000, pp. 49–56.
  * [32] ARM, “AMBA AXI-Stream Protocol Specification,” 2010, accessed on January 10, 2023. [Online]. Available: https://developer.arm.com/documentation/ihi0051/latest/
  * [33] M. C. McFarland, A. C. Parker, and R. Camposano, “The High-Level Synthesis of Digital Systems,” _Proceedings of the IEEE_ , vol. 78, no. 2, pp. 301–318, 1990.
  * [34] A. Krizhevsky, V. Nair, and G. Hinton, “The CIFAR-10 Dataset,” 2014, accessed on November 21, 2021. [Online]. Available: http://www.cs.toronto.edu/kriz/cifar.html
  * [35] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [36] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-Mnist: A Novel Image Dataset for Benchmarking Machine Learning Algorithms,” arXiv:1708.07747, 2017.
  * [37] D. Harrison Jr and D. L. Rubinfeld, “Hedonic Housing Prices and the Demand for Clean Air,” _Journal of Environmental Economics and Management_ , vol. 5, no. 1, pp. 81–102, 1978.
  * [38] R. K. Pace and R. Barry, “Sparse Spatial Autoregressions,” _Statistics & Probability Letters_, vol. 33, no. 3, pp. 291–297, 1997.
  * [39] Dataport, “Pecan Street,” 2017, accessed on August 14, 2022. [Online]. Available: https://www.pecanstreet.org/
  * [40] W. Wu, L. He, W. Lin, R. Mao, C. Maple, and S. Jarvis, “SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low Overhead,” _IEEE Transactions on Computers_ , vol. 70, no. 5, pp. 655–668, 2020.
  * [41] A. T. Thorgeirsson and F. Gauterin, “Probabilistic Predictions With Federated Learning,” _Entropy_ , vol. 23, no. 1, p. 41, 2020.
  * [42] F. Wang, H. Zhu, R. Lu, Y. Zheng, and H. Li, “A Privacy-Preserving and Non-Interactive Federated Learning Scheme for Regression Training With Gradient Descent,” _Information Sciences_ , vol. 552, pp. 183–200, 2021\.
  * [43] K. Zhang, “Tiresias: A Peer-to-Peer Platform for Privacy Preserving Machine Learning,” Ph.D. dissertation, Massachusetts Institute of Technology, 2020.
  * [44] N. Rodríguez-Barroso, G. Stipcich, D. Jiménez-López, J. A. Ruiz-Millán, E. Martínez-Cámara, G. González-Seco, M. V. Luzón, M. A. Veganzones, and F. Herrera, “Federated Learning and Differential Privacy: Software Tools Analysis, the Sherpa. AI FL Framework and Methodological Guidelines for Preserving Data Privacy,” _Information Fusion_ , vol. 64, pp. 270–292, 2020.
  * [45] V. Venkataramanan, S. Kaza, and A. M. Annaswamy, “Der Forecast Using Privacy Preserving Federated Learning,” arXiv:2107.03248, 2021.


We introduce the homogenized coefficients in the limit energy which  gives the convergence result for the homogenized energy. From the Lemma <ref>, the quadratic form associated to $\GJ_{vK}^{hom}$ is coercive.

We are ready to give the main result of this subsection.

Under the assumption on the forces (<ref>) and (<ref>), the problem

\min_{\Uc\in\D_0} \GJ_{vK}^{hom^*}(\Uc)

admits at least a solution. Moreover, one has

$$ m^*=	\lim_{\e\to0}{m^*_\e \over \e^5} = \min_{(\Uc,\wh{\fu},\Gu^M)\in \D} \GJ^*(\Uc,\wh{\fu},\Gu^M)= 	\min_{\Uc\in\D_0} \GJ_{vK}^{hom^*}(\Uc).$$

We remark the following

* The existence of a minimizer of the limit homogenized energy (<ref>) is given by the fact that the associated quadratic form is coercive and lower semi-continuous.

* Note that the homogenized energy (<ref>) has an addition term due to the matrix pre-strain, which is absent on the homogenized energy (<ref>), showing that for the soft matrix to have any effect in the homogenized model pre-strain is necessary.

* We have

$$\GJ^*(0,0,0)= \int_{\o}\int_{Y_\kappa^M} a(y)\;Sym(B) :\;Sym(B)dydx'= {1\over 2}\int_{\o}d^{M,hom}_{i'j'k'l'}Sym(B)Sym(B)dx'.$$

We end this subsection by showing that our homogenized vK plate, obtained through the incorporation of small matrix pre-strain (Case 2), exhibits orthotropic behavior[We know that this is true for Von-Kármán and linear regime, since the homogenized coefficients depend only on the cell problems.].

We know that for isotropic homogenized materials, we have from [37] that

$$ a_{ijkl}(y)=\lambda\delta_{ik}\delta_{jl}+\mu\delta_{ij}\delta_{kl}+\mu\delta_{il}\delta_{kj},\quad \text{for all}\;\;y\in\Yc.$$

where $\lambda>0$, $\mu>0$ are the material Lamé constants and $\delta_{ij}$ is the Kronecker symbol.

Let us assume that the plate is made from isotropic and homogeneous material, then we have

b^{B,hom}_{\alpha\beta\alpha^\prime\beta^\prime} =0,\qquad \forall\;(\alpha\beta\alpha^\prime\beta^\prime)\in \{1,2\}^4,

and also have

\begin{aligned}
a^{B, hom}_{1111}=a^{B, hom}_{2222}\quad\text{and}\quad a_{\alpha\alpha12}^{B, hom}=0,\quad \alpha\in\{1,2\},\\
c_{1111}^{B, hom}=c_{2222}^{B, hom}\quad\text{and}\quad c^{B, hom}_{\alpha\alpha12}=0,\quad \alpha\in\{1,2\},\\
a^{M,hom}_{1111}=a^{M,hom}_{2222}\quad \text{and} \quad a^{M,hom}_{\alpha\alpha12}=0,\quad \alpha\in\{1,2\}.
\end{aligned}

The proof is in line with Lemma 6.9 of [31].

§ EXTENSION TO GENERAL PERIODIC COMPOSITE PLATE

In this section, the extension of our results to a more general periodic composite plate is undertaken. The key concept to emphasize is that our results can be extended to any periodic perforated plate with holes filled by a soft matrix. This extension yields similar results as presented in Lemma <ref>, with the holes being bounded domains featuring Lipschitz boundaries, thereby allowing for the attainment of the estimates (<ref>).

In the following discussion, we explore cases where the general holes take the form of (<ref>) (refer to Figure <ref>).

§.§ General periodic composite plate

Our general domain is a periodic composite plate in which the part filled with soft matrix is connected. We set the following:

* $Y^B_\kappa$ be an  open connected  subset of $\Yc$ with Lipschitz boundary (see for example Figure <ref>a) such that interior $\big(\overline{Y^B_\kappa}\cup(\overline{Y^B_\kappa}+\Ge_\alpha)\big)$  is connected for $\alpha=1,2$;

* $Y^M_\kappa=\Yc\setminus\overline{Y}^B_\kappa$ is such that  interior$\big(\overline{Y^M_\kappa}\cup(\overline{Y^M_\kappa}+\Ge_\alpha)\big)$ is connected for $\alpha=1,2$;

* $Y^B_\kappa$, $Y^M_\kappa$ are  the reference cells of the stiff and matrix part respectively;

* $\Xi_\e=\{\xi\in\Z^2\X\{0\}\;|\;\e(\xi+\Yc)\cap\O_\e\neq \emptyset\}$;

* $\wt{\O}_\e^\xi=\e(\xi+Y^B_\kappa)$, and $\O_\e^\xi=\e(\xi+Y^M_\kappa)$ for all $\xi\in\Xi_\e$;

* $\O^B_\e= \bigcup_{\xi\in\Xi_\e}\wt{\O}_\e^\xi $, see for example Figure <ref>a.

We assume that the domain $\O_\e^M$ is filled with soft matrix and is given by  $\O_\e^M=\O_\e\setminus\overline{\O_\e^B}$ . Then, we have

\begin{equation}\label{GenHole}
\O_\e^M=\hbox{interior} \bigcup_{\xi\in\Xi_\e}\overline{\O_\e^\xi}.
\end{equation}

Here, we note the following

* Here, we have $\bigcup_{\alpha=1}^2\Xi_{\e,\alpha}\subset\Xi_\e$ and that the domain $\O_\e^B$ is connected and made of stiff material.

* The domain filled with soft matrix $\O_\e^M$ is also connected.

* This structure is (like the one in Figure <ref>a) is $3$-dimensional and only periodic in two directions, in the third direction it is thin that is, its thickness is of same order $r=\kappa\e$ as the period of the other two directions[This particular structure has been extensively investigated in the literature, specifically in the works of [27], [28], and [31]. In these studies, they consider the relationship $r=\kappa\varepsilon$ with $\kappa\in(0,1/3]$ as stated in Remark A.1 of [27].].

[remember picture,
fill opacity = 1, scale=0.58]

[fill = gray] (0,0,5)–(0,0,6)–(0,1,6)–(0,1,5)–cycle;
[name path = frontlow]  (6,1,5)–(5,1,5) to[out=180,in=0] (1,0,5) – (0,0,5) ;
[name path = fronthigh]  (6,2,5)–(5,2,5) to[out=180,in=0] (1,1,5) – (0,1,5);
[name path = backlow]   (6,1,6)–(5,1,6) to[out=180,in=0] (1,0,6) – (0,0,6) ;
[name path = backhigh] (6,2,6)–(5,2,6) to[out=180,in=0] (1,1,6) – (0,1,6) ;

[of=frontlow and fronthigh] left color=gray!20!white, right color = gray, shading angle=60,;
[of=fronthigh and backhigh] left color=gray, right color = gray!20!white, shading angle=180,;

[name path = frontlow] (6,1,5)–(5,1,5) to[out=180,in=0] (1,0,5) – (0,0,5) ;
[name path = fronthigh] (6,2,5)–(5,2,5) to[out=180,in=0] (1,1,5) – (0,1,5 );
[name path = backhigh]  (6,2,6)–(5,2,6) to[out=180,in=0] (1,1,6) – (0,1,6) ;

[fill = gray] (6,0,5)–(6,0,6)–(6,1,6)–(6,1,5)–cycle;

[fill=gray] (1,2,6)–(1,1,6)–(0,1,6)–(0,2,6)–cycle;
[name path = righthigh] (1,1,0)–(1,1,1) to[out=55,in=-135] (1,2,5)–(1,2,6) ;
[name path = rightlow] (1,0,0)–(1,0,1) to[out=45,in=-125] (1,1,5)–(1,1,6) ;
[name path = lefthigh] (0,1,0)–(0,1,1) to[out=55,in=-135] (0,2,5)–(0,2,6) ;
[name path = leftlow] (0,0,0)–(0,0,1) to[out=45,in=-125] (0,1,5)–(0,1,6) ;
[fill=gray] (0,0,0)–(0,1,0)–(1,1,0)–(1,0,0)–cycle;

[of=rightlow and righthigh] left color=gray!20!white, right color = gray, shading angle=60,;
[of=lefthigh and righthigh] left color=gray, right color = gray!20!white, shading angle=180,;

[name path = righthigh] (1,1,0)–(1,1,1) to[out=55,in=-135] (1,2,5)–(1,2,6) ;
[name path = rightlow] (1,0,0)–(1,0,1) to[out=45,in=-125] (1,1,5)–(1,1,6) ;
[name path = lefthigh] (0,1,0)–(0,1,1) to[out=55,in=-135] (0,2,5)–(0,2,6) ;
[] (0,2,6)–(1,2,6)–(1,1,6);

[name path = frontlow]  (0,1,0)–(1,1,0) to[out=0,in=180] (5,0,0) – (6,0,0) ;
[name path = fronthigh] (0,2,0)–(1,2,0) to[out=0,in=180] (5,1,0) – (6,1,0) ;
[name path = backlow]    (0,1,1)–(1,1,1) to[out=0,in=180] (5,0,1) – (6,0,1) ;
[name path = backhigh]   (0,2,1)–(1,2,1) to[out=0,in=180] (5,1,1) – (6,1,1);

[fill=gray ] (0,1,0)–(0,2,0)–(0,2,1)–(0,1,1)–cycle;

[of=frontlow and fronthigh] left color=gray!20!white, right color = gray, shading angle=60,;
[of=fronthigh and backhigh] left color=gray, right color = gray!20!white, shading angle=180,;

[name path = frontlow]   (0,1,0)–(1,1,0) to[out=0,in=180] (5,0,0) – (6,0,0) ;
[name path = fronthigh]  (0,2,0)–(1,2,0) to[out=0,in=180] (5,1,0) – (6,1,0) ;
[name path = backhigh]  (0,2,1)–(1,2,1) to[out=0,in=180] (5,1,1) – (6,1,1);

[fill=gray] (6,0,0)–(6,1,0)–(6,1,1)–(6,0,1)–cycle;

[] (6,0,6)–(6,1,6)– (5,1,6)– (5,0,6)–cycle;

[name path = rightlow] (6,1,0)–(6,1,1) to[out=45,in=-145] (6,0,5) – (6,0,6);
[name path = righthigh] (6,2,0)–(6,2,1) to[out=35,in=-145] (6,1,5) – (6,1,6);
[name path = leftlow] (5,1,0)–(5,1,1) to[out=45,in=-135] (5,0,5) – (5,0,6);
[name path = lefthigh] (5,2,0)–(5,2,1) to[out=35,in=-145] (5,1,5) – (5,1,6);

[of=rightlow and righthigh] left color=gray!20!white, right color = gray, middle color=black!40!gray,;
[of=righthigh and lefthigh] top color=gray, bottom color = gray!20!white, middle color=black, shading angle=0;

[name path = rightlow] (6,1,0)–(6,1,1) to[out=45,in=-145] (6,0,5) – (6,0,6);
[name path = righthigh] (6,2,0)–(6,2,1) to[out=35,in=-145] (6,1,5) – (6,1,6);

[name path = lefthigh] (5,2,0)–(5,2,1) to[out=35,in=-145] (5,1,5) – (5,1,6);

[fill=gray] (5,1,0)–(6,1,0)–(6,2,0)–(5,2,0)–cycle;

[fill = gray] (0,0,5)–(0,0,6)–(0,1,6)–(0,1,5)–cycle;
[name path = frontlow]  (6,1,5)–(5,1,5) to[out=180,in=0] (1,0,5) – (0,0,5) ;
[name path = fronthigh]  (6,2,5)–(5,2,5) to[out=180,in=0] (1,1,5) – (0,1,5);
[name path = backlow]   (6,1,6)–(5,1,6) to[out=180,in=0] (1,0,6) – (0,0,6) ;
[name path = backhigh] (6,2,6)–(5,2,6) to[out=180,in=0] (1,1,6) – (0,1,6) ;

[of=frontlow and fronthigh] left color=gray!20!white, right color = gray, shading angle=60,;
[of=fronthigh and backhigh] left color=gray, right color = gray!20!white, shading angle=180,;

[name path = frontlow] (6,1,5)–(5,1,5) to[out=180,in=0] (1,0,5) – (0,0,5) ;
[name path = fronthigh] (6,2,5)–(5,2,5) to[out=180,in=0] (1,1,5) – (0,1,5 );
[name path = backhigh]  (6,2,6)–(5,2,6) to[out=180,in=0] (1,1,6) – (0,1,6) ;

[fill = gray] (6,1,5)–(6,1,6)–(6,2,6)–(6,2,5)–cycle;

[|-|,thick] (0.5,-0.6,0) – (5.5,-0.6,0) node [midway, below, sloped] (TextNode1) $\e/2$;
[dashed,thick] (0.5,-0.6,0) – (0.5,0.5,0);
[dashed,thick] (5.5,-0.6,0) – (5.5,1.5,0);

[|-|,thick] (0,-0.16,0) – (1,-0.16,0) node [pos=.25,yshift=-1pt,xshift=-2pt, below, sloped] (TextNode2) $2r$;
[|-|,thick] (-0.16,0,0) – (-0.16,1,0) node [midway, above, sloped] (TextNode3) $2r$;

[|-|,thick] (6,-0.5,0.5) – (6,-0.5,5.5) node [midway, below, sloped] (TextNode4) $\e/2$;
[dashed,thick] (6,-0.5,0.5)– (6,0.5,0.5);
[dashed,thick] (6,-0.5,5.5)– (6,1.5,5.5);

(a) The domain is a quarter of the periodicity cell of the full structure. (b) $2$D periodic composite plate: Blue part is with stiff material and red part is with soft matrix.

Here, we have a similar extension result like in Lemma <ref>, which is a consequence of the extension result given in see [[31], Lemma 2.1].

For every deformation $v\in H^1(\O_\e^B)^3$, there exist a deformation $\Gv\in H^1(\O_\e)^3$ satisfying

\begin{equation}\label{ExDef3}
\begin{aligned}
\Gv_{|\O_\e^B}=v,\quad\text{and}\quad
\|dist(\nabla\Gv,SO(3))\|_{L^2(\O_\e)}\leq C\|dist(\nabla v,SO(3))\|_{L^2(\O_\e^B)}.
\end{aligned}
\end{equation}

The constant does not depend on $\e$.

Note that $\wt{\O}_\e^\xi$ and $\O_\e^\xi$ are the periodicity cell of the stiff part and the matrix part respectively. Here, we observe that $Y_\kappa^M$ (resp. $\O_\e^\xi$) is a bounded domain, but we do not assume it has a Lipschitz boundary (see Figure <ref>a). For, this reason we proceed differently. The deformation $v$ is defined in $\O_\e$ and satisfies

$$\|dist(\nabla v, SO(3))\|_{L^{2}(\O^B_\e)}+\e\|dist(\nabla v, SO(3))\|_{L^{2}(\O^M_\e)}\leq C\e^{5/2}\|f\|_{L^2(\o)}.$$

The restriction of $v$ to $\O^M_\e$ is naturally extended by itself and satisfies

$$\|dist(\nabla v, SO(3))\|_{L^{2}(\O_\e)}\leq C\e^{3/2}\|f\|_{L^2(\o)}.$$
Then, we can decompose $v$ since it is a deformation of the plate $\O_\e$. After passing to the limit, we can use the estimate $\|dist(\nabla v, SO(3))\|_{L^{2}(\O^B_\e)}\leq C\e^{5/2}\|f\|_{L^2(\o)}$ to show that the elementary limit of the extended deformation vanishes in $\o\X Y_\kappa^B$. Hence all our results are valid for the general composite plate with soft matrix.

Our results are also valid for composite plate $\O_\e$ when the part with soft matrix $\O_\e^M$ is not connected, we define such structure as

* Let $\GC^*\subset Y$ be a bounded  domain with Lipschitz boundary such that the interior$\left(\overline{\GC}\cup(\overline{\GC}+\Ge_\alpha)\right)$  for $\alpha=1,2$ are connected  with Lipschitz boundary, where $\GC=Y\setminus\overline{\GC^*}$, for example see Figure <ref>b;

* $Y_\kappa^B=\GC\X(-\kappa,\kappa)$, and $Y_\kappa^M=\Yc\setminus\overline{Y_\kappa^B}$.

Using $Y_\kappa^B$ and $Y_\kappa^M$ as the reference periodicity cell of the stiff part and matrix part respectively, we define $\O_\e^B$ and $\O_\e^M$. Further analysis is same as the case with connected $\O_\e^M$.  Hence all our results are valid for this type of composite plate with soft matrix. [The periodic perforated plate $\O_\e^B$ is studied in [30] with respect to a linear elasticity problem starting with displacements.].

§ CONCLUSION

This paper focuses on the asymptotic analysis of a periodic composite plate with matrix pre-strain within the framework of non-linear elasticity. Two cases are considered: one without pre-strain and one with pre-strain. In both cases, the total elastic energy follows the von-Kármán (vK) regime. The re-scaling unfolding operator is utilized to derive the asymptotic behavior of the Green St. Venant's strain tensor. The existence of a minimizer for the limit energy is shown through $\Gamma$-convergence. The discussion of both cases aims to demonstrate that the soft matrix only has an impact on the homogenized plate when pre-strain is present. This behavior is attributed to the relative weakness of the matrix compared to the stiff part, as well as the stability of the structure (due to similar orders of thickness $r$ and periodicity $\varepsilon$). Furthermore, it is shown that the cell problems are linear elastic and the homogenized von-Kármán plate exhibits orthotropic behavior for isotropic homogenized material. The findings of this study have practical applications in the fields of 3D printing, textiles, and aerospace industries.

§ ACKNOWLEDGMENT

The research was funded by DFG, German Research Foundation, project number OR 190/10-1 and AIF project OptiDrape, both in the collaboration with the textile institute ITA in Aachen, Germany.

#1ISBN #1#1#1 #1#1 #1#1#1#1 #1#1 #1#1#1#1 #1#1 #1#1 #1#1 #1#1 #1#1 et al. #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 #1#1 <><#>1#1 #1#1 #1#1 #1#1 #1#1 #1#1 ˜˜126

[1]

H.C. Koch,
D. Schmelzeisen and
T. Gries,
4D textiles made by additive manufacturing on pre-stressed textiles:
An overview,
in: Actuators,
Vol. 10,
p. 31.

[2]

D. Cioranescu,
A. Damlamian and
G. Griso,
Periodic unfolding and homogenization,
Comptes Rendus Mathematique

[3]

D. Cioranescu,
A. Damlamian and
G. Griso,
The periodic unfolding method in homogenization,
SIAM Journal on Mathematical Analysis

[4]

L. Tartar,
The general theory of homogenization: a personalized introduction,
Vol. 7,
Springer Science & Business Media,

[5]

A. Bensoussan,
J.-L. Lions and
G. Papanicolaou,
Asymptotic analysis for periodic structures,
Vol. 374,
American Mathematical Soc.,

[6]

D. Cioranescu and
P. Donato,
An introduction to homogenization,
Oxford University Press,

[7]

G. Nguetseng,
A general convergence result for a functional related to the theory of
SIAM Journal on Mathematical Analysis

[8]

G. Allaire,
Homogenization and two-scale convergence,
SIAM Journal on Mathematical Analysis

[9]

G.P. Panasenko,
Multi-scale modelling for structures and composites,
Vol. 3,

[10]

G. Friesecke,
R.D. James and
S. Müller,
A theorem on geometric rigidity and the derivation of nonlinear plate
theory from three-dimensional elasticity,
Communications on Pure and Applied Mathematics: A Journal Issued by
the Courant Institute of Mathematical Sciences

[11]

G.D. Maso,
An Introduction to Gamma-convergence,
Springer Science+Business Media,

[12]

A. Braides,
Gamma-Convergence for Beginners,
Oxford University Press,
ISBN 9780198507840.

[13]

G. Friesecke,
R.D. James and
S. Müller,
A hierarchy of plate models derived from nonlinear elasticity by
Archive for rational mechanics and analysis

[14]

G. Friesecke,
R.D. James and
S. Müller,
The Föppl–von Kármán plate theory as a low energy
$\Gamma$-limit of nonlinear elasticity,
Comptes Rendus Mathematique

[15]

P.G. Ciarlet,
Three-dimensional elasticity,

[16]

P.G. Ciarlet and
C. Mardare,
An Introduction to Shell Theory,

[17]

M. de Benito Delgado and
B. Schmidt,
A hierarchy of multilayered plate models,
ESAIM: Control, Optimisation and Calculus of Variations

[18]

D. Blanchard,
A. Gaudiello and
G. Griso,
Junction of a periodic family of elastic rods with a 3d plate. Part
Journal de mathématiques pures et appliquées

[19]

D. Blanchard,
A. Gaudiello and
G. Griso,
Junction of a periodic family of elastic rods with a thin plate. Part
Journal de mathématiques pures et appliquées

[20]

D. Cioranescu,
A. Damlamian and
G. Griso,
The periodic Unfolding Method: Theory and Applications to Partial
Differential Problems,
ISBN 10.1007/978-981-13-3032-2.

[21]

D. Blanchard and
G. Griso,
Asymptotic behavior of a structure made by a plate and a straight
Chinese Annals of Mathematics, Series B

[22]

S. Neukamm and
I. Velčić,
Derivation of a homogenized von-Karman plate theory from 3D nonlinear
Mathematical Models and Methods in Applied Sciences

[23]

K. Böhnlein,
S. Neukamm,
D. Padilla-Garza and
O. Sander,
A homogenized bending theory for prestrained plates,
Journal of Nonlinear Science

[24]

I. Velčić,
On the general homogenization of von Kármán plate equations
from three-dimensional nonlinear elasticity,
Analysis and Applications

[25]

M. Lewicka,
Quantitative immersability of Riemann metrics and the infinite
hierarchy of prestrained shell models,
Archive for Rational Mechanics and Analysis

[26]

M. Lewicka,
A. Raoult and
D. Ricciotti,
Plates with incompatible prestrain of high order,
in: Annales de l'Institut Henri Poincaré C, Analyse non
Vol. 34,
pp. 1883–1912.

[27]

G. Griso,
J. Orlik and
S. Wackerle,
Asymptotic behavior for textiles,
SIAM Journal on Mathematical Analysis

[28]

J. Orlik,
R. Falconi,
G. Griso and
S. Wackerle,
Asymptotic behavior for textiles with loose contact,
Mathematical Methods in the Applied Sciences

[29]

G. Griso,
M. Hauck and
J. Orlik,
Asymptotic analysis for periodic perforated shells,
ESAIM: Mathematical Modelling and Numerical Analysis

[30]

G. Griso,
L. Khilkova,
J. Orlik and
O. Sivak,
Homogenization of perforated elastic structures,
Journal of Elasticity

[31]

G. Griso,
J. Orlik and
S. Wackerle,
Asymptotic behavior for textiles in von-Kármán regime,
Journal de Mathématiques Pures et Appliquées

[32]

D. Blanchard and
G. Griso,
Decomposition of the deformations of a thin shell. Asymptotic behavior
of the Green-St Venantś strain tensor,
Journal of Elasticity

[33]

D. Blanchard and
G. Griso,
Decomposition of deformations of thin rods: application to nonlinear
Analysis and Applications

[34]

G. Griso,
Decomposition of plate displacements via Kirchhoff-Love displacements

[35]

C. Eckart,
The thermodynamics of irreversible processes. III. Relativistic theory
of the simple fluid,
Physical review

[36]

E.H. Lee,
Elastic-plastic deformation at finite strains

[37]

O.A. Oleïnik,
A. Shamaev and
G. Yosifian,
Mathematical problems in elasticity and homogenization,


# VidereX: A Navigational Application inspired by ants

Nam Ho Koh1, Doran Amos2, Paul Graham3, Andrew Philippides3  1Computer Science
& Engineering
University of Michigan - Ann Arbor, MI. Email<EMAIL_ADDRESS>2School of
Life Sciences
University of Sussex, Brighton. Email<EMAIL_ADDRESS>3School of Life
Sciences
University of Sussex, Brighton. Email<EMAIL_ADDRESS>4School of
Engineering and Informatics
University of Sussex, Brighton. Email<EMAIL_ADDRESS>
###### Abstract

Navigation is a crucial element in any person’s life, whether for work,
education, social living or any other miscellaneous reason; naturally, the
importance of it is universally recognised and valued. One of the critical
components of navigation is vision, which facilitates movement from one place
to another. Navigating unfamiliar settings, especially for the blind or
visually impaired, can pose significant challenges, impacting their
independence and quality of life [3]. Current assistive travel solutions have
shortcomings, including GPS limitations and a demand for an efficient, user-
friendly, and portable model [3, 6]. Addressing these concerns, this paper
presents VidereX [13], a smartphone-based solution using an ant-inspired
navigation algorithm. Emulating ants’ ability to learn a route between nest
and feeding grounds after a single traversal, VidereX enables users to rapidly
acquire navigational data using a one/few-shot learning strategy. A key
component of VidereX is its emphasis on active user engagement. Like ants with
a scanning behaviour to actively investigate their environment, users wield
the camera, actively exploring the visual landscape. Far from the passive
reception of data, this process constitutes a dynamic exploration, echoing
nature’s navigational mechanisms.

##  I Introduction

Many visually impaired and blind suffer from being excluded from activities
that strongly depend on navigation and mobility. Consequently, they are
discouraged from further participation in social interactions, which may also
result in isolation and depression [16]. In 2020, the World Health
Organization estimated that the total number of visually impaired people stood
at 285 million worldwide, of which 29 million are blind, and 246 million have
low vision [18]. For a sighted person to navigate an environment, one can
easily acquire enough visual cues consisting of building names, street signs
and distinct landmarks, which is crucial in guiding them to their destination.
On the other hand, the blind and visually impaired are at a significant
disadvantage as they do not have direct access to the same type of
information.

According to Brambring’s locomotion model of the blind, perception, landmarks,
and orientation are critical to their environmental navigation [7]. Building
on this model, this paper demonstrates that visual cues can be an effective
navigational tool when embodied in the VidereX system, which instantiates a
variant of our ant-inspired navigation algorithm [1] for use with a standard
mobile phone. In particular, VidereX allows for rapid one/few-shot learning of
routes because users actively interact with their environment, similar to how
the ants the algorithm is based on scan to explore their surroundings actively
[19]. Here we present the VidereX system and show that it is possible for
users, even with limited visual input, can rapidly learn and navigate their
surroundings through active engagement and exploration, leveraging visual cues
for efficient and independent navigation.

Background: visual guidance aids. Navigating their environment, the blind and
visually impaired often resort to traditional aids such as the long cane,
guide dogs, and sighted guides to overcome mobility challenges. However, these
methods have their shortcomings. Guide dogs, while helpful, are not
universally adopted and can impose considerable costs due to potential
mismatching issues [9]. The long cane offers valuable ground-level information
but can lead to awkward reaching behaviours [12], cause pedestrian tripping in
crowded spaces, and offer limited protection above the waist. Furthermore,
navigating with impaired sight increases the risk of head-level and fall
accidents, often requiring medical treatment [10], consequently reducing
confidence in mobility. As for sighted guides, they provide reliable
assistance but require significant trust and entail additional costs. Adding
to these challenges is the inherent inadequacy of GPS technology in indoor
navigation. GPS signals struggle to penetrate buildings, rendering them less
effective indoors. This lack of accurate positioning can pose considerable
difficulties for the visually impaired when navigating indoor spaces, further
compounding their mobility challenges. Thus, there is a pressing need for more
effective and reliable navigation aids for the blind and visually impaired.

Extensive research has been conducted in assistive technology and electronic
travel aids [12, 10, 2, 17] for the visually impaired and blind. This may be
attributed to the scope of research extending from the physiological factors
pertaining to vision loss to human factors associated with mobility, access to
information and various technological developmental aspects in the forms of
navigation and way-finding. Due to this expansive research space, authors in
[2] state that it is a challenge to capture the true essence of this field
into one single concept. However, despite the progress in the domain, many of
the navigation systems proposed for the blind and visually impaired are
brought with various limitations, and only a mere few can truly deliver a
dynamic, seamless and robust solution [17]. From analyzing the different
solutions present in addressing navigation challenges for the visually
impaired and blind, it is clear that many tools often require the user to
carry additional equipment and hardware, which might not be as comfortable nor
inexpensive [6]. Furthermore, the emergence of several GPS-based solutions may
provide specific degrees of accuracy and reliability for blind and visually
impaired users’ daily navigational tasks. However, due to the inherent flaws
of GPS, an alternative solution might be needed.

Background: Familiarity-based navigation. In this paper, we introduce a
navigation model inspired by the behaviours of ants. In their natural
environment, ants set out, return via path integration, and on this first
route, traversal can acquire the navigational knowledge to repeat this journey
using only visual information [11]. This ability is primarily thought to be
due to their active strategy of using views as a ”visual compass”. That is,
the ants scan their surroundings, compare what they see to the views they
remembered during the first PI-mediated route traversal, and then move in the
most familiar direction (i.e. most similar when compared to their memories).
This active scanning approach forms the basis for our navigation algorithm.

Background: Overview. The challenge we address is translating this ant-
inspired strategy to a handheld camera. Unlike ants’ panoramic perception, a
camera offers a limited field of view, necessitating a sweeping motion to
capture the surroundings. We developed a mobile application and algorithm
replicating this ant-like active scanning and one-shot learning. With VidereX,
a user walks a path, and the visual data is captured and uploaded.
Subsequently, a second user actively navigates by continually sampling the
visual environment. This paper presents the application and shows proof in
principle that this model can be effectively implemented in a mobile
application. Specifically, we demonstrate the feasibility and potential of
active, visual navigation by showing that a user can actively regain the
correct heading by a sweep through the environment.

##  II Methods

Methods: Familiarity-based navigation. As detailed in [1], ant-inspired
familiarity-based navigation is a two-phase algorithm encompassing training
and execution stages. In the training phase, the agent traverses the route and
captures panoramic views to create a comprehensive route memory. In the
execution phase, these captured views are used as ”visual compasses”, where
the agent compares their current view to the memories and moves in the most
familiar direction by seeing which direction is most similar to its stored
memories. The agent identifies the most familiar direction by rotating in
place, comparing each rotated version of the current view with its stored
snapshots. This process replicates ants’ saccadic head movements to find
familiar views and decide their next move, allowing them to retrace their path
reliably [1, 14].

Methods: VidereX algorithm. The algorithm used in VidereX is a variant of the
Perfect Memory algorithm [8]. In our version, the user performs the training
sequence by stepping through the route while recording it on their mobile
device. Subsequently, the image frames will be extracted and stored as the
main route. When returning to the same route, the user will scan the path with
their camera, where the images in memory will be compared against the current
frames being passed in real-time. Moreover, the difference between each frame
in memory and the real-time frame being viewed by the user is denoted by an
auditory tone, where a higher tone signifies the best match. Therefore, by
listening to the tone alone, the user could head towards the correct heading
along the trained route. This interaction is represented in figure 3. To
ascertain the similarity between the current view and each of the training
views, we follow Zeil et al. [20] and use an image difference function (IDF)
which gives the difference between a remembered training view or ‘snapshot’
and the current world view (X, Y) respectively [15].

$\operatorname{IDF}(X,Y)=\frac{1}{P}\sqrt{\sum_{i}\sum_{j}(X(i,j)-Y(i,j))^{2}}$
(1)

where X(i,j) represents the pixel in the ith row and the jth column of image
X. This simple RMS measure of pixel difference has been shown to be a reliable
cue for visual homing [20, 14]. However, [15] outlined potential limitations
present due to the variation in illumination and environmental changes
relating to the time of day and variability of vegetation.

Figure 1: Example of training sequence performed on device. The user is
scanning the environment to determine the correct heading from the learnt
views from training.

Methods: VidereX Application. The final prototype of the application (VidereX)
contains a rich set of features to augment the BVI navigational experience. In
essence, the application will allow the user to perform a single matching
action with a single image taken as the reference point, record an entire
route via a video, store it on-device and via the cloud using Firebase and
follow any route from the start. In addition, the application carries a robust
on-device NLU (Natural Language Utterance) system, which facilitates hands-
free interactions for increased accessibility user experience.

Methods: Platform and tools. A Samsung GalaxyS10+ and GalaxyS21 Ultra were
during the development and testing cycle. The OpenCV4Android (C++ Native 3.0)
was used for all image processing, The ASR (Automatic Speech Recognition) was
handled with TensorFlow Lite and the TTS (text-to-speech) feature used
Android’s native NLU model.

Figure 2: MainMenuActivity screenshot of VidereX. The user is able to execute
single match, record route or follow route activities. The menu also holds a
speech execution button which will allow the user to perform all actions
through natural language commands.

Methods: Single Matching. The single match activity allows the user to take a
single image and utilize it for visual matching against the environment. The
application informs the user via auditory and tactile cues that a match has
occurred. The former is set through a tone generator library in which the
maximum and minimum difference values from the calculation are normalized to
the range of the tone generation frequency. In this case, a low difference
value denoting a good match will be met with a high frequency, and in
contrast, a high difference value will return a low frequency. The latter
takes advantage of the Android vibration library, which triggers when a good
match has occurred. The implementation of both signals seemed appropriate, as
recommended in [5].

Methods: Record Route. The record route activity allows the user to step and
record through an entire route via video recording. Once the video has been
recorded, the user can name the route as demonstrated below:

Once the user has done so, they can extract the frames from the video, which
will be used for visual guidance; these frames are stored locally; however, in
future implementations, the images will be stored on a cloud instance to
minimize storage and increase accessibility. Moreover, once the user has been
notified that the frames have been extracted and stored, they can either
follow the route directly or store the route for later access.

Methods: Follow Route. The follow route activity allows users to select a
stored route on the cloud for navigation. Once a specific route has been
chosen via voice or touch, the application will call a data retrieval method
to the selected route, as shown in Figure 6.4. Furthermore, if the route
exists, the application will pass the respective image paths to the next
activity for the images to be used for navigation. Moreover, the route data is
collectively stored on Firebase via Firestore [4], a scalable database aimed
at mobile and web applications. Its ability to perform atomic batch operations
offers a lightweight solution ideal for a smartphone application that relies
on low latency and high performance.

Figure 3: Outside route demonstration.

Methods: Dataset. In order to test the algorithm, we collected 20,418 image
samples in varying locations by taking a 180° sweep at respective positions
along each route, as shown in Figure 4 below. In addition, 4 unique test
positions were taken 7cm apart (approximately one step) with the camera facing
directly forwards in parallel.

Figure 4: Enhanced data collection route. A denotes the starting position, B
represents the goal position. The dotted circles represent the rotational
sweeps and the unmarked circles left-right of the sweeps represent the
individual test positions taken.

The purpose of this data collection method was to augment the data by
capturing different angular positions and distances as well as allowing the
algorithm to be tested across different environments and scenery.

##  III Results

Results: Testing environment. Our initial test of the algorithm sought to
assess its capability in a real-world setting. We were particularly interested
in understanding whether a user, actively searching, could successfully
identify the correct path with the aid of our algorithm. Furthermore, to gauge
the robustness and accuracy of our navigation solution, we tested positions at
different distances from the training path. The results of these initial
trials are shown in fig 5. Here we see that a general heading can be derived,
as shown by the widened minima of the RIDFs. This suggests that each instance
captures synonymous visual cues and information from the environment at each
position.

Figure 5: RIDF plot taken from route rotational sweep against the test
position. General heading can be derived shown by the widened minima of the
functions with similar patterns. This suggests that each instance captures
synonymous visual cues and information from the environment at each position.

We found that the angular error slightly increased as the user deviated from
the centre of the sweep. However, the deviations remained within acceptable
ranges, thus demonstrating the potential for practical yet reliable
navigation. The Rotational Image Difference Function (RIDF) depicted in figure
5 confirms the presence of the necessary visual information that permits the
user to find the true heading despite the sweep being performed manually, as
we can see the clear minimum near the true heading. In addition, these results
remain consistent across different environments as the error distribution,
visualised in the box plots of figure 6, shows the algorithm’s effectiveness
and robustness at different distances from the route.

Figure 6: Boxplots of error distribution across rotational sweeps and test
positions along the different routes.

It is crucial to mention that although the algorithm is extremely promising in
its wide applicability in navigation, continuous rigorous testing is needed to
prove the algorithms worth.

##  IV Conclusion

Support and care for the blind and visually impaired have been progressively
improving worldwide, mainly due to technological advancements and the
development of assistive navigational solutions. While these improvements are
noteworthy, our vision, often taken for granted due to its constant presence,
is an invaluable tool that is profoundly missed when absent. This paper shows
that studying the visual behaviour of ants in their natural environment can
provide a solution in a very different context by translating their remarkable
navigation strategies into a smartphone application for the blind and visually
impaired.

Exploring this algorithm’s implementation deepened the authors’ understanding
of assistive technology solutions and shed light on the various user
constraints in this underrepresented global population. Navigation, whether
for sighted or non-sighted individuals, remains a fundamental principle that
empowers and liberates individuals. However, as we look to the future, there
are improvements to make to VidereX to make it more robust. Collecting more
data and conducting detailed testing is the first step to allow us to refine
the algorithm, enhancing its precision and reliability. Additionally,
incorporating additional modalities, such as an inertial measurement unit
(IMU), could enhance the system’s navigational capabilities by restricting
matches to only those where images have similar attitudes.

In summary, while our insect-inspired navigation algorithm implemented in a
smartphone application is functional and promising, it represents a starting
point. The continued exploration, development, and improvement of such
assistive technology solutions could be crucial to promoting the independence
and quality of life of the blind and visually impaired.

## Acknowledgments

We thank the EPSRC (Engineering and Physical Sciences Research Council) for
funding Andy Philippides and the CCNR (Centre for Computational Neuroscience
and Robotics) for the support.

## References

  * Baddeley et al. [2012] Bart Baddeley, Paul Graham, Philip Husbands, and Andrew Philippides.  A model of ant route navigation driven by scene familiarity.  _PLoS Comput Biol_ , 8(1):e1002336, 2012.
  * Bhowmick and Hazarika [2017] Alexy Bhowmick and Shyamanta M Hazarika.  An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends.  _Journal on Multimodal User Interfaces_ , 11(2):149–172, 2017.
  * Chanana et al. [2017] Piyush Chanana, Rohan Paul, M Balakrishnan, and PVM Rao.  Assistive technology solutions for aiding travel of pedestrians with visual impairment.  _Journal of rehabilitation and assistive technologies engineering_ , 4:2055668317725993, 2017.
  * FireBase [2019] FireBase.  Cloud firestore, October 2019.  URL https://firebase.google.com/docs/firestore.
  * Giudice and Legge [2008] Nicholas A Giudice and Gordon E Legge.  Blind navigation and the role of technology.  _The engineering handbook of smart technology for aging, disability, and independence_ , 8:479–500, 2008.
  * Helal et al. [2008] Abdelsalam Helal, Mounir Mokhtari, and Bessam Abdulrazak.  _The engineering handbook of smart technology for aging, disability, and independence_.  John Wiley & Sons, 2008.
  * Hersh and Johnson [2010] Marion Hersh and Michael A Johnson.  _Assistive technology for visually impaired and blind people_.  Springer Science & Business Media, 2010.
  * Knight et al. [2019] James C Knight, Daniil Sakhapov, Norbert Domcsek, Alex DM Dewar, Paul Graham, Thomas Nowotny, and Andrew Philippides.  Insect-inspired visual navigation on-board an autonomous robot: Real-world routes encoded in a single layer network.  In _ALIFE 2019: The 2019 Conference on Artificial Life_ , pages 60–67. MIT Press, 2019.
  * Lloyd et al. [2016] Janice Lloyd, Claire Budge, Steve La Grow, and Kevin Stafford.  An investigation of the complexities of successful and unsuccessful guide dog matching and partnerships.  _Frontiers in veterinary science_ , 3:114, 2016.
  * Manduchi and Kurniawan [2011] Roberto Manduchi and Sri Kurniawan.  Mobility-related accidents experienced by people with visual impairment.  _AER Journal: Research and Practice in Visual Impairment and Blindness_ , 4(2):44–54, 2011.
  * Mangan and Webb [2012] Michael Mangan and Barbara Webb.  Spontaneous formation of multiple routes in individual desert ants (cataglyphis velox).  _Behavioral Ecology_ , 23(5):944–954, 2012.
  * Marston and Golledge [2003] James R Marston and Reginald G Golledge.  The hidden demand for participation in activities and travel by persons who are visually impaired.  _Journal of Visual Impairment & Blindness_, 97(8):475–488, 2003.
  * NamhoKoh [2017] NamhoKoh.  Viderex repository, May 2017.  URL https://github.com/namhkoh/VidereX.
  * Philippides et al. [2011] Andrew Philippides, Bart Baddeley, Ken Cheng, and Paul Graham.  How might ants use panoramic views for route navigation?  _Journal of Experimental Biology_ , 214(3):445–451, 2011.
  * Philippides et al. [2012] Andrew Philippides, Bart Baddeley, Philip Husbands, and Paul Graham.  How can embodiment simplify the problem of view-based navigation?  In _Conference on Biomimetic and Biohybrid Systems_ , pages 216–227. Springer, 2012.
  * Popescu et al. [2012] Mihaela L Popescu, Hélène Boisjoly, Heidi Schmaltz, Marie-Jeanne Kergoat, Jacqueline Rousseau, Solmaz Moghadaszadeh, Fawzia Djafari, and Ellen E Freeman.  Explaining the relationship between three eye diseases and depressive symptoms in older adults.  _Investigative ophthalmology & visual science_, 53(4):2308–2313, 2012.
  * Ran et al. [2004] Lisa Ran, Sumi Helal, and Steve Moore.  Drishti: an integrated indoor/outdoor blind navigation system and service.  In _Second IEEE Annual Conference on Pervasive Computing and Communications, 2004. Proceedings of the_ , pages 23–30. IEEE, 2004.
  * who [2007] who.  Global data on visual impairment, May 2007.  URL https://www.who.int/blindness/publications/globaldata/en/.
  * Wystrach et al. [2014] Antoine Wystrach, Andrew Philippides, Amandine Aurejac, Ken Cheng, and Paul Graham.  Visual scanning behaviours and their role in the navigation of the australian desert ant melophorus bagoti.  _Journal of Comparative Physiology A_ , 200:615–626, 2014\.
  * Zeil et al. [2003] Jochen Zeil, Martin I Hofmann, and Javaan S Chahl.  Catchment areas of panoramic snapshots in outdoor scenes.  _JOSA A_ , 20(3):450–469, 2003.