# Bayesian optimization for sparse neural networks with trainable activation
functions

Mohamed Fakhfakh1,2 and Lotfi Chaari2 1 University of Sfax, MIRACL, Tunisia
<EMAIL_ADDRESS>2 Toulouse INP, IRIT, University of Toulouse,
France
<EMAIL_ADDRESS>
###### Abstract

In the literature on deep neural networks, there is considerable interest in
developing activation functions that can enhance neural network performance.
In recent years, there has been renewed scientific interest in proposing
activation functions that can be trained throughout the learning process, as
they appear to improve network performance, especially by reducing
overfitting. In this paper, we propose a trainable activation function whose
parameters need to be estimated. A fully Bayesian model is developed to
automatically estimate from the learning data both the model weights and
activation function parameters. An MCMC-based optimization scheme is developed
to build the inference. The proposed method aims to solve the aforementioned
problems and improve convergence time by using an efficient sampling scheme
that guarantees convergence to the global maximum. The proposed scheme is
tested on three datasets with three different CNNs. Promising results
demonstrate the usefulness of our proposed approach in improving model
accuracy due to the proposed activation function and Bayesian estimation of
the parameters.

###### Index Terms:

Activation function, Deep neural networks, Optimization, MCMC, Hamiltonian
dynamics

##  I Introduction

Classification is a machine-learning task that identifies which objects are
present in an image or video. It is critical for various applications such as
computer vision [1, 2, 3], medical diagnostics [4], signal processing [5], and
others. The process involves learning significant or nontrivial relationships
from a set of training data and extending these relationships to interpret new
test data [6]. The task involves categorizing elements into one of the finite
set of classes by comparing the measured attributes of a given object with the
known properties of objects to determine whether the object belongs to a
specific category.
Convolutional Neural Networks (CNNs) [7, 8, 9, 10, 11] have become the
industry standard in numerous applications over the past two decades, as they
can process complex high-dimensional input data into simple low-dimensional
concepts through a series of nonlinear transformations. Each feature layer in
CNNs comprises features from the layer below, creating a hierarchical
organization of ever-more-abstract concepts. They are particularly effective
at capturing high-level abstractions in real-world observations, making them a
popular choice for image classification tasks. CNNs can learn features from
raw image pixels, reducing the need for manual feature engineering. This makes
them well-suited for tasks such as object recognition, where the goal is to
identify the presence and location of specific objects in an image. In this
context, the activation function plays a critical role in learning
representative features. The Rectified Linear Unit (ReLU) [12] is currently
the most widely used activation function for neural networks. When provided
with a positive argument, the ReLU activation function is either zero or the
identity. ReLUs have the additional benefit of alleviating the vanishing
gradient problem, in addition to providing sparse codes [13]. According to the
state of the art, activation functions can be fixed or trainable during a
learning phase [14]. In most cases, gradient descent is the most widely used
method in the literature for parameter estimation.
From another side, Bayesian methods have advanced significantly in many
domains over time and have numerous useful applications. The primary idea is
to represent all uncertainties in the model using probabilities. Bayesian
techniques are distinctive in that they treat the problem as an inference
problem [15]. One of the most significant advantages is the ability to
incorporate prior information about the model parameters and hyperparameters.
Recent advancements in Markov Chain Monte Carlo (MCMC) methods [16, 17, 18,
19] make it easier to use Bayesian analyses in complex datasets with missing
observations and to handle multidimensional outcomes. Recent studies have
demonstrated that using a Bayesian framework in CNNs for the optimization
process leads to more promising performances than standard gradient descent
[20, 21].
In this study, we introduce a new trainable activation function. The
parameters of the proposed function are automatically estimated from the data.
For doing so, a Bayesian framework is used where these parameters as well as
the network weights are assumed to be realizations of random variables. With
an adequate likelihood, a hierarchical Bayesian model is built with priors and
hyperpriors. An MCMC-based inference is then use, specifically a Gibbs
sampler, to derive estimators from the target distributions. Our method is an
extension of our previous work described in [22], where we employed non-smooth
Hamiltonian methods to fit sparse artificial neural networks. Our main
objective with the Bayesian scheme is to minimize the target cost function of
the learning model. The use of non-smooth Hamiltonian techniques enables us to
perform efficient and fast sampling, even when dealing with non-differentiable
energy functions that arise due to the use of sparse regularization functions.
The contribution of this paper is therefore twofold: _i)_ proposing a new
trainable activation function, and _ii)_ a more general Bayesian formulation
than in [22] to integrate the estimation of all parameters from the data.
The rest of this paper is organized as follows. The state of the art is the
focus of Section II. Then, the Problem statement is in Section III. In section
IV we detail the adopted hierarchical Bayesian model. The proposed Bayesian
inference scheme is developed in Section V and validated in Section VI.
Finally, The conclusion and future work are drawn in Section VII.

##  II Related work

Finding the best activation function to integrate into an architecture is a
challenging task. This section proposes a taxonomy of different activation
functions described in the literature. A primary established classification is
based on the ability to modify the shape of the activation function during the
training phase, resulting in two major categories that can be distinguished
[14].

###  II-A Fixed-shape activation functions

This category pertains to the use of activation functions in neural network
research, including sigmoid [23], hyperbolic tangent (tanh) [24], and ReLU,
all of which have a defined form. The introduction of rectified functions,
particularly ReLU, has led to a marked enhancement in neural network
performance and heightened scientific interest. Consequently, this category
can be subdivided into subcategories based on their distinctive
characteristics.

Classic activation functions:
The findings presented in [25] demonstrated that a feed-forward and shallow
network can efficiently manage any continuous function defined on a compact
subset. For many years, bounded activation functions like Sigmoid and tanh
were the preferred choices for neural networks, with researchers demonstrating
their efficacy, particularly in shallow network architectures [26]. Although
Sigmoid, Bipolar sigmoid [27], Hyperbolic tangent, Absolute value [28], and
other bounded activation functions are commonly used, their efficacy is
limited when training multi-layer neural networks due to the vanishing
gradient problem [29].

Rectifier-based activation functions:
The primary advantage of using rectified activation functions is to alleviate
the problem of vanishing gradient. The success of ReLU [30] has inspired the
development of many new activation functions during the last years [31, 32].
While ReLU has numerous benefits such as solving the vanishing gradient issue
and making sparse coding easier [33], it is not without flaws. The ”dying”
ReLU problem [34] and non-differentiability at zero are the main concerns.
To address these issues, several variations of ReLU have been developed, such
as Leaky ReLU (LReLU) [34], Truncated rectified [35], softplus [36],
Exponential linear unit (ELU) [37], E-swish [38], and Flatten-T Swish [39].

###  II-B Trainable activation functions

The concept of using trainable activation functions is not new in the field of
neural network research, with many studies published on this topic as early as
the 1990s [40, 41, 42]. However, the growing interest in neural networks in
recent years has led researchers to reconsider the potential benefits of
trainable activation functions in improving network performance.
In this section, we discuss the main strategies proposed in the literature for
learning activation functions from data. Based on their primary
characteristics, these strategies can be classified into three families.

Parameterized standard activation functions:
In [40], a generalized hyperbolic tangent function was proposed by introducing
two trainable parameters $\alpha$ and $\beta$, which adjust the saturation
level and slope, respectively. Similarly, a sigmoid function with two
trainable parameters was used in [41] to modify the activation function’s
shape. These parameters are learned along with the network weights using the
backpropagation algorithm. More recently, the work in [43] aimed to avoid
manually setting the parameter of the ELU unit by proposing an alternative
based on two trainable parameters. The proposed activation function, called
PELU, is defined as follows:

$PELU(x)=\begin{cases}\frac{\beta}{\gamma}x&\text{if $x\geq 0$}\\\
\beta\times(exp(\frac{x}{\gamma})-1)&\text{otherwise,}\end{cases}$ (1)

where $\beta$ is a trainable parameter.

A flexible ReLU function has been proposed in [44]:

$frelu(x)=ReLU(x+\alpha)+\beta,$ (2)

where $\alpha$ and $\beta$ are parameters learned from data. This is done to
capture negative information that is lost in the classic ReLU function [12].

The activation function introduced in [45] is another type of ReLU function
that partially learns its shape from the training set. Indeed, it can modify
the negative part of the data via the parameter $\alpha$. This function is
called Parametric ReLU (PReLU) and can be defined as follows:

$PReLU(x)=\begin{cases}x&\text{if $x>0$}\\\ \alpha\times
x&\text{otherwise}\end{cases}$ (3)

where the parameter $\alpha$ is learned jointly with the model using a
gradient method.

Functions based on ensemble methods:
Functions based on ensemble methods involve combining multiple basic
activation functions to form a more complex function. In [46], a method for
investigating activation functions built as compositions of various basic
activation functions are proposed. Similarly, [47] introduces a similar method
using a genetic algorithm composed of two new activation functions:
Exponential Linear Sigmoid SquasHing (EliSH) and HardELiSH. EliSH is defined
as follows:

$EliSH(x)=\begin{cases}x/(1+e^{-x})&\text{if $x\geq 0$}\\\
(e^{x}-1)/(1+e^{-x})&\text{otherwise}.\end{cases}$ (4)

The negative part of EliSH is a multiplication of two functions, ELU and
Sigmoid, while the positive part is shared with Swish. HardELiSH is defined as
a multiplication of HardSigmoid and ELU in the negative portion and
HardSigmoid and Linear in the positive part. In [48], another interesting
activation function is proposed, called the Mexican Hat Linear Unit (MeLU).
This activation function solves the problems of unstable learning related to
trainable parameters. Unstable learning can cause a decrease in accuracy and
an increase in generalization error when the model’s performance varies
significantly in response to slight changes in the data or parameters. Mexican
hat-type functions have a smoother curve than ReLU, which prevents saturation
and allows for optimal performance.
Let $f$ be the function defined by

$f_{\gamma,\lambda}(x)=max(\lambda-|x-\gamma|,0),$ (5)

where $\lambda$, $\gamma$ are real numbers. This function returns zero when
$\lvert x-\gamma|>\lambda$. Moreover, it increases with a derivative of 1
between $\gamma-\lambda$ and $\gamma$, then decreases using a derivative of -1
between $\gamma$ and $\gamma+\lambda$. MeLU is defined for each layer as

$MeLU(x)=PReLU(x)+\sum_{j=1}^{k-1}c_{j}f_{\gamma_{j},\lambda_{j}}(x),$ (6)

where $k$ represents the number of parameters that can be learned in each
neuron. The $c_{j}$ parameters are learnable real numbers, while $\gamma_{j}$,
$\lambda_{j}$ are fixed parameters.
In [49], the authors introduced a new function called Adaptive Blending Units
(ABU) as a trainable linear combination of a set of activation functions

$ABU(x)=\sum_{i=1}^{k}\alpha_{i}\times f_{i}(x),$ (7)

where ($\alpha_{1}$, $\alpha_{2}$,…, $\alpha_{k}$) are parameters to be
learned, and {$f_{1}(.)$, $f_{1}(.)$, …, $f_{k}(.)$} is a set of activation
functions. The parameters $\alpha_{i}$ are all initialized to the value
$\frac{1}{k}$ and are trained using the gradient descent method.
Likewise, similar methods were introduced recently like Kernel-based
activation function [50, 51] and Trained activation function [52].

Activation functions based on other techniques:
In [42], a method using polynomial functions with adjustable coefficients is
proposed. Similarly, in the context of fuzzy activation functions, a neural
unit based on Type-2 fuzzy logic [53] was developed. Other works have proposed
functions using interpolation and spline approaches [54]. However, depending
on the chosen technique, these strategies may require additional input.

###  II-C Comparison and analysis

Learning activation functions is a popular topic in the field of machine
learning because the performance of learning architectures can be improved by
using more suitable activation functions. Various approaches have been
explored to enhance these performances. Most trainable activation functions
are variants of standard activation functions, whose shape is adjusted using
trainable parameters. Several studies on trainable activation functions have
reported substantial performance improvements compared to neural network
architectures equipped with classic fixed activation functions such as ReLU or
sigmoid.
Other trainable activation functions can be expressed as sub-networks nested
within the main architectures or those based on different approaches than
classical activation functions, such as fuzzy logic.

While these activation functions have significant potential to improve neural
network model performance, their implementation can be more complex and
require more time and resources for learning.
Despite encouraging results, it is still challenging to identify a strategy
for automatic learning of an activation function that would solve different
problems and significantly improve performance. Most trainable activation
functions use the gradient descent method for hyperparameter estimation. The
main limitations of these techniques lie in the computation time and gradient
vanishing. This process prevents the network from learning deep features and
may even lead to excessive processing capacity during training.
Indeed, neural networks can get stuck in local minima [55], which can harm the
model’s performance. This phenomenon is partly due to the gradient vanishing
that occurs as derivatives decrease as the model deepens. This gradient
decrease makes it harder to optimize the model, leading to a decrease in
performance.

##  III Problem statement

In the previous Section, we have examined different activation functions
presented in the literature. In this paper, we introduce a modification of the
MeLU activation function [56], while integrating the estimation of its
parameters into a global Bayesian optimization framework. The choice of this
function is mainly justified by its promising performance, as well as its form
which promotes non-linearity and sparsity. However, the limits of the MeLU
function are mainly related to memory requirements. This limit indicates that
using this function may require a larger amount of memory compared to other
activation functions, which can affect computational efficiency.
As a first contribution of this paper, the Modified Mexican ReLU (MMeLU)
activation function is proposed to solve the complexity problem and improve
model performance. The specificity of this new activation function is that it
requires fewer parameters to estimate than MeLU. The second contribution is
related to integrating the parameters estimation of the MMeLU function into a
Bayesian optimizer [57], rather than using a standard optimization procedure
such as the ADAM optimizer.
To define MMeLU, let

$f_{\gamma,b}(x)=max(b-|x-\gamma|,0),$ (8)

where $\gamma$ and $b$ are real numbers.
Using $f_{\gamma,b}$, the proposed activation function MMeLU can be defined as

$\textit{MMeLU}(x)=c\times f_{\gamma,b}(x)+(1-c)\times\textit{ReLU}(x)$ (9)

where $c$ is a real number belonging to the interval $[0,1]$. For the proposed
MMeLU function, $c$, $b$, and $\gamma$ are the parameters to be estimated.

The shape of the proposed MMeLU function, as well as those of competing
functions ReLU, FReLU, PReLU, and MeLU, are illustrated in Figure 1. The
activation function proposed in this paper is made up of a mixture of the ReLU
and the Mexican hat functions. The curves in Figure 1[(a)-(d)] clearly show
the flexibility and non-linearity of our MMeLU function with different
configurations of the parameters $\gamma$, $b$, and $c$.
The Mexican hat function is often used as an activation function in neural
networks due to its advantages. Mexican hat functions are continuous, which
means that changes in inputs produce continuous changes in outputs. This is
important in neural networks because it allows a gradual update of weights.
Additionally, they have high representation capacity, which means that they
can accurately model complex functions. This allows neural networks to learn
non-linear relationships between inputs and outputs.
The Mexican hat function looks like a bell but with a peak in the center that
makes it more pronounced for small values, as shown by the MMeLU curves in
Figures 1[a] and [b]. This means that for small input values, the Mexican hat
function will have a stronger response than other activation functions, such
as ReLU or FReLU. This behavior can be useful in certain situations, for
example when the input data has a restricted value range or when the neural
network’s response needs to be more sensitive to small variations in the input
data.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 1: Illustrations of ReLU, FreLU, PReLU, MeLU, and MMeLU curves with
different configurations.

Let us now consider the convolutional neural network in Figure 2. The MMeLU
activation function is applied after each convolutional layer, replacing the
ReLU function. This strategy has the potential to increase the non-linearity
of the model and improve its ability to represent features in images. It is
worth noting that when $c$ is estimated close to 0 ($c\sim 0$), the MMeLU
function tends to the ReLU behavior.

---
Figure 2: A general diagram of a convolutional neural network with the MMeLU
activation function.

Regarding the model fitting, let us assume that the estimated label (or
numerical value) is obtained by applying the proposed activation function
MMeLU(x,$W$), where x is the input data and $W\in\mathbb{R}^{N}$ denotes the
weights vector. The model parameters (weight vector and parameters of the
activation function) can be determined during the training phase using a
generic error function $D$ (Euclidean, Minkowski, etc.). Akin to [21], the
target CNN is assumed to be sparse. We also use the same Bayesian formulation
of the optimization problem. For the $M$ input data, we can write

$\displaystyle\widehat{W}=\arg\min_{W}\mathcal{L}(W)$
$\displaystyle=\arg\min_{W,b,\gamma,c}\sum_{m=1}^{M}D(MMeLU(x^{m};W)-y^{m})+\sum_{l=1}^{L}\lambda_{l}\|W^{l}\|_{1},$
(10)

where $y^{m}$ is the ground truth for input data $x^{m}$, $L$ is the number of
layers in the network, and $\lambda_{l}$ is a regularization parameter to be
estimated for layer $l$ that balances the solution between the data attachment
term and the $\ell_{1}$ sparse regularization terms.
In the following Section, we formulate the adopted hierarchical Bayesian model
to conduct the inference and fit the model weights and the activation function
parameters.

##  IV Hierarchical Bayesian Model

The problem of estimating the parameters of the MMeLU activation function is
formulated within a Bayesian framework. In this sense, all parameters and
hyperparameters are supposed to follow probability distributions. A likelihood
is defined to model the relationship between the target weight vector, the
activation function parameters, and the data. A prior distribution is defined
to model the prior knowledge about the target weights and all activation
function parameters.

###  IV-A Likelihood

Following the principle of minimizing the error between the reference vector y
(labels or continuous values) and its estimate $\widehat{\textbf{y}}$, we
define the likelihood distribution as

$\displaystyle f(\textbf{y},\textbf{x};W,c,\gamma,b)\propto$
$\displaystyle\prod_{m=1}^{M}\exp\left[-D;(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right].$
(11)

It is worth noting that when a Euclidean distance is used for $D$, the adopted
likelihood is nothing but a Gaussian distribution.

###  IV-B Priors

In our model, the unknown parameters are grouped in the unknown vector
$\theta$ = {$W$, $c$, $\gamma$, $b$, $\lambda$}, where
$\lambda=\\{\lambda_{1},\ldots,\lambda_{L}\\}$.

Prior for $W$:
To promote sparsity in the neural network, we use a Laplace distribution for
the weight vector $W$ akin to [21]:

$f(W;\lambda)\propto\prod_{l=1}^{L}\prod_{k=1}^{K_{l}}\left[\frac{1}{\lambda_{l}}\exp\left(-\frac{\rvert
W_{k}^{l}\rvert}{\lambda_{l}}\right)\right],$ (12)

where $K_{l}$ is the number of weights in layer $l$ of the network and
$\lambda_{l}$ is a parameter to be estimated.

Prior for $\lambda_{l}$:
Since $\lambda_{l}\in\mathbb{R}_{+}$, we chose to use an inverse gamma (IG)
distribution:

$\displaystyle
f(\lambda_{l};\delta,\mu)=IG(\lambda_{l};\delta,\mu)\propto(\lambda_{l})^{-1-\delta}\exp\left(-\dfrac{\mu}{\lambda_{l}}\right),$
(13)

where $\delta$ and $\mu$ are positive parameters that were fixed at $10^{-3}$
to have a non-informative prior.

Prior for $c$:
Regarding the parameter $c$, we consider a uniform distribution over the
interval $[0,1]$, denoted as

$c\sim U_{[0,1]}(c).$ (14)

Prior for $\gamma$ :
Since $\gamma$ is a real value, a Gaussian distribution is used as follows

$f(\gamma;\sigma^{2})=\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{\gamma^{2}}{2\sigma^{2}}\right),$
(15)

where $\sigma^{2}$ is a hyperparameter to be estimated.

Priori for $b$:
Since $b$ is a positive real number, an exponential distribution is used as
follows:

$f(b;\lambda_{b})\propto\begin{cases}\frac{1}{\lambda_{b}}\exp\left(-\dfrac{b}{\lambda_{b}}\right);\text{if};b\geq
0\\\ 0;\text{otherwise}.\end{cases}$ (16)

where $\lambda_{b}$ is a hyperparameter to be estimated. This prior penalizes
large values of $b$.

###  IV-C Hyperpriors

Since $\lambda_{b}$ and $\sigma^{2}$ are positive real numbers, an inverse
gamma (IG) distribution was used as a hyper-_a priori_ :

$\displaystyle
f(\lambda_{b};\delta,\mu)=IG(\lambda_{b};\delta,\mu)\propto(\lambda_{b})^{-1-\delta}\exp\left(-\dfrac{\mu}{\lambda_{b}}\right)$
(17)

and

$\displaystyle
f(\sigma^{2};\delta,\mu)=IG(\sigma^{2};\delta,\mu)\propto(\sigma^{2})^{-1-\delta}\exp\left(-\dfrac{\mu}{\sigma^{2}}\right),$
(18)

where $\delta$ and $\mu$ are positive parameters that were fixed at $10^{-3}$.

##  V Inference scheme

By adopting a Maximum _a Posteriori_ (MAP) approach, we first need to express
the posterior distribution. Let $\Phi_{e}$ be the hyperparameters to be
estimated, represented by $\Phi_{e}$ = {$\sigma^{2}$, $\lambda_{b}$}, and
$\Phi_{m}$ be the hyperparameters to be fixed, $\Phi_{m}$ = {$\delta$, $\mu$}.
Using the likelihood, the prior distributions, and the defined hyperpriors, we
can write the posterior distribution as:

$\displaystyle f(\theta,\Phi_{e};y,\Phi_{m})\propto
f(y;\theta)f(\theta;\Phi_{e})f(\Phi_{e};\Phi_{m})$

which can be reformulated in a detailed version as

$\displaystyle f(\theta,\Phi_{e};\textbf{y},\textbf{x},\Phi_{m})\propto$
$\displaystyle\prod_{m=1}^{M}\exp\left[-D\;(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right]\times$
$\displaystyle\prod_{l=1}^{L}\frac{1}{\lambda_{l}^{K_{l}}}\prod_{k=1}^{K_{l}}\left[\exp\left(-\frac{\rvert
W_{k}^{l}\rvert}{\lambda_{l}}\right)\right]\times(\lambda_{l})^{-1-\delta}\exp\left(-\dfrac{\mu}{\lambda_{l}}\right)\times$
$\displaystyle\exp\left(-\frac{\gamma^{2}}{2\sigma^{2}}\right)\times\frac{1}{\lambda_{b}}\exp\left(-\dfrac{b}{\lambda_{b}}\right){1}_{\mathbb{R}_{+}}(b)\times{1}_{[0,1]}(c)\times$
$\displaystyle(\lambda_{b})^{-1-\delta}\exp\left(-\dfrac{\mu}{\lambda_{b}}\right)\times(\sigma^{2})^{-1-\delta}\exp\left(-\dfrac{\mu}{\sigma^{2}}\right).$

One can clearly notice that the posterior in (LABEL:eq:posterior) is
complicated to deal with in order to derive close-form estimators. We,
therefore, resort to numerical approximations using a Markov Chain Monte Carlo
technique (MCMC) [17, 58]. Specifically, we use a Gibbs sampler to
sequentially sample according to the conditional posteriors. To calculate the
conditional distributions associated with each parameter of the model, one
needs to integrate the joint posterior distribution in (LABEL:eq:posterior)
with respect to all the other parameters.

Regarding the parameter $W$, calculations based on (LABEL:eq:posterior) lead
to the following form:

$\displaystyle
f(W;c,\gamma,b,\lambda)\propto\exp\left[-\sum_{l=1}^{L}\sum_{k=1}^{K_{l}}\frac{\rvert
W_{k}^{l}\rvert}{\lambda_{l}}\right]\times$
$\displaystyle\exp\left[-\sum_{m=1}^{M}\left(D;(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right)\right].$
(21)

The conditional distribution for the parameter $c$ is given by:

$\displaystyle f(c;W,b,\gamma)\propto{1}_{[0,1]}(c)\times$
$\displaystyle\exp\left[-\sum_{m=1}^{M}\left(D;(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right)\right].$
(22)

For the parameter $b$, the condition distribution is given by:

$\displaystyle
f(b;W,c,\gamma,\lambda_{b})\propto\exp\left(-\frac{b}{\lambda_{b}}\right)\times$
$\displaystyle\exp\left[-\sum_{m=1}^{M}\left(D;(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right)\right].$
(23)

As regards $\gamma$, the conditional distribution writes:

$\displaystyle
f(\gamma;W,b,c,\sigma^{2})\propto\exp\left(-\frac{\gamma^{2}}{2\sigma^{2}}\right)\times$
$\displaystyle\exp\left[-\sum_{m=1}^{M}\left(D(MMeLU(x^{m};W,c,\gamma,b)-y^{m})\right)\right].$
(24)

The conditional distribution for the parameter $\lambda_{l}$ is given by:

$\displaystyle f(\lambda_{l};\delta,\mu)$
$\displaystyle\propto\lambda_{l}^{-1-(\delta+K_{l})}\exp\left(-\dfrac{\mu}{\lambda_{l}}\right)$
$\displaystyle\propto IG(\delta+K_{l},\mu).$ (25)

For the hyperparameter vector $\Phi_{e}$, it is necessary to calculate the
conditional distributions from which it is possible to sample based on the
likelihood and adopted priors.

The conditional distribution for the hyperparameter $\lambda_{b}$ is given by:

$\displaystyle f(\lambda_{b};,b,\mu,\delta)$
$\displaystyle\propto\lambda_{b}^{-2-\delta}\exp\left(-\frac{b+\mu}{\lambda_{b}}\right)$
$\displaystyle\propto IG(\delta+1,b+\mu)$ (26)

The conditional distribution for the hyperparameter $\sigma^{2}$ is given by:

$\displaystyle f(\sigma^{2};\mu,\gamma,\delta)$
$\displaystyle\propto(\sigma^{2})^{-1-\delta}\exp\left(-\frac{\gamma^{2}+2\mu}{2\sigma^{2}}\right)$
$\displaystyle\propto IG(\delta,\gamma+2\mu).$ (27)

The sampling scheme is summarized in Algorithm 1, where the model weights $W$
and the parameters of the proposed MMeLU function are sampled.

Fix the hyperparameters $\Phi_{m}$ ;

for  _$r=1,\ldots,S$_ do

* Sample $c$ according to $f(c;W,b,\gamma)$ ;

* Sample $\gamma$ according to $f(\gamma;W,b,c,\sigma^{2})$ ;

* Sample $b$ according to $f(b;W,c,\gamma,\lambda_{b})$ ;

* Sample $\sigma^{2}$ according to $f(\sigma^{2};\mu,\gamma,\delta)$ ;

* Sample $\lambda_{b}$ according to $f(\lambda_{b};,b,\mu,\delta)$ ;

* Sample $\lambda_{l}$ according to $f(\lambda_{l};\delta,\mu)$ $\forall\;l\in\\{1,\ldots,L\\}$ ;

* Sample $W$ as described in [22] ;

end for

Algorithm 1 Gibbs sampler for the proposed method.

In Algorithm 1, $S$ denotes the number of MCMC sampling iterations. After the
burn-in period, the sampled coefficients are used to calculate the estimators
$\widehat{W}$, $\widehat{c}$, $\widehat{b}$, $\widehat{\gamma}$, in addition
to $\widehat{\sigma^{2}}$, $\widehat{\lambda}$ and $\widehat{\lambda_{b}}$.

##  VI Experimental validation

In order to validate the proposed method, three image classification
experiments are conducted using different datasets: COVID-19 dataset including
Computed tomography (CT) images for challenging classification [59], and two
standard datasets, namely Fashion-MNIST [60] and CIFAR-10 [61]. Table I
illustrates the setting details of the different datasets.

TABLE I:  Setting details of the used datasets. Dataset | Training set | Test set | # Classes
---|---|---|---
CT images classification | 566 | 180 | 2
Fashion-MNIST | 48000 | 12000 | 10
CIFAR-10 | 50000 | 10000 | 10

To compare the proposed method with the state of the art, two types of
optimizers are used: _i)_ our previous Bayesian optimizer that uses non-smooth
Hamiltonian methods described in [22] with the standard ReLU activation
function, and _ii)_ the standard Adam optimizer (with a learning rate of
$10^{-3}$) with eight of the most well-known activation functions: ReLU,
LReLU, ELU, PReLU, SeLU [62], swish, FReLU, and MeLU. As regards coding, we
used python programming language with Keras and Tensorflow libraries on an
Intel(R) Core(TM) i7-2720QM CPU 2.20GHZ architecture with 16 Go memory.

###  VI-A ConvNet models

In this work, three CNN architectures are utilized. Similar to the LeNet model
[63], the first one has two fully-connected and three convolutional (Conv-32,
Conv-64, and Conv-128) layers (FC-64 and FC-softmax). The second architecture
has nine convolutional (3XConv-32, 3XConv-64, and 3XConv-128) and three FC
layers (FC-128, FC-64 and FC-softmax) which are organized similarly to VGG-Net
[64]. These architectures are shown in Table II. The third model is a deeper
CNN with 25 convolutional layers and 4 FC layers (for more information, see
section VI-F). Each one uses convolutional layers with a stride size of 1 and
$3\times 3$ filters in addition to $2\times 2$ max-pooling.
Deep neural networks are expanded with three regularizing strategies since
they can easily overfit when trained on small datasets: Batch Normalization
[65], $\ell_{1}$ Regularization [66] and Dropout [67].
We chose to use different architecture depths in the experiments, mainly to
test the ability of our proposed method to achieve better performance in the
shallow model. In this sense, training with large and complex data can be
expensive.

TABLE II:  Convnet with regularization techniques. CNN_1 |  CNN_2
---|---
Conv3x3-32:stride=1 |  3 X Conv3x3-32:stride=1
BatchNormalization |  BatchNormalization
MaxPool 2x2 |  MaxPool 2x2
Dropout(0.2) |  Dropout(0.3)
Conv3x3-64:stride=1 |  3 X Conv3x3-64:stride=1
BatchNormalization |  BatchNormalization
MaxPool 2x2 |  MaxPool 2x2
Dropout(0.3) |  Dropout(0.3)
Conv3x3-128:stride=1 |  3 X Conv3x3-128:stride=1
BatchNormalization |  BatchNormalization
MaxPool 2x2 |  MaxPool 2x2
Dropout(0.4) |  Dropout(0.4)
Flattening |  Flattening
FC-64 |  FC-128
Dropout(0.3) |  Dropout(0.35)
|  FC-64
|  Dropout(0.35)
FC-softmax |  FC-softmax

###  VI-B Sampling Results

After using the proposed Bayesian optimization method to train the CNN models
detailed above for the classification of Covid-19 CT images, we analyzed the
convergence behavior. Figure 3 presents the sampling chains for the $\gamma$,
$b$, and $c$ parameters of the proposed MMeLU function (a-c), as well as the
histograms of the corresponding samples (d-f). The sampling chains and
histograms of the sampled coefficients confirm the good convergence properties
of the designed Gibbs sampler. After a burn-in period of 350 iterations, the
algorithm achieves stable convergence and exhibits a good mixing rate of the
sampled chains.

|
---|---
(a) | (b)
|
(c) | (d)
|
(e) | (f)
Figure 3: Sampling of parameters $c$ (a,b), $b$ (c,d), and $\gamma$ (e,f):
chains and histograms.

###  VI-C Experiment 1: COVID-19 classification using CT images

This section examines the effectiveness of our approach in classifying
Covid-19 infections from other pneumonia in CT data. The COVID-CT
dataset111https://www.kaggle.com/luisblanche/covidct includes 397 images that
are negative for COVID-19 and 349 images that are positive for COVID-19, and
belong to 216 patients. We used 566 images for the train and 180 images for
the test.
Table III presents compelling evidence that the proposed Bayesian method
outperforms all other activation functions for both the $CNN_{1}$ and
$CNN_{2}$ architectures. Moreover, except with respect to ”ReLU +[22]”, the
computational time for convergence is shorter that all the other activation
functions. With respect to our previous work (”ReLU +[22]”), it is worth
noting that the proposed method performs better in terms of accuracy and loss
value, which confirms the usefulness of the used trainable activation
function. However, due to the use of additional parameters, the proposed
method is approximately 15$\%$ slower. These conclusions hold for both
$CNN_{1}$ and $CNN_{2}$.
Notably, when regularization is employed, a considerable drop in performance
is observed for all competing activation functions, which is largely
attributable to the inherent difficulty of classifying CT images due to their
content richness and the similarities between Covid-19 infection and other
types of pneumonia.

TABLE III:  Experiment 1: CT classification results with $CNN_{1}$ and $CNN_{2}$ (activation functions (Act Fcts), computation time in minutes, accuracy (Acc), loss, sensitivity (Sens) and specificity (Spec)). | $CNN_{1}$ | $CNN_{2}$
---|---|---
Act. Fcts | Time | Acc. | Loss | Sens. | Spec. | Time | Acc. | Loss | Sens. | Spec.
MMeLU | 46.21 | 0.90 | 0.23 | 0.87 | 0.86 | 61.92 | 0.91 | 0.21 | 0.87 | 0.87
ReLU+[22] | 40 | 0.84 | 0.26 | 0.82 | 0.80 | 53 | 0.88 | 0.24 | 0.86 | 0.85
ReLU | 58 | 0.73 | 0.43 | 0.69 | 0.68 | 81 | 0.77 | 0.39 | 0.74 | 0.72
LReLU | 65.18 | 0.73 | 0.52 | 0.71 | 0.69 | 105 | 0.78 | 0.44 | 0.76 | 0.75
ELU | 63 | 0.75 | 0.47 | 0.75 | 0.74 | 97 | 0.76 | 0.46 | 0.75 | 0.75
PReLU | 71.28 | 0.68 | 0.72 | 0.64 | 0.62 | 119 | 0.70 | 0.76 | 0.68 | 0.67
SeLU | 64.75 | 0.77 | 0.78 | 0.74 | 0.72 | 107 | 0.76 | 0.69 | 0.74 | 0.73
Swish | 83.41 | 0.68 | 0.58 | 0.65 | 0.62 | 132 | 0.73 | 0.55 | 0.71 | 0.70
FReLU | 77.8 | 0.76 | 0.59 | 0.76 | 0.75 | 123 | 0.77 | 0.52 | 0.76 | 0.75
MeLU | 95.89 | 0.77 | 0.43 | 0.77 | 0.76 | 146 | 0.80 | 0.38 | 0.80 | 0.80

Learning and test curves (accuracy and loss) illustrated in Figures 4 and 5
confirm the good behavior of the proposed method, which is not necessarily the
case of the other competing models where a marked difference between the
precision and loss curves can be noticed. For example, while the LReLU
function introduces a negative bias that suppresses excessive activations, an
inappropriate bias value can lead to underfitting. Similarly, although the ELU
function allows negative activation, its exponential form can lead to an
explosion of the activation value for large values of $x$.
The Swish function is known to accelerate learning convergence, but it can
also lead to overfitting by being more sensitive to outliers. Likewise,
although the FReLU function can capture complex data patterns, it can also
suffer from overfitting if the parameters are not well chosen.
These remarkable differences confirm the interest and efficiency of our MMeLU
function, which outperforms all competing activation functions in terms of
accuracy and robustness to regularization.

|  |
---|---|---
(a): ReLU | (b): ELU | (c): FReLU
|  |
(d): LReLU | (e): SeLU | (f): PReLU
|  |
(g): Swish | (h): MeLU | (i): MMeLU
Figure 4: Experiment 1: Train and test curves using $CNN_{1}$ for all competing activation functions.  |  |
---|---|---
(a): ReLU | (b): ELU | (c): FReLU
|  |
(d): LReLU | (e): SeLU | (f): PReLU
|  |
(g): Swish | (h): MeLU | (i): MMeLU
Figure 5: Experiment 1: Train and test curves using $CNN_{2}$ for all
competing activation functions.

###  VI-D Experiment 2: Fashion-MNIST image classification

The learning performance of the competing activation function algorithms is
assessed in this scenario using the standard Fashion-MNIST dataset. A training
set of 60,000 images is employed, with a test set of 10,000 images. Each
example is a $28$ grayscale image paired with a label from one of ten classes,
with 7,000 images in each class. We used 48,000 images for the train set and
12,000 for the test set for model training.
Table IV reports the results obtained for the Fashion-MNIST dataset. Our
proposed method outperformed all other competing activation functions,
achieving a minimum accuracy of 93% for both CNN models. Additionally, Table
IV shows that the processing time for competing methods is greater than 150
minutes for $CNN_{1}$, while only almost 112 minutes are required for our
method. Similar conclusions hold with $CNN_{2}$.
Moreover, our proposed MMeLU function showed similar global performance
compared to ”ReLU + [22]” for the two CNN models. One more advantage of MMeLU
is its fully automatic estimation of parameters, which is not the case for
ReLU + [22], where some parameters need to be manually set. Furthermore, our
proposed method has demonstrated its ability to achieve remarkable
performances in challenging cases, as shown in the first experiment.

TABLE IV:  Experiment 2: Fashion-MNIST classification results with $CNN_{1}$ and $CNN_{2}$ (activation functions (Act Fcts), computation time in minutes, accuracy (Acc), loss, sensitivity (Sens) and specificity (Spec)). | $CNN_{1}$ | $CNN_{2}$
---|---|---
Act. Fcts | Time | Acc. | Loss | Sens. | Spec. | Time | Acc. | Loss | Sens. | Spec.
MMeLU | 111.6 | 0.93 | 0.20 | 0.90 | 0.89 | 319.8 | 0.94 | 0.19 | 0.92 | 0.91
ReLU+[22] | 90.5 | 0.92 | 0.22 | 0.90 | 0.88 |  308.4 | 0.93 | 0.19 | 0.91 | 0.89
ReLU | 145.4 | 0.90 | 0.38 | 0.85 | 0.82 | 421.3 | 0.91 | 0.32 | 0.88 | 0.87
LReLU | 157.2 | 0.86 | 0.36 | 0.85 | 0.83 | 409.7 | 0.87 | 0.34 | 0.85 | 0.84
ELU | 154.5 | 0.87 | 0.33 | 0.86 | 0.84 | 389 | 0.88 | 0.32 | 0.86 | 0.85
PReLU | 158.7 | 0.87 | 0.32 | 0.86 | 0.85 | 400.3 | 0.85 | 0.33 | 0.83 | 0.82
SeLU | 150 | 0.85 | 0.34 | 0.84 | 0.83 | 383.8 | 0.87 | 0.33 | 0.86 | 0.85
Swish | 148.5 | 0.86 | 0.36 | 0.83 | 0.81 | 423 | 0.87 | 0.35 | 0.84 | 0.82
FReLU | 149.8 | 0.87 | 0.35 | 0.85 | 0.83 | 395.7 | 0.86 | 0.34 | 0.85 | 0.84
MeLU | 169 | 0.89 | 0.33 | 0.86 | 0.84 | 452.5 | 0.89 | 0.31 | 0.88 | 0.87

###  VI-E Experiment 3: CIFAR-10 image classification

In this experiment, the learning performance of the proposed model (made up of
the trainable activation function and the Bayesian sparse optimization) is
evaluated using the standard CIFAR-10 dataset. The CIFAR-10 dataset contains
60000 $32\times 32$ color images divided into 10 classes with 6000 images per
class. There are 50,000 training and 10,000 test images.
Table V presents the classification results for the CIFAR-10 dataset. It can
be observed that the proposed Bayesian model performed well overall, even when
multiple classes were used. In contrast, all competing activation functions
(with Adam optimizer) had similar accuracy, around 88%, with a loss rate
almost double that of the proposed model for both architectures.
Moreover, the MMeLU activation function takes significantly less time for
training than other activation functions + Adam, which often take a long time
before convergence. For the competing functions when used with Adam, we
noticed similar performance between the trainable and standard activation
functions, with a slight superiority observed with the trainable function ELU.
The same conclusions can be drawn by examining the results of ”ReLU + [21]” on
this dataset. These findings suggest that the proposed MMeLU function provides
better flexibility for the activation task, even for the multi-class case.

TABLE V:  Experiment 3: CIFAR-10 classification results with $CNN_{1}$ and $CNN_{2}$ (activation functions (Act Fcts), computation time in minutes, accuracy (Acc), loss, sensitivity (Sens) and specificity (Spec)). | $CNN_{1}$ | $CNN_{2}$
---|---|---
Act. Fcts | Time | Acc. | Loss | Sens. | Spec. | Time | Acc. | Loss | Sens. | Spec.
MMeLU | 120.7 | 0.91 | 0.21 | 0.89 | 0.88 | 332.5 | 0.93 | 0.20 | 0.90 | 0.88
ReLU+[22] | 100.7 | 0.90 | 0.25 | 0.89 | 0.87 | 331 | 0.92 | 0.21 | 0.90 | 0.87
ReLU | 161 | 0.87 | 0.42 | 0.83 | 0.81 | 429 | 0.90 | 0.36 | 0.87 | 0.86
LReLU | 170 | 0.87 | 0.52 | 0.85 | 0.84 | 437.9 | 0.85 | 0.55 | 0.83 | 0.81
ELU | 165 | 0.88 | 0.41 | 0.86 | 0.85 | 438 | 0.88 | 0.39 | 0.86 | 0.85
PReLU | 198.2 | 0.82 | 0.48 | 0.84 | 0.82 | 466.8 | 0.84 | 0.47 | 0.81 | 0.79
SeLU | 173.6 | 0.84 | 0.43 | 0.85 | 0.84 | 454.3 | 0.86 | 0.39 | 0.87 | 0.86
Swish | 209.7 | 0.83 | 0.49 | 0.82 | 0.79 | 478.3 | 0.85 | 0.46 | 0.83 | 0.81
FReLU | 172.3 | 0.84 | 0.41 | 0.82 | 0.81 | 431.7 | 0.86 | 0.38 | 0.83 | 0.81
MeLU | 219.9 | 0.86 | 0.40 | 0.83 | 0.83 | 485.6 | 0.90 | 0.34 | 0.88 | 0.87

###  VI-F Experiment 4: Comparison on Deep CNN

This section investigates the performance of our algorithm on the standard
Fashion-MNIST dataset using a deep CNN. This CNN is claimed to be deeper than
$CNN1$ and $CNN2$. It has 25 convolutional layers (5 X Conv3x3-32, 5 X
Conv3x3-64, 5 X Conv3x3-128, 5 X Conv3x3-256 and 5 X Conv3x3-512) and four
layers (FC-512, FC-256, Fc-128 and FC-softmax). All of them use convolutional
layers with $3\times 3$ kernel filters as well as $2\times 2$ max-pooling and
stride size of 1.
The results demonstrate that our proposed MMeLU method maintains good
performance on this deep architecture, as shown in table VI. The same
conclusions can be drawn as from previous experiments regarding the
superiority of MMeLU over competing activation functions, including our
previous work in [22]. Interestingly, in addition to better scores in
comparison to [22], the proposed method provides convergence time which is
only 2 $\%$ slower. It turns out that when the depth of the network increases
(similarly the number of parameters), the additional cost to estimate the
activation function parameters is no longer significant while reaching better
performance.

TABLE VI:  Results of Fashion-MNIST classification with a deep CNN (activation functions (Act Fcts), computation time in minutes, accuracy, loss, sensitivity (Sens), and specificity (Spec) ). Act. Fcts | Time (min) | Accuracy | Loss | Sens | Spec
---|---|---|---|---|---
MMeLU | 651.4 | 0.94 | 0.19 | 0.92 | 0.92
ReLU+[22] | 638 | 0.93 | 0.21 | 0.91 | 0.90
ReLU | 854 | 0.89 | 0.34 | 0.87 | 0.85
LReLU | 847.9 | 0.86 | 0.38 | 0.84 | 0.82
ELU | 870 | 0.87 | 0.35 | 0.85 | 0.84
PReLU | 855.6 | 0.86 | 0.36 | 0.83 | 0.83
SeLU | 867.3 | 0.88 | 0.33 | 0.87 | 0.86
Swish | 880 | 0.87 | 0.34 | 0.86 | 0.85
FReLU | 873.4 | 0.85 | 0.36 | 0.84 | 0.82
MeLU | 890 | 0.90 | 0.32 | 0.86 | 0.85

##  VII Conclusion

In this paper, we proposed a Bayesian approach for training sparse deep neural
networks with trainable activation functions. Our method learns the weights
and parameters of the proposed activation function directly from the data
without any user configuration. Compared to competing algorithms, our approach
achieves promising results with better classification accuracy and high
generalization performance, while also being computationally efficient,
particularly in challenging cases. Our experiments on standard datasets such
as Fashion-MNIST and CIFAR-10 demonstrate the precision and stability of our
approach, as well as its general applicability.
Future work will focus on parallelizing the algorithm to enable GPU
calculations and further reduce computational time.

## References

  * [1] A. Alsarhan, M. Alauthman, E. Alshdaifat, A.-R. Al-Ghuwairi, A. Al-Dubai, Machine learning-driven optimization for svm-based intrusion detection system in vehicular ad hoc networks, Journal of Ambient Intelligence and Humanized Computing (2021) 1–10.
  * [2] Z. Wang, J. Chen, S. C. Hoi, Deep learning for image super-resolution: A survey, IEEE transactions on pattern analysis and machine intelligence 43 (10) (2020) 3365–3387.
  * [3] C. Dong, C. C. Loy, K. He, X. Tang, Image super-resolution using deep convolutional networks, IEEE transactions on pattern analysis and machine intelligence 38 (2) (2015) 295–307.
  * [4] V. Sree, J. Mapes, S. Dua, O. S. Lih, J. E. Koh, E. J. Ciaccio, U. R. Acharya, et al., A novel machine learning framework for automated detection of arrhythmias in ecg segments, Journal of Ambient Intelligence and Humanized Computing (2021) 1–18.
  * [5] S. N. B. Jaini, D. Lee, S. Lee, M. Kim, Y. Kwon, Tool monitoring of end milling based on gap sensor and machine learning, Journal of Ambient Intelligence and Humanized Computing (2021) 1–13.
  * [6] R. S. Mitchell, R. A. Sherlock, L. A. Smith, An investigation into the use of machine learning for determining oestrus in cows, Computers and electronics in agriculture 15 (3) (1996) 195–213.
  * [7] A. Drewek-Ossowicka, M. Pietrołaj, J. Rumiński, A survey of neural networks usage for intrusion detection systems, Journal of Ambient Intelligence and Humanized Computing 12 (1) (2021) 497–514.
  * [8] T. K. Sajja, H. K. Kalluri, Image classification using regularized convolutional neural network design with dimensionality reduction modules: Rcnn–drm, Journal of Ambient Intelligence and Humanized Computing (2021) 1–12.
  * [9] M. Fakhfakh, B. Bouaziz, F. Gargouri, L. Chaari, Prognet: Covid-19 prognosis using recurrent and convolutional neural networks, The Open Medical Imaging Journal 12 (1) (2020).
  * [10] Y. Li, H. Zhang, X. Xue, Y. Jiang, Q. Shen, Deep learning for remote sensing image classification: A survey, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (6) (2018) e1264.
  * [11] S. Ji, W. Xu, M. Yang, K. Yu, 3d convolutional neural networks for human action recognition, IEEE transactions on pattern analysis and machine intelligence 35 (1) (2012) 221–231.
  * [12] K. Hara, D. Saito, H. Shouno, Analysis of function of rectified linear unit used in deep learning, in: 2015 international joint conference on neural networks (IJCNN), IEEE, 2015, pp. 1–8.
  * [13] M. M. Lau, K. H. Lim, Review of adaptive activation function in deep neural network, in: 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), IEEE, 2018, pp. 686–690.
  * [14] A. Apicella, F. Donnarumma, F. Isgrò, R. Prevete, A survey on modern trainable activation functions, Neural Networks 138 (2021) 14–32.
  * [15] C. T. Butts, Network inference, error, and informant (in) accuracy: a bayesian approach, social networks 25 (2) (2003) 103–140.
  * [16] C. Andrieu, A. Doucet, R. Holenstein, Particle markov chain monte carlo methods, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 72 (3) (2010) 269–342.
  * [17] C. Robert, G. Casella, Monte Carlo statistical methods, Springer Science & Business Media, 2013.
  * [18] L. Chaari, H. Batatia, N. Dobigeon, J.-Y. Tourneret, A hierarchical sparsity-smoothness bayesian model for l0+l1+l2 regularization, in: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 1901–1905.
  * [19] L. Chaari, A bayesian grouplet transform, Signal, Image and Video Processing 13 (2019) 871–878.
  * [20] M. Fakhfakh, B. Bouaziz, L. Chaari, F. Gargouri, Efficient bayesian learning of sparse deep artificial neural networks, in: International Symposium on Intelligent Data Analysis, Springer, 2022, pp. 78–88.
  * [21] M. Fakhfakh, B. Bouaziz, F. Gargouri, L. Chaari, Bayesian optimization using hamiltonian dynamics for sparse artificial neural networks, in: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), IEEE, 2022, pp. 1–4.
  * [22] M. Fakhfakh, L. Chaari, B. Bouaziz, F. Gargouri, Non-smooth bayesian learning for artificial neural networks, Journal of Ambient Intelligence and Humanized Computing (2022).
  * [23] U. of Illinois at Urbana-Champaign. Center for Supercomputing Research, Development, G. Cybenko, Continuous valued neural networks with two hidden layers are sufficient, 1988.
  * [24] F. Xiao, Y. Honma, T. Kono, A simple algebraic interface capturing scheme using hyperbolic tangent function, International journal for numerical methods in fluids 48 (9) (2005) 1023–1040.
  * [25] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of control, signals and systems 2 (4) (1989) 303–314.
  * [26] B. Ding, H. Qian, J. Zhou, Activation functions and their characteristics in deep neural networks, in: 2018 Chinese control and decision conference (CCDC), IEEE, 2018, pp. 1836–1841.
  * [27] P. d. B. Harrington, Sigmoid transfer functions in backpropagation neural networks, Analytical Chemistry 65 (15) (1993) 2167–2168.
  * [28] S. Chadha, Fractional programming with absolute-value functions, European Journal of Operational Research 141 (1) (2002) 233–238.
  * [29] Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE transactions on neural networks 5 (2) (1994) 157–166.
  * [30] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 315–323.
  * [31] C. Gulcehre, M. Moczulski, M. Denil, Y. Bengio, Noisy activation functions, in: International conference on machine learning, PMLR, 2016, pp. 3059–3068.
  * [32] X. Nie, J. Cao, Multistability of second-order competitive neural networks with nondecreasing saturated activation functions, IEEE Transactions on Neural Networks 22 (11) (2011) 1694–1708.
  * [33] A. Montalto, G. Tessitore, R. Prevete, A linear approach for sparse coding by a two-layer neural network, Neurocomputing 149 (2015) 1315–1323.
  * [34] A. L. Maas, A. Y. Hannun, A. Y. Ng, et al., Rectifier nonlinearities improve neural network acoustic models, in: Proc. icml, Vol. 30, Atlanta, Georgia, USA, 2013, p. 3.
  * [35] K. Konda, R. Memisevic, D. Krueger, Zero-bias autoencoders and the benefits of co-adapting features, arXiv preprint arXiv:1402.3337 (2014).
  * [36] C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, R. Garcia, Incorporating second-order functional knowledge for better option pricing, Advances in neural information processing systems 13 (2000).
  * [37] A. Shah, E. Kadam, H. Shah, S. Shinde, S. Shingade, Deep residual networks with exponential linear unit, in: Proceedings of the third international symposium on computer vision and the internet, 2016, pp. 59–65.
  * [38] M. ÇELEBİ, M. CEYLAN, New complex valued activationfunctions: Complex modifiedswish, complex e-swish andcomplex flatten-tswish., International Journal of Advanced Research in Computer Science 11 (2) (2020).
  * [39] H. H. Chieng, N. Wahid, O. Pauline, S. R. K. Perla, Flatten-t swish: a thresholded relu-swish-like activation function for deep learning, International Journal of Advances in Intelligent Informatics 4 (2) (2018) 76–86.
  * [40] C.-T. Chen, W.-D. Chang, A feedforward neural network with function shape autotuning, Neural networks 9 (4) (1996) 627–641.
  * [41] S. Guarnieri, Multilayer neural networks with adaptive spline-based activation functions, in: Proceedings of Word Congress on Neural Networks WCNN’95, Washington, DC, July, 1995, pp. 17–21.
  * [42] F. Piazza, A. Uncini, M. Zenobi, Artificial neural networks with adaptive polynomial activation function (1992).
  * [43] L. Trottier, P. Giguere, B. Chaib-Draa, Parametric exponential linear unit for deep convolutional neural networks, in: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, 2017, pp. 207–214.
  * [44] S. Qiu, X. Xu, B. Cai, Frelu: flexible rectified linear units for improving convolutional neural networks, in: 2018 24th international conference on pattern recognition (icpr), IEEE, 2018, pp. 1223–1228.
  * [45] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026–1034.
  * [46] A. Nader, D. Azar, Searching for activation functions using a self-adaptive evolutionary algorithm, in: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, 2020, pp. 145–146.
  * [47] M. Basirat, P. M. Roth, The quest for the golden activation function, Proceedings of the ARW & OAGM Workshop (2019) 1–16.
  * [48] G. Maguolo, L. Nanni, S. Ghidoni, Ensemble of convolutional neural networks trained with different activation functions, Expert Systems with Applications 166 (2021) 114048.
  * [49] L. R. Sutfeld, F. Brieger, H. Finger, S. Fullhase, G. Pipa, Adaptive blending units: Trainable activation functions for deep neural networks, in: Science and Information Conference, Springer, 2020, pp. 37–50.
  * [50] S. Scardapane, S. Van Vaerenbergh, S. Totaro, A. Uncini, Kafnets: Kernel-based non-parametric activation functions for neural networks, Neural Networks 110 (2019) 19–32.
  * [51] S. Scardapane, S. Van Vaerenbergh, A. Hussain, A. Uncini, Complex-valued neural networks with nonparametric activation functions, IEEE Transactions on Emerging Topics in Computational Intelligence 4 (2) (2018) 140–150.
  * [52] Ö. F. Ertuğrul, A novel type of activation function in artificial neural networks: Trained activation function, Neural Networks 99 (2018) 148–157.
  * [53] O. Castillo, P. Melin, A review on interval type-2 fuzzy logic applications in intelligent control, Information Sciences 279 (2014) 615–631.
  * [54] S. Scardapane, M. Scarpiniti, D. Comminiello, A. Uncini, Learning activation functions from data using cubic spline interpolation, in: Italian Workshop on Neural Nets, Springer, 2017, pp. 73–83.
  * [55] X. Wang, Y. Qin, Y. Wang, S. Xiang, H. Chen, Reltanh: An activation function with vanishing gradient resistance for sae-based dnns and its application to rotating machinery fault diagnosis, Neurocomputing 363 (2019) 88–98.
  * [56] G. Maguolo, L. Nanni, S. Ghidoni, Ensemble of convolutional neural networks trained with different activation functions, Expert Systems with Applications 166 (2021) 114048.
  * [57] M. Fakhfakh, B. Bouaziz, H. Batatia, L. Chaari, Bayesian optimization for sparse artificial neural networks: Application to change detection in remote sensing, in: Proceedings of International Conference on Information Technology and Applications, Springer, 2022, pp. 39–49.
  * [58] L. Chaari, J.-Y. Tourneret, C. Chaux, H. Batatia, A Hamiltonian Monte Carlo method for non-smooth energy sampling, IEEE Trans. on Signal Process. 64 (21) (2016) 5585 – 5594.
  * [59] P. Afshar, S. Heidarian, N. Enshaei, F. Naderkhani, M. J. Rafiee, A. Oikonomou, F. B. Fard, K. Samimi, K. N. Plataniotis, A. Mohammadi, Covid-ct-md, covid-19 computed tomography scan dataset applicable in machine learning and deep learning, Scientific Data 8 (1) (2021) 121.
  * [60] M. Kayed, A. Anter, H. Mohamed, Classification of garments from fashion mnist dataset using cnn lenet-5 architecture, in: 2020 international conference on innovative trends in communication and computer engineering (ITCE), IEEE, 2020, pp. 238–243.
  * [61] R. C. Çalik, M. F. Demirci, Cifar-10 image classification with convolutional neural networks for embedded systems, in: 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), IEEE, 2018, pp. 1–2.
  * [62] G. Klambauer, T. Unterthiner, A. Mayr, S. Hochreiter, Self-normalizing neural networks, Advances in neural information processing systems 30 (2017).
  * [63] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  * [64] U. Muhammad, W. Wang, S. P. Chattha, S. Ali, Pre-trained vggnet architecture for remote-sensing image scene classification, in: 24th International Conference on Pattern Recognition (ICPR), 2018, pp. 1622–1627.
  * [65] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International conference on machine learning, PMLR, 2015, pp. 448–456.
  * [66] Z. Xu, H. Zhang, Y. Wang, X. Chang, Y. Liang, L 1/2 regularization, Science China Information Sciences 53 (6) (2010) 1159–1169.
  * [67] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research 15 (1) (2014) 1929–1958.


# Federated Learning in Wireless Networks via Over-the-Air Computations*
††thanks: * This work has been funded by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC
2002/1 “Science of Intelligence” – project number 390523135. ††thanks: 1 H.Y.
Oksuz, F. Molinari, and J. Raisch are with the Control Systems Group at
Technische Universität Berlin, Germany<EMAIL_ADDRESS><EMAIL_ADDRESS> ††thanks: 2 H.Y. Oksuz, H. Sprekeler,
and J. Raisch are with Exzellenzcluster Science of Intelligence, Technische
Universität Berlin, Marchstr. 23, 10587, Berlin, Germany. ††thanks: 3 H.Y.
Oksuz and H. Sprekeler are with the Modelling Cognitive Processes Group at
Technische Universität Berlin, Germany.

Halil Yigit Oksuz1,2,3, Fabio Molinari1, Henning Sprekeler2,3, Jörg Raisch1,2

###### Abstract

In a multi-agent system, agents can cooperatively learn a model from data by
exchanging their estimated model parameters, without the need to exchange the
locally available data used by the agents. This strategy, often called
federated learning, is mainly employed for two reasons: (i) improving
resource-efficiency by avoiding to share potentially large data sets and (ii)
guaranteeing privacy of local agents’ data. Efficiency can be further
increased by adopting a beyond-5G communication strategy that goes under the
name of Over-the-Air Computation. This strategy exploits the interference
property of the wireless channel. Standard communication schemes prevent
interference by enabling transmissions of signals from different agents at
distinct time or frequency slots, which is not required with Over-the-Air
Computation, thus saving resources. In this case, the received signal is a
weighted sum of transmitted signals, with unknown weights (fading channel
coefficients). State of the art papers in the field aim at reconstructing
those unknown coefficients. In contrast, the approach presented here does not
require reconstructing channel coefficients by complex encoding-decoding
schemes. This improves both efficiency and privacy.

##  I INTRODUCTION

Over the last decade, topics related to machine learning have attracted a
great deal of attention due to their success in many application areas (e.g.,
[1, 2, 3]). As computational power increases, we have smaller but more
intelligent and powerful devices, which are able to handle big volumes of data
and more complex computations.

Data is often distributed over these intelligent devices or agents, such as
smartphones, personal computers and distributed data centers. In a centralized
learning approach, all data are collected in a computationally powerful
machine, such as a cloud-based unit or a server, on which the model is trained
[4]. However, this approach may not be suitable for specific applications
requiring privacy, resource efficiency, and low-latency. For instance, due to
privacy concerns, smartphone users might not be willing to share their data;
moreover, with an increasing number of agents, more communication resources
must be allocated to share the agents’ data with the central unit. However, it
is also possible to train a model without centralized data collection. In this
case, agents individually train models on their respective data and share
their individually trained models (in the form of parameter vectors) with the
central unit. Sharing parameter vectors is, in general, less expensive in
terms of communication resources than sharing all the local data sets. This
approach is often referred to as federated learning, and allows more private
and communication efficient learning (see [5, 6, 7, 8]).

Even though federated learning allows improved privacy by avoiding to share
individual data sets, sharing local model parameters with the central unit
might still reveal sensitive information [9, 10, 11]. It is possible to tackle
this privacy problem with popular techniques like encryption or differential
privacy [12], but they come at the price of lower efficiency and performance
[9, 13].

As the number of agents and the number of model parameters increase, the
communication load on the overall system increases as well [14, 11, 10]. To
cope with this problem, one can carry out local training on agents for
multiple steps or utilize quantization and/or sparsification on the model
updates in order to accomplish an efficient compression; another approach is
allowing only a subset of agents to transmit at specific time steps [15, 16,
17, 18, 19]. However, most of these techniques are not resource efficient in
the sense that they increase the need of bandwidth or the number of
communication rounds, which in general leads to a decrease in total throughput
and learning speed as also observed in [20, 21].

Instead, we consider the so-called Over-The-Air computation approach, e.g.,
[22], to improve communication efficiency. When multiple agents transmit at
the same time and in the same frequency band, signals are affected by the
physical phenomenon of interference. Standard communication protocols prevent
interference by transmitting signals orthogonally: in TDMA (Time Division
Multiple Access), agents are assigned different time slots when they can
transmit, whereas in FDMA (Frequency Division Multiple Access), different
frequency bands are allocated to different users.

The philosophy of Over-The-Air Computation is to exploit interference rather
than combat it, to increase communication efficiency. For example, in a system
composed of $N$ agents, each of which has to transmit an $m$-dimensional
parameter vector $\theta\in\Re^{m}$ to a central unit, TDMA or FDMA would
require $mN$ orthogonal transmissions (multiplexed in time or frequency),
whereas Over-The-Air Computation requires only $m$ orthogonal transmissions.
Due to its advantages, federated learning has been carried out by [23, 24, 25]
via over-the air computation.

With this in mind, the main contributions of this paper can be summarized as
follows:

  * •

Unlike the studies by [23] and [25], we do not assume to know (nor to be able
to reconstruct) channel coefficients, but we present an algorithm that can
deal with their unknown nature. We will not need extra pre-processing to
reconstruct the channel, which makes the proposed scheme more time and
resource efficient.

  * •

Privacy is inherently guaranteed by the unknown nature of the channel. The
central unit will not be able to reconstruct information transmitted by
individual agents.

The remainder of this paper is organized as follows: we describe the problem
setup in Section II. In Section III, we present the proposed federated
learning algorithm and prove its convergence. A numerical example is presented
in Section IV. Concluding remarks are given in Section V.

### Notation

The set of real numbers is denoted by $\Re$, $\Re^{m}$ represents
$m$-dimensional Euclidean space. $\mathbb{N}$ and $\mathbb{N}_{0}$
respectively denote the set of natural numbers and the set of nonnegative
integers. For a vector $x\in\Re^{m}$, $x^{T}$ denotes its transpose. The
2-norm of vector $x$ is denoted by $||x||$. The expected value of a random
variable $p$ is denoted by $\mathbb{E}[p]$. Given a finite set $S$, its
cardinality is denoted by $|S|$. For a differentiable function
$f:\Re^{m}\to\Re$, $\nabla f(\mathbf{x})$ represents the gradient of the
function $f$ at $x\in\Re^{m}$. The projection of $x\in\Re^{m}$ onto a nonempty
closed convex set $S\subset\Re^{m}$ is denoted by $\mathbf{P}_{S}(x)$, where
$\mathbf{P}_{S}(x)=\operatorname*{arg\,min}_{s\in S}||s-x||$.

##  II Problem Description

###  II-A Federated Learning with Constraints

Consider a system of $N$ agents connected to a server or cloud based unit,
whose objective is to carry out a machine learning task. Each agent can access
different portions of the dataset, and has an individual local cost function.
We denote the set of data available to the $i$-th agent by
$D_{i}=\\{d^{n}_{i}\\}_{n=1}^{|D_{i}|}$ and use
$\mathcal{L}_{i}(d^{n}_{i},\mathbf{\theta})$ to represent the value of the
cost function of a model with parameter $\mathbf{\theta}\in\mathcal{R}^{m}$.
In a supervised learning setting, the dataset $D_{i}$ consists of pairs of
inputs and targets, i.e., $d^{n}_{i}=(u^{n}_{i},z^{n}_{i})$, where $u^{n}_{i}$
and $z^{n}_{i}$ represent, respectively, input and target data. Moreover, the
private local cost function of agent $i$ can be expressed as

$f_{i}(\theta)=\frac{1}{|D_{i}|}\sum_{n=1}^{|D_{i}|}\mathcal{L}_{i}(d^{n}_{i},\theta),$
(1)

If the global cost function is defined as the average of all local cost
functions, i.e.,

$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta),$ (2)

then, the objective of the overall system is to cooperatively solve the
following constrained optimization problem

$\theta^{*}=\operatorname*{arg\,min}_{\theta\in\Theta}\mathcal{L}(\theta)=\operatorname*{arg\,min}_{\theta\in\Theta}\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta),$
(3)

where $\Theta\subset\Re^{m}$ is a nonempty constraint set. In what follows, we
denote $\mathcal{L}(\theta^{*})$ by $\mathcal{L}^{*}$. Moreover, the set of
optimal solutions is defined as

$\Theta^{*}=\\{\theta\in\Theta\textbf{\hskip 2.84526pt}|\textbf{\hskip
2.84526pt}\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta)=\mathcal{L}^{*}\\}.$

If the entire dataset, i.e.,

$D=D_{1}\cup D_{2}\cup\vspace{1mm}\cdots\cup D_{N}$

were known to the server, one could employ a centralized gradient descent
based optimization to solve the above global learning task [3]. However, in
the considered federated learning approach, each agent can access only its own
dataset, on which it trains its own model. After local optimization steps, the
current versions of the local parameter estimates are transmitted to the
server, where they are aggregated. The aggregated version is then transmitted
to the agents for the next optimization step (see Fig. 1).

Figure 1: An illustration of a federated learning setting.

###  II-B Over-the-Air Communication

In wireless communication systems, the wireless multiple access channel (WMAC)
model has been extensively used to characterize communication between multiple
transmitters and a single receiver over fading channels, e.g., [26, 27].
Throughout this paper, we employ the WMAC-based communication model described
by [20, 28]. In this model, multiple (say $N$) agents simultaneously send
information $\\{\mathbf{s}_{i}(k)\in\Re^{m}\\}_{i=1}^{N}$ in the same
frequency band. This information is corrupted by the channel and superimposed
by the receiver, i.e., the received information is

$\mathbf{r}^{rec}(k)=\sum_{i=1}^{N}\alpha_{i}(k)\mathbf{s}_{i}(k),$ (4)

where the $\alpha_{i}(k)$ are unknown time-varying positive channel
coefficients.

As indicated above, employing Over-the-Air computation for federated learning
provides the following benefits:

  * •

privacy: the channel coefficients $\\{\alpha_{i}(k)\\}_{i=1}^{N}$ in (4) are
unknown, thus it is impossible for the central unit to individually
reconstruct $\\{s_{i}(k)\\}_{i=1}^{N}$ from $\mathbf{r}^{rec}(k)$;

  * •

efficiency: as shown by [21], Over-the-Air Computation can achieve better-
than-linear scaling of the communication cost as the number of transmitters
grows.

Unlike the algorithm suggested by [25], our approach does not require any
knowledge of the channel coefficients, nor do they need to be reconstructed at
each time step, thus removing one source of complexity.

Algorithm 1: FedCOTA

0: $\theta(0)\in\Theta$.

0:

1: for each time step $k\in\mathbb{N}_{0}$ do

2: The server broadcasts $\theta(k)$

3: Each agent $i$ updates its local variables:
$\displaystyle\theta_{i}(k)=\theta(k)-\eta(k)\nabla
f_{i}\big{(}\theta(k)\big{)}$ (5) $\displaystyle\varrho_{i}(k)=1$ (6)

4: Each agent $i$ transmits $\theta_{i}(k)$ and $\varrho_{i}(k)$

5: The server receives: $\displaystyle\theta^{\text{rec}}(k)$
$\displaystyle=\sum_{i=1}^{N}\alpha_{i}(k)\theta_{i}(k)$ (7)
$\displaystyle\varrho^{\text{rec}}(k)$
$\displaystyle=\sum_{i=1}^{N}\alpha_{i}(k)\varrho_{i}(k)$ (8)

6: The server updates: $\displaystyle\theta(k+1)$
$\displaystyle=\mathbf{P}_{\Theta}\Bigg{(}\frac{\theta^{\text{rec}}(k)}{\varrho^{\text{rec}}(k)}\Bigg{)}$
(9)

7: end for

##  III Federated Constrained Over-the-Air Learning (FedCOTA) Algorithm

In this section, we first introduce the proposed algorithm that allows agents
to carry out federated learning via over-the-air communication. Then, we will
investigate the convergence properties of the proposed algorithm.

###  III-A Algorithm

The FedCOTA algorithm is summarized in Algorithm 1. At the beginning, the
server broadcasts to agents $\theta(0)\in\Theta$. Then, through iterations,
each agent computes its own parameter vector $\theta_{i}(k)$ by using the
local update rule (5). Afterwards, all agents transmit simultaneously (and in
the same frequency band) their local parameter vectors $\theta_{i}(k)$ to the
central unit. Then, again simultaneously and in the same frequency band a
constant $\varrho_{i}(k)=1$ is transmitted by all agents. Because of the
superposition property of the wireless channel (see Section II-B), the server
receives (7) and (8) at time step $k$. Finally, the server computes (9) thus
obtaining $\theta(k+1)$, which can be written as

$\displaystyle\theta(k+1)$
$\displaystyle=\mathbf{P}_{\Theta}\Bigg{(}\frac{\theta^{\text{rec}}(k)}{\varrho^{\text{rec}}(k)}\Bigg{)}=\mathbf{P}_{\Theta}\Bigg{(}\sum_{i=1}^{N}\frac{\alpha_{i}(k)}{\sum_{i=1}^{N}\alpha_{i}(k)}\theta_{i}(k)\Bigg{)}$
$\displaystyle=\mathbf{P}_{\Theta}\Bigg{(}\sum_{i=1}^{N}h_{i}(k)\theta_{i}(k)\Bigg{)}$
(10)

where $\mathbf{P}_{\Theta}$ is the projection operator onto the set $\Theta$,
the $\alpha_{i}(k)$ are the unknown time-varying positive real channel
coefficients, and the
$h_{i}(k)=\frac{\alpha_{i}(k)}{\sum_{i=1}^{N}\alpha_{i}(k)}$ are the
corresponding normalized channel coefficients. By construction, the $h_{i}(k)$
are positive and

$\sum_{i=1}^{N}h_{i}(k)=1,$ (11)

for all $k\geq 0$.

###  III-B Preliminaries

The following assumptions, also made in similar papers in the field, will hold
throughout the paper.

###### Assumption 1

The constraint set $\Theta\subset\Re^{m}$ is convex and compact. As a
consequence (see [29, Theorem 2.41, p.40]), $\Theta$ is then also closed and
bounded.

###### Assumption 2

The cost functions $f_{i}(\theta)$ are continuously differentiable and
strongly convex with $L$-Lipschitz continuous gradients, i.e., for any
$\theta_{1},\theta_{2}\in\Theta\subset\Re^{m}$ and for all $i=1,2,\ldots,N$,
the following inequalities hold:

  * $(i)$

$\exists\mu_{i}>0$ such that

$\displaystyle f_{i}(\theta_{2})-f_{i}(\theta_{1})\geq\nabla
f_{i}(\theta_{1})^{T}(\theta_{2}-\theta_{1})+\frac{\mu_{i}}{2}||\theta_{2}-\theta_{1}||^{2},$
(12)

  * $(ii)$

$\exists L_{i}>0$ such that

$\displaystyle||\nabla f_{i}(\theta_{1})-\nabla f_{i}(\theta_{2})||\leq
L_{i}||\theta_{1}-\theta_{2}||.$ (13)

###### Remark 1

Note that the global cost function $\mathcal{L}(\theta)$ is then also
differentiable and strongly convex with $L$-Lipschitz continuous gradient,
i.e.,

$\displaystyle\centering\mathcal{L}(\theta_{2})-\@add@centering$
$\displaystyle\mathcal{L}(\theta_{1})\geq\nabla\mathcal{L}(\theta_{1})^{T}(\theta_{2}-\theta_{1})+\frac{\mu}{2}||\theta_{2}-\theta_{1}||^{2}$
$\displaystyle||\nabla\mathcal{L}(\theta_{1})-\nabla\mathcal{L}(\theta_{2})||\leq
L||\theta_{1}-\theta_{2}||,$

where $\mu=\frac{1}{N}\sum_{i=1}^{N}\mu_{i}$ and
$L=\frac{1}{N}\sum_{i=1}^{N}L_{i}$ (see [30]).

###### Remark 2

Note that Assumptions 1 and 2 imply that the local and global cost functions
have bounded gradients on $\Theta$.

Note that Assumptions 1 and 2 have been widely utilized in the federated
learning and distributed optimization literature to illustrate the existence
of a solution to problem (3) and convergence properties of the proposed
algorithms (see [31, 32, 33, 34, 35, 25, 3], and the references therein).

###### Assumption 3

For all $k\geq 0$, the step size in Algorithm 1 is chosen as
$\eta(k)=\frac{\eta_{c}}{\sqrt{k+1}}$, where $0<\eta_{c}\leq\frac{1}{L}$.

###### Remark 3

Under Assumption 3, the step size $\eta(k)$ is decreasing and satisfies
$\sum_{k=0}^{\infty}\eta(k)=\infty$ and $\lim_{k\to\infty}\eta(k)=0$.

Conditions similar to Assumption 3 on the step size have also been utilized in
many previous studies (e.g., [35, 36, 3, 37, 38, 34, 39, 40], and the
references therein) to show the exact convergence to an optimal solution.

###### Lemma 1

If $f(\cdot)$ is continuously differentiable, convex with $L$-Lipschitz
continuous gradients, then for any $\theta_{1},\theta_{2}\in\Re^{m}$, the
following inequality holds:

$\displaystyle 0\leq f(\theta_{2})-f(\theta_{1})-\nabla
f(\theta_{1})^{T}(\theta_{2}-\theta_{1})\leq\frac{L}{2}||\theta_{1}-\theta_{2}||^{2}.$

###### Proof:

The result is a direct consequence of [35, Theorem 2.1.5]. ∎

###### Remark 4

From Lemma 1 and the definition of strong convexity (12), we have $\mu\leq L$
if a function is continuously differentiable, strongly convex with
$L$-Lipschitz continuous gradients. Hence,
$\eta_{c}\leq\frac{1}{L}\leq\frac{1}{\mu}$ holds by Assumption 3.

###### Assumption 4

The unknown time-varying positive real channel coefficients are assumed to be
independent realizations (across time and agents) of the same probability
distribution, i.e., $\forall k\in\mathbb{N}_{0}$,
$\alpha_{i}(k)\sim\mathcal{D}\big{(}\bar{\alpha},Var(\alpha)\big{)}$, where
$\bar{\alpha}$ and $Var(\alpha)$ respectively denote the mean and variance of
the distribution $\mathcal{D}$.

As in [20, 28], we refer here to [41, Ch 2.3, Ch 2.4] and [42, Ch 5.4], thus
considering channel coefficients independent realizations of the same
probability distribution (see [43]).

###### Lemma 2

Under Assumption 4, $\mathbb{E}[h_{i}(k)]=\frac{1}{N}$ holds for all
$i=1,2,\ldots,N$ and $k\geq 0$.

###### Proof:

From (11), we have
$h_{i}(k)=\frac{\alpha_{i}(k)}{\sum_{i=1}^{N}\alpha_{i}(k)}$ and
$\sum_{i=1}^{N}h_{i}(k)=1$. Since each $\alpha_{i}(k)$ is sampled from the
same distribution as stated in Assumption 4, the underlying distribution of
each $h_{i}(k)$ is the same, i.e., $\mathbb{E}[h_{i}(k)]=\mathbb{E}[h_{j}(k)]$
holds for all $i,j\in\\{1,2,\cdots,N\\}$ and $\forall k\geq 0$. Moreover, due
to the linearity of expectation, we have

$1=\mathbb{E}\Big{[}\sum_{i=1}^{N}h_{i}(k)\Big{]}=\sum_{i=1}^{N}\mathbb{E}[h_{i}(k)]=N\mathbb{E}[h_{i}(k)],$

which implies $\mathbb{E}[h_{i}(k)]=\frac{1}{N}$ for all $i=1,2,\ldots,N$. ∎

###  III-C Convergence Analysis of the FedCOTA Algorithm

First, we present preparatory results needed to show the convergence of the
FedCOTA algorithm. Consider

$\displaystyle C(k)=C_{k}=1-\eta(k)\mu.$ (14)

Note that, under Assumption 3, $C_{k}\geq 0$ holds $\forall
k\in\mathbb{N}_{0}$. As $\eta(k)$ is decreasing, $C_{k}$ is increasing in $k$,
hence it suffices to show $C_{0}\geq 0$. This immediately follows from
$C_{0}=1-\eta_{c}\mu$ and $\eta_{c}\leq\frac{1}{\mu}$ (see Remark 4), hence
$C_{0}\geq 0$. Note furthermore that, as $C_{k}$ is increasing, $C_{k}>0$
holds for all $k\geq 1$.

###### Lemma 3

Under Assumption 3, $\lim_{k\to\infty}(C_{k})^{k}=0$.

###### Proof:

By letting $C_{k}=1-\frac{Q}{\sqrt{k+1}}$, where $Q=\eta_{c}\mu>0$, we write

$\displaystyle\lim_{k\to\infty}(C_{k})^{k}$
$\displaystyle=\lim_{k\to\infty}\Big{(}1-\frac{Q}{\sqrt{k+1}}\Big{)}^{k}.$
(15)

Note that $\forall x\in\Re$, we have $1+x\leq e^{x}$. Hence, $\forall
k\in\mathbb{N}_{0}$

$\displaystyle C_{k}=1-\frac{Q}{\sqrt{k+1}}\leq e^{-\frac{Q}{\sqrt{k+1}}}.$
(16)

Moreover, as noted above, $C_{k}\geq 0$, hence

$\displaystyle 0\leq(C_{k})^{k}\leq e^{-\frac{Qk}{\sqrt{k+1}}}.$ (17)

As the term on the right hand side of (17) goes to zero for $k\to\infty$, we
have established that $\lim_{k\to\infty}(C_{k})^{k}=0$. ∎

###### Lemma 4

Under Assumption 3, ${\prod}_{k=0}^{\infty}C_{k}=0$.

###### Proof:

By using the relation $1+x\leq e^{x}$, $\forall x\in\Re$, we write

$\displaystyle{\prod}_{t=0}^{k}C_{t}$
$\displaystyle={\prod}_{t=0}^{k}\big{(}1-\eta(t)\mu\big{)}$
$\displaystyle\leq{\prod}_{t=0}^{k}\mathrm{e}^{-\eta(t)\mu}=\mathrm{e}^{-{\sum}_{t=0}^{k}\eta(t)\mu}$
(18)

Taking the limit as $k\to\infty$ on both sides gives

$\displaystyle\lim_{k\to\infty}{\prod}_{t=0}^{k}C_{t}\leq\mathrm{e}^{-\mu{\sum}_{t=0}^{\infty}\eta(t)}=0$
(19)

since by Assumption 3, $\mu{\sum}_{t=0}^{\infty}\eta(t)=\infty$. ∎

###### Lemma 5

Under Assumption 3, for $k\to\infty$,

${\sum}_{t=0}^{k-2}\Big{(}{\prod}_{l=t+1}^{k-1}C_{l}\Big{)}\eta^{2}(t)+\eta^{2}(k-1)$

converges to an arbitrarily small positive value $\epsilon$.

###### Proof:

See Appendix. ∎

We are now ready to present the main result.

###### Theorem 1

Let Assumptions 1, 2, 3, and 4 hold. Then, $\exists\theta^{*}\in\Theta^{*}$
such that for $k\to\infty$,
$\mathbb{E}\big{[}||\theta(k)-\theta^{*}||^{2}\big{]}$ is arbitrarily small.

###### Proof:

For any $\theta^{*}\in\Theta^{*}$, by using (5), (III-A), (11), and the non-
expansive111$||\mathbf{P}_{\Theta}(x)-\mathbf{P}_{\Theta}(y)||\leq||x-y||$
holds for all $x,y\in\Re^{m}$ if $\Theta$ is a nonempty closed convex set (see
[44]). property of the projection $\mathbf{P}_{\Theta}$, we write

$\displaystyle||\theta(k+1)-\theta^{*}||^{2}$
$\displaystyle=\bigg{|}\bigg{|}\mathbf{P}_{\Theta}\Bigg{(}\sum_{i=1}^{N}h_{i}(k)\theta_{i}(k)\Bigg{)}-\mathbf{P}_{\Theta}\big{(}\theta^{*}\big{)}\bigg{|}\bigg{|}^{2}$
$\displaystyle\leq\bigg{|}\bigg{|}\sum_{i=1}^{N}h_{i}(k)\bigg{(}\theta(k)-\eta(k)\nabla
f_{i}\big{(}\theta(k)\big{)}\bigg{)}-\theta^{*}\bigg{|}\bigg{|}^{2}$
$\displaystyle=\big{|}\big{|}\theta(k)-\theta^{*}\big{|}\big{|}^{2}$
$\displaystyle\qquad-2\eta(k)\sum_{i=1}^{N}h_{i}(k)\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}$
$\displaystyle\qquad\qquad\qquad+\Big{|}\Big{|}\eta(k)\sum_{i=1}^{N}h_{i}(k)\nabla
f_{i}\big{(}\theta(k)\big{)}\Big{|}\Big{|}^{2}.$ (20)

Note that boundedness of the constraint set $\Theta$ and Lipschitz continuity
of $\nabla f_{i}\big{(}\theta(k)\big{)}$ imply that $\exists D_{\Theta},M>0$
such that

$\displaystyle\big{|}\big{|}\theta(k)-\theta^{*}\big{|}\big{|}$
$\displaystyle\leq D_{\Theta},$ (21) $\displaystyle\big{|}\big{|}\nabla
f_{i}\big{(}\theta(k)\big{)}\big{|}\big{|}$ $\displaystyle\leq M.$ (22)

Then, an upper-bound for the last term on the right hand side of (III-C) can
be written as

$\displaystyle\Big{|}\Big{|}\eta(k)\sum_{i=1}^{N}h_{i}(k)\nabla
f_{i}\big{(}\theta(k)\big{)}\Big{|}\Big{|}^{2}$
$\displaystyle=\eta^{2}(k)\Big{|}\Big{|}\sum_{i=1}^{N}h_{i}(k)\nabla
f_{i}\big{(}\theta(k)\big{)}\Big{|}\Big{|}^{2}$
$\displaystyle\leq\eta^{2}(k)\sum_{i=1}^{N}h_{i}(k)\Big{|}\Big{|}\nabla
f_{i}\big{(}\theta(k)\big{)}\Big{|}\Big{|}^{2}$
$\displaystyle\leq\eta^{2}(k)M^{2},$ (23)

which follows from (11), (22), and the convexity of the function
$||\cdot||^{2}$. Subsequently, taking the expectations of both sides of
(III-C) and using the linearity of expectation results in

$\displaystyle\mathbb{E}\big{[}||\theta(k+1)-\theta^{*}||^{2}\big{]}$
$\displaystyle\leq\mathbb{E}\big{[}\big{|}\big{|}\theta(k)-\theta^{*}\big{|}\big{|}^{2}\big{]}$
$\displaystyle-2\eta(k)\mathbb{E}\big{[}\sum_{i=1}^{N}h_{i}(k)\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle\qquad\qquad\qquad+\eta^{2}(k)M^{2},$ (24)

where the second term on the right hand side can be written as

$\displaystyle-2\eta(k)\mathbb{E}\big{[}\sum_{i=1}^{N}h_{i}(k)\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle=-2\eta(k)\sum_{i=1}^{N}\mathbb{E}\big{[}h_{i}(k)\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}.$
(25)

Note that the statistics of $h_{i}(k)$ ($i=1,2,\cdots,N$) are independent of
$h_{i}(t)$ for $t<k$, which implies that $\theta(k)$ and $h_{i}(k)$ are
statistically independent at time $k$ since the statistics of $\theta(k)$ are
dependent only of $h_{i}(t)$ for $t<k$ and $i=1,2,\cdots,N$. Hence, by using
(2), Lemma 2, and the linearity of the expectation, we can further write
(III-C) as

$\displaystyle-2\eta(k)\sum_{i=1}^{N}\mathbb{E}\big{[}h_{i}(k)\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle=-2\eta(k)\sum_{i=1}^{N}\mathbb{E}\big{[}h_{i}(k)\big{]}\mathbb{E}\big{[}\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle=-2\eta(k)\sum_{i=1}^{N}\frac{1}{N}\mathbb{E}\big{[}\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle=-2\eta(k)\mathbb{E}\big{[}\sum_{i=1}^{N}\frac{1}{N}\nabla
f^{T}_{i}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}$
$\displaystyle=-2\eta(k)\mathbb{E}\big{[}\nabla\mathcal{L}^{T}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}.$
(26)

Moreover, by Assumption 2 (strong convexity of $f_{i}(\theta)$ and
$\mathcal{L}\big{(}\theta(k)\big{)}$), we have

$\displaystyle-2\eta(k)\nabla\mathcal{L}^{T}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\leq-$
$\displaystyle
2\eta(k)\big{(}\mathcal{L}\big{(}\theta(k)\big{)}-\mathcal{L}\big{(}\theta^{*}\big{)}\big{)}$
$\displaystyle{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}-}\eta(k)\mu||\theta(k)-\theta^{*})||^{2}.$
(27)

Note that
$\mathcal{L}\big{(}\theta(k)\big{)}-\mathcal{L}\big{(}\theta^{*}\big{)}\geq 0$
holds for all $k\geq 0$ since $\theta^{*}\in\Theta^{*}$ (optimal point in the
constraint set), which together with taking the expectations of both sides of
(III-C) results in

$\displaystyle-2\eta(k)\mathbb{E}\big{[}\nabla\mathcal{L}^{T}\big{(}\theta(k)\big{)}\big{(}\theta(k)-\theta^{*}\big{)}\big{]}\leq-\eta(k)\mu\mathbb{E}\big{[}||\theta(k)-\theta^{*})||^{2}\big{]}.$
(28)

By using (III-C)-(28), (III-C) can be rewritten as

$\displaystyle\mathbb{E}\big{[}||\theta(k+1)-\theta^{*}||^{2}\big{]}$
$\displaystyle\leq\Big{(}1-\eta(k)\mu\Big{)}\mathbb{E}\big{[}\big{|}\big{|}\theta(k)-\theta^{*}\big{|}\big{|}^{2}\big{]}$
$\displaystyle\qquad\qquad\qquad+\eta^{2}(k)M^{2}$
$\displaystyle=C_{k}\mathbb{E}\big{[}||\theta(k)-\theta^{*}||^{2}\big{]}+\eta^{2}(k)M^{2}.$
(29)

The recursive relation (III-C) can be written as

$\displaystyle\mathbb{E}\big{[}\big{|}\big{|}\theta(k)-\theta^{*}\big{|}\big{|}^{2}\big{]}$
$\displaystyle\leq\bigg{(}{\prod}_{t=0}^{k-1}C_{t}\bigg{)}\mathbb{E}\big{[}\big{|}\big{|}\theta(0)-\theta^{*}\big{|}\big{|}^{2}\big{]}$
$\displaystyle+M^{2}\Bigg{(}{\sum}_{t=0}^{k-2}\Big{(}{\prod}_{l=t+1}^{k-1}C_{l}\Big{)}\eta^{2}(t)+\eta^{2}(k-1)\Bigg{)}.$
(30)

Note that by Lemma 4, ${\prod}_{t=0}^{\infty}C_{t}=0$ holds. Moreover, by
Lemma 5, Assumption 3, taking the limits of both sides of (III-C) completes
the proof.∎

##  IV Numerical Example

We now apply the FedCOTA algorithm to a federated logistic regression problem,
where a system of agents, each with access to only its own local dataset,
tries to accomplish a global binary classification task. Let the dataset
available to the $i$-th agent be $D_{i}=\\{d^{n}_{i}\\}_{n=1}^{|D_{i}|}$,
where $d^{n}_{i}=(u^{n}_{i},z^{n}_{i})\in\Re^{m}\times\\{0,1\\}$, and
$u^{n}_{i}$ and $z^{n}_{i}$ respectively represent input data and labels
available to $i$-th agent. Notice that all the agents have 2 different classes
of data, labeled by 0 or 1, and their objective is to find a separating
hyperplane in $\Re^{m}$ by using the existing data so that the agent is able
to separate some unseen data from different classes. In order to accomplish
this in a federated manner, each agent utilizes cross-entropy as its local
cost function, which can be expressed as

$\displaystyle
f_{i}(\theta,d_{i})=\lambda||\theta||_{2}^{2}-\frac{1}{|D_{i}|}\Bigg{(}$
$\displaystyle\sum_{n=1}^{|D_{i}|}z^{n}_{i}\log\big{(}S(\theta^{T}u^{n}_{i})\Big{)}$
$\displaystyle+(1-z^{n}_{i})\log\big{(}1-S(\theta^{T}u^{n}_{i})\Big{)}\Bigg{)}$
(31)

where $\hat{z}^{n}_{i}=S(\theta^{T}u^{n}_{i})$ is the local estimate of the
label $z^{n}_{i}$ by the $i$-th agent, $S(x)=\frac{1}{1+e^{-x}}$ is the
sigmoid function, $||\theta||_{2}$ is the $L_{2}$-norm of the parameter vector
$\theta$, and $\lambda$ is the regularization hyper-parameter, where
$\lambda=0$ means no regularization while $\lambda=1$ represents maximum level
of regularization. Note that $\lambda=0.0001$ has been chosen in simulations,
and the overall cost function given in (IV) is strongly convex. Moreover,
since the parameter vector $\theta$ is always in a closed and bounded
(constraint) set $\Theta$, the second order derivative of (IV) is bounded and
therefore has a Lipschitz continuous gradient. Additionally, for each agent,
the step size is identically chosen as $\eta(k)=\frac{1}{\sqrt{k+1}}$, and the
overall cost function is then

$\displaystyle\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta,d_{i})$
(32)

where $N$ is the number of agents in the system. We consider a system of $10$
agents, each having a total number of $|D_{i}|=100$ training samples. The
parameter vector has dimension $m=3$, i.e., $\theta\in\Re^{3}$, which also
includes a bias term. The objective is to find
$\theta^{*}=\operatorname*{arg\,min}_{\theta\in\Theta}\mathcal{L}(\theta)$,
where $\Theta=\\{\theta\in\Re^{3}|\hskip 2.84526pt||\theta(k)||\leq 15,\forall
k\geq 0\\}$ is the constraint set.

Figure 2: Comparison of a global logistic regression and federated logistic
regression models. While the former is trained over the entire dataset, the
latter is trained via FedCOTA and FedAVG.

In order to obtain a desired parameter vector $\theta_{d}$, we have trained a
global logistic regression model, which has access to the entire dataset. We
would expect that the parameter vector $\theta(k)$ tends to converge
$\theta_{d}$. This can be assessed by computing
$\epsilon(k)=\log_{10}\Big{(}\frac{\theta(k)-\theta_{d}}{\theta(0)-\theta_{d}}\Big{)}$
for each communication round $k\geq 0$. As it can be seen in Fig. 2, the
parameter vectors of agents utilizing the FedCOTA algorithm tend to converge
to $\theta_{d}$.

Note that Fig. 2 also includes a comparison of the FedCOTA algorithm and the
standard FedAVG algorithm [5], for which the TDMA scheme is used for
communication between agents and the central unit. In this case, individual
time slots are allocated for each agent to transmit its parameter vector at
each communication round. After receiving parameter vectors, the server
computes the average of them and sends it back to the agents. Since we
consider a system with $N=10$ agents, it takes $10$ time slots per
communication round for each agent to transmit its updated parameter vector
while only $2$ are needed for the FedaOTA algorithm (see (7) and (8)), which
makes it 5 times faster than the FedAVG algorithm (see Fig. 2).

##  V Conclusion

In this paper we have investigated the federated learning problem via Over-
the-Air Computation, which provides significant improvements in terms of
communication efficiency and privacy. We have investigated the convergence
properties of the proposed gradient-based algorithm by considering time-
varying step sizes. Subsequently, we have presented some numerical examples to
illustrate our theoretical results.

Our current research is on the use of stochastic gradient descent, which is
more efficient when we have large number of training samples. In addition,
future work will also include cases where the communication between agents is
fully distributed and data distribution among users are not independent and
identically distributed (non-iid).

##  VI Appendix: Proof of Lemma 5

Due to Assumption 3, $C_{k}$ is increasing and nonnegative. Then, we have

$\displaystyle{\sum}_{t=0}^{k-2}\Big{(}{\prod}_{l=t+1}^{k-1}C_{l}\Big{)}\eta^{2}(t)+\eta^{2}(k-1)\leq{\sum}_{t=0}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t).$
(33)

Moreover, again by Assumption 3, $\eta(k)$ is decreasing and
$\lim_{k\to\infty}\eta(k)=0$. Thus, for an arbitrarily given $\epsilon>0$,
there exists a time step $k_{0}>0$ such that $\eta(k)\leq\epsilon$, $\forall
k\geq k_{0}$. Since $\eta(k)$ is decreasing, we have

$\displaystyle\eta(k)<\eta(k-1)=\frac{1-C_{k-1}}{\mu}.$ (34)

Multiplying both sides of (34) with $\eta(k)$ provides for all $k\geq k_{0}$

$\displaystyle\eta^{2}(k)<\frac{(1-C_{k-1})\epsilon}{\mu}.$ (35)

For $k>k_{0}+1$, the right hand side of (33) can be bounded as

$\displaystyle{\sum}_{t=0}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t)\leq{\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t)$
$\displaystyle\hskip
99.58464pt+{\sum}_{t=k_{0}+1}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t).$

Decreasingness of $\eta(k)$ implies $\eta^{2}(0)\geq\eta^{2}(k)$ for $k\geq
0$. From (14), $C_{k-1}\leq 1$ for all $k\geq 1$. Hence, the first term on the
right hand side of (VI) can be written as

$\displaystyle{\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t)$
$\displaystyle\leq\eta^{2}(0){\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{k-t-1}$
$\displaystyle=\eta^{2}(0){\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{k-k_{0}+t-1}$
$\displaystyle=\eta^{2}(0)\big{(}C_{k-1}\big{)}^{-k_{0}}\big{(}C_{k-1}\big{)}^{k-1}{\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{t}$
$\displaystyle\leq\eta^{2}(0)\big{(}C_{k-1}\big{)}^{-k_{0}}\big{(}C_{k-1}\big{)}^{k-1}(k_{0}+1),$
(37)

where $\lim_{k\to\infty}(C_{k-1})^{k-1}=0$ holds by Lemma 3. Thus, we have

$\displaystyle\lim_{k\to\infty}{\sum}_{t=0}^{k_{0}}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(t)=0.$
(38)

In addition, (35) holds for $k>k_{0}+1$, which allows us to find an upper-
bound for the second term on the right hand side of (VI) as

$\displaystyle{\sum}_{t=k_{0}+1}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}\eta^{2}(k)$
$\displaystyle\leq\frac{(1-C_{k-1})\epsilon}{\mu}{\sum}_{t=k_{0}+1}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}$
$\displaystyle=\frac{(1-C_{k-1})\epsilon}{\mu}\Bigg{(}\frac{1-\big{(}C_{k-1}\big{)}^{k-k_{0}-1}}{1-C_{k-1}}\Bigg{)}$
$\displaystyle\leq\frac{\epsilon}{\mu}.$ (39)

By using (VI)-(VI), and taking the limit of both sides of (VI) results in

$\displaystyle\lim_{k\to\infty}{\sum}_{t=0}^{k-1}\big{(}C_{k-1}\big{)}^{k-t-1}\eta(t)\leq\frac{\epsilon}{\mu}.$
(40)

Since $\epsilon$ is arbitrary, we complete the proof.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [3] A. Nedic, “Distributed gradient methods for convex machine learning problems in networks: Distributed optimization,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 92–101, 2020.
  * [4] J. Chen and X. Ran, “Deep learning with edge computing: A review,” _Proceedings of the IEEE_ , vol. 107, no. 8, pp. 1655–1674, 2019.
  * [5] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [6] V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [7] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 10, no. 2, pp. 1–19, 2019.
  * [8] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, B. McMahan _et al._ , “Towards federated learning at scale: System design,” _Proceedings of Machine Learning and Systems_ , vol. 1, pp. 374–388, 2019.
  * [9] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” _arXiv preprint arXiv:1710.06963_ , 2017\.
  * [10] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021\.
  * [11] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 50–60, 2020.
  * [12] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in _Theory of cryptography conference_. Springer, 2006, pp. 265–284.
  * [13] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 1175–1191.
  * [14] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” _arXiv preprint arXiv:1610.05492_ , 2016.
  * [15] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “Qsgd: Communication-efficient sgd via gradient quantization and encoding,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [16] D. Alistarh, T. Hoefler, M. Johansson, N. Konstantinov, S. Khirirat, and C. Renggli, “The convergence of sparsified gradient methods,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [17] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2020, pp. 2021–2031.
  * [18] M. Chen, N. Shlezinger, H. V. Poor, Y. C. Eldar, and S. Cui, “Communication-efficient federated learning,” _Proceedings of the National Academy of Sciences_ , vol. 118, no. 17, 2021.
  * [19] A. Mitra, R. Jaafar, G. Pappas, and H. Hassani, “Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [20] F. Molinari, N. Agrawal, S. Stańczak, and J. Raisch, “Max-consensus over fading wireless channels,” _IEEE Transactions on Control of Network Systems_ , vol. 8, no. 2, pp. 791–802, 2021.
  * [21] M. Frey, I. Bjelaković, and S. Stańczak, “Over-the-air computation in correlated channels,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 5739–5755, 2021.
  * [22] G. Zhu, J. Xu, K. Huang, and S. Cui, “Over-the-air computing for wireless data aggregation in massive iot,” _IEEE Wireless Communications_ , vol. 28, no. 4, pp. 57–65, 2021.
  * [23] K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 3, pp. 2022–2035, 2020.
  * [24] M. M. Amiri and D. Gündüz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 2155–2169, 2020.
  * [25] T. Sery, N. Shlezinger, K. Cohen, and Y. C. Eldar, “Over-the-air federated learning from heterogeneous data,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 3796–3811, 2021.
  * [26] R. Ahlswede, “Multi-way communication channels,” in _Second International Symposium on Information Theory: Tsahkadsor, Armenia, USSR, Sept. 2-8, 1971_ , 1973.
  * [27] A. Giridhar and P. Kumar, “Toward a theory of in-network computation in wireless sensor networks,” _IEEE Communications Magazine_ , vol. 44, no. 4, pp. 98–107, apr 2006.
  * [28] F. Molinari, N. Agrawal, S. Stańczak, and J. Raisch, “Over-the-air max-consensus in clustered networks adopting half-duplex communication technology,” _IEEE Transactions on Control of Network Systems_ , 2022.
  * [29] W. Rudin _et al._ , _Principles of mathematical analysis_. McGraw-hill New York, 1976, vol. 3.
  * [30] K. Yuan, Q. Ling, and W. Yin, “On the convergence of decentralized gradient descent,” _SIAM Journal on Optimization_ , vol. 26, no. 3, pp. 1835–1854, 2016.
  * [31] C. Xi, V. S. Mai, R. Xin, E. H. Abed, and U. A. Khan, “Linear convergence in optimization over directed graphs with row-stochastic matrices,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 10, pp. 3558–3565, 2018.
  * [32] R. Xin, S. Pu, A. Nedić, and U. A. Khan, “A general framework for decentralized optimization with first-order methods,” _Proceedings of the IEEE_ , vol. 108, no. 11, pp. 1869–1889, 2020.
  * [33] V. S. Mai and E. H. Abed, “Distributed optimization over weighted directed graphs using row stochastic matrix,” in _2016 American Control Conference (ACC)_. IEEE, 2016, pp. 7165–7170.
  * [34] H. Li, Q. Lü, and T. Huang, “Distributed projection subgradient algorithm over time-varying general unbalanced directed graphs,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 3, pp. 1309–1316, 2018.
  * [35] Y. Nesterov, _Introductory lectures on convex optimization: A basic course_. Springer Science & Business Media, 2003, vol. 87.
  * [36] A. Nedić and A. Olshevsky, “Distributed optimization over time-varying directed graphs,” _IEEE Transactions on Automatic Control_ , vol. 60, no. 3, pp. 601–615, 2014.
  * [37] T. Yang, X. Yi, J. Wu, Y. Yuan, D. Wu, Z. Meng, Y. Hong, H. Wang, Z. Lin, and K. H. Johansson, “A survey of distributed optimization,” _Annual Reviews in Control_ , vol. 47, pp. 278–305, 2019.
  * [38] P. Wang, P. Lin, W. Ren, and Y. Song, “Distributed subgradient-based multiagent optimization with more general step sizes,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 7, pp. 2295–2302, 2017.
  * [39] P. Xie, K. You, R. Tempo, S. Song, and C. Wu, “Distributed convex optimization with inequality constraints over time-varying unbalanced digraphs,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 12, pp. 4331–4337, 2018.
  * [40] V. S. Mai and E. H. Abed, “Distributed optimization over directed graphs with row stochasticity and constraint regularity,” _Automatica_ , vol. 102, pp. 94–104, 2019.
  * [41] D. Tse and P. Viswanath, _Fundamentals of Wireless Communication_. Cambridge University Press, 2012.
  * [42] A. F. Molisch, _Wireless communications_. John Wiley & Sons, 2012.
  * [43] B. Sklar, “Rayleigh fading channels in mobile digital communication systems. i. characterization,” _IEEE Communications magazine_ , vol. 35, no. 7, pp. 90–100, 1997.
  * [44] D. Bertsekas, A. Nedic, and A. Ozdaglar, _Convex analysis and optimization_. Athena Scientific, 2003, vol. 1.