HARP

    High-frequency Acoustic Recording Package
MNLL

    mean negative log likelihood
MASINT

    measurement and signature intelligence
NPS

    Naval Postgraduate School
SRWBR

    short range wide band radio
TCP

    Transmission Control Protocol
MARS

    Monterey accelerated research system
MMSI

    maritime mobile service identities
USN

     United States (U.S.) Navy
NN

    neural network
DOA

    direction of arrival
AIS

    automatic identification system
BAL

    Bayesian active learning
UATR

    underwater acoustic target recognition
NPS

    Naval Postgraduate School
ML

    machine learning
FL

    federated learning
HFL

    hierarchical federated learning (FL)
BDL

    Bayesian deep learning
DL

    deep learning
CL

    continual learning
AL

    active learning
CNN

    convolutional neural network
GDPR

    General Data Protection Regulation
BFL

    Bayesian federated learning
IID

    independent and identically distributed
non-IID

    non-independent and identically distributed
VI

    variational inference
NWA

    Naive Weighted Averaging
WS

    Weighted Sum of Normal Distributions
LP

    Linear Pooling
IV

    Inverse Variance Weighting
DWC

    Distributed Weight Consolidation
CLP

    Centered Linear Pool
LLP

    Log-Linear Pooling
WC

    Weighted Conflation
BNN

    Bayesian neural network
MC

    Monte Carlo
KL

    Kullback-Leibler
NLL

    negative log likelihood
SGD

    Stochastic Gradient Descent
ELBO

    Evidence Lower Bound
TL

    transfer learning
LOFAR

    low-frequency analysis and recording
ResNet

    residual network
mAP

    mean average precision
AUC

    area under the curve
VI-PANN

    variational inference pre-trained audio neural network
PANN

    pre-trained audio neural network
MCMC

    Markov Chain Monte Carlo
STFT

    Short-time Fourier transform
MFCC

    mel frequency cepstral coefficients
CCTZ

    Chroma, Contrast, Tonnetz, and Zero-cross rate
ResNet

    residual network
RNN

    recurrent neural network
NLP

    natural language processing
LLM

    large language model
COCO

    Common Objects in Context
MLOps

    Machine Learning Operations
EABO

    Expeditionary Advanced Base Operations
EWC

    Elastic Weight Consolidation
VCL

    Variational Continual Learning
BALD

    Bayesian active learning by disagreement
DARPA

    Defense Advanced Research Projects Agency
FFT

    federated fine-tuning
CI

    confidence interval
OOD

    out-of-distribution
TL

    transfer learning
NOAA

    National Oceanic and Atmospheric Administration
USCG

    United States Coast Guard
SanctSound

    Sanctuary Soundscapes
MMSI

    Maritime Mobile Service Identities
IMO

    International Maritime Organization
LTSA

    long-term spectral average
UPAD

    Underwater Passive Acoustic Dataset
SOTA

    state-of-the-art
AI

    artificial intelligence

# Federated Bayesian Deep Learning: The Application of Statistical Aggregation
Methods to Bayesian Models

John Fischer
Department of Computer Science
Naval Postgraduate School
Monterey, CA 93954
<EMAIL_ADDRESS>
&Marko Orescanin
Department of Computer Science
Naval Postgraduate School
Monterey, CA 93954
<EMAIL_ADDRESS>
&Justin Loomis
Department of Computer Science
Naval Postgraduate School
Monterey, CA 93954
&Patrick McClure
Department of Computer Science
Naval Postgraduate School
Monterey, CA 93954

(March 22, 2024)

###### Abstract

Federated learning (FL) is an approach to training machine learning models
that takes advantage of multiple distributed datasets while maintaining data
privacy and reducing communication costs associated with sharing local
datasets. Aggregation strategies have been developed to pool or fuse the
weights and biases of distributed deterministic models; however, modern
deterministic deep learning (DL) models are often poorly calibrated and lack
the ability to communicate a measure of epistemic uncertainty in prediction,
which is desirable for remote sensing platforms and safety-critical
applications. Conversely, Bayesian DL models are often well calibrated and
capable of quantifying and communicating a measure of epistemic uncertainty
along with a competitive prediction accuracy. Unfortunately, because the
weights and biases in Bayesian DL models are defined by a probability
distribution, simple application of the aggregation methods associated with FL
schemes for deterministic models is either impossible or results in sub-
optimal performance. In this work, we use independent and identically
distributed (IID) and non-IID partitions of the CIFAR-10 dataset and a fully
variational ResNet-20 architecture to analyze six different aggregation
strategies for Bayesian DL models. Additionally, we analyze the traditional
federated averaging approach applied to an approximate Bayesian Monte Carlo
dropout model as a lightweight alternative to more complex variational
inference methods in FL. We show that aggregation strategy is a key
hyperparameter in the design of a Bayesian FL system with downstream effects
on accuracy, calibration, uncertainty quantification, training stability, and
client compute requirements.

_Keywords_ Bayesian deep learning $\cdot$ federated learning $\cdot$ Monte
Carlo dropout $\cdot$ uncertainty decomposition $\cdot$ uncertainty
quantification $\cdot$ variational inference

##  1 Introduction

###  1.1 Motivation

Remote sensors are ubiquitous in the world today. This is due to the continual
advancement in hardware technology, which supports their deployment in ever-
shrinking, affordable devices. The mass deployment of remote sensors has
resulted in an explosion in the amount of data collected in support of various
research groups, corporations, and government entities [1]. The overwhelming
task of analyzing this data has resulted in a desire to automate or to augment
data analysis using deep learning (DL) techniques. In many remote sensing
applications, the cost of collecting, labeling, communicating, and storing
local data may be acceptable; however, when deploying sensors all over the
world, these costs can breach the threshold of acceptability. In other
applications, the transfer and storage of user data raises data privacy
concerns and regulations, such as the General Data Protection Regulation
(GDPR) [2], and imposes significant restrictions on the handling of such data.
In order to address these issues, McMahan et al. [3] developed a technique
that they termed FL, whereby client models train, or update, an instance of a
global model using locally captured data and periodically send updated model
parameters to a central server for aggregation. By using FL, it is possible to
take advantage of numerous distributed datasets while maintaining privacy and
reducing communication costs associated with training on a collection of these
individual datasets.

Since the seminal work by McMahan et al. [3], a significant amount of research
has been done to address the numerous challenges associated with implementing
an FL framework. This list of challenges includes, but is not limited to,
efficiency of communications [4, 5, 6], device heterogeneity [7, 8, 9],
statistical heterogeneity [10, 11, 12, 13], and privacy preservation/security
[14, 15, 16, 17]. Often, when addressing the aforementioned problems in the FL
literature, a deterministic model architecture is assumed [3, 7, 11]. While
deterministic deep learning models are capable of achieving high accuracy,
they are often uncalibrated [18] (i.e., model-predicted probabilities for each
class do not match the empirical frequency for that class) and do not provide
access to epistemic (model) uncertainty in prediction. The inability to access
epistemic uncertainty is problematic in many remote sensing and other safety-
critical applications, where it is important that the underlying model is
capable of communicating a measure of epistemic uncertainty in prediction and
that the uncertainty be calibrated.

Although the statistics, meteorological, and multi-sensor fusion literature
are rich with probabilistic pooling or fusion approaches [19, 20, 21], there
does not exist a detailed study of these probabilistic pooling/fusion
strategies in the FL setting. Recognizing that there are numerous ways of
aggregating, or fusing, probability distributions and that there is no agreed
upon technique, Koliander et al. [22] provide a comprehensive review of fusion
techniques and the motivation for utilizing various fusion techniques and
client/site weighting schemes. Utilizing several of the strategies from [22]
and additional strategies from [3] and [23], we analyze the effect of fusion
techniques in the Bayesian federated learning (BFL) setting, with a focus on
the quality of uncertainty. To align with the nomenclature common in FL
literature throughout this manuscript, we use the term “aggregate" in place of
“fuse” or “pool.”

###  1.2 Contributions

In this paper, we make the following contributions:

  1. 1.

We re-frame statistical aggregation strategies in the context of FL,
specifically for Bayesian deep learning (BDL).

  2. 2.

We evaluate six aggregation strategies for BDL models in the FL setting on
independent and identically distributed (IID) and non-IID partitions of a
popular computer vision dataset commonly used to benchmark performance in
machine learning (ML) and FL applications. We show that aggregation strategy
is a crucial hyperparameter in the design of a BFL algorithm.

  3. 3.

We evaluate three client weighting strategies in conjunction with each of the
six aggregation strategies under evaluation.

  4. 4.

We examine a popular lightweight Bayesian approximation, Monte Carlo (MC)
dropout, in the FL setting and compare its performance to more complex
variational inference (VI) models.

  5. 5.

We evaluate not only the prediction accuracy, but also the calibration and
uncertainty (total and decomposed) of both mean-field Gaussian VI and MC
dropout VI models used in the FL setting.

  6. 6.

We share our Flower Federated Learning Framework [24] implementation to
facilitate further experimentation with aggregation methods and client
weighting techniques on additional datasets.111https://github.com/meekus-
fischer/BayesianFederatedLearning

##  2 Background and Related Work

###  2.1 Federated Learning

In FL, client models train, or update, an instance of a global model using
locally captured data and periodically send updated model parameters to a
central server for aggregation [3]. Often, this technique is utilized to take
advantage of numerous local datasets while maintaining privacy. Additionally,
in remote sensing platforms and other edge devices, FL significantly reduces
the communication costs associated with training on these individual datasets.
We use the following notation throughout this manuscript: $k\in\\{1,...,K\\}$
represents the individual clients,
$\mathcal{D}^{k}=\\{(x_{1},y_{1}),...,(x_{|\mathcal{D}^{k}|},y_{|\mathcal{D}^{k}|})\\}$
represents the local dataset of client $k$ with $|\mathcal{D}^{k}|$ local
training examples, $\mathcal{D}$ comprises the entire training dataset split
amongst $K$ clients, $\theta_{r}^{k}$ is the set of parameters for client $k$
and FL round $r$, $\theta^{g}_{r}$ is the set of parameters for the global
model at round $r$, $\omega_{k}$ is the aggregation weight for client $k$, B
is the local minibatch size, E is the number of local epochs, and $\eta$ is
the learning rate. In general, we can describe the federated optimization
problem of $K$ clients with the following:

$\underset{\theta\in\mathbb{R}^{d}}{\text{min}}\hskip
3.0ptf(\theta^{g}),\hskip 15.0pt\text{where}\hskip
20.0ptf(\theta^{g})=\sum_{k=1}^{K}\omega_{k}f_{k}(\theta^{g}).$ (1)

In the ML/FL context, $f_{k}(\theta)$ often corresponds to
$\mathcal{L}_{k}(\mathcal{D}^{k},\theta)$, the local loss of client $k$ on
$\mathcal{D}^{k}$ with model parameters $\theta$. We place the following
constraint on client aggregation weights: $\sum_{k=1}^{K}\omega_{k}=1$. In
practice, this optimization is done, as described in Alg. 1, over multiple
optimization/communication rounds. First, the server initializes the global
model (or distributes a pre-trained baseline model) and distributes the model
parameters to each of the clients. During each optimization round $r$, clients
train the global model, using a variant of Stochastic Gradient Descent (SGD),
for $E$ epochs before returning the client parameters to the server. The
server then uses an aggregation function (e.g., FedAvg [3]) to combine the
model weights and biases of each client into a new global model, and
distributes the new global model parameters to each of the clients. This
process repeats for $R$ communication rounds, or until convergence.

Server Executes:

initialize $\theta^{g}_{0}$

foreach  _round r = 0,1,…,R-1_ do

foreach  _client $k\in\\{1,...,K\\}$ in parallel_ do

$\theta_{r+1}^{k}\leftarrow$ ClientUpdate($k,\theta^{g}_{r}$)

$\omega=$ CalculateClientWeights() … [Compute $w_{k}$ using Eq. (18), (19),
(20), or (21)]

$\theta^{g}_{r+1}\leftarrow$
AggregateClients($\\{\theta^{1}_{r+1},...,\theta^{K}_{r+1}\\},\omega$) … [Eq.
(10), (11), (12), (15), (16), or (17) ]

ClientUpdate($k,\theta^{g}$): // Run on client $k$

$\mathcal{MB}\leftarrow$ (split $\mathcal{D}^{k}$ into minibatches of size B)

$\theta^{k}\leftarrow\theta^{g}$

foreach  _local epoch i from 1 to E_ do

for  _minibatch b $\in\mathcal{MB}$_ do

$\theta^{k}\leftarrow\theta^{k}-\eta\nabla\ell$($\theta^{k};b$)

return $\theta^{k}$ to the server

Algorithm 1 Federated Learning.

###  2.2 Bayesian Deep Learning

To instill sufficient confidence in the performance, or capability, of ML
models for deployment on remote sensing platforms, or safety-critical
applications, and to ensure the models are able to adapt to dynamic
environments, it is critical that the models are capable of communicating a
measure of epistemic uncertainty in prediction [25, 26]. BDL models apply
Bayesian methods in order to provide access to the uncertainty measure
(epistemic uncertainty), which deterministic ML models are incapable of
providing. Instead of utilizing point estimates (deterministic ML), BDL learns
a probability distribution over model parameters [27]. BDL has its roots in
Bayesian statistics, where inference about model parameters, $\theta$, given
data, $\mathcal{D}$, involves the calculation of the posterior
$p(\theta|\mathcal{D})$. Placing a prior $p(\theta)$ on model parameters
(e.g., the weights and biases of a BDL architecture) enables the calculation
of the posterior distribution of model weights and biases utilizing Bayes’
rule:

$p\left(\theta|\mathcal{D}\right)=\frac{p\left(\mathcal{D}|\theta\right)p(\theta)}{p(\mathcal{D})}.$
(2)

Ultimately, prediction amounts to calculating

$p\left(y^{*}|x^{*},\mathcal{D}\right)=\int_{\Theta}p\left(y^{*}|x^{*},\theta\right)p\left(\theta|\mathcal{D}\right)d\theta,$
(3)

where $x^{*}$ is the input feature vector, $y^{*}$ is a corresponding
prediction, $\Theta$ represents the parameter space of the model, and
$p\left(y^{*}|x^{*},\mathcal{D}\right)$ is the probability of that prediction
given a trained model. Due to the fact that $p(\mathcal{D})$ in Eq. (2) is not
directly accessible, it must be calculated using known quantities.

$p(\mathcal{D})=\int_{\Theta}p(\mathcal{D},\theta)d\theta.$ (4)

High-dimensional integrals, such as the one in Eq. (4), often have no closed
form and are computationally intractable [28]. In practice, approximate
inference is used. There are many methods for approximate inference; however,
two of the most common approaches are Markov Chain Monte Carlo (MCMC) sampling
and VI [29]. Although MCMC is often described as the “gold standard” for
approximate inference, VI is commonly favored in BDL, due to the increased
speed and the ability to scale with large data and large models [30]. There
are a number of approaches to implementing VI; two of the most common of
implementations—Flipout [31] and Local Reparameterization Trick [32]—use
Gaussian distributions and effectively double the number of model parameters.
Another common approach, MC dropout, was shown to be an effective Bayesian
approximation that does not double the number of model parameters [33]. In
many situations, each of these methods has been empirically shown to lead to
useful approximation of $p(\theta|\mathcal{D})$, providing access to improved
uncertainty estimation when compared to the uncertainty provided by
deterministic models. In this work, we utilize Flipout VI models and MC
dropout models as our BDL models.

The essence of VI is that optimization is used to approximate the true
posterior $p(\theta|\mathcal{D})$ by identifying the closest distribution
$q_{\phi}(\theta)$, parameterized by $\phi$, to the posterior among a family
of pre-determined (variational) distributions (commonly a Gaussian). Often,
the “closeness” is determined using Kullback-Leibler (KL) divergence, a common
information-theoretic measure of similarity between two distributions.

$KL[q_{\phi}(\theta)\mid\mid
p(\theta|\mathcal{D})]=\int_{\Theta}q_{\phi}(\theta)log\frac{q_{\phi}(\theta)}{p(\theta|\mathcal{D})}d\theta.$
(5)

The optimization objective for VI then becomes

$\mathcal{L}_{q}=\text{KL}[q_{\phi}(\theta)\mid\mid
p(\theta)]-\mathbb{E}_{q}\left[\log p\left(\mathcal{D}|\theta\right)\right],$
(6)

where $\mathbb{E}_{q}$ represents the expected value under the probability
distribution $q_{\phi}(\theta)$. In this form, also known as the negative
Evidence Lower Bound (ELBO), it becomes clear that the optimization objective
is the sum of a data-dependent portion (likelihood cost), and a prior
dependent portion (complexity cost) [34].

The quality of the approximation in VI is highly dependent on the choice of
the variational distribution family. As mentioned in the previous paragraph,
we consider two distinct approaches, a mean-field Gaussian distribution with a
Flipout MC estimator of KL-divergence [31] and MC dropout, which uses a
Bernoulli distribution over the model weights and biases [33]. In the first,
the use of Gaussian distributions, parameterized by the mean and variance of
the distribution, doubles the number of parameters in the model architecture,
when compared to its deterministic counterpart. The use of the Gaussian
distribution results in a more challenging optimization problem relative to MC
dropout. In the MC dropout approach, there is no additional increase in the
number of model parameters, and minimal changes are required to the model
architecture [35, 25, 33]. It is these simplifications that motivate our
analysis of the MC dropout model in the FL framework as a lightweight Bayesian
approximation.

###  2.3 Uncertainty Quantification and Decomposition

In BDL models used for classification, predictive probability $p(y=c\mid x)$
is approximated by using MC integration with $M$ samples [36]. The mean
probability per class $\bar{p}_{c}$ is calculated using the prediction
probabilities, $\hat{p}_{c_{m}}$, where $\hat{p}_{c_{m}}=p(y=c\mid
x,\theta^{m})$ and $\theta^{m}$ is sampled from an approximation of
$p(\theta|\mathcal{D})$.

$\bar{p}_{c}=\frac{1}{M}\sum_{m=1}^{M}\hat{p}_{c_{m}}.$ (7)

The predicted class for each sample is chosen by selecting the class that
yields the highest mean probability. In addition to a predicted class, the $M$
MC samples allow for the calculation of predictive entropy and variance.
Predictive entropy has several interpretations; however, in information
theory, entropy is interpreted as the amount of information contained in a
predictive distribution [28]. Normalized predictive entropy ($\tilde{H}$), for
the multi-class classification problem, is given by

$\tilde{H}_{p}(y\mid\textbf{x})=-\sum_{c\in C}p(y=c\mid x)\frac{\log p(y=c\mid
x)}{\log C},$ (8)

where $C$ represents all possible classes [37]. Although entropy is commonly
used as a measure of aleatoric uncertainty in deterministic and Bayesian deep
learning, Depeweg et al. [38] interpret entropy as a measure of predictive
uncertainty, which could be further broken down into its aleatoric and
epistemic uncertainty components. Aleatoric uncertainty is known as data
uncertainty and cannot be reduced with an increase in training data. Epistemic
uncertainty is known as model uncertainty and can be reduced by increasing the
amount of training data [28]. Epistemic uncertainty is not accessible using
deterministic models.

For BDL models, variance provides another measure of total uncertainty, which
can be broken down into epistemic and aleatoric uncertainty. Kendall et al.
[26] and Kwon et al. [39] separately propose methods for estimating aleatoric
and epistemic uncertainty with Kwon et al. modifying the approach in [26] to
give the following estimator:

$\underbrace{\frac{1}{M}\sum_{m=1}^{M}\textnormal{diag}(\hat{p}_{m})-\hat{p}_{m}^{\otimes
2}}_{aleatoric}+\underbrace{\frac{1}{M}\sum_{m=1}^{M}(\hat{p}_{m}-\bar{p}_{c})^{\otimes
2}}_{epistemic},$ (9)

where $M$ is the number of MC samples, $\hat{p}_{m}$ is a $c$-dimensional
vector containing the predictive probabilities of each class
($\hat{p}_{{c}_{m}}$), $\textnormal{diag}(\hat{p}_{m})$ is a diagonal matrix,
${\hat{p}_{m}}^{\otimes 2}=\hat{p}_{m}{\hat{p}_{m}}^{T}$, and
$({\hat{p}_{m}}-\overline{p}_{c})^{\otimes
2}=({\hat{p}_{m}}-\overline{p}_{c})({\hat{p}_{m}}-\overline{p}_{c})^{T}$.

###  2.4 Federated Bayesian Learning

In scenarios involving big data with many distributed remote sensing
platforms, the need to adapt to dynamic environments, and the need to benefit
from the ability to communicate a measure of uncertainty, the application of
BDL to FL is a natural one. Several works have applied Bayesian techniques to
the FL problem. In [40], the authors apply the Laplace approximation at both
the client and server to reduce aggregation errors and regularize, or guide,
local model training. Notably, in this work, the authors choose to aggregate
client models using a mixture of Gaussians. FedBE [41] constructs a global
model by fitting a distribution (Gaussian or Dirichlet) to the client models
and sampling from this distribution to arrive at a higher-quality global
model. Similarly, FedAG [42] applies uncertainty in the aggregation step by
using the client parameters to fit a Gaussian distribution at the server. In
[43], the authors present a BFL framework where each client performs
approximate Bayesian inference, using MCMC sampling, and distills the clients’
posterior distribution into a single deep neural network via knowledge
distillation techniques for aggregation at the server. pFedBayes [12] is a
personalized FL model that uses VI to learn a personalized local model at each
client while learning a shared global model.

With the exception of pFedBayes [12], none of these works employ VI. While
pFedBayes [12] utilizes VI, the global model uses a simple averaging method
for client parameter aggregation, and no additional methods are discussed, or
analyzed. Additionally, in pFedBayes [12], the authors utilize a model with a
single variational layer (i.e., with the exception of the output layer, all
layers are deterministic). This work is the first time, to our knowledge, that
anyone has utilized a fully variational model (i.e., all model layers are
variational) to analyze the effects of the choice of aggregation function and
client weighting scheme on BDL models in FL.

The use of BDL in FL introduces several unique problems. First, the model
weights and biases in a VI model are no longer a point estimate. Each weight
and bias is defined by the parameters of a distribution. In the case of a
Gaussian, this is the mean and variance of the distribution. In the following
sub-sections, we present multiple aggregation strategies for univariate
Gaussian distributions and their underlying theory. Next, we discuss the
selection of client weights in the aggregation process. Often, the size of the
local training dataset is utilized to weight each client. However, this is not
the only method, and we present several other weighting schemes for analysis.
Finally, in mean-field Gaussian VI we often select priors with a fixed mean of
zero and a fixed variance, $p(\theta)=N(0,\sigma_{prior}^{2}I)$, for the
regularization term in the optimization problem. However, in this work, we
pose the question of whether or not to update the prior after each round of
optimization. We analyze the effect of updating the prior each round vs.
maintaining a fixed prior throughout the training process.

###  2.5 Aggregation Strategies

Although there are a significant number probabilistic aggregation approaches
that have been presented in the literature [19, 20, 21], we focus our study on
six select strategies from [3, 21, 22, 44, 45] due to their suitability for
implementation in the FL construct. In the following sub-sections, we discuss
each of these approaches using the univariate Gaussian distribution for each
parameter ($\theta_{i}$), parameterized by its mean ($\mu_{i}$) and variance
($\sigma_{i}^{2}$), as the model distribution. We use univariate Gaussians
because of our use of mean-field Gaussian VI, where we assume each parameter
is independent. The six approaches are Naive Weighted Averaging (NWA) [3],
Weighted Sum of Normal Distributions (WS), Linear Pooling (LP) [22],
Conflation/ Weighted Conflation (WC) [21, 44], and Distributed Weight
Consolidation (DWC) [23]. We use the following notation in our discussion: $K$
is the set of client models, $q_{k}$ is the distribution for client $k$,
$q_{g}$ is the global (server) distribution, and $\omega_{k}$ is the weight of
client $k$ with $\sum_{k=1}^{K}\omega_{k}=1$.

####  2.5.1 Naive Weighted Averaging

This aggregation method is similar to the FedAvg algorithm [3] and is simplest
of the aggregation techniques. In the case where
$\omega_{k}=\frac{|\mathcal{D}_{k}|}{|\mathcal{D}|}$, NWA is equivalent to
FedAvg. Although this method was shown to be effective for aggregating
deterministic model weights and biases in [3], independently aggregating the
mean and variance parameters in this manner ignores the statistical properties
of the Gaussian Distribution. This method is familiar to the statistical
literature, however, and is described as Centered Linear Pool (CLP) in [46],
and used extensively in [46, 47, 48]. Clements defends the use of this
variance calculation in [47] noting that the variance calculated using LP is
often inflated by the “disagreement” term (described in Section II.G.3), and
the NWA method is equal to the expected variance of a randomly selected
client.

$\mu_{q_{g}}=\sum_{k=1}^{K}\omega_{k}\mu_{q_{k}},\hskip
10.0pt\sigma_{q_{g}}^{2}=\sum_{k=1}^{K}\omega_{k}\sigma_{q_{k}}^{2}.$ (10)

Ultimately, this method involves taking the weighted average of each of the
clients’ individual model parameters to form the global model without regard
for what each of the parameters represents.

####  2.5.2 Weighted Sum of Normal Distributions

If the mean and variance of each model parameter are considered as part of a
univariate Gaussian distribution, instead of being considered individually as
in NWA, it is possible to take the weighted average of each of the client
Gaussian distributions. This strategy assumes the weights of each client are
mutually independent.

$\mu_{q_{g}}=\sum_{k=1}^{K}\omega_{k}\mu_{q_{k}},\hskip
10.0pt\sigma_{q_{g}}^{2}=\sum_{k=1}^{K}\omega_{k}^{2}\sigma_{q_{k}}^{2}.$ (11)

The resultant Gaussian distribution in WS has the same mean as in NWA, but the
variance parameter differs. Because the $\omega$ parameter falls in the range
of $[0,1]$, and we square each client’s weight parameter $\omega_{k}$ in WS,
the variance parameter in WS will always be less than or equal to the variance
parameter in NWA.

####  2.5.3 Linear Pooling

The LP technique is studied extensively in the forecasting literature [49, 46,
48, 50, 51, 52] and is connected to model averaging and multiple model
adaptive estimation (MMAE) [53], as described by Koliander et al. in [22]. To
derive the LP operator for univariate Gaussian random variables, we define
each client distribution as
$q_{k}{(\theta)}=\mathcal{N}(\theta;\mu_{q_{k}},\sigma_{q_{k}})$ where
$\mu_{q_{k}}=\mathbb{E}_{q_{k}}[\theta]$ and
$\sigma_{q_{k}}^{2}=\mathbb{E}_{q_{k}}[\theta-\mu_{q_{k}}]^{2}$. LP is then
defined as the linear combination of each of the client distributions:

$q_{g}{(\theta)}=\sum_{k=1}^{k}\omega_{k}\mathcal{N}(\theta;\mu_{q_{k}},\sigma_{q_{k}}).$
(12)

The global mean is then defined by $\mu_{q_{g}}=\mathbb{E}_{q_{g}}[\theta]$,
which is just the weighted average of the client means ($\mu_{q_{k}}$):

$\mu_{q_{g}}=\sum_{k=1}^{K}\omega_{k}\mu_{q_{k}}.$ (13)

The global variance can similarly be defined as
$\sigma_{q_{g}}^{2}=\mathbb{E}_{q_{g}}[\theta-\mu_{q_{g}}]^{2}$, which is
broken down by Lahiri et al. [54] into two components. The first component is
the weighted average variance of each client $\sigma_{q_{k}}^{2}$. The second
component is the “disagreement” component of the global variance and captures
the discrepancy between clients.

$\begin{split}\sigma_{q_{g}}^{2}=\underbrace{\sum_{k=1}^{K}\omega_{k}\sigma_{q_{k}}^{2}}_{\text{avg
client
var}}+\underbrace{\sum_{k=1}^{K}\omega_{k}(\mu_{q_{k}}-\mu_{q_{g}})^{2}}_{\text{disagreement
term}}\\\
=\sum_{k=1}^{K}\omega_{k}(\sigma_{k}^{2}+(\mu_{k}-\mu_{q_{g}})^{2}).\end{split}$
(14)

In [46], the authors show that LP imposes an upward bias on the global
variance resulting in under-confident predictions. We briefly mentioned one of
the ways in which researchers have dealt with this upward bias in the NWA
subsection. CLP, which is equivalent to NWA, is a variant of LP which ignores
the disagreement term in the LP variance calculation and reduces this upward
bias.

####  2.5.4 Conflation and Weighted Conflation

The Conflation and WC methods [21, 44] are the normalized product of client
distributions and are closely related to Log-Linear Pooling (LLP) [22]. LLP is
a weighted geometric average of client distributions and is so named because
it is a linear function of client distributions in the log-domain [22]. When
combining Gaussian distributions, LLP differs from WC only in the numerator of
the global variance term, with LLP replacing $\omega_{max}$ with a value of 1.
As shown in [21, 44], Conflation/WC has several desirable properties. These
properties include:

  * •

Conflation/WC is commutative, associative, and iterative.

  * •

The resulting distribution of the Conflation/WC of Gaussian distributions is a
Gaussian.

  * •

The result of the Conflation/WC of Gaussian distributions matches the result
of the weighted-least-squares method.

  * •

Conflation/WC minimizes the loss of Shannon information as a result of the
consolidation of multiple Gaussian distributions into a single Gaussian
distribution.

  * •

Conflation/WC is the best linear unbiased estimate.

  * •

Conflation/WC yields a maximum likelihood estimator.

We provide the formulas for both Conflation, Eq. (15), and WC, Eq. (16),
below.

$\displaystyle\mu_{q_{g}}=\frac{\sum_{k=1}^{K}\frac{\mu_{q_{k}}}{\sigma_{q_{k}}^{2}}}{\sum_{k=1}^{K}\frac{1}{\sigma_{q_{k}}^{2}}},\hskip
10.0pt\sigma_{q_{g}}^{2}=\frac{1}{\sum_{k=1}^{K}\frac{1}{\sigma_{q_{k}}^{2}}}.$
(15)

Readers familiar with the Inverse Variance Weighting (IV) technique [45] may
notice that the Conflation formula, Eq. (15), is identical to the formula for
IV. IV is a well-known method for combining random variables in such a way as
to minimize the variance of the resulting distribution.

$\displaystyle\mu_{q_{g}}=\frac{\sum_{k=1}^{K}\frac{\omega_{k}\mu_{q_{k}}}{\sigma_{q_{k}}^{2}}}{\sum_{k=1}^{K}\frac{\omega_{k}}{\sigma_{q_{k}}^{2}}},\hskip
10.0pt\sigma_{q_{g}}^{2}=\frac{\omega_{max}}{\sum_{k=1}^{K}\frac{\omega_{k}}{\sigma_{q_{k}}^{2}}}.$
(16)

Notably, WC produces a Gaussian distribution with a mean value which is
proportional to the client weights, inversely proportional to client
variances, and has a variance which is never greater than the variance of the
client with the greatest weight.

####  2.5.5 Distributed Weight Consolidation

DWC was introduced by McClure et al. [23] as a continual learning technique to
combine or consolidate the model weights and biases of multiple Bayesian
neural networks trained on independent datasets. This method builds on the
concept of Variational Continual Learning [55] and Bayesian Incremental
Learning [56]. However, both of these methods require that the underlying
models be trained sequentially. In DWC, multiple disparate models (clients)
can be trained independently, and their weights and biases can be consolidated
into a single network (global model), similar to the FL setting. DWC is
related to the Supra-Bayesian framework described in [22], which involves the
selection of a prior distribution, $p(\theta)$, and the update of the prior
using a Bayesian update rule. In DWC, we are using the global model,
$q_{\circ}(\theta)$, as the prior, $p(\theta)$, and applying the same Bayesian
update rule. In Eq. (17), we provide the formulas for the global mean and
variance, the derivation of which can be found in [23].

$\mu_{q_{g}}=\frac{\sum_{k=1}^{K}\frac{\mu_{q_{k}}}{\sigma_{q_{k}}^{2}}-\sum_{k=1}^{K-1}\frac{\mu_{q_{\circ}}}{\sigma_{q_{\circ}}^{2}}}{\sum_{k=1}^{K}\frac{1}{\sigma_{q_{k}}^{2}}-\sum_{k=1}^{K-1}\frac{1}{\sigma_{q_{\circ}}^{2}}},\hskip
10.0pt\sigma_{q_{g}}^{2}=\frac{1}{\sum_{k=1}^{K}\frac{1}{\sigma_{q_{k}}^{2}}-\sum_{k=1}^{K-1}\frac{1}{\sigma_{q_{\circ}}^{2}}}.$
(17)

where $\mu_{q_{\circ}}$ and $\sigma_{q_{\circ}}^{2}$ correspond to the global
mean and variance from the previous round, respectively. It should be noted
that DWC was designed to start with a base model trained on some initial
dataset, and not a random initialization. Additionally, in the DWC setting,
each “client” trains asynchronously on their local dataset before aggregating
client parameters. This is the first time that this method has been applied to
and analyzed in the BFL setting.

###  2.6 Determining Weight Parameters

In the seminal work on FL, McMahan et al. [3] weighted client updates based on
the size of each client’s local dataset relative to the datasets of all other
clients. Although this method is common in FL, there have been several works
[57, 22] which explore various techniques for weighting clients to achieve
varying objectives (robustness against outliers, reducing communications,
training efficiency, etc.). We treat the client weighting scheme as a
hyperparameter and analyze the results (with a focus on uncertainty) of using
four different techniques for selecting client weights. Notably, the
implementation of several of these methods require that the underlying models
be Bayesian. As such, we only apply each weighting technique to models for
which it is applicable.

####  2.6.1 Equal Weight

The first, and simplest, method is to give equal weight to each client in the
update process using Eq. (18). In addition to its simplicity, it does not
require any a priori knowledge of the client datasets. As a result, it is
completely independent of the amount of training data, or the quality of
training data, at each client site, and is not resistant to outliers or
extreme updates. This method is applicable to both deterministic and
probabilistic models.

$\omega_{k}=\frac{1}{K}.$ (18)

####  2.6.2 Local Train Dataset Size

This method was introduced in [3], and is widely used in the FL literature. In
order to apply this weighting scheme, the server either needs to have a priori
knowledge of the size of the client datasets, or the client needs to
communicate the size of their training dataset with the model update. When the
data is distributed evenly amongst clients, this method, Eq. (19), is
equivalent to Equal Weight. This method is applicable to both deterministic
and probabilistic models. Since each of our dataset partitions results in
clients having an equal amount of data, this method is equivalent to Equal
Weight.

$\omega_{k}=\frac{|\mathcal{D}_{k}|}{|\mathcal{D}|}.$ (19)

####  2.6.3 Maximum Discrepancy Weighting

Koliander et al. described maximum discrepancy weighting in [22]. This
technique involves treating the client weights as a measure of distance. In
Eq. (20), we use Kullback Leibler Divergence. However, other measures of
distance or divergence could be used. By computing client weights in this
manner, the server places higher importance on model weights and biases that
are not extreme (relative to the other clients). The gamma value
($\gamma_{k}$) and omega value ($\omega_{k}$) in Eq. (20) are the un-
normalized and normalized weights of each client, respectively. This method is
only applicable to probabilistic models. However, a similar technique could be
used in deterministic models using the L2 norm, or another measure of
distance. We only apply this technique to the probabilistic (VI) models.

$\gamma_{k}=\underset{j\in\\{1,...,K\\}}{\mathrm{max}}\frac{1}{\text{KL}[\,q_{k}\mid\mid
q_{j}]\,},\hskip
10.0pt\omega_{k}=\frac{\gamma_{k}}{\sum_{j=1}^{K}\gamma_{j}}.$ (20)

####  2.6.4 Distance to a Fixed Point

This method is similar to maximum discrepancy weighting in that it also treats
client weights as a measure of distance. Instead of calculating the
divergences between the distributions of the model weights and biases for
individual clients, we use the divergence between the distributions of the
model weights and biases for individual clients and the global model from the
previous round. In effect, this places a greater importance on clients that do
not diverge significantly from the current global model. Similar to maximum
discrepancy weighting, the gamma value ($\gamma_{k}$) and omega value
($\omega_{k}$) in Eq. (21) are the un-normalized and normalized weights of
each client, respectively. Again, this method is only applicable to
probabilistic models; however, a similar technique could be used in
deterministic models using the L2 norm, or another measure of distance. We
only apply this technique to the probabilistic (VI) models.

$\gamma_{k}=\frac{1}{\text{KL}[\,q_{g}\mid\mid q_{k}]\,},\hskip
10.0pt\omega_{k}=\frac{\gamma_{k}}{\sum_{j=1}^{K}\gamma_{j}}.$ (21)

##  3 Experiments

###  3.1 Datasets

We utilize the CIFAR-10 [58] image classification dataset to benchmark our
models and model parameter aggregation rules. The CIFAR-10 dataset consists of
50,000 training images and 10,000 test images of size 32x32x3 pixels, which
belong to one of 10 classes. Similar to [3, 10, 59], we adopt both an IID and
non-IID partitioning strategy. In both strategies, we partition the training
dataset amongst 10 clients. For the IID strategy, we assign each client a
uniform distribution over the 10 classes. For the non-IID strategy, we follow
the strategy introduced by Li et al. in [59] and adopt both a quantity-based
label imbalance and a distribution-based label imbalance. In the quantity-
based label imbalance scenario, we first sort the data by class and divide the
data into 20 partitions of size 2,500, and randomly assign each client two
partitions from different classes (i.e., each client training dataset contains
examples from two classes with 2,500 examples of each class). In the
distribution-based label imbalance scenario, each client is allocated a
proportion of the samples of each label according to the Dirichlet
distribution, which is parameterized by $\alpha$ [60]. In our experiment, we
use $\alpha$ parameters of 0.5 and 5.0 to simulate different levels of class
imbalance across client training datasets.

###  3.2 Architecture

We utilize the Flower Federated Learning Framework [24] to implement the FL
setup. In order to conduct a fair comparison of performance across model
configurations, we utilize the ResNet-20 [61] architecture in a deterministic,
MC dropout [33], and Flipout [31] (VI) configuration, as shown in Fig. 1.
Notably, when trained on a centralized training dataset, all variants of the
ResNet-20 model used in this paper achieved comparable performance to the
benchmark performance reported by He et al. [61] on the CIFAR-10 dataset, see
Table 1. The models were coded using Tensorflow v2.9 and Tensorflow
Probability v0.16 [62] and all experiments were performed on NVIDIA RTX 8000
graphics processing units.

Figure 1: ResNet Model Block Structure. (a) depicts the structure of a ResNet block in the deterministic configuration. (b) depicts the structure of the same ResNet block in the MC dropout configuration with the addition of a dropout layer after each convolutional and linear layer. These dropout layers are enabled during the inference phase. (c) depicts the ResNet block in the Flipout configuration. Each convolutional and linear layer is replaced by the Tensorflow Probability implementation of the corresponding Flipout Layer. Changes from the deterministic configuration are highlighted in orange. Table 1: ResNet-20 Centralized Training Results Model | Accuracy
---|---
Deterministic | 90.14
MC Dropout | 90.91
Flipout | 90.42

###  3.3 General Experiment Setup and Evaluation Metrics

For all experiments, we utilize $K=10$ clients, $B=32$ local minibatch size,
and $\eta=.001$ initial local learning rate with an exponential schedule that
reduces the learning rate by a factor of .04 every round after a silent period
of 50 aggregation rounds. Additionally, we use a dropout rate of 0.2 for MC
dropout models and our Flipout models use $p(\theta)=N(0,100I)$.

We present the results of each experiment using accuracy and negative log
likelihood (NLL). NLL is equivalent to the KL Divergence between a model’s
predicted distribution and a one-hot encoded true distribution [28], with a
lower NLL corresponding to a better representation. In terms of calibration,
Filos et al. [36] demonstrated that if a model’s performance increases as the
number of discarded high-uncertainty predictions increases, the model is well-
calibrated. As a result, we utilize accuracy vs. data retained curves to
assess the calibration of models based on normalized predictive entropy,
aleatoric uncertainty, and epistemic uncertainty.

###  3.4 Evaluation of Aggregation Methods - 1 Local Epoch

In this experiment, we evaluate each of the proposed aggregation strategies
with $E=1$ local epochs. Each simulation is run three times for 400 rounds,
and the results, shown in Table 2, are averaged across experiments.

In terms of accuracy and NLL, no aggregation strategy outperforms other
aggregation strategies on all data distributions. Notably, NWA and LP do not
perform as well as WS, WC, and conflation, which all perform similarly to the
deterministic model baseline (NWA). LP performs especially poorly on all data
distributions.

Uncertainty calibration plots are depicted in Fig. 2. When looking at the bulk
uncertainty metric (normalized entropy), in the IID and non-IID Dirichlet (0.5
and 5.0) cases, all of the global models display a calibrated uncertainty.
Even though the global models are calibrated, the models trained using WS, WC,
and conflation appear to be better calibrated than the models trained using LP
and NWA (i.e., test accuracy increases more rapidly as the most uncertain
samples are discarded from the test data). By breaking down the bulk
uncertainty into its epistemic and aleatoric components, the performance
difference is amplified. Specifically, when filtering based on epistemic
uncertainty, models trained using WS, WC, and conflation perform much better
than those trained using LP and NWA. In the non-IID 2-Class case, the
relationships between aggregation strategies is similar. However, not all the
global models have a calibrated epistemic uncertainty component.

In Fig. 3, we plot learning curve comparisons (test set accuracy vs federation
round) for each of the data distributions. Again, models trained using WS, WC,
and conflation exhibit similar behavior. The models trained using each of
these aggregation methods converge much faster than the LP and NWA strategies.

Table 2: Simulation Results - Site Epochs ($E$ = 1)

| IID | Non-IID 2-Class | Non-IID Dir ($\alpha=0.5$) | Non-IID Dir ($\alpha=5.0$)
---|---|---|---|---
Model | Agg. Fn. | Client Weighting | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL
Deterministic | NWA | Train Size | $87.66\pm 0.001$ | $0.370\pm 0.005$ | $51.45\pm 0.020$ | $1.55\pm 0.040$ | $83.45\pm 0.003$ | $0.485\pm 0.008$ | $85.78\pm 0.003$ | $0.435\pm 0.001$
MC Dropout | NWA | Train Size | $89.19\pm 0.001$ | $0.330\pm 0.006$ | $47.20\pm 0.020$ | $1.65\pm 0.020$ | $85.52\pm 0.002$ | $0.431\pm 0.004$ | $87.45\pm 0.002$ | $0.373\pm 0.005$
Flipout (VI) | NWA | Train Size | $81.67\pm 0.002$ | $0.576\pm 0.004$ | $43.04\pm 0.025$ | $1.68\pm 0.006$ | $74.77\pm 0.008$ | $0.721\pm 0.015$ | $79.89\pm 0.002$ | $0.62\pm 0.005$
Max Disc | $81.28\pm 0.001$ | $0.581\pm 0.002$ | $47.81\pm 0.024$ | $1.65\pm 0.013$ | $75.45\pm 0.009$ | $0.722\pm 0.005$ | $80.06\pm 0.003$ | $0.608\pm 0.012$
Distance | $81.76\pm 0.003$ | $0.576\pm 0.007$ | $40.00\pm 0.030$ | $1.80\pm 0.036$ | $75.92\pm 0.011$ | $0.717\pm 0.003$ | $80.04\pm 0.001$ | $0.614\pm 0.002$
WS | Train Size | $87.41\pm 0.002$ | $0.388\pm 0.006$ | $53.52\pm 0.056$ | $1.41\pm 0.076$ | $83.74\pm 0.002$ | $0.508\pm 0.014$ | $85.17\pm 0.004$ | $0.465\pm 0.01$
Max Disc | $87.81\pm 0.004$ | $0.377\pm 0.004$ | $47.72\pm 0.049$ | $1.49\pm 0.048$ | $84.17\pm 0.004$ | $0.487\pm 0.008$ | $84.90\pm 0.002$ | $0.476\pm 0.001$
Distance | $87.55\pm 0.003$ | $0.391\pm 0.004$ | $51.84\pm 0.014$ | $1.47\pm 0.021$ | $83.23\pm 0.003$ | $0.51\pm 0.005$ | $85.07\pm 0.002$ | $0.464\pm 0.005$
LP | Train Size | $77.21\pm 0.004$ | $0.683\pm 0.007$ | $35.50\pm 0.048$ | $1.85\pm 0.046$ | $67.67\pm 0.008$ | $0.925\pm 0.02$ | $74.7\pm 0.007$ | $0.744\pm 0.016$
Max Disc | $77.35\pm 0.004$ | $0.681\pm 0.012$ | $37.86\pm 0.018$ | $1.79\pm 0.014$ | $69.35\pm 0.009$ | $0.895\pm 0.017$ | $76.01\pm 0.011$ | $0.717\pm 0.023$
Distance | $77.69\pm 0.002$ | $0.674\pm 0.005$ | $32.48\pm 0.064$ | $1.90\pm 0.085$ | $68.47\pm 0.015$ | $0.914\pm 0.034$ | $75.94\pm 0.005$ | $0.721\pm 0.009$
WC | Train Size | $87.95\pm 0.001$ | $0.377\pm 0.006$ | $53.00\pm 0.052$ | $1.44\pm 0.096$ | $83.18\pm 0.004$ | $0.510\pm 0.009$ | $85.09\pm 0.003$ | $0.478\pm 0.001$
Max Disc | $87.81\pm 0.002$ | $0.384\pm 0.004$ | $50.71\pm 0.025$ | $1.49\pm 0.044$ | $83.70\pm 0.002$ | $0.514\pm 0.004$ | $84.86\pm 0.002$ | $0.470\pm 0.005$
| Distance | $87.52\pm 0.002$ | $0.389\pm 0.002$ | $46.41\pm 0.037$ | $1.52\pm 0.037$ | $83.17\pm 0.001$ | $0.514\pm 0.006$ | $85.23\pm 0.002$ | $0.464\pm 0.006$
| Conflation | N/A | $87.42\pm 0.001$ | $0.385\pm 0.003$ | $53.09\pm 0.008$ | $1.46\pm 0.027$ | $83.18\pm 0.001$ | $0.502\pm 0.008$ | $84.94\pm 0.005$ | $0.471\pm 0.008$

(a)

(b)

(c)

(d)

Figure 2: Uncertainty calibration plots for $E=1$. VI aggregation method
comparison plot of test set accuracy vs. ratio of data retained for IID and
non-IID 2-class data distributions. Non-IID Dirichlet distribution plots show
trends and relationships that mirror IID distribution results. Normalized
entropy and epistemic uncertainty plots are shown, aleatoric uncertainty plots
show trends and relationships that mirror normalized entropy.  Figure 3:
Learning curve plots for $E=1$. Plot of test set accuracy vs. federation round
for VI aggregation methods for each data distribution.

###  3.5 Evaluation of Aggregation Methods - 5 Local Epochs

Next, we increase the amount of client computation and evaluate each of the
proposed aggregation strategies with $E=5$ local epochs. Each simulation is
run three times for 200 rounds, and the results, shown in Table 3, are
averaged across experiments.

The results, depicted in Table 3, show much more parity between aggregation
strategies in terms of accuracy and NLL. In fact, models trained using the NWA
strategy perform as well as, or better, than the other aggregation strategies
across the board. When compared to the $E=1$ experiment results (Table 2),
models employing WS, WC, and conflation show minimal improvement when client
computation is increased. In some cases (WC/conflation - non-IID 2-class), the
models trained using $E=5$ actually perform worse. Notably, none of the VI
aggregation methods perform as well as the deterministic baseline on the IID
or non-IID Dirichlet data distributions. However, all of them significantly
improve on the deterministic baseline in the non-IID 2-class case.

Similar to the accuracy and NLL results, the calibration differences across
aggregation strategies are drastically reduced in the IID and non-IID
Dirichlet cases. However, as is the case in the $E=1$ experiments, NWA and LP
do not display a calibrated epistemic uncertainty component. The learning
curve comparison plots in Fig. 5 display the same relationships as their $E=1$
counterparts in Fig. 3.

Table 3: Simulation Results - Site Epochs ($E$ = 5)

| IID | Non-IID 2-Class | Non-IID Dir ($\alpha=0.5$) | Non-IID Dir ($\alpha=5.0$)
---|---|---|---|---
Model | Agg. Fn. | Client Weighting | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL
Deterministic | NWA | Train Size | $90.74\pm 0.003$ | $0.371\pm 0.009$ | $48.50\pm 0.040$ | $1.56\pm 0.060$ | $87.33\pm 0.002$ | $0.456\pm 0.008$ | $89.06\pm 0.002$ | $0.456\pm 0.008$
MC Dropout | NWA | Train Size | $91.16\pm 0.002$ | $0.264\pm 0.003$ | $44.90\pm 0.040$ | $1.58\pm 0.030$ | $87.91\pm 0.001$ | $0.357\pm 0.002$ | $89.53\pm 0.002$ | $0.316\pm 0.001$
Flipout (VI) | NWA | Train Size | $88.76\pm 0.002$ | $0.334\pm 0.005$ | $56.36\pm 0.017$ | $1.55\pm 0.021$ | $85.26\pm 0.001$ | $0.439\pm 0.001$ | $87.46\pm 0.001$ | $0.380\pm 0.004$
Max Disc | $88.87\pm 0.001$ | $0.336\pm 0.002$ | $55.91\pm 0.061$ | $1.55\pm 0.083$ | $85.43\pm 0.001$ | $0.435\pm 0.002$ | $87.30\pm 0.003$ | $0.380\pm 0.002$
Distance | $88.98\pm.003$ | $0.332\pm 0.004$ | $45.60\pm 0.129$ | $1.59\pm 0.098$ | $85.35\pm 0.003$ | $0.436\pm 0.008$ | $87.38\pm 0.001$ | $0.377\pm 0.001$
WS | Train Size | $88.37\pm 0.004$ | $0.446\pm 0.008$ | $56.96\pm 0.004$ | $1.36\pm 0.015$ | $84.53\pm 0.002$ | $0.585\pm 0.004$ | $85.51\pm 0.002$ | $0.545\pm 0.002$
Max Disc | $88.23\pm 0.005$ | $0.429\pm 0.002$ | $49.84\pm 0.036$ | $1.42\pm 0.012$ | $84.68\pm 0.004$ | $0.587\pm 0.005$ | $84.92\pm 0.009$ | $0.554\pm 0.006$
Distance | $88.34\pm 0.003$ | $0.444\pm 0.003$ | $48.49\pm 0.021$ | $1.42\pm 0.017$ | $84.91\pm 0.004$ | $0.581\pm 0.005$ | $85.75\pm 0.007$ | $0.545\pm 0.014$
LP | Train Size | $88.05\pm 0.002$ | $0.358\pm 0.002$ | $55.41\pm 0.023$ | $1.60\pm 0.040$ | $83.59\pm 0.001$ | $0.478\pm 0.006$ | $86.36\pm 0.001$ | $0.408\pm 0.003$
Max Disc | $88.19\pm 0.001$ | $0.356\pm 0.001$ | $48.87\pm 0.031$ | $1.65\pm 0.041$ | $84.00\pm 0.003$ | $0.472\pm 0.008$ | $86.62\pm 0.002$ | $0.404\pm 0.004$
Distance | $87.91\pm 0.001$ | $0.363\pm 0.004$ | $41.95\pm 0.050$ | $1.65\pm 0.049$ | $84.03\pm 0.001$ | $0.471\pm 0.004$ | $86.47\pm 0.001$ | $0.408\pm 0.004$
WC | Train Size | $88.21\pm 0.002$ | $0.443\pm 0.002$ | $52.23\pm 0.027$ | $1.41\pm 0.040$ | $84.49\pm 0.004$ | $0.583\pm 0.006$ | $86.01\pm 0.006$ | $0.542\pm 0.014$
Max Disc | $88.34\pm 0.002$ | $0.430\pm 0.007$ | $49.21\pm 0.051$ | $1.49\pm 0.059$ | $83.52\pm 0.002$ | $0.589\pm 0.012$ | $84.47\pm 0.014$ | $0.548\pm 0.012$
| Distance | $88.28\pm 0.005$ | $0.437\pm 0.012$ | $48.25\pm 0.026$ | $1.44\pm 0.043$ | $85.17\pm 0.005$ | $0.574\pm 0.009$ | $85.37\pm 0.007$ | $0.578\pm 0.006$
| Conflation | N/A | $88.16\pm 0.008$ | $0.444\pm 0.006$ | $52.47\pm 0.028$ | $1.39\pm 0.031$ | $84.86\pm 0.003$ | $0.586\pm 0.015$ | $85.59\pm 0.006$ | $0.550\pm 0.003$

(a)

(b)

(c)

(d)

Figure 4: Uncertainty calibration plots for $E=5$. VI aggregation method
comparison plot of test set accuracy vs. ratio of data retained for IID and
non-IID 2-class data distributions. Non-IID Dirichlet distribution plots show
trends and relationships that mirror IID distribution results. Normalized
entropy and epistemic uncertainty plots are shown, aleatoric uncertainty plots
show trends and relationships that mirror normalized entropy.  Figure 5:
Learning curve plots for $E=5$. Plot of test set accuracy vs. federation round
for VI aggregation methods for each data distribution.

###  3.6 Evaluation of Aggregation Methods - DWC and Model Pre-training

The results of the DWC experiments are presented separately due to significant
differences in the training setup required by the DWC aggregation strategy. In
Table 4, we present DWC results with and without global model pre-training.
For comparison, we employ the same pre-training strategy using the WS
aggregation method. Each of the DWC and model pre-training experiments is run
three times for a single federation round, where each client trains for 400
local epochs. Before distributing the global model to each client, the server
pre-trains the global model on 10% of the CIFAR-10 dataset for a maximum of
100 local epochs. To avoid overfitting in our global model pre-training, we
employ an early stopping strategy which monitors validation accuracy with a
patience of 10 epochs.

In Table 4, the first row holds the results for DWC without pre-training the
global model. In this case, DWC does not hold up as a viable aggregation
method. One of the potential explanations for the poor performance of DWC in
this setup, is the use of a randomly initialized model as the initial global
model. This is due to fact that the global model is heavily weighted in DWC.
As a result, we experimented with pre-training the global model on a small
subset of the overall training dataset. This setup more closely resembles the
experimental setup in [23]. Additionally, this is a likely scenario for a
network of distributed sensors where a model is pre-trained on a centralized
dataset and deployed on a group of sensors for collaborative learning.

In terms of accuracy and NLL, our pre-training setup does not perform
comparably to the $E=1$ or $E=5$ experiments in the IID and Non-IID Dirichlet
cases. On the other hand, in the non-IID 2-class case, the models that were
pre-trained perform exceedingly well. In addition to a significant improvement
in accuracy and NLL, the uncertainty calibration plots in Fig. 6 display well-
calibrated bulk (normalized entropy) and decomposed (aleatoric and epistemic)
uncertainty components.

Table 4: Simulation Results - DWC

| IID | Non-IID 2-Class | Non-IID Dir ($\alpha=0.5$) | Non-IID Dir ($\alpha=5.0$)
---|---|---|---|---
Model | Agg. Fn. | Pre-train | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL | Accuracy | NLL
Flipout | DWC | No | $34.95\pm 1.230$ | $1.73\pm 0.031$ | $11.64\pm 1.73$ | $2.69\pm 0.220$ | $17.25\pm 3.14$ | $2.440\pm 0.210$ | $31.83\pm 1.05$ | $1.801\pm 0.029$
Flipout | DWC | Yes | $71.42\pm 0.270$ | $0.83\pm 0.007$ | $66.35\pm 1.53$ | $0.97\pm 0.038$ | $68.81\pm 0.42$ | $0.918\pm 0.007$ | $65.87\pm 1.36$ | $0.972\pm 0.026$
Flipout | WS | Yes | $72.70\pm 0.008$ | $0.80\pm 0.017$ | $64.63\pm 1.02$ | $1.01\pm 0.051$ | $62.65\pm 0.02$ | $1.061\pm 0.061$ | $68.18\pm 0.02$ | $0.913\pm 0.051$

(a)

(b)

(c)

(d)

Figure 6: Uncertainty calibration plots for global model pre-training
experiments. DWC and WS aggregation method comparison plot of test set
accuracy vs. ratio of data retained for IID and non-IID 2-class data
distributions. Non-IID Dirichlet distribution plots show trends and
relationships that mirror IID distribution results. Normalized entropy and
epistemic uncertainty plots are shown, aleatoric uncertainty plots show trends
and relationships that mirror normalized entropy.

###  3.7 Updating the Prior Every Federation Round

In this experiment, we assess the effect of updating the prior after every
federation round. Each of these experiments are run twice on the IID data
partition, and the results are averaged across experiments. The results in
Table 5 show a precipitous drop in accuracy and NLL when compared to the
results in Tables 2 and 3, demonstrating that updating the prior every
federation round is not an effective strategy.

Table 5: Simulation Results - Prior Updated Every Federation Round ($E=1$)

| IID
---|---
Model | Agg. Fn. | Client Weighting | Accuracy | NLL
Flipout (VI) | NWA | Train Size | $41.84\pm.59$ | $1.55\pm.009$
WS | Train Size | $43.54\pm.54$ | $1.53\pm.017$
LP | Train Size | $46.14\pm.50$ | $1.46\pm.011$
WC | Train Size | $44.23\pm.52$ | $1.51\pm.002$
Conflation | N/A | $42.62\pm.58$ | $1.56\pm.007$

###  3.8 Client Weighting Strategies

Results of the $E=1$ and $E=5$ experiments with different client weighting
strategies are depicted in Tables 2 and 3. These experiments were conducted in
conjunction with the $E=1$ and $E=5$ experiments, and utilized the same setup.
Unfortunately, this experiment was inconclusive, and the results did not
elucidate the importance of the different client weighting techniques.

###  3.9 MC Dropout as a Lightweight Alternative to Mean-field Gaussian VI

The MC dropout experiments were conducted in conjunction with the $E=1$ and
$E=5$ experiments and the setup is the same. Accuracy and NLL results are
presented in Tables 2 and 3.

For both $E=1$ and $E=5$, in terms of accuracy/NLL, the MC dropout models
using NWA perform similarly to the deterministic baseline. In the IID and non-
IID Dirichlet cases, the MC dropout models outperform all the VI aggregation
methods. On the other hand, in the non-IID 2-class case, the MC dropout model
does not perform nearly as well as the top-performing VI aggregation methods.

In Fig. 7, we present uncertainty calibration comparison plots for $E=1$. Like
the previous uncertainty calibration plots, the non-IID Dirichlet plots depict
the same trends and relationships as the IID plots and are not shown. The
$E=5$ calibration plots also follow similar relationships and trends to the
$E=1$ plots and are not shown. In both cases, the models trained on IID and
non-IID Dirichlet data distributions appear to be well-calibrated. In
contrast, in the non-IID 2-class case, the MC dropout models show poor
calibration of the epistemic uncertainty.

Figures 8 and 9 depict the learning curve comparisons for $E=1$ and $E=5$.
Similar to the calibration plots, WS is used as a reference for comparison.
For $E=1$, the MC dropout models converge at a marginally slower rate. In the
non-IID 2-class case for $E=5$, shown in Fig. 9, the MC dropout model
converges at a significantly slower rate than the mean-field Gaussian VI model
trained using WS.

(a)

(b)

(c)

(d)

Figure 7: Uncertainty calibration plots for MC dropout experiments ($E=1$). MC
dropout (NWA) and VI (WS) aggregation method comparison plot of test set
accuracy vs. ratio of data retained for IID and non-IID 2-class data
distributions. Non-IID Dirichlet distribution plots show trends and
relationships that mirror IID distribution results. Normalized entropy and
epistemic uncertainty plots are shown, aleatoric uncertainty plots show trends
and relationships that mirror normalized entropy. Figure 8: Learning curve
plots for MC dropout experiments ($E=1$). Plot of test set accuracy vs.
federation round for MC dropout (NWA) and VI (WS) aggregation methods for each
data distribution. Figure 9: Learning curve plots for MC dropout experiments
($E=5$). Plot of test set accuracy vs. federation round for MC dropout (NWA)
and VI (WS) aggregation methods for each data distribution.

##  4 Discussion

In our evaluation of six different aggregation methods, and three client
weighting schemes, we demonstrated that the results vary based on the training
data distribution and number of local training epochs per federation round. We
showed that the selection of the aggregation method is a critical
hyperparameter, with downstream effects on model performance (accuracy),
calibration, and training time, which must be selected based on the machine
learning model type, dataset, and other hyperparameters. Notably, we
established that often these aggregation strategies can be placed into two
groups, with each group exhibiting similar performance (accuracy/NLL),
calibration, and convergence rate. The first group consists of WS, WC, and
conflation. The second group consists of NWA and LP. Models employing the
aggregation strategies in the first group tend to exhibit relatively high
accuracy, low NLL, and rapid convergence with a single local epoch of training
during each federation round. These models are also better calibrated than
their counterparts from the second group, which require five local epochs of
training per federation round to reach the same performance level. These
relationships may facilitate a targeted hyperparameter search. Specifically,
the similarities amongst strategies in each group may allow the testing of a
single strategy from each group to narrow the search.

Although our experiments on client weighting techniques were not fruitful, we
believe that this may be due to the CIFAR-10 dataset and the model we
selected. In FL implementations with more complex architectures and complex,
real-world, datasets, where distributions are highly non-IID, these weighting
techniques may help to smooth training and reduce the effects of a non-
representative client or an adversary attempting to disrupt the training
process. Further experimentation with larger and more complex models and
datasets is necessary to validate these beliefs.

The DWC algorithm required some significant adjustments to our experimental
setup; however, it drove us to experiment with pre-training. In many remote
sensing tasks, like visual target recognition or passive sonar target
classification [63], this setup is likely. Prior to deployment on a remote
sensing platform, the model needs to be trained and evaluated on a
representative dataset until it demonstrates an acceptable level of
performance. The random initialization is not realistic in these remote
sensing applications. Once the model is deployed on a group of sensors, it can
be fine-tuned using FL in a setup that mirrors our experimental setup. In this
application, DWC is a viable solution and displays comparable performance to
WS.

The cost of deploying VI models to remote sensing platforms and training those
models, in many cases, is prohibitive. MC dropout is a lightweight alternative
to mean-field Gaussian VI which, prior to this work, has not been directly
compared to VI in the FL environment. In our experiments, we showed that in
the IID and non-IID Dirichlet cases, MC dropout performs on par with one of
the top performing VI aggregation methods (WS). In fact, in terms of accuracy
and NLL, the MC dropout models outperform the mean-field Gaussian VI models.
Although the mean-field Gaussian VI models tended to converge faster, the MC
dropout models have half the number of parameters, which require fewer client
resources and lower communication requirements between client and server. The
non-IID 2-class data distribution case was the exception: the MC dropout
models converged slower, showed a reduction in accuracy, and displayed poor
calibration. In some cases, the use of an MC dropout model utilizing the NWA
aggregation strategy may be an acceptable lightweight alternative to mean-
field Gaussian VI models employing one of the aggregation strategies discussed
in this manuscript.

##  5 Conclusion and Future Work

In remote sensing and safety-critical applications, the ability of a model to
communicate a measure of epistemic uncertainty is essential. BDL models are
capable of communicating a measure of epistemic uncertainty while
simultaneously delivering competitive prediction accuracy when compared to
their deterministic counterparts. However, to fully realize the benefits of
BDL in the FL setting, it is important to understand the downstream effects of
the selected model aggregation method. In this manuscript, we used four
different partitions of the CIFAR-10 dataset and a fully variational ResNet-20
architecture to analyze six BFL model aggregation methods for mean-field
Gaussian VI models. This is the first time, to our knowledge, that a fully
variational architecture (i.e., all model layers are variational) has been
utilized in this type of analysis. Additionally, when trained on a centralized
dataset, each of the variants of our ResNet-20 architecture achieved
comparable performance to the benchmark performance reported by He et al.
[61], see Table 1. As such, we were able to compare our FL results to an
important ML benchmark and provide a more detailed look at uncertainty.
Ultimately, we identified the aggregation method as a critical hyperparameter
in the development of a BFL strategy, and we showed that, in certain
applications, MC dropout is an acceptable lightweight alternative to mean-
field Gaussian VI. In both cases (mean-field Gaussian VI and MC dropout), our
global models (Table 3, $E=5$) achieved test set accuracy within $\pm{2-3\%}$
of the centralized benchmark.

##  6 Disclaimer

The views and conclusions contained herein are those of the authors and should
not be interpreted as necessarily representing the official policies or
endorsements, either expressed or implied, of the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright annotations thereon.

## References

  * [1] Patricia Mabry,  “Making sense of the data explosion: The promise of systems science,”  American Journal of Preventive Medicine, vol. 40, pp. S159–61, May 2011.
  * [2] Paul Voigt and Axel von dem Bussche,  The EU General Data Protection Regulation (GDPR): A Practical Guide,  Springer Publishing Company, Incorporated, 1st edition, 2017.
  * [3] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas,  “Communication-efficient learning of deep networks from decentralized data,”  in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Apr. 2017, vol. 54 of Proceedings of Machine Learning Research, pp. 1273–1282, PMLR.
  * [4] Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon,  “Federated learning: Strategies for improving communication efficiency,”  arXiv preprint arXiv:1610.05492, 2016.
  * [5] Hangyu Zhu and Yaochu Jin,  “Multi-objective evolutionary federated learning,”  IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 4, pp. 1310–1322, 2019.
  * [6] Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek,  “Robust and communication-efficient federated learning from non-I.I.D. data,”  IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3400–3413, 2020.
  * [7] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith,  “Federated optimization in heterogeneous networks,”  in Proceedings of Machine Learning and Systems, 2020, vol. 2, pp. 429–450.
  * [8] Xiaohui Xu, Sijing Duan, Jinrui Zhang, Yunzhen Luo, and Deyu Zhang,  “Optimizing federated learning on device heterogeneity with a sampling strategy,”  in 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS), 2021, pp. 1–10.
  * [9] Yassine Laguel, Krishna Pillutla, Jerôme Malick, and Zaid Harchaoui,  “A superquantile approach to federated learning with heterogeneous devices,”  in 2021 55th Annual Conference on Information Sciences and Systems (CISS), 2021, pp. 1–6.
  * [10] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra,  “Federated learning with non-IID data,”  arXiv preprint arXiv:1806.00582, 2018.
  * [11] Canh T. Dinh, Nguyen Tran, and Josh Nguyen,  “Personalized federated learning with Moreau envelopes,”  in Advances in Neural Information Processing Systems, 2020, vol. 33, pp. 21394–21405.
  * [12] Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao,  “Personalized federated learning via variational Bayesian inference,”  in Proceedings of the 39th International Conference on Machine Learning. Jul. 2022, vol. 162 of Proceedings of Machine Learning Research, pp. 26293–26310, PMLR.
  * [13] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar,  “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,”  in Advances in Neural Information Processing Systems, 2020, vol. 33, pp. 3557–3568.
  * [14] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth,  “Practical secure aggregation for privacy-preserving machine learning,”  in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, CCS ’17.
  * [15] Robin C. Geyer, Tassilo Klein, and Moin Nabi,  “Differentially private federated learning: A client level perspective,”  arXiv preprint arXiv:1712.07557, 2017.
  * [16] Abhishek Bhowmick, John C. Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers,  “Protection against reconstruction and its applications in private federated learning,”  arXiv preprint arXiv:1812.00984, 2018.
  * [17] Liping Li, Wei Xu, Tianyi Chen, Georgios B. Giannakis, and Qing Ling,  “RSA: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets,”  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 1544–1551, Jul. 2019.
  * [18] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger,  “On calibration of modern neural networks,”  in Proceedings of the 34th International Conference on Machine Learning. Aug. 2017, vol. 70 of Proceedings of Machine Learning Research, pp. 1321–1330, PMLR.
  * [19] Zhansheng Duan, X. Rong Li, and U. D. Hanebeck,  “Multi-sensor distributed estimation fusion using minimum distance sum,”  in 17th International Conference on Information Fusion (FUSION), 2014, pp. 1–8.
  * [20] Christian Genest and James V. Zidek,  “Combining probability distributions: A critique and an annotated bibliography,”  Statistical Science, vol. 1, no. 1, pp. 114 – 135, 1986.
  * [21] Theodore Hill,  “Conflations of probability distributions,”  Transactions of the American Mathematical Society, vol. 363, Aug. 2008.
  * [22] Gunther Koliander, Yousef El-Laham, Petar Djuric, and Franz Hlawatsch,  “Fusion of probability density functions,”  Proceedings of the IEEE, vol. 110, pp. 404–453, 04 2022.
  * [23] Patrick McClure, Charles Y Zheng, Jakub Kaczmarzyk, John Rogers-Lee, Satra Ghosh, Dylan Nielson, Peter A Bandettini, and Francisco Pereira,  “Distributed weight consolidation: A brain segmentation case study,”  in Advances in Neural Information Processing Systems, 2018, vol. 31.
  * [24] Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D Lane,  “Flower: A friendly federated learning research framework,”  arXiv preprint arXiv:2007.14390, 2020.
  * [25] Yarin Gal,  Uncertainty in Deep Learning,  Ph.D. thesis, Dept. of Engineering, University of Cambridge, Cambridge, England, 2016.
  * [26] Alex Kendall and Yarin Gal,  “What uncertainties do we need in Bayesian deep learning for computer vision?,”  in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., 2017, vol. 30.
  * [27] David M Blei, Alp Kucukelbir, and Jon D McAuliffe,  “Variational inference: A review for statisticians,”  Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017.
  * [28] Oliver Durr, Beate Sick, and Elvis Murina,  Probabilistic Deep Learning With Python, Keras and TensorFlow Probability, chapter 8,  Manning Publications Co., 2020.
  * [29] Christopher M Bishop,  Pattern Recognition and Machine Learning,  Springer, New York, NY, USA, 2006.
  * [30] Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley,  “Stochastic variational inference,”  Journal of Machine Learning Research, vol. 14, no. 4, pp. 1303–1347, 2013.
  * [31] Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse,  “Flipout: Efficient pseudo-independent weight perturbations on mini-batches,”  in International Conference on Learning Representations, 2018.
  * [32] Durk P Kingma, Tim Salimans, and Max Welling,  “Variational dropout and the local reparameterization trick,”  in Advances in Neural Information Processing Systems, 2015, vol. 28.
  * [33] Yarin Gal and Zoubin Ghahramani,  “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,”  in Proceedings of The 33rd International Conference on Machine Learning. 20–22 Jun 2016, vol. 48 of Proceedings of Machine Learning Research, pp. 1050–1059, PMLR.
  * [34] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra,  “Weight uncertainty in neural networks,”  in Proceedings of the 32nd International Conference on Machine Learning. Jul. 2015, vol. 37 of Proceedings of Machine Learning Research, pp. 1613–1622, PMLR.
  * [35] Yarin Gal, Riashat Islam, and Zoubin Ghahramani,  “Deep Bayesian active learning with image data,”  in Proceedings of the 34th International Conference on Machine Learning. Aug. 2017, vol. 70 of Proceedings of Machine Learning Research, pp. 1183–1192, PMLR.
  * [36] Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal,  “A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks,” 2019.
  * [37] Laurence A. F. Park and Simeon Simoff,  “Using entropy as a measure of acceptance for multi-label classification,”  in Advances in Intelligent Data Analysis XIV. 2015, pp. 217–228, Springer International Publishing.
  * [38] Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft,  “Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning,”  in Proceedings of the 35th International Conference on Machine Learning. Jul. 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 1184–1193, PMLR.
  * [39] Yongchan Kwon, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik,  “Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation,”  Computational Statistics & Data Analysis, vol. 142, 2020.
  * [40] L. Liu, X. Jiang, F. Zheng, H. Chen, G. Qi, H. Huang, and L. Shao,  “A Bayesian federated learning framework with online Laplace approximation,”  IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 46, no. 01, pp. 1–16, Jan. 2024.
  * [41] Hong-You Chen and Wei-Lun Chao,  “FedBE: Making Bayesian model ensemble applicable to federated learning,”  in International Conference on Learning Representations, 2021.
  * [42] Adam Thor Thorgeirsson and Frank Gauterin,  “Probabilistic predictions with federated learning,”  Entropy, vol. 23, no. 1, Dec. 2020.
  * [43] Shrey Bhatt, Aishwarya Gupta, and Piyush Rai,  “Bayesian federated learning via predictive distribution distillation,” 2022.
  * [44] Theodore P. Hill and Jack Miller,  “How to combine independent data sets for the same quantity,”  Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 21, no. 3, Sep. 2011.
  * [45] William G. Cochran and Sarah Porter Carroll,  “A sampling investigation of the efficiency of weighting inversely as the estimated variance,”  Biometrics, vol. 9, no. 4, pp. 447–459, 1953.
  * [46] Malte Knüppel and Fabian Krüger,  “Forecast uncertainty, disagreement, and the linear pool,”  Journal of Applied Econometrics, vol. 37, no. 1, pp. 23–41, 2022.
  * [47] Michael P. Clements,  “Are macroeconomic density forecasts informative?,”  International Journal of Forecasting, vol. 34, no. 2, pp. 181–198, 2018.
  * [48] Victor Zarnowitz and Louis A. Lambros,  “Consensus and uncertainty in economic prediction,”  Journal of Political Economy, vol. 95, no. 3, pp. 591–621, 1987.
  * [49] M. Stone,  “The opinion pool,”  The Annals of Mathematical Statistics, vol. 32, no. 4, pp. 1339 – 1342, 1961.
  * [50] Michael P. Clements,  “Forecast uncertainty—ex ante and ex post: U.S. inflation and output growth,”  Journal of Business & Economic Statistics, vol. 32, no. 2, pp. 206–216, 2014.
  * [51] Kajal Lahiri and Xuguang Sheng,  “Measuring forecast uncertainty by disagreement: The missing link,”  Journal of Applied Econometrics, vol. 25, Feb. 2008.
  * [52] Christian Genest and Kevin McConway,  “Allocating the weights in the linear opinion pool,”  Journal of Forecasting, vol. 9, pp. 53–73, 1990.
  * [53] P.D. Hanlon and P.S. Maybeck,  “Multiple-model adaptive estimation using a residual correlation Kalman filter bank,”  IEEE Transactions on Aerospace and Electronic Systems, vol. 36, no. 2, pp. 393–406, 2000.
  * [54] Kajal Lahiri, Christie Teigland, and Mark Zaporowski,  “Interest rates and the subjective probability distribution of inflation forecasts,”  Journal of Money, Credit and Banking, vol. 20, no. 2, pp. 233–248, 1988.
  * [55] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner,  “Variational continual learning,”  in International Conference on Learning Representations, 2018.
  * [56] Max Kochurov, Timur Garipov, Dmitry Podoprikhin, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov,  “Bayesian incremental learning for deep neural networks,” 2018.
  * [57] Jonatan Reyes, Lisa Di-Jorio, Cécile Low-Kam, and Marta Kersten-Oertel,  “Precision-weighted federated learning,”  arXiv preprint arXiv:2107.09627, 2021.
  * [58] Alex Krizhevsky,  “Learning multiple layers of features from tiny images,”  M.s. thesis, University of Toronto, Toronto, Ontario, Canada, 2009.
  * [59] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He,  “Federated learning on non-IID data silos: An experimental study,”  in 2022 IEEE 38th International Conference on Data Engineering (ICDE), 2022, pp. 965–978.
  * [60] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown,  “Measuring the effects of non-identical data distribution for federated visual classification,”  arXiv preprint arXiv:1909.06335, 2019.
  * [61] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [62] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng,  “TensorFlow: Large-scale machine learning on heterogeneous distributed systems,” 2015.
  * [63] John Fischer, Marko Orescanin, Paul Leary, and Kevin B. Smith,  “Active Bayesian deep learning with vector sensor for passive sonar sensing of the ocean,”  IEEE Journal of Oceanic Engineering, vol. 48, no. 3, pp. 837–852, 2023.

  *[U.S.]: United States
  *[FL]: federated learning
  *[DL]: deep learning
  *[GDPR]: General Data Protection Regulation
  *[BFL]: Bayesian federated learning
  *[BDL]: Bayesian deep learning
  *[IID]: independent and identically distributed
  *[ML]: machine learning
  *[MC]: Monte Carlo
  *[VI]: variational inference
  *[SGD]: Stochastic Gradient Descent
  *[MCMC]: Markov Chain Monte Carlo
  *[KL]: Kullback-Leibler
  *[ELBO]: Evidence Lower Bound
  *[NWA]: Naive Weighted Averaging
  *[WS]: Weighted Sum of Normal Distributions
  *[LP]: Linear Pooling
  *[WC]: Weighted Conflation
  *[DWC]: Distributed Weight Consolidation
  *[CLP]: Centered Linear Pool
  *[LLP]: Log-Linear Pooling
  *[IV]: Inverse Variance Weighting
  *[NLL]: negative log likelihood