# Split personalities in Bayesian Neural Networks:
the case for full marginalisation

David Yallup
Kavli Institute for Cosmology
Cavendish Laboratory
University of Cambridge
<EMAIL_ADDRESS>
&Will Handley
Kavli Institute for Cosmology
Cavendish Laboratory
University of Cambridge
<EMAIL_ADDRESS>
&Mike Hobson
Astrophysics Group
Cavendish Laboratory
University of Cambridge
<EMAIL_ADDRESS>
&Anthony Lasenby
Astrophysics Group
Cavendish Laboratory
University of Cambridge
<EMAIL_ADDRESS>
&Pablo Lemos
Department of Physics and Astronomy
University of Sussex
<EMAIL_ADDRESS>

Corresponding Author

###### Abstract

The true posterior distribution of a Bayesian neural network is massively
multimodal. Whilst most of these modes are functionally equivalent, we
demonstrate that there remains a level of real multimodality that manifests in
even the simplest neural network setups. It is only by fully marginalising
over all posterior modes, using appropriate Bayesian sampling tools, that we
can capture the split personalities of the network. The ability of a network
trained in this manner to reason between multiple candidate solutions
dramatically improves the generalisability of the model, a feature we contend
is not consistently captured by alternative approaches to the training of
Bayesian neural networks. We provide a concise minimal example of this, which
can provide lessons and a future path forward for correctly utilising the
explainability and interpretability of Bayesian neural networks.

##  1 Introduction

The emergence of the deep learning paradigm to tackle high dimensional
inference problems has driven numerous striking advances in recent years. In
most practical settings this is approached as a high dimensional Neural
Network (NN) optimisation problem, with an extensive industry built up to
attempt to make this reliably solvable. Bayesian Neural Networks (BNNs) are a
parallel program to bring probabilistic reasoning over the same classes of
models, realised through the application of Bayes theorem to the training
procedure. A core attraction of performing Bayesian inference with NNs is that
it provides a probabilistic interpretation of the action of a network,
allowing clear expression of the increasingly important question: _How certain
are the predictions we make with ML?_ Vital issues such as quantifying the
potential for bias to occur in network predictions are expressed in a
principled Bayesian framework; imbalances in input training data are fed back
as reduced certainty in predictions for under-represented population members
in the training sample [1, 2].

Despite the attraction of a BNN framework, progress in this area has been
marred by a looming issue: performing Bayesian inference over the vast
parameter spaces a typical deep NN covers. The increased utility of a Bayesian
calculation suffers a trade-off in the dimensionality of parameter space that
the inference can be drawn over. When analytic approaches break down, as is
the case in any real-world scenario, approximations are needed to make the
Bayesian computations tractable [3]. Particular attention is generally paid to
Bayesian approximations that are best suited to scaling to the very high
dimensional space that maximum likelihood based approaches typically operate
in. In general this results in an approximation for the posterior network
probability distribution that holds under some potentially strict assumptions
[4]. In particular it is often assumed that the posterior over the free
parameters of a network is suitably captured by a unimodal distribution
(typically around the maximum a posteriori value). In this work we propose a
minimal problem that directly challenges this viewpoint by demonstrating that
even the simplest possible Neural Networks have a posterior distribution that
is truly multimodal.

In the example we provide in this work, we present the case for focusing on a
“compromise free” numerical marginalisation of the likelihood as a way to
guarantee that the training procedure accounts for, and accurately reflects,
the posterior multimodality. We perform this explicit marginalisation using
numerical sampling tools designed to directly calculate the full marginal
likelihood, or _Evidence_. A suitable sampling tool for such a calculation
should have the ability to handle integrals over likelihoods with potentially
multiple strongly peaked structures, which in turn affords formal guarantees
that the posterior distribution inferred from the samples accurately captures
these structures. Some preliminary work in marginalising over NNs in this
manner has taken place [5, 6], where in general a focus has been placed on the
role of the evidence for Bayesian Model comparison between networks. Whilst
this is an interesting avenue to build further on the work presented here, we
make the case that by closely examining the split personalities of networks
emerging from these calculations, a fundamental challenge to all probabilistic
NN descriptions is exposed.

In this work the case for, and an example of, explicit marginalisation over
NNs is presented. In Section 2 a brief overview of Bayesian Neural Networks is
given, particularly covering the challenges facing numerical Bayesian methods.
In Section 3 we present a minimal challenge that exposes the truly split
behaviour of a NN posterior in a clear fashion, as well as discussing the
implications of this observation. Lastly in Section 4 we provide some
concluding remarks, and higher-level motivation for future work.

##  2 Bayesian formulation of Neural Network training

This work focuses on a subset of the myriad of modern NN realisations: Fully
Connected Feedforward Neural Networks. These networks were some of the
earliest proposed forms of Artificial Neural Network, and in spite of the rise
of symmetry respecting transformation layers [7], these _Dense_ (synonymous
with fully connected) layers are still abundant in modern architectures.

A dense layer is composed of a linear mapping defined by a weight matrix
$\mathcal{W}$ taking a vector of $M$ inputs to a vector of $N$ outputs,
$\mathcal{W}:\mathbb{R}^{M}\rightarrow\mathbb{R}^{N}$. The elements of this
matrix, $w_{ij}$, connect every input element $i\in M$ to every output $j\in
N$, a constant _bias_ vector, $b_{j}$, is also usually added. If the output
nodes of a dense layer are internal (or hidden) a non-linear _activation_
function is applied to each node. Repeated action of $K$ (indexed in this work
as a superscript $k\in\\{1,\dots,K\\}$) parameterised linear maps with
activation builds up a deep NN, mapping an input data vector, $x_{i}$, to a
prediction vector, $y^{\prime}_{j}$. A neural network would be considered deep
if it has more than one hidden layer ($K\geq 3$), in this work we examine
_shallow_ networks ($K=2$).

In this study we focus on a simple binary classification network, predicting
class probabilities of a categorical distribution likelihood. The network is
then trained by comparing labelled data, $D=\\{x,y\\}$, to predictions,
$y^{\prime}$, made with a choice of the free parameters of the network,
$\theta=\\{w,b\\}$. The comparison is realised through the joint likelihood
for making a set of observations conditioned on a choice of $\theta$,
$\mathcal{L}\mathopen{}\left(D\,\rvert\,\theta\right)\mathclose{}$. The
Bayesian approach to training is then using Bayes theorem to invert the
conditional arguments of this likelihood,

$\mathcal{P}\mathopen{}\left(\theta\,\rvert\,D\right)\mathclose{}=\frac{\mathcal{L}\mathopen{}\left(D\,\rvert\,\theta\right)\mathclose{}\Pi(\theta)}{\mathcal{Z}(D)}\,,$
(1)

where in addition to the likelihood, the following distributions are
introduced: the prior over the network parameters $\Pi$, the posterior
distribution of the network parameters $\mathcal{P}$ and the evidence for the
network $\mathcal{Z}$. The evidence is the marginal likelihood, found by
marginalising over all the network parameters,

$\mathcal{Z}(D)=\int\mathcal{L}\mathopen{}\left(D\,\rvert\,\theta\right)\mathclose{}\Pi(\theta)d\theta\,.$
(2)

A trained network in this sense is the posterior distribution of the weights
and biases given an input set of data points. A sample from this posterior
distribution can then be used to predict a sample of predicted labels for a
new unseen set of data points, referred to as the model prediction through the
remainder of this article. The role of the Bayesian evidence in tasks such as
model selection is long established [8], and has been demonstrated as a tool
for tuning hyperparameters such as number of nodes in a hidden layer of a
network [5, 6]. What is less clearly established thus far in the literature
are the implications of obtaining a convergent evidence calculation in
relation to the resulting network posterior distribution.

###  2.1 Priors for Dense Neural Networks

Since the early days of NNs it has been understood that there are many
functionally equivalent, or _degenerate_ , choices of $\theta$ for any given
network architecture [9]. For any given dense layer outputting $N$ nodes,
arguments arising from combinatoric reordering of nodes and potential
compensating translations across activation functions, lead to $2^{N}N!$
degenerate choices of weight for every posterior mode [10]. This presents a
strict challenge for any Bayesian method. Either you sample the posterior and
find at least $2^{N}N!$ degenerate modes per layer (noting this is clearly
impossible due to the combinatorial growth of this factor and the typical
value of $N$ in modern NNs), or you don’t and hence have only explored some
undefined patch of the total parameter space. In order to isolate and examine
this issue in detail we employ a novel definition for the solution space a
dense NN occupies. We propose that a constrained form of weight prior can be
employed that retains the flexibility of a dense layer, without admitting
functionally equivalent solutions.

We initially restrict our analysis to mappings with $M=N$, _i.e._ only
admitting square weight matrices. A square weight matrix can be trivially
written as a sum of symmetric and skew symmetric pieces,

$\mathcal{W}=\frac{1}{2}(\mathcal{W}+\mathcal{W}^{T})+\frac{1}{2}(\mathcal{W}-\mathcal{W}^{T})=\mathcal{W}\mathrm{sym}+\mathcal{W}\mathrm{skew}\,.$
(3)

We can impose a condition that $\mathcal{W}\mathrm{sym}$ is positive definite.
This implicitly removes $N!$ permutations by requiring that the weight matrix
be pivot free. Further to this requiring positive definiteness imposes that
the scaling part of the linear map cannot flip sign, giving consistently
oriented scaling transformations. The remaining degrees of freedom in
$\mathcal{W}\mathrm{skew}$ can also be constrained, $\mathcal{W}\mathrm{skew}$
forms the Lie algebra of $SO(N)$, with the structure constant being the
totally anti-symmetric tensor $\epsilon$. By convention we can take the normal
ordering, $\epsilon_{1,\dots,N}=1$, which in practice can be enforced by
appropriate sign choices in $\mathcal{W}\mathrm{skew}$.

For simplicity in this work we use a uniform prior over weights and biases in
the range $(-5,5)$, rejecting any samples where $\mathcal{W}\mathrm{sym}$ is
not positive definite, and $\mathcal{W}\mathrm{skew}$ is not consistently
oriented. It would perhaps be more familiar to choose a Gaussian prior, or
equivalently L2 regularisation, to promote sparsity. This is less important as
we are not operating in the over-parameterized regime with the examples we
study, although implications of non flat priors are briefly discussed
alongside our results. Starting from any typical symmetric prior and rejecting
to a particular subset of mappings is useful for visualising this proposal on
the concise problem in this work, however these constraints can be more
efficiently encoded as a prior that has an appropriate support. The $N\times
N$ matrix can be effectively split into a lower triangle (including diagonal),
$L$, and upper triangle, $U$. $\mathcal{W}\mathrm{sym}$ can then be generated
as $\mathcal{W}\mathrm{sym}=L\cdot L^{T}$ with a prior over $L$ to reflect the
desired sparsity promotion (typically employed in the form of regularisation
terms for optimized NN approaches). The upper triangle provides the skew piece
as, $\mathcal{W}\mathrm{skew}=U-U^{T}$, with appropriate signed priors over
$U$ to respect a choice of structure constant for $SO(N)$.

The generalisation of this to cases where $M\neq N$ is not trivial,
particularly to construct in the rejection sampling format used in this work.
A potential construction of a prior with this support could come by
considering the reverse of a singular value decomposition. It is typical for
such a decomposition of $\mathcal{W}$ to implicitly perform the required
pivoting to return a sorted and positive set of singular values, as well as
orthogonal rotations in the $M$ and $N$ planes. To remove spurious degeneracy
that emerges due to the flexibility of these rotations a similar construction
to the one motivated for appropriate $\mathcal{W}\mathrm{skew}$ in square
matrices is likely necessary. Additionally we note that the symmetry
respecting equivariant layers being increasingly deployed in modern NN
applications are naturally better suited than dense layers to the
marginalisation we attempt in this work [11]. It is quite natural to
incorporate such layers into the picture we build here.

###  2.2 Tools for Neural Network marginalisation

In Section 2.1 we considered the potentially troublesome sampling problem a
dense NN layer poses. In order to sample these spaces we can consider which
numerical Bayesian tools will be sufficient for adequately exploring the
parameter space.

Limited exploration is acceptable if every mode is functionally equivalent,
however if there is genuine multimodality to the solution, there is no
guarantee that the sampled posterior will be representative of the true
posterior unless these degeneracies can be resolved before sampling. Bayesian
methods that arise from an optimized point estimate (Variational Inference
being a common example of this [12, 13, 14]), by construction cannot reflect
any split behaviour. Sampling methods such as Hamiltonian Monte Carlo do give
formal guarantees to sample a representation of the full density [15, 16],
however in practice it is well known that multimodal targets are a common
failure mode for Markov Chain approaches. Ensemble approaches (either of
Markov chains, or optimized point estimate solutions) are a potential method
to extend the previously mentioned tools to work with multimodal targets.
Whilst there is promise in these approaches, in general they lack formal
probabilistic guarantees to obtain the true posterior [17]. In practice then,
it is often considered sufficient to either implicitly (or explicitly) assume
the posterior distribution is unimodal.

A correctly weighted ensemble of unimodal methods would adequately define the
correct posterior, however the challenge is in how to construct this
weighting. In this work we contend that an effective approach can be found by
employing Nested Sampling [18] (NS). NS is a sampling algorithm uniquely
designed to target numerical estimation of the evidence integral. It is often
bracketed with other Markov Chain sampling algorithms, but its ability to
navigate strongly peaked multimodal distributions with little to no tuning
makes it uniquely able to perform a full marginalisation of a Neural Network,
in a manner necessary for this work. As a full numerical integration
algorithm, the dimensionality of space that the sampling can cover is
typically limited to $\mathcal{O}\left(1000\right)$ parameters. Whilst this is
much smaller than desired for full networks, probabilistic techniques with
similar dimensionality limitations, such as Gaussian Processes, have been
successfully embedded in scalable frameworks [19, 20].

In this work we use the PolyChord implementation of NS [21]. The resolution of
the algorithm is defined by the number of live points used and is set to
$n_{\text{live}}=1000$, the number of initial prior samples is also boosted to
ensure exploration of the space and set to
$n_{\text{prior}}=n_{\text{live}}\times 10$. All other hyperparameters are set
to the defaults. The chain files are analysed using the anesthetic package
[22], which is used to create the corner plots. The NNs were built on the flax
[23] NN library within the jax ecosystem [24].

##  3 A concise example of multimodality

In this section we propose a minimal classification problem, that can be fit
with a minimal NN. We choose to examine a modified noisy version of an XOR
logic gate problem. Samples forming the training set are drawn from cardinal
signal points $(x_{1},x_{2})=\\{(1,1),(-1,-1),(1,-1),(-1,1)\\}$, with diagonal
points being assigned a class $y=1$ and anti-diagonal points classified as
$y=0$. The samples have two sources of noise added to remove any actual
symmetry from the problem (with the intention of isolating degenerate
symmetries in the weight space): samples are drawn unevenly from each cardinal
point with $\\{30,100,10,80\\}$ draws respectively, and to each sampled point
Gaussian noise with a mean of 0 and a variance of 0.5 is added. Arbitrary test
datasets can be made either by repeating the sampling with different cardinal
populations or alternative noise seeds. A minimal neural network is chosen to
fit this composed of a single hidden layer with $N=2$ nodes, passed through
sigmoid activations with an added bias vector. Such a set up is typically
identified as a logistic regression problem. This introduces a network with 9
free parameters, illustrated in Figure 1a. This network is trained with a
standard binary cross entropy loss function111 The binary cross entropy being
the log of a categorical likelihood with two classes. Both the proposed
problem and trained network solution are shown in Figure 1b.

(a) The minimal Neural Network used in this example.

(b) The target training data, and trained model on the noisy XOR problem.

Figure 1: The chosen network architecture (Figure 1a) and target function to
fit in this challenge (Figure 1b). The left panel of Figure 1b shows the
initial cardinal signal points, with noisy uneven sampled data along side
this. The right panel of the same Figure retains the same noisy training data
overlaid on a prediction from the trained NN across the whole parameter space.

The network predictions shown in Figure 1b are built by taking the mean
prediction of the binary classifier $y^{\prime}$ over 1000 posterior samples
of $\theta$. These samples are drawn from the posterior calculated as the
result of using Nested Sampling to fully marginalise the network likelihood,
as detailed in Section 2.2. In the model prediction figures throughout this
work we display the mean of the vector of predicted $y^{\prime}$ over a
regular grid of input data $(x_{1},x_{2})$ with spacing 0.1 covering the full
set of training points. We can further interrogate the structure of these
solutions by examining how the posterior samples of network parameters,
$\theta$, are distributed. The corner plots of these posteriors are shown in
Figure 2. For simplicity only the parameters of the weight matrix connecting
the inputs to the hidden nodes are shown. This figure displays posterior
samples for networks trained with two different choices of prior, with an
important additional note that both choices of prior give functionally
equivalent model predictions (_e.g._ the right panel in Figure 1b). The Nested
Sampling algorithm realised in PolyChord keeps track of the number of
posterior modes it finds during its evolution. We define a prominent node as
being any mode that has a local evidence estimate greater than a factor of
$10^{-3}$ smaller than the mode with the highest local evidence. This
definition implies that we only select modes that contribute on average at
least 1 sample to the 1000 posterior samples used to infer the figures in this
work. By this definition the full prior finds 14 distinct prominent modes,
whereas the proposed prior constraint yields only 2 prominent posterior modes.

(a)

(b)

Figure 2: Posterior samples from two choices of prior for a dense layer, where
the parameters shown are only the weight matrix parameters connecting the
input data to the hidden nodes. A Uniform distribution from (-5,5) forms the
base weight prior of both figures, with Figure 2b rejecting samples not
following the recipe given in Section 2.1. The distinct modes in Figure 2b
have been separated and coloured differently, the sum of these densities gives
a functionally equivalent network to the red posterior in Figure 2a.

The proposed form of prior for dense layers greatly simplified the composition
of the target function, from being a weighted superposition of 14 modes to
only 2, with no apparent change in the quality of the predictions. This can be
further investigated by now examining the individual modes found against the
full solution already presented, this is shown in Figure 3. The rows of this
figure display three comparison diagnostics, with the columns replicating the
diagnostics for each mode as well as the full solution. The top row of panels
display the average log likelihood, $\langle\log\mathcal{L}\rangle$, across
the training and test dataset. A test dataset has been formed for this purpose
by taking ten repeated samples of the same cardinal point proportions as in
the training set, but with a different Gaussian noise seed for each set. The
violin density estimate is again derived from an ensemble of predictions made
using 1000 posterior samples (from the individual modes and the full posterior
respectively), with the error bars in black showing the mean
$\langle\log\mathcal{L}\rangle$ from these posterior set and extending to the
most extreme values in the ensemble. Overlaid across all three panels are the
mean values from the full solution. The maximum a posteriori mode (containing
as well the maximum likelihood solution in this set), is Mode 1. Whilst this
mode gives the best performing (highest average
$\langle\log\mathcal{L}\rangle$) solution on the training set it poorly
generalises to the test set, under-performing the full solution on average.
Mode 2 gives a more robust and generalisable solution, outperforming the full
solution on the test set on average however it less successfully captured the
training set. It is worth noting that this kind of “local minima” in the loss
landscape is deliberately suppressed in nearly all NN training schemes. The
full solution, as an evidence weighted superposition of the two modes combines
the best features of both. The second row of panels display the log evidence
of each solution, $\log\mathcal{Z}$. The log evidence of each mode effectively
defines the full solution, with the ratio of evidences giving the ratio of
contribution of each mode to the full solution. The bottom row of panels show
the predictions of each solution across the input data space, using the same
scale and format as the corresponding panel in Figure 1b.

Figure 3: Comparison diagnostics for the two posterior modes found when using
the constrained prior, as well as the corresponding full solution. The
$\langle\log\mathcal{L}\rangle$ and $\log\mathcal{Z}$ diagnostics for each
solution are colour coded corresponding to the colours used in Figure 2, with
the Model Prediction diagnostic corresponding to colouring and scale used in
Figure 1b.

###  3.1 Discussion

The noisy XOR gate problem demonstrated in this work provides a clear
illustration of multimodal behaviour in NN posteriors, present in even the
most minimal network. This poses challenges both for future work using the
principled marginalisation approach we proposed in this study, and more
generally for robust inference with any NN. There are however some marked
differences between the stylised challenge presented here and the generally
accepted standard practices for NN inference, here we consider how some of
these standard practices would impact the patterns demonstrated.

A notable feature that only emerges from the full solution as demonstrated in
Figure 3, is the complexity of the model achieved. It is not possible to
represent the decision boundary that the full multimodal solution constructs
with any single point estimate network weights given the fixed finite network
size. Correctly posterior weighting two candidate models gives an inferred
decision boundary that would only be achievable with a point estimate solution
for a higher dimensional network. It is not uncommon for people to consider
Bayesian inference as simply giving an uncertainty on a prediction, but in
this case it is doing something grander, allowing construction of higher
dimension solutions by a mixture of models. A larger model with standard
regularization terms (or equivalently non-flat priors) could be proposed as a
way of constructing a more unimodal solution. Reshaping the prior will
influence the evidence calculation, and hence the ratio of local evidences
between modes, but there is little motivation to think that this will
consistently decide in favour of a particular solution. Furthermore we contend
that increasing the dimensionality of the model will likely make the potential
for competitive models at scale worse.

Reading further into the diagnostic plot, the maximum likelihood solution
(Mode 1) is demonstrated to have worse generalisability than the full
solution. By allowing a weighted sum of two distinct models to contribute to
our full solution, a network that is more robust to noise is constructed.
Whilst this in and of itself is a useful trait for any network, there is a
secondary fact about principled Bayesian inference on display here, robust
performance in low data/high noise environments. Taking the typical argument
that Neural Networks are data hungry, this challenge could be refuted by
requiring a much larger set of data samples for training, implicitly averaging
over more samples of data noise at training time. Whilst this would resolve
much of the challenge in this case, this is an ad-hoc requirement for
applicability of NNs and furthermore an inadequate defence if we want to make
sensitive statements in the massive model and data limits employed in cutting
edge large model NN inference [25].

Lastly, a potential refutation of this challenge is to invoke some kind of
ensemble based method as a vehicle to capture a set of candidate alternative
models. The correct weighting of the two proposed solutions in our fully
marginalised approach is the result of a convergent calculation of the exact
volume under the likelihood surface over the prior. Any method that does not
guarantee that the ratio of densities is correct will have a vanishingly small
chance of getting this ratio correct, the rate of attraction of a gradient
boosted path (whether that be via an optimizer or an HMC chain) will not
necessarily be proportional to the posterior mass of a mode.

##  4 Conclusion

In this work we presented a concise example of multimodal posteriors in Neural
Networks and the case for capturing them with numerical marginalisation. By
reducing the Neural Network to its most minimal form, and choosing a minimal
simple learning problem, a paradox for inference using Neural Networks was
posed. In this paradox, a learning scheme that focuses on network parameters
that maximise the likelihood on the training data is at odds with the aim of
generalisability and explainability of the network. It is only by both
abandoning point estimate optimisation based approaches, and adopting a
sampling scheme that can handle multimodal posterior distributions, that the
most robust solution on this problem can be obtained. Two particularly
compelling features of the solution a full numerical marginalisation can
uniquely obtain were highlighted; the ability to reason between multiple
candidate models and reason even when there is limited data with sizeable
noise. Both of these are compelling features that show characteristics of what
could be called intelligence.

The current prevailing wind of Machine Learning research is following the
direction of increasingly large models and large datasets. There are clear
advantages, and some incredibly compelling results, emerging from this work at
large scale. The orthogonal viewpoint we presented in this work offers
features that can complement the striking successes of modern Machine Learning
at scale. A more principled understanding, offered by a marginal likelihood
based approach, of some parameters in a larger model could be a path to bring
together principled probabilistic models with inference at massive scale.

## Acknowledgments and Disclosure of Funding

DY, WH and ANL were supported by STFC grant ST/T001054/1. DY & WH were also
supported by a Royal Society University Research Fellowship and enhancement
award.

PolyChord is licensed to PolyChord Ltd., PolyChordLite is free for academic
use.

This work was performed using resources provided by the Cambridge Service for
Data Driven Discovery (CSD3) operated by the University of Cambridge Research
Computing Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using
Tier-2 funding from the Engineering and Physical Sciences Research Council
(capital grant EP/T022159/1), and DiRAC funding from the Science and
Technology Facilities Council (www.dirac.ac.uk).

## References

  * [1] Nado, Z., N. Band, M. Collier, et al.  Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning.  _arXiv e-prints_ , arXiv:2106.04015, 2021.
  * [2] Maddox, W. J., P. Izmailov, T. Garipov, et al.  A simple baseline for bayesian uncertainty in deep learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * [3] Foong, A., D. Burt, Y. Li, et al.  On the expressiveness of approximate inference in bayesian neural networks.  _Advances in Neural Information Processing Systems_ , 33:15897–15908, 2020.
  * [4] Wenzel, F., K. Roth, B. S. Veeling, et al.  How Good is the Bayes Posterior in Deep Neural Networks Really?  _arXiv e-prints_ , arXiv:2002.02405, 2020.
  * [5] Higson, E., W. Handley, M. Hobson, et al.  Bayesian sparse reconstruction: a brute-force approach to astronomical imaging and machine learning.  _Monthly Notices of the Royal Astronomical Society_ , 483(4):4828–4846, 2018.
  * [6] Javid, K., W. Handley, M. Hobson, et al.  Compromise-free bayesian neural networks, 2020.
  * [7] Bronstein, M. M., J. Bruna, T. Cohen, et al.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  _CoRR_ , abs/2104.13478, 2021.
  * [8] MacKay, D. J. C.  _Information Theory, Inference & Learning Algorithms_.  Cambridge University Press, USA, 2002.
  * [9] Kůrková, V., P. C. Kainen.  Functionally Equivalent Feedforward Neural Networks.  _Neural Computation_ , 6(3):543–558, 1994.
  * [10] Bishop, C. M.  _Pattern Recognition and Machine Learning (Information Science and Statistics)_.  Springer-Verlag, Berlin, Heidelberg, 2006.
  * [11] Lim, L.-H., B. J. Nelson.  What is an equivariant neural network?, 2022.
  * [12] Tomczak, M., S. Swaroop, R. Turner.  Efficient low rank gaussian variational inference for neural networks.  _Advances in Neural Information Processing Systems_ , 33:4610–4622, 2020.
  * [13] Rudner, T. G., Z. Chen, Y. W. Teh, et al.  Tractable function-space variational inference in bayesian neural networks.
  * [14] Wu, A., S. Nowozin, E. Meeds, et al.  Deterministic Variational Inference for Robust Bayesian Neural Networks.  _arXiv e-prints_ , arXiv:1810.03958, 2018.
  * [15] Cobb, A. D., A. G. Baydin, I. Kiskin, et al.  Semi-separable hamiltonian monte carlo for inference in bayesian neural networks.  In _Advances in Neural Information Processing Systems Workshop on Bayesian Deep Learning_. 2019.
  * [16] Cobb, A. D., B. Jalaian.  Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting.  In _Uncertainty in Artificial Intelligence_ , pages 675–685. PMLR, 2021.
  * [17] Wilson, A. G.  The Case for Bayesian Deep Learning.  _arXiv e-prints_ , arXiv:2001.10995, 2020.
  * [18] Skilling, J.  Nested sampling for general Bayesian computation.  _Bayesian Analysis_ , 1(4):833–859, 2006.
  * [19] Wilson, A. G., Z. Hu, R. Salakhutdinov, et al.  Deep kernel learning.  _CoRR_ , abs/1511.02222, 2015.
  * [20] Lee, J., Y. Bahri, R. Novak, et al.  Deep Neural Networks as Gaussian Processes.  _arXiv e-prints_ , arXiv:1711.00165, 2017.
  * [21] Handley, W. J., M. P. Hobson, A. N. Lasenby.  polychord: next-generation nested sampling.  _Mon. Not. Roy. Astron. Soc._ , 453(4):4385–4399, 2015.
  * [22] Handley, W.  anesthetic: nested sampling visualisation.  _The Journal of Open Source Software_ , 4(37):1414, 2019.
  * [23] Heek, J., A. Levskaya, A. Oliver, et al.  _Flax: A neural network library and ecosystem for JAX_ , 2020\.
  * [24] Bradbury, J., R. Frostig, P. Hawkins, et al.  _JAX: composable transformations of Python+NumPy programs_ , 2018.
  * [25] Balestriero, R., J. Pesenti, Y. LeCun.  Learning in high dimension always amounts to extrapolation.  _CoRR_ , abs/2110.09485, 2021.


# Comparison of Static Analysis Architecture Recovery Tools
for Microservice Applications

Simon Schneider  Hamburg University of TechnologyHamburg, Germany ,  Alexander
Bakhtin  University of OuluOulu, Finland ,  Xiaozhou Li  University of
OuluOulu, Finland ,  Jacopo Soldani  University of PisaPisa, Italy ,  Antonio
Brogi  University of PisaPisa, Italy ,  Tomas Cerny  University of
ArizonaTucson, USA ,  Riccardo Scandariato  Hamburg University of
TechnologyHamburg, Germany and  Davide Taibi  University of OuluOulu, Finland
Tampere UniversityTampere, Finland

(2024)

###### Abstract.

Architecture recovery tools help software engineers obtain an overview of
their software systems during all phases of the software development
lifecycle. This is especially important for microservice applications because
their distributed nature makes it more challenging to oversee the
architecture. Various tools and techniques for this task are presented in
academic and grey literature sources. Practitioners and researchers can
benefit from a comprehensive overview of these tools and their abilities.
However, no such overview exists that is based on executing the identified
tools and assessing their outputs regarding effectiveness. With the study
described in this paper, we plan to first identify static analysis
architecture recovery tools for microservice applications via a multi-vocal
literature review, and then execute them on a common dataset and compare the
measured effectiveness in architecture recovery. We will focus on static
approaches because they are also suitable for integration into fast-paced
CI/CD pipelines.

microservices, static analysis, architecture recovery

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Mining Software Repositories; 2024;
Lisbon††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Static analysis tools can support developers with valuable feedback on their
work without the need to run and test their systems. Tools such as SonarQube
111sonarsource.com/products/sonarqube/, PMD 222pmd.github.io/, or IntelliJ
333jetbrains.com/de-de/idea/ are examples of widely popular solutions that
perform real-time analyses for different aspects of development. Static
analysis architecture recovery tools can support developers by showing the
implemented system’s high-level architectural design, helping them adhere to
the intended design and avoid architectural issues such as violations of
security rules (Bambhore Tukaram et al., 2022), bad smells (Ponce et al.,
2022), antipatterns (Taibi et al., 2020), or non-conformances (Cao et al.,
2024). Providing accessibility of the systems’ architecture is additionally
important for distributed systems, where it is challenging for developers to
oversee the complete application.

The microservice architecture is an architectural style that often entails a
distributed codebase. It has been increasingly adopted in the last years and
continues to gain popularity in software development. Applications employing
the microservice architecture split their business logic into multiple
microservices. The individual microservices communicate over lightweight
communication channels (Dragoni et al., 2016; Lewis and Fowler, 2014). The
architecture has many benefits for software engineering activities. However,
the distributed nature of the codebase also poses challenges, since it is more
difficult to gain and maintain an overview of the application’s architectural
design (Di Francesco et al., 2019; Soldani et al., 2018).

Architecture recovery tools and techniques can support developers and analysts
in this regard by creating a representation of the implemented system. In the
field of program comprehension, it has been shown that such representations
foster better and easier analysis, maintainability, and usability, and support
software engineers during development (e.g., (Arisholm et al., 2006; Budgen et
al., 2011; Gravino et al., 2010, 2015; Schneider et al., 2024)). By allowing
to assess the adherence of the implemented architecture to the designed one,
issues such as architectural drift are also mitigated.

With the growing adoption of the microservice architecture, the need for
static analysis tools that specialize in microservice applications rises.
Consequently, various approaches for architecture recovery for microservices
have been proposed in academic literature (Alshuqayran et al., 2018;
Granchelli et al., 2017; Kleehaus et al., 2018; Quéval and Zdun, 2023; Soldani
et al., 2021). Some authors also provide tools to show the feasibility of
their presented approaches. Further tools can be found in the grey literature.

In this paper, we present a study that we plan to conduct to identify and
compare static analysis tools for architecture recovery of microservice
applications. A core contribution is the execution of all identified tools on
a common dataset and the comparison of their effectiveness in performing
architecture recovery. We measure effectiveness in terms of precision and
recall of extracted application characteristics, compared to a manually
created ground truth.

In the context of this work, we refer to a microservice application’s
architecture as a representation of its architectural design in terms of
components and their logical connections. The microservice architecture offers
such a system decomposition by definition since the individual microservices
are meant to be self-standing, independently deployable units. The
communication links between them that are necessary to fulfill the system’s
business logic form the connections between components.

The results we expect to obtain from our study can be beneficial to both
researchers and practitioners. For researchers, an overview of existing tools
can prevent the creation of yet another approach for which a similar technique
has already been proposed. Consequently, the results of the study can shed
light on the directions for future work and help accelerate research on the
topic. For practitioners, the results of the planned study can provide a
reference for the tools and for comparing their actual capabilities. Industry
adoption of tools and techniques presented in academic literature is
notoriously challenging. Our study is geared towards fostering visibility of
tools for microservice architecture recovery and showing their effectiveness
in basic architecture recovery (i.e., the extraction of components and
connections) as well as additional characteristics.

##  2\. Related Work

Several literature reviews have been presented that address domains similar to
those of our planned study. For example, Dragoni et al. (Dragoni et al., 2016)
and Soldani et al. (Soldani et al., 2018) conducted literature reviews for the
microservice architecture in general. Others present reviews of approaches for
a wide variety of specific use cases. For example, Abdelfattah et al.
(Abdelfattah and Cerny, 2023) present a review of approaches for
reconstruction, reasoning, and evolution, and Bushong et al. (Bushong et al.,
2021a) a review of reconstruction, architectural degradation, and technical
debt, among others. Neri et al. (Neri et al., 2020) and Ponce et al. (Ponce et
al., 2022) conducted literature reviews with more narrowly defined scopes,
both focusing on smells that are specific to microservice applications.
Fritzsch et al. (Fritzsch et al., 2019) presented a review of microservice
refactoring approaches for migration from monolithic to microservice
architecture. Gortney et al. (Gortney et al., 2022) presented a review of
approaches for microservice reconstruction. The authors focused on dynamic
analysis approaches, which are based on, e.g., log analysis, tracing
technologies, or monitoring technologies. Finally, Cerny et al. (Cerny et al.,
2022) also presented a review of microservice architecture reconstruction
approaches without restricting the search to dynamic approaches. However,
different from our planned study, these publications all present literature
reviews of techniques instead of tools, meaning, that they do not consider
whether the found approaches are supported by implementations.

Some work has also been published on reviews of tools. Although not focussed
on microservice applications, Emanuelsson and Nilsson (Emanuelsson and
Nilsson, 2008), Lenarduzzi et al. (Lenarduzzi et al., 2023), and Mantere and
Uusitalo (Mantere et al., 2009) compared static analysis tools for different
use cases. Bakhtin et al. (Bakhtin et al., 2022) performed a systematic grey
literature review of tools detecting microservice API patterns, and Giamatti
et al. (Giamattei et al., 2024) presented the results of a systematic grey
literature review of monitoring and DevOps tools for microservice
applications, thus focusing on the same domain we intend to target with our
planned study.

The planned study is based on and extends the work presented by Bakhtin et al.
(Bakhtin et al., 2023). The authors conducted a systematic mapping study based
on the academic literature to identify tools for the architecture recovery of
microservice applications. The review is based on information reported by the
corresponding publications, and not on insights gained from executing the
tools, which we plan to do instead. Reviewing tools based on their reported
evaluations is insufficient for a comprehensive comparison because they are
not applied to the same dataset and further, they are often not evaluated
thoroughly, as noted by Schneider and Scandariato (Schneider and Scandariato,
2023) as well as by Akkaya and Ovatman (Akkaya and Ovatman, 2022). Instead,
evaluations are often based on executing the corresponding tool on less than
five applications.

Notably, Akkaya and Ovatman (Akkaya and Ovatman, 2022) performed a similar
study to the one we propose. They identified three tools for microservice
extraction in the literature and executed them on four applications. However,
they only considered the detection of microservices and neglected all other
characteristics. Consequently, they do not review a holistic architecture
recovery process, as we intend to do.

In conclusion, to the best of our knowledge, no review of static analysis
microservice architecture recovery tools has been presented in the literature
that provides a comprehensive comparison of the tools. With the planned study,
we aim at addressing this gap in the literature and provide a comprehensive
comparison of such tools that can be found in academic as well as grey
literature. The comparison shall be based on the tools’ observed effectiveness
in architecture recovery on a shared dataset.

##  3\. Objectives and Research Questions

In this work, we aim to provide researchers and practitioners with an overview
of the state-of-the-art, freely available, static analysis architecture
recovery tools for microservice applications. Insights on the quality of such
tools, as well as on the specific characteristics they extract, are valuable
for both groups of stakeholders. With the comparison of the tools’
effectiveness in architecture recovery instead of an overview of their
approaches, our study fills a gap in the literature that has not been
addressed before. Especially in academia, where published tools are often
prototypes created for the sake of showing the feasibility of a presented
approach and where subsequent maintenance is often neglected, such an
evaluation is crucial for properly judging the tools’ qualities.

In pursuing to fulfill the above objectives, we will answer the following
research questions:

  * •

RQ1: Which freely available, static analysis tools for architecture recovery
of microservice applications exist?
Several architecture recovery approaches have been proposed in the literature,
which are often supported by prototypes implementing the techniques.
Additionally, grey literature sources give pointers to further tools that fit
this scope. We will curate a list of such tools in a multivocal literature
review, which will then be evaluated and compared on a common benchmark.

  * •

RQ2: Which characteristics do the tools extract in addition to the basic
architecture (i.e., components and connections between them)?
All architecture recovery tools extract the analyzed application’s basic
architecture (which, in the context of our work, are the application
_characteristics_ services and connections between them). Many tools extract
additional application characteristics, i.e., properties beyond the basic
architecture, such as information about implemented security mechanisms, links
to design requirements, or trust boundaries. An overview of these additional
characteristics helps to identify tools for specific use cases.

  * •

RQ3: Which are the most commonly considered characteristics extracted by the
tools?
Based on the presentation of the characteristics extracted by the identified
tools, we will analyze the tools’ overlaps and differences in their extraction
scopes. The results will show what the tools mostly focus on and also where
possible gaps lie.

  * •

RQ4: Which is the identified tools’ effectiveness in architecture recovery?
To compare the identified tools based on their effectiveness in architecture
recovery, we will execute them on a common benchmark and measure their
precision and recall concerning the following properties:

    * –

RQ4.1: Which is the identified tools’ effectiveness in detecting the
components that form a microservice application?
The individual microservices of an application constitute the building blocks
that the application’s microservice architecture consists of. We follow the
distinction of Schneider et al. (Schneider and Scandariato, 2023), which says
that internal microservices realize the application’s business logic and are
implemented specifically for an application. Infrastructural microservices
instead use mainly existing code libraries that are adjusted to the
application’s requirements (e.g., API gateways, authorization servers, or
message brokers). The components are often the easiest characteristics to
extract for microservice applications since deployment technologies such as
Docker Compose or Kubernetes ease their detection. Nevertheless, this is the
foundational step of architecture recovery and will thus be evaluated.

    * –

RQ4.2: Which is the identified tools’ effectiveness in detecting connections
between the components forming a microservice application?
The second group of application characteristics in the core architecture of
microservice applications is that of logical connections between the
components over which requests are made and data is exchanged. Such
connections can be realized in different ways, for example via direct API
calls, asynchronous communication techniques, or implicit invocations by
infrastructural components such as the communication for registering services
in a service registry. Due to this added complexity, the detection of
connections is harder than that of the components and will be evaluated
separately.

    * –

RQ4.3: Which is the tools’ effectiveness in detecting the additionally
extracted characteristics?
For those tools that extract characteristics in addition to the basic
architecture (see RQ2), we will measure their effectiveness in extracting this
extra information. We will directly compare tools that extract the same
additional characteristics.

##  4\. Methodology

Figure 1. Methodology of the planned study.

The planned study consists of broadly two stages, (i) performing a multi-vocal
literature review to identify static analysis architecture recovery tools for
microservice applications, and (ii) comparing the identified tools’
effectiveness in architecture recovery by executing them on a common dataset
and evaluating the outputs they produce. Figure 1 shows the complete
methodology of the planned study structured into five steps. Each step is
further described in the following sections, in the order indicated by Figure
1.

###  4.1. Identification of Tools

We will repeat the literature review of academic sources performed by Bakhtin
et al. (Bakhtin et al., 2023) to identify tools published after the authors
performed their search. Further, we will apply our in- and exclusion criteria
to the tools already identified by the authors to select those relevant to us.
Since the scope of tools considered in the planned study is a subset of the
ones identified by Bakhtin et al. (Bakhtin et al., 2023), and since the
methodology we apply is adapted from theirs, we can assume that no relevant
tools will be missed in this process. Additionally to academic sources, we
will adopt the methodology and apply it to grey literature sources as well,
thus extending the work into a multivocal literature review (Garousi et al.,
2019) (see step 1 in Figure 1).

For the repetition of the literature review of Bahtin et al.(Bakhtin et al.,
2023), we will use the same search string (given below) and search for it in
the same four scientific databases (Scopus,444The Scopus database:
https://www.scopus.com., IEEEXplore555The IEEEXplore database:
https://ieeexplore.ieee.org/., ACM Digital Library,666The ACM Digital Library:
https://dl.acm.org. and Web of Science777Web of Science:
https://www.webofscience.com/wos/woscc/basic-search). We will only consider
results published after their reported search date.

(Microservice* OR Micro-service* OR "micro-service*")
AND Architect*
AND (Reconstr* OR Mining OR Reverse engineering
OR Recover* OR Extract* OR Discover*)
AND (Tool* OR Prototype OR Implementation OR GitHub OR
Proof of concept OR POC OR Proof-of-concept)

The fourth AND-group of the search string is used for in-text search if the
database functionality allows it.

For grey literature sources, we will apply the search string to the four
popular grey literature websites Google Search888Google Search:
https://google.com, Twitter (X)999Twitter (X): https://x.com,
Reddit101010Reddit: https://reddit.com, and Medium111111Medium:
https://medium.com. These are popular websites and are used as sources of grey
literature by other reviews, such as (Moreschini et al., 2023; Peltonen et
al., 2021). Duplicates will be removed during result aggregation.

After compiling the list of initial sources, we will apply the following
inclusion and exclusion criteria, adapted from (Bakhtin et al., 2023):

  * •

Inclusion criteria

    * –

Mentions a tool for microservice architecture recovery

    * –

A reference to the freely available tool is made or the tool can be found by
searching for its name

    * –

The tool follows a static analysis or hybrid approach

  * •

Exclusion criteria

    * –

Source is not in English

    * –

Out of topic - relevant terms are used in a different context

    * –

Source describes different aspects of microservice recovery (not dealing with
tools)

    * –

Source describes closely related tasks such as monolith to microservice
migration

For the academic sources, we will apply the criteria in two phases, as it is
common practice when conducting SLRs. In the first phase, sources are excluded
based on reading the title and abstract of the paper; in the second phase, the
complete paper is examined. For the grey literature sources, we will apply the
criteria on the complete sources directly while also assessing the sources’
quality regarding the producer’s authority, the applied methodology,
objectivity of the source, the date of publishing, and novelty, as per
(Garousi et al., 2019). If sources are deemed to be of insufficient quality,
they will be excluded independent of their adherence to the inclusion
criteria.

All steps will be performed by two authors independently, and disagreements
solved via discussion with a third author. How well authors agree with each
other will also be analyzed with Cohen’s kappa coefficient (Emam, 1999). The
kappa coefficient considers the observed frequency of agreement relative to
the expected probability of agreement, assuming authors decide randomly and
independently.

Finally, we will perform forward and backward snowballing (Wohlin, 2014) on
the academic sources, i.e., reviewing all references and citations that occur
in relevant places of the paper in the same way described above. For the grey
literature sources, we will instead check whether contained links to other
resources refer to tools that are in scope.

We will examine the identified sources in detail to identify all presented and
mentioned tools for microservice architecture recovery. Specifically, we will
look for any references to source code repositories, web applications, Docker
images, or other ways of providing a tool. In line with the objective of the
study, only tools that follow a static analysis approach or hybrid approach
where the static part can be run independently are considered. The
identification will be performed by the first author. In cases where no tools
will be found in a source, another author will check the source as well for
confirmation. The resulting list of static analysis microservice architecture
recovery tools will serve as the answer to RQ1.

###  4.2. Tool Characterization

For all identified tools, we will further extract from the sources and from
the resource where the tool is provided (e.g., the source code repository)
their general properties (platform, language, static/hybrid approach, output
format, etc.), as was done by Bakhtin et al. (Bakhtin et al., 2023) (step 2 in
Figure 1). Here, those characteristics extracted by the tools that go beyond
the basic architecture of the analyzed systems are of particular interest. The
results will be presented in a matrix listing all identified tools as well as
the characteristics they extract. This matrix will later determine which tools
are compared with each other based on each characteristic. To this end, we
expect to generalize some characteristics to allow a comparison between tools.
For example, if one tool extracts information about implemented authorization
mechanisms while others extract information about other security features,
these characteristics will all be generalized into a common group of _security
mechanisms_. In addition to guiding the comparison of tools’ extraction
effectiveness, the created matrix will also be the basis to answer RQ2 and
RQ3.

###  4.3. Extension of Dataset

To compare the identified tools’ correction in architecture recovery under
controlled circumstances, they need to be executed on the same dataset. We
will use a dataset of 17 dataflow diagrams (DFDs) of open-source microservice
applications (Schneider et al., 2023) for this purpose. The DFDs depict the
corresponding applications’ architecture as well as additional properties.
Their nodes represent the application’s components, i.e., (internal and
infrastructural) microservices, databases, and external entities; their edges
represent connections between any two components. As such, the nodes and edges
are used as ground truth for the basic architecture (i.e., RQ4.1 and RQ4.2
will be answered based on the tools’ effectiveness in extracting these
characteristics). The applications corresponding to the DFDs in the dataset
are typical, small- to medium-sized open-source microservice applications
written in Java with a focus on the Spring framework. On average, the DFDs
contain 11 nodes and 22 edges. According to reports, Java is the most popular
language for developing microservice applications (JetBrains, 2022) and Spring
is the most used framework for Java microservice applications (JRebel, 2022).
The applications in the dataset were selected from sources in the literature
as well as popular repositories on GitHub. According to the authors,
established design patterns for microservice applications using the Java
Spring framework are prevalent in the applications.

Concerning the additional characteristics extracted by the tools (the basis
for RQ4.3), the DFDs in the dataset contain extensive annotations that
represent security mechanisms, deployment information, and other system
properties (on average, 84 annotations per DFD). We expect that this
information will be in line with some of the tools’ extracted characteristics
but that it will not be sufficient to serve as ground truth for all identified
tools. Likely, some of them extract characteristics not currently contained in
the DFDs. For these cases, we will manually extend the dataset with the
required information (step 3 in Figure 1). The methodology’s details for how
to detect the additional characteristics in the code depend on what exactly is
to be extracted. In general, at least four authors will take part in manually
extracting the characteristics and cross-validating their results. For this,
the process will be performed by multiple authors independently. Then,
possible discrepancies between the authors’ results will be solved by
discussion with another author. The extraction will be based on a manual
analysis of the source code of the applications. For this step, we will define
code artifacts that indicate the existence of the characteristics for each of
them. These indicators will guide the manual extraction. They will be
formulated based on the tools’ descriptions and the corresponding publications
(for academic sources).

A further extension of the dataset will be needed in case tools are identified
that analyze applications not written in Java. In this case, we will manually
create DFDs for applications in the required language. For this, we will
follow the same process for creating the DFDs in the dataset described by
Schneider et al. (Schneider et al., 2023). However, from our experience and
the feasibility study (see Section 5), most tools focus on Java. This step
might thus not be required.

As a result of the described process, an extended dataset will be an
additional contribution of the planned work. It will contain extensive
information on different characteristics that are useful to benchmark a
variety of different architecture recovery approaches.

###  4.4. Execution of Identified Tools

To obtain outputs from the identified tools for their evaluation, we will run
them on the applications in the extended dataset. Naturally, the tools will
need to be executed successfully to create outputs. It is possible that there
will be obstacles in terms of reproducibility, i.e., that it will not be
trivial to execute some of the tools. To achieve a fair comparison, a
methodology for executing them was established (step 4 in Figure 1) that
ensures that the effort invested into attempting to run each tool will be
comparable. We will first attempt to run a tool based on available
instructions (documentation, information in the source, etc.), possibly on a
virtual machine if specific requirements for the execution environment are
mentioned. If this will not be successful, we will analyze the code for
indicators of how to run the tool (code comments, hints by identifiers, etc.).
If all steps fail, we will abandon the tool and exclude it from the
comparison. As an indicator of successful execution, we will check whether the
tool produces any output for any model item it is supposed to extract and
which is present in the analyzed application.

###  4.5. Comparing Tools’ Effectiveness

We will use precision, recall, and execution time as quantitative measures for
comparison. These are common and objective metrics used for such evaluations.
Although we do not dictate a specific use case for the tools, their ability to
perform their core functionality correctly is the most important basis for
evaluation. A tool should extract all existing characteristics it is supposed
to detect and not falsely produce results for more than these. Precision and
recall serve as measures to indicate these two aspects. The execution time is
more dependent on the intended use case but is important for most scenarios as
well. Lightweight static analysis tools that show quick execution times lend
themselves to being integrated into automated pipelines such as fast-paced
CI/CD pipelines. For example, the output of architecture recovery tools could
be used by model-based analysis tools in a deployment pipeline.

To quantify the tools’ output, we will manually count (step 5 in Figure 1), by
comparing to the ground truth, the number of correctly extracted
characteristics (true positives, TP), the number of falsely extracted
characteristics (false positives, FP), and the number of undetected
characteristics (false negatives, FN). Since the tools have different output
formats, quantifying the results manually is deemed the safest method for a
correct representation. The process will be performed by two authors
independently and disagreements solved in discussion with a third author.
Precision and recall will be calculated from these measures with the following
formulas:

$\text{Precision}=\frac{\text{Correct characteristics}}{\text{Correct
characteristics}+\text{False characteristics}}$
$\text{Recall}=\frac{\text{Correct characteristics}}{\text{Correct
characteristics}+\text{Undetected characteristics}}$

We will answer RQ4 based on the above measures. Specifically, we will compare
different subsets of the complete list of tools against each other. The
overview of each tool’s extracted characteristics (see Section 4.2) will
determine which tools are compared with each other. For each characteristic,
we will compare the effectiveness achieved by all tools that are supposed to
extract it in extracting this characteristic. All tools will be compared on
the characteristics _components_ and _connections_ since these form the basic
architecture.

Table 1. Static analysis architecture recovery tools identified in the feasibility study. Pub. = Publication; Upd. = last update Name | GitHub Repository | Pub. | Upd.
---|---|---|---
Attack Graph Generator | tum-i4/attack-graph-generator | (Ibrahim et al., 2019) | 01/2021
Code2DFD | tuhh-softsec/code2DFD | (Schneider and Scandariato, 2023) | 02/2024
MicroDepGraph | clowee/MicroDepGraph | (Rahman et al., 2019) | 11/2021
microMiner | di-unipi-socc/microMiner | (Muntoni et al., 2021) | 11/2020
microTOM | di-unipi-socc/microTOM | (Soldani et al., 2023) | 01/2023
Prophet | cloudhubs/prophet | (Bushong et al., 2021b) | 08/2023
RAD | cloudhubs/rad-analysis | (Das et al., 2021) | 01/2021

##  5\. Feasibility Study

To ensure the feasibility and insightfulness of the planned study, we
conducted a small preliminary study. First, we selected all static analysis
microservice architecture recovery tools from the list of tools presented by
Bakhtin et al. (Bakhtin et al., 2023). Table 1 shows the seven tools that fit
the inclusion criteria of the planned study. Secondly, we attempted to run
these tools following the presented methodology (see Section 4.4). We were
successful in executing five of them within minutes. The other two require a
deeper analysis of the code (as specified in the methodology). This step was
omitted for now. Without measuring the tools’ effectiveness, this small
preliminary study ensures the availability of data for comparison. Note that
the _Attack Graph Generator_ tool only performs architecture recovery as a
means to achieve the generation of attack graphs. However, it produces an
architectural representation as an intermediate result.

##  6\. Potential Risks

Some unforeseeable factors could influence the planned study. They will have
to be addressed when and if they manifest. We present here factors that we
identified and how they could be addressed:

#### Inability to Execute Tools

By design, the planned study relies on the successful execution of all
identified tools. It is possible that we will not be able to do so. Some tools
could show a lack of reproducibility that does not allow us to execute them
successfully. This is a realistic risk, especially for academic tools, which
are often created as prototypes to prove the feasibility of presented
approaches and not further maintained afterward. As a consequence, we
performed the feasibility study (see Section 5) to verify that at least some
tools are executable and will be part of the comparison.

#### Tools’ Extraction Scopes too Distinct for Comparison

The overview of existing tools, the characteristics they extract, and their
effectiveness in doing so is of high value on its own. However, a core
contribution of the planned study lies in comparing the tools’ effectiveness.
If the tools are too distinct in their extraction scopes, we will present
their effectiveness individually for the additional characteristics and will
focus on comparing the results for nodes and edges, characteristics that all
tools should extract by definition.

#### Characteristics not existent in Dataset

We may find tools that extract characteristics that are not contained in the
initial dataset and that are too profound to be addressed by an extension of
the dataset. As described in Section 4.3, we will extend the used dataset to
serve as ground truth for all identified tools. However, the additional
characteristics may not occur in the applications in the dataset. In such
cases, we will assess the feasibility of extending the dataset with additional
applications that show the missing characteristic.

##  7\. Threats to Validity

Some threats to validity will apply to the planned study’s results (Wohlin et
al., 2012).

#### Internal Validity

The identification of tools via a multivocal literature review entails the
authors’ subjective judgment when deciding whether to include sources or not
and when examining the selected sources to identify the tools. To address this
possible bias, the methodology adheres to established standards in performing
literature reviews, e.g., by including multiple authors in the decision
process. Still, we might not be successful in identifying all tools in our
target scope, for example, because the designed methodology might not cover
all relevant sources of information. However, the search is done on a broad
body of sources and is extensive enough to reasonably claim to provide a
comprehensive overview of all relevant sources. The measurement of the tools’
effectiveness will rely on manual work, both in the creation of the ground
truth and in the analysis of the outputs. As for the identification of tools,
this carries the risk of biases, which we try to mitigate by including
multiple authors. At this point of the study, not all details of this step can
be predicted. Consequently, this threat to validity will be discussed further
after conducting the study.

#### External validity

We will compare the identified tools based on their measured effectiveness in
architecture extraction. This measurement could show different results in
other setups. The used dataset contains mainly models of showcase applications
of small to medium size, which are rather homogeneous concerning their
architectures and the used technologies. Thus, the identified tools could
perform differently on another dataset and other conclusions could be drawn
concerning their comparison. However, the dataset is the largest one currently
available in the literature for this purpose. Future work could include the
creation of a dataset containing more industry-near applications, or a
replication of the work if such a dataset is published by others. The human
factor in executing the tools could also affect the external validity. We plan
to mitigate this factor with the designed methodology, where extensive and
equal effort will be put into attempting to run each tool.

##  8\. Conclusion

This paper presents a study that we plan to conduct to identify and compare
static analysis architecture recovery tools for microservice applications. To
identify existing tools, we will replicate an existing study (Bakhtin et al.,
2023), re-run it to identify new tools that appeared since its publication,
and extend it into a multi-vocal literature review. The comparison will be
based on executing all identified tools on a common dataset. Such an overview
of tools and their effectiveness in architecture recovery has not been
presented before, to the best of our knowledge. The results can have
implications for researchers and practitioners alike by presenting and making
accessible the state-of-the-art and its capabilities, as well as by
identifying gaps and thereby future research directions.

## Acknowledgements

This work is based on work supported by a grant from the Research Council of
Finland (grants n. 349487 and 349488 - MuFAno) and by the National Science
Foundation under Grant No. 2245287.

## References

  * (1)
  * Abdelfattah and Cerny (2023) Amr S. Abdelfattah and Tomas Cerny. 2023.  Roadmap to Reasoning in Microservice Systems: A Rapid Review.  _Applied Sciences_ 13, 3 (2023).
  * Akkaya and Ovatman (2022) Kerem Akkaya and Tolga Ovatman. 2022.  A Comparative Study of Meta-Data-Based Microservice Extraction Tools.  _IJSSMET_ (2022).
  * Alshuqayran et al. (2018) Nuha Alshuqayran, Nour Ali, and Roger Evans. 2018.  Towards Micro Service Architecture Recovery: An Empirical Study. In _ICSA_.
  * Arisholm et al. (2006) Erik Arisholm, Lionel C. Briand, Siw E. Hove, and Yvan Labiche. 2006.  The impact of UML documentation on software maintenance: an experimental evaluation.  _TSE_ (2006).
  * Bakhtin et al. (2022) Alexander Bakhtin, Abdullah Al Maruf, Tomas Cerny, and Davide Taibi. 2022.  Survey on Tools and Techniques Detecting Microservice API Patterns. In _SCC_.
  * Bakhtin et al. (2023) Alexander Bakhtin, Xiaozhou Li, Jacopo Soldani, Antonio Brogi, Tomas Cerny, and Davide Taibi. 2023.  Tools Reconstructing Microservice Architecture: A Systematic Mapping Study.  _AMP, co-located with ECSA_.
  * Bambhore Tukaram et al. (2022) Anusha Bambhore Tukaram, Simon Schneider, Nicolás E. Díaz Ferreyra, Georg Simhandl, Uwe Zdun, and Riccardo Scandariato. 2022\.  Towards a Security Benchmark for the Architectural Design of Microservice Applications. In _ARES_. ACM, New York, NY, USA.
  * Budgen et al. (2011) David Budgen, Andy J. Burn, Perl Brereton, Ann B. Kitchenham, and Rialette Pretorius. 2011.  Empirical evidence about the UML: a systematic literature review.  _Software: Practice and Experience_ (2011).
  * Bushong et al. (2021a) Vincent Bushong, Amr S. Abdelfattah, Abdullah A. Maruf, Dipta Das, Austin Lehman, Eric Jaroszewski, Michael Coffey, Tomas Cerny, Karel Frajtak, Pavel Tisnovsky, and Miroslav Bures. 2021a.  On Microservice Analysis and Architecture Evolution: A Systematic Mapping Study.  _Applied Sciences_ (2021).
  * Bushong et al. (2021b) Vincent Bushong, Dipta Das, Abdullah Al Maruf, and Tomas Cerny. 2021b.  Using Static Analysis to Address Microservice Architecture Reconstruction. In _ASE_.
  * Cao et al. (2024) Clinton Cao, Simon Schneider, Nicolas Diaz Ferreyra, Sicco Verweer, Annibale Panichella, and Riccardo Scandariato. 2024.  CATMA: Conformance Analysis Tool For Microservice Applications. In _ICSE-Companion_.
  * Cerny et al. (2022) Tomas Cerny, Amr S. Abdelfattah, Vincent Bushong, Abdullah Al Maruf, and Davide Taibi. 2022\.  Microservice Architecture Reconstruction and Visualization Techniques: A Review. In _SOSE_.
  * Das et al. (2021) Dipta Das, Andrew Walker, Vincent Bushong, Jan Svacina, Tomas Cerny, and Vashek Matyas. 2021\.  On automated RBAC assessment by constructing a centralized perspective for microservice mesh.  _PeerJ Computer Science_ 7 (2021).
  * Di Francesco et al. (2019) Paolo Di Francesco, Patricia Lago, and Ivano Malavolta. 2019\.  Architecting with microservices: A systematic mapping study.  _JSS_ (2019).
  * Dragoni et al. (2016) Nicola Dragoni, Saverio Giallorenzo, Alberto Lluch-Lafuente, Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina. 2016\.  _Microservices: yesterday, today, and tomorrow_.  Springer International Publishing.
  * Emam (1999) Khaled El Emam. 1999\.  Benchmarking Kappa: Interrater Agreement in Software ProcessAssessments.  _EMSE_ (1999).
  * Emanuelsson and Nilsson (2008) Pär Emanuelsson and Ulf Nilsson. 2008.  A Comparative Study of Industrial Static Analysis Tools.  _Electronic Notes in Theoretical Computer Science_.  SSV.
  * Fritzsch et al. (2019) Jonas Fritzsch, Justus Bogner, Alfred Zimmermann, and Stefan Wagner. 2019.  From Monolith to Microservices: A Classification of Refactoring Approaches. In _Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment_. Springer International Publishing, Cham.
  * Garousi et al. (2019) Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2019\.  Guidelines for including grey literature and conducting multivocal literature reviews in software engineering.  _IST_ (2019).
  * Giamattei et al. (2024) L. Giamattei, A. Guerriero, R. Pietrantuono, S. Russo, I. Malavolta, T. Islam, M. Dînga, A. Koziolek, S. Singh, M. Armbruster, J.M. Gutierrez-Martinez, S. Caro-Alvaro, D. Rodriguez, S. Weber, J. Henss, E. Fernandez Vogelin, and F. Simon Panojo. 2024.  Monitoring tools for DevOps and microservices: A systematic grey literature review.  _JSS_ (2024).
  * Gortney et al. (2022) Mia E. Gortney, Patrick E. Harris, Tomas Cerny, Abdullah Al Maruf, Miroslav Bures, Davide Taibi, and Pavel Tisnovsky. 2022.  Visualizing Microservice Architecture in the Dynamic Perspective: A Systematic Mapping Study.  _IEEE Access_ (2022).
  * Granchelli et al. (2017) Giona Granchelli, Mario Cardarelli, Paolo Di Francesco, Ivano Malavolta, Ludovico Iovino, and Amleto Di Salle. 2017.  Towards Recovering the Software Architecture of Microservice-Based Systems. In _ICSAW_.
  * Gravino et al. (2015) Carmine Gravino, Giuseppe Scanniello, and Genoveffa Tortora. 2015\.  Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication.  _Journal of Visual Languages & Computing_ (2015).
  * Gravino et al. (2010) Carmine Gravino, Genoveffa Tortora, and Giuseppe Scanniello. 2010\.  An Empirical Investigation on the Relation between Analysis Models and Source Code Comprehension. In _SAC_. ACM.
  * Ibrahim et al. (2019) Amjad Ibrahim, Stevica Bozhinoski, and Alexander Pretschner. 2019\.  Attack graph generation for microservice architecture. In _Symposium on Applied Computing_. ACM.
  * JetBrains (2022) JetBrains. 2022\.  _The State of Developer Ecosystem 2022_.  Technical Report. JetBrains.  https://www.jetbrains.com/lp/devecosystem-2022/microservices/ Accessed on 09.02.2024.
  * JRebel (2022) JRebel. 2022.  _2022 Java Developer Productivity Report_.  Technical Report. JRebel.  https://www.jrebel.com/resources/java-developer-productivity-report-2022 Accessed on 09.02.2024.
  * Kleehaus et al. (2018) Martin Kleehaus, Ömer Uludag, Patrick Schäfer, and Florian Matthes. 2018.  MICROLYZE: A Framework for Recovering the Software Architecture in Microservice-Based Environments. In _Information Systems in the Big Data Era_. Springer International Publishing.
  * Lenarduzzi et al. (2023) Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023\.  A critical comparison on six static analysis tools: Detection, agreement, and precision.  _JSS_ (2023).
  * Lewis and Fowler (2014) James Lewis and Martin Fowler. 2014.  Microservices: a definition of this new architectural term.  https://martinfowler.com/articles/microservices.html
  * Mantere et al. (2009) Matti Mantere, Ilkka Uusitalo, and Juha Roning. 2009\.  Comparison of Static Code Analysis Tools. In _SECURWARE_.
  * Moreschini et al. (2023) Sergio Moreschini, Gilberto Recupito, Valentina Lenarduzzi, Fabio Palomba, David Hästbacka, and Davide Taibi. 2023.  Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Review.  _ArXiv_ (2023).
  * Muntoni et al. (2021) Giuseppe Muntoni, Jacopo Soldani, and Antonio Brogi. 2021\.  Mining the Architecture of Microservice-Based Applications from their Kubernetes Deployment. In _Advances in Service-Oriented and Cloud Computing_. Springer International Publishing, Cham.
  * Neri et al. (2020) Davide Neri, Jacopo Soldani, Olaf Zimmermann, and Antonio Brogi. 2020.  Design principles, architectural smells and refactorings for microservices: a multivocal review.  _SICS_ (2020).
  * Peltonen et al. (2021) Severi Peltonen, Luca Mezzalira, and Davide Taibi. 2021\.  Motivations, benefits, and issues for adopting Micro-Frontends: A Multivocal Literature Review.  _IST_ (2021).
  * Ponce et al. (2022) Francisco Ponce, Jacopo Soldani, Hernán Astudillo, and Antonio Brogi. 2022.  Smells and refactorings for microservices security: A multivocal literature review.  _JSS_ (2022).
  * Quéval and Zdun (2023) Pierre-Jean Quéval and Uwe Zdun. 2023.  Extracting the Architecture of Microservices: An Approach for Explainability and Traceability. In _ECSA_. Springer Nature Switzerland, Cham.
  * Rahman et al. (2019) Mohammad Imranur Rahman, Sebastiano Panichella, and Davide Taibi. 2019.  A curated Dataset of Microservices-Based Systems.
  * Schneider et al. (2024) Simon Schneider, Nicolas E. Diaz Ferreyra, Pierre-Jean Queval, Georg Simhandl, Uwe Zdun, and Riccardo Scandariato. 2024.  How Dataflow Diagrams Impact Software Security Analysis: an Empirical Experiment. In _SANER_.
  * Schneider et al. (2023) Simon Schneider, Tufan Özen, Michael Chen, and Riccardo Scandariato. 2023.  microSecEnD: A Dataset of Security-Enriched Dataflow Diagrams for Microservice Applications. In _MSR_.
  * Schneider and Scandariato (2023) Simon Schneider and Riccardo Scandariato. 2023.  Automatic extraction of security-rich dataflow diagrams for microservice applications written in Java.  _JSS_ (2023).
  * Soldani et al. (2023) Jacopo Soldani, Javad Khalili, and Antonio Brogi. 2023\.  Offline Mining of Microservice-Based Architectures (Extended Version).  _SN Comput. Sci._ (2023).
  * Soldani et al. (2021) Jacopo Soldani, Giuseppe Muntoni, Davide Neri, and Antonio Brogi. 2021.  The mTOSCA toolchain: Mining, analyzing, and refactoring microservice‐based architectures.  _Software: Practice and Experience_ (2021).
  * Soldani et al. (2018) Jacopo Soldani, Damian Andrew Tamburri, and Willem-Jan Van Den Heuvel. 2018.  The pains and gains of microservices: A Systematic grey literature review.  _JSS_ (2018).
  * Taibi et al. (2020) Davide Taibi, Valentina Lenarduzzi, and Claus Pahl. 2020\.  _Microservices Anti-patterns: A Taxonomy_.  Springer International Publishing, Cham.
  * Wohlin (2014) Claes Wohlin. 2014\.  Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In _EASE_. ACM.
  * Wohlin et al. (2012) Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012.  _Experimentation in software engineering_.  Springer Science & Business Media.