Recently, several methods have been proposed for

estimating the mutual information from sample data using deep neural networks and without the knowledge of closed form distribution of the data.

This class of estimators is referred to as neural mutual information estimators.

Although very promising, such techniques have yet to be rigorously bench-marked so as to establish their efficacy, ease of implementation, and stability for capacity estimation which is joint maximization frame-work.

In this paper, we compare the different techniques proposed in the literature for estimating  capacity  and provide a practitioner perspective on their effectiveness.

In particular, we study the performance of  mine,
and  dine
and provide insights on

We evaluated these algorithms in terms of their ability to learn the input distributions that are capacity-approaching for the AWGN channel, the optical intensity channel, and peak power-constrained AWGN channel.

For both scenarios, we provide insightful comments on various aspects of the training process, such as stability, sensitivity to initialization.

Neural capacity estimators;  AWGN channel; the optical intensity channel;  peak power-constrained AWGN channel; optimal input distribution.

§ INTRODUCTION

Determining the capacity of a channel and the optimal input distribution are problem of  fundamental importance in many communication scenarios of practical relevance.

The value of capacity and the optimal input distribution provide critical insight on the choice of coding rates and input constellation shape, respectively.

Generally speaking, solving the capacity problem is analytically challenging. For this reason, much research effort has been dedicated to the design of algorithms that provide approximate numerical solutions to the capacity problem.

The scenarios in which the analytical computation of capacity is intractable and numerical approaches are necessary are cases in which
(i) the channel transition probability expression has a complex analytical expression – as in fiber optics
(ii) the channel transition probability is to be estimated from a set of pilot measurements.

The advent of deep neural networks (DNNs) holds the promise of mitigating the inherent complexity of the capacity estimation.

In this paper, we try to benchmark the neural capacity estimators proposed in the literature so far and provide insights on their performance for channels of relevance in the information theory literature.

We also provide consideration on the practical issues in training the neural capacity estimators using channel samples.

Relevant Literature:
The computation of capacity using algorithmic and numerical methods most notably dates back to the well-known works of Blahut and Arimoto [1, 2]. This algorithm consists of an iterative alternating maximization method using the primal and dual formulation of the capacity problem <cit.>.
Various authors have considered extension and refinements to this original algorithm.
In [4], the authors provide an algorithm to maximize mutual information in a finite state indecomposable (noise free) channel with Markov source.
In [5],  the approach of [4] is extended to the case of a general finite state channels.
An algorithm based on deterministic annealing is proposed for determining the capacity of a Poisson channel through gradient descent in [6].
Finally, [7] provides a extension of Blahut-Arimoto for estimating directed mutual information and use it for estimating capacity of channels with feedback.

Recently, a few works have considered deep learning as a tool for estimating the capacity [8, 9], and channel coding based on mutual information maximization [10, 11, 12] and probability and geometric shaping [13]. This new approach has been partly motivated by neural network based estimators of mutual information [14, 15, 16, 17, 18, 19, 20], and can be particularly effective at estimating capacity from sample data, as deep networks can be used to  express complex relationship between channel input and outputs.  The authors of [9] are the first to specifically consider deep learning methods for capacity computations. They focus, in particular, in determining the feedback capacity of finite state channels through reinforcement learning.
In [21] the approach of [15] is used to produce efficient joint encoder and decoders for modulation with low probability of error by producing sequences with maximum mutual information between inputs and outputs of channel. Finally, [9] presents dine, which relies on directed information for estimating the capacity of continuous channels with and without feedback.

In this work, we explore how the choice of neural network-based estimators of mutual information can affect capacity estimation. There have been several neural estimators proposed over the past few years for estimating mutual information [14, 15, 16, 17, 18, 8, 19], and it is not clear what are the advantages and disadvantages of each approach. We focus on the problem of finding capacity of memoryless point-to-point channels with continuous alphabets. As in [8], our approach uses neural architectures for approximating the optimal input distribution from an initial distribution by iteratively estimating the mutual information using a maximization, and then optimizing the input distribution using the estimated mutual information. We explore how the choice of neural network based estimators of mutual information can effect the capacity estimation.

We start the paper by introducing the different neural mutual information estimators. Specifically, there are two general classes of estimators. Those that estimate the mutual information directly, such as mine [15] and smile [18], and those that use entropy estimation based on reference random variables [17] such as dine [8] and the lower bound based on InfoNCE [14]. Note that these entropy-based methods can be used for estimation of other information theoretic measures besides mutual information such as directed information and conditional mutual information, and that was the reason they were employed in [8] and [19]. For evaluation, we consider two continuous channels, the well-known average power constraint AWGN where the optimal input is continuous, and the optical intensity channel where the optimal input has finite mass points. Based on our numerical evaluations we observe that for the point-to-point channels, neural mutual information estimators based on direct estimation of the mutual information give more accurate estimates of the capacity as well as the optimal input distribution. We also show that neural capacity estimation can give tighter bounds for channels with only upper and lower bounds on capacity such as the optical intensity channel.

§ BOUNDS ON KL DIVERGENCE

In this section, we provide a lower and an upper bound on KL-divergence and in the next section we will use these bounds to present various methods that have been proposed in the literature for estimating mutual information using neural networks.

Let us denote the sample space of the random variable $X$ as $\mathcal{X}$, and $\mathcal{P}(\mathcal{X})$ the set of all probability measures over the Borel $\sigma$-algebra on $\mathcal{X}$.

As we consider only absolutely continuous distributions, we shall assume that  the density of the random variables (RVs) $X$ and $Y$ have a joint density $P(X,Y)$, and marginal distributions $P(X)$ and $P(Y)$.

In general, we will not explicitly indicate the independent variables of a distribution when they are clear from the context.

In the remainder of the paper, we consider three measures of information between RVs: the Kullback–Leibler (KL) divergence, mutual information (MI), and the $\chi^{2}$ divergence, defined as

D(P || Q)      = ∫P(x) logP(x)Q(x)   x,

I(X,Y)      = D(P(x,y) || P(x) P(y)),

χ^2(P||Q)     =  ∫_^(P(x)-Q(x))^2/Q(x)x,

respectively, where $ 0 \log 0 / 0=0$ by definition.

We present two bounds on KL divergence, which are used in
estimating mutual information. First bound is the Donsker-Varadhan [22] bound rewritten in the Theorem below for convenience.

Donsker-Varadhan representation, [22].

Consider the RV $X$ and two probability measures over $\Xcal$, $P$,$Q$. The KL-divergence can then be represented as

\begin{equation}\label{imj}
D(P || Q) = \sup_{T \in \mathcal{T}} \mathbb{E}_{P}[T(X)] - \log \lb \mathbb{E}_Q\left[e^{T(X)}\right] \rb,
\end{equation}

where $\mathcal{T}$ is the set of all functions with finite expectations in (<ref>).

Because using the gradients of the second term leads to a bias estimate, in [15] exponential moving average is used to reduce the bias.

Another lower bound for KL-divergence is [16, 19]:

\begin{equation}
\label{eq:tubalower}
D(P || Q) \geq \mathbb{E}_{P}[T(X)]-\lsb \frac{\mathbb{E}_Q\left[e^{T(X)}\right]}{\alpha}+\log(\alpha)-1\rsb
\end{equation}

where $\alpha$ is a positive constant.

Using these bounds, in the next section we present different neural mutual estimation approaches.

§  NEURAL ESTIMATION OF MUTUAL INFORMATION

In recent years, there have been many different methods proposed for estimating mutual information using neural networks [14, 15, 16, 17, 18, 19].

We refer to these approaches as nmie.

We review these methods in this section and then in the next section, we discuss how they can be employed to estimate capacity and the optimal input distribution.

These methods can be divided into groups. Those that evaluate the mutual information directly by using (<ref>) and bounds on KL divergence, and those that break the mutual information into entropy terms [17, 18, 19] and estimate each of those separately using bounds on KL divergence. We start by describing the direct methods.

§.§ Direct Mutual Information Estimation

The earliest attempt to NMIE estimation can be found in [15], where the
Mutual Information Neural Estimator (MINE) is proposed.

The MINE is obtained from the the Donsker-Varadhan representation in Th. <ref> when a DNN is used to represent $T(X,Y)$ in (<ref>). Note that here the KL divergence is between the joint distribution and the product of the marginals as in (<ref>), and hence the function $T(.)$ is a function of $X$ and $Y$.

It is noted in [15] that the optimization of $T(X,Y)$ using a DNN through stochastic gradient descent (SGD) is generally challenging.

This is because using a naive gradient estimate over the samples of the mini-batch leads to a biased estimate of the full gradient. Exponential moving average is proposed for mitigating this problem. However, this leads to an MI estimator with large variance, as noted in [16].

In [15], the MINE is shown to be effective in preventing the MODE collapse in generative adversarial networks (GANs).

To address the large variance of the MINE, the authors of [18] propose the smile. In this method, the Donsker-Varadhan representation in Theorem <ref> is rewritten as in the following theorem.

SMILE [18]
Consider the RV $X$ and two probability measures over $\Xcal$, $P$ and $Q$. The KL-divergence can then be approximated as

D(P || Q) ≈sup_T ∈𝒯 𝔼_P[T(X)] - log𝔼_Q[(e^T(X),τ,-τ)],

where $\clip(x,\tau,-\tau) \triangleq \max\{\min\{x,-\tau\},\tau\}$.

SMILE leverages the bound in (<ref>) as the $\clip$ function is equivalent to clipping the log density ratio estimator in the interval $[-\tau,+\tau]$. Similar to mine mutual information can be estimated by using a neural network to represent $T(.)$, and optimizing the neural network using gradient ascent.

The choice of $\tau$ crucially affects the bias-variance trade-off. With smaller $\tau$, the variance of the estimator is reduced at the cost of increasing its bias.

InfoNCE Contrastive Lower Bound:
The InfoNCE bound provides a method for computing MI directly using $K$ samples of joint distribution of $X,Y$. This bound is given by

$$I(X; Y) \ge \mathbb{E}_{p^K(x,y)}\left[\frac{1}{K} \sum_{i=1}^K \log \frac{e^{\color{black}f(x_i, y_i)}}{\frac{1}{K}\sum_{j=1}^K e^{\color{black} f(x_j, y_i)}}\right].$$

where $K$ is mini-batch size and $(x_i,y_i)$ are samples of joint distribution.

§.§ Entropy-based Mutual Information Estimation

We begin the section by presenting variational bounds on entropy.

To estimate this quantity using the variational methods  [17], a reference (and arbitrary) distribution $Q$ over the random variable $X$ with pdf $q(x)$ is used in place of the true and unknown distribution $P$ with pdf $p(x)$. Using $Q$, the entropy of the random variable $X$ can be written as:

\begin{equation}\label{mes}
h(X)=\mathbb{E}_P [-\log(q(x))]-D(P||Q).
\end{equation}

Note that the first term is the cross-entropy term $h_{\text{CE}}(P,Q)$.

Using (<ref>), and i.i.d. reference random variables $X^{'}$ and $Y^{'}$, mutual information is represented as

\begin{align}
I(X;Y) &= h(X) + h(Y) - h(X,Y),  \nonumber \\
& =D(P_{X,Y}||Q_{X^{'},Y^{'}})-D(P_X||Q_{Y^{'}})-D(P_Z||Q_{Y^{'}}). \label{eq:decompose}
\end{align}

Note that since we can choose the reference distributions to be i.i.d., the cross-entropy terms in (<ref>) will cancel out leaving only the KL-divergence terms. We now present the methods that use this appraoch for estimating mutual information.

The dine [8] estimates the directed information, which is used to compute the capacity of channels with feedback. It can also be used to estimate the mutual information and the capacity of memoryless channels.  Instead of using (<ref>) to decompose mutual information, dine uses $h(Y)$ and $h(Y|X)$ and hence has two KL divergence terms to estimate instead of three in equation (<ref>). Each KL divergence term is then computed using (<ref>) adopted from[17]. Since the KL divergence term corresponding to $h(Y|X)$ is positive while the KL divergence term corresponding to $h(Y)$ is negative, using (<ref>) estimates a quantity which is not a lower bound. To support feedback and channels with memory, dine uses a recurrent neural network to estimate the directed information and hence it is slower to train for memoryless channels without feedback. This can be easily resolved however by using fully connected layers to represent the function $T$ as is done in [19].

Chi-Square bound:
This method is used to compute an explicit lower bound on mutual information, and was employed in [19] to compute the boundaries of the capacity region of the multiple access channels. This method uses the decomposition in (<ref>).

For the positive KL divergence term in (<ref>),  (<ref>)
is used where $T$ is represented and optimized using a neural networks. For the negative KL divergence terms in (<ref>), the upper bound $\chi^{2}_{\mathsf{UP}}(P||Q)$ is used, where the $\chi^2$ divergence terms are estimated using histogram-based density estimations [19]. The following theorem summarizes this approach:

Chi-Square Bound [19]
The MI between the RVs $X$ and $Y$ is lower bounded as

I(X;Y)≥    𝔼_P[T(X,Y)] - 𝔼_Q[e^T(X^',Y^')]/α-log(α)+1

where $\alpha > 0$ and $X^\prime, Y^\prime$ are uniform random variables.

We refer to this as $\chi^{2}$ bound in the Table. <ref>.

In the next section we present how each of these neural mutual information estimators can be used to estimate the capacity and the optimal input distribution. Then we will compare each method numerically.

After extensive studies we have observed that Infonce lower bound and reverse-jensen shows a high variance (and bias) in estimation of the capacity value and optimal input distribution. So we have not included the results along with other methods. It should be noted that although chi-square bound give rises to tight lower bounds in cases considered, its optimal input approximates have high varinace and is not accurate so we have excluded their results for optimal input approximation as well.

§ NEURAL CAPACITY ESTIMATION

In this section, we use the nmie approach in Sec. <ref>  to estimate  the capacity $\Csf$ of memoryless point-to-point channels, as described by conditional probability $P_{Z|X} \in \Xcal \times \Zcal$ assumed continuous, that is

=sup_P ∈()  I(X,Z)

where $X \in \Xcal$ is the channel input, $Z \in \Zcal$ is the channel output, and the supremum is taken over all continuous probability measures $P \in \Pcal(\Xcal)$ over the input alphabet $\Xcal$. The architecture we employ for the optimization in (<ref>) is similar to [8] and uses two different neural networks, the NMIE to estimate the mutual information and one called the nit to find the capacity approaching input distribution. We describe each of these networks next.

For the NMIE, any of the the four methods presented in the previous sections as well as other NMIE can be used. In this work, the NMIE is uses a number fully connected layers since we are considering memoryless channels. The nit also uses fully connected layers.  The input to nit is a standard Gaussian RV $N_1$ and the output of the nit is the channel input $X$. Hence the nit transforms the standard Gaussian RV into any other distribution.

The training of nit and nmie are performed iteratively (similar to GANs). A single training iteration has 3 phases. In phase 0, the NMIE is trained exclusively for a few iterations an initialization learning for the algorithm. This is only done only once at the beginning. The algorithm then enters the main optimization loop where in phase 1 the weights of the NIT network are kept constant and the NMIE network is trained and in phase 2 the weights of the NIT network are kept constant and the NMIE network is trained. The loss function used for training both networks is the negative of the estimated mutual information and is different for each of the four nmie methods considered in this paper as described in the previous section. The main training loop continues in this fashion until the estimates $I(X;Z)$ converge or until a specific number of iterations are reached. Algorithm <ref> summarizes this training procedure.

Neural capacity (achievable rate) estimation
Channel model or its GAN approximation
Estimate of the capacity or an achievable rates for the channel
Initialize parameters of NMIE and NIT randomly

Phase 0: perform initial training of NMIE network using randomly generated samples

not converged or max iteration not reached

Generate $B$ sample of $N_1$: $\{(n_1^{(i)})\}_{i=1}^B$

Generate $\{(x_i, z_i)\}_{i=1}^B$ using NIT and channel

Use channel and NMIE to estimate:

Phase 1: Train NMIE

Train NMIE networks using stochastic gradient decent and $-\hat{I}$ as loss

Phase 2: Train NIT

Train NIT network using stochastic gradient decent and $-\hat{I}$ as loss

Perform final evaluation on all  or subset of data

Return: $ \hat{I}(X;Z)$

§ NUMERICAL RESULTS

This section evaluates the proposed method on different channels with different noise models

We focus, in particular, on two model of broad interest:   (i) the Gaussian Additive White Noise (AWGN)  with power constraint and (ii) the optical intensity cahnnel.

Both channels are described by the input/output relationship

Z    = X + N.

For (i) we have the constraint  $\Ebb[X^{2}] \leq \ep$ and $X \in \mathbb{R}$, while for (ii) we have the additional constraint  $X \in [0,A]$.

In the case of the AWGN channel with an input power constraint, the capacity and the optimal input distribution are both well-known analytically. We use this channel to validate and compare the performance of each nmie for capacity estimation.

For the case of the AWGN channel with a peak input constraint, it is known that optimal input distribution is discrete but capacity and exact optimal distribution don't have analytical formations and must be evaluated numerically.

In the following, we shall compare the performance of the four approaches in Sec. <ref> for first channel and a variant of second channel where $A \rightarrow \infty$. We begin the section by describing the neural network architectures used for each method and then present the performance results.

§.§ Architectures:

For all the experiments in this section, with the exception of dine, nmie network used in computation of mutual information is base on feed-forward fully connected layers. For dine, we are using the code that was released with  [8], which uses a modified LSTM. We use ReLu activation functions in all layers. Based on the method used for computation, 4 to 10 layer neural networks have been chosen for training. The hidden layers dimension is between 64 to 256 for different methods. The nit architecture uses 5 fully connected layers with the hidden layer dimension of 64.

We have experimented with the convolutional neural networks and no significant improvement in capacity estimation observed. One of the possible causes is related to inherent difficulty of MI estimation regardless of methods used for f of its quantity.

§.§ Data Generation and Hyperparameter Choice

We have used a batch size of 256 for training our algorithms with SMILE and MINE method.

It is worth to mention that all algorithms are trained in a few minutes at each SNR. The learning rate is set to .0001 for each of NIT and NMIE networks. We have used Adam optimizer and for SMILE method the clipping parameters $\tau$ is set to $0.2$. The gradient clipping is set to 0.2 in all cases. Results are obtained using a 10 round estimate and averaging between rounds to produce smooth and reliable estimates for different quantities.

§.§ Numerical Experiments

SNR ($\rm dB$)     DINE     MINE      SMILE     True

2     $0.39 \pm 0.12$     $\mathbf{0.476\pm 0.05}$     $0.42\pm 0.09$      0.474

20      $2.21 \pm 0.16$     $\mathbf{2.29\pm 0.06}$     $2.12\pm 0.07$     2.307

40     $4.11\pm 0.23$       $\mathbf{4.49\pm 0.08}$     $4.42\pm 0.17$      4.605

Estimated capacity of AWGN using different methods across 10 trials.

AWGN with average power constraint:

In Table. <ref> we provide a comparison between the capacity of this model and its neural estimate for different SNR values and variance of capacity estimation.

We note that the relative accuracy of the decreases as the SNR grows but the loss of accuracy does not affect all methods equally.

The MINE gives a more accurate estimate of the capacity compared to other methods, while dine seems to diverge from the true capacity at higher SNRs.

Fig. <ref> shows the histogram of the nit

learned optimal input distribution at SNR=40dB for the DINE, MINE, and SMILE.

We observe that at higher SNRs the mine and smile achieve a more accurate approximation of the optimal input distribution, i.e. Gaussian.

While dine can also learn accurate input distributions at low SNRs, at higher SNRs, it seems to deviate from the optimal input. It is evident from the the table that MINE gives the least variance compared to the other methods. Although there are other method than MINe which yield lower variance in estimating MI, we have observed that in the joint optimization setting where two neural networks optimzed jointly these observation does not hold. In fact SMILE gives higher variance at different SNR regimes compared with MINR, but smaller variance compared to DINE in capacity estimation.

Note that all three methods are trained for the same number of iterations here for a fair comparison.

Optical intensity channel:

The capacity of optical intensity is generally unknown [23], [24]. Since the exact capacity with closed form solution is not available for this channel we compare our results with bounds of [23], which are some of the best known bounds in the literature.

Table. <ref> shows the estimated value of the capacity and achievable rates using different nmie methods for the optical intensity channel with only the average power constraint.

Here we did not include the results of dine since the code released with [8] does not support this channel. Also since MINE and SMILE yield very similar results, we only show the results of SMILE in Table. <ref>. We  observe that both nmie methods that are considered, estimate values that are between the lower and upper bound, with SMILE estimating values that are closer to the upper bound. Moreover, we consider some special cases where optimal input is known for optical intensity is known. Fig. 3 and Fig.5 illustrate optimal input distribution in two different SNR regime where optimal input is known from[25]. Fig. 3 illustrates an SNR range where optimal input has four mass point, and Fig.5 shows low SNR regime where it is known that optimal inputs is binary (like One-Off keying).
Peak power constraint Gaussian channel

There is no a closed form formula for the capacity of peak constraint AWGN channel. Although the accurate shape of optimal distributions is not known in general, It is proved that optimal distributions have discrete structure[26]. In order to compare and show the effectiveness of our approach, we compare our results with the lower and upper bounds provided in the [27]. In their setting they set $A=\sqrt{P}$. Our results and comparison with upper and lower bounds in [27] demonstrate that estimated values fill the gap between lower and upper bounds for this channel.

Joint optimization framework for neural capacity computations.

group style=
group name=my plots,
group size=3 by 3,
xlabels at=edge bottom,
xticklabels at=edge bottom,
vertical sep=0pt,
horizontal sep=5pt

(top) at (axis cs:1,/pgfplots/ymax);

[name path=f,red,samples=100,domain=-4:4,samples=500,fill=red!120, opacity=0.4](1/(1.25*sqrt(2*pi)))*exp(-(x^2/(2*1.58)));

+[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5, ]
table[col sep=comma]./data_m/data_batch_hs-dine-awgn-2.txt;

(top) at (axis cs:1,/pgfplots/ymax);

[red,domain=-30:30,samples=500,fill=red!120, opacity=0.4](1/(10*sqrt(2*pi)))*exp(-(x^2/(2*100)));

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5]
table[col sep=comma]./data_m/data_batch_hs-dine-awgn-20.txt ;

(top) at (axis cs:1,/pgfplots/ymax);

[red,domain=-500:500,samples=500,fill=red!120, opacity=0.4]

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5,]
table[col sep=comma]./data_m/data_batch_hs-dine-awgn-40.txt;

[name path=f,red,samples=100,domain=-4:4,samples=500,fill=red!120, opacity=0.4](1/(1.25*sqrt(2*pi)))*exp(-(x^2/(2*1.58)));

+[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5]
[restrict x to domain=-4:4]
table[col sep=comma]./data_m/data_batch_hs-mine-awgn-2.txt;

(top) at (axis cs:1,/pgfplots/ymax);

[red,domain=-30:30,samples=500,fill=red!120, opacity=0.4](1/(10*sqrt(2*pi)))*exp(-(x^2/(2*100)));

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5]
table[col sep=comma]./data_m/data_batch_hs-mine-awgn-20.txt ;

[red,domain=-500:500,samples=500,fill=red!120, opacity=0.4]

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5,]
table[col sep=comma]./data_m/data_batch_hs-mine-awgn-40.txt;

[ylabel=SMILE, xlabel=  $2 \ \rm dB$,xtick=-4,-3,...,4,]

[name path=f,red,samples=100,domain=-4:4,samples=500,fill=red!120, opacity=0.4](1/(1.25*sqrt(2*pi)))*exp(-(x^2/(2*1.58)));;

+[ybar interval,mark=no,
fill=blue!120,draw=blue, opacity=0.5,]
[restrict x to domain=-4:4]
table[col sep=comma]./data_m/data_batch_hs-smile-awgn-2.txt;

(bot) at (axis cs:1,/pgfplots/ymin);

[xtick=-30,-20,...,30, xlabel=$20 \ {\rm dB}$ ]

(top) at (axis cs:1,/pgfplots/ymax);

[red,domain=-30:30,samples=500,fill=red!120, opacity=0.4](1/(10*sqrt(2*pi)))*exp(-(x^2/(2*100)));

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5]
table[col sep=comma]./data_m/data_batch_hs-smile-awgn-20.txt ;

(bot) at (axis cs:1,/pgfplots/ymin);

[xlabel=$40 \ {\rm dB}$,xtick=-500,-250,...,500]

[red,domain=-500:500,samples=500,fill=red!120, opacity=0.4]

[ybar interval,mark=no,fill=blue!120,draw=blue, opacity=0.5,]
table[col sep=comma]./data_m/data_batch_hs-smile-awgn-40.txt;

Comparison of different methods for computing the optimal input distribution for Gaussian channel. Each row corresponds to SNR equal to $ 2 \ {\rm dB}$, $20 \ {\rm dB}$ ,and $40 \ {\rm dB}$ respectively.

Based on the numerical evaluations, with respect to the considerations in Sec. <ref>, we conclude that

using nmie approaches that estimate the MI directly can result in more accurate estimates of the channel capacity and the optimal input distribution. While more investigation is needed to gain a deeper understanding of this observation, we suspect that this is because the error terms in entropy-base estimations

compound through the learning process, resulting in less accurate estimates of MI and, consequently, capacity.

A repository containing the code used to derive the above results is available online here

SNR (db)    5    10     15     20

MINE     $.424\pm .077$     $.84 \pm .080 $     $1.39\pm .12 $     $1.81\pm .09$

Chi-Square     $.422\pm .08555$    $.7611\pm .131$    $ 1.2839\pm .099$     $1.7687\pm.14$

SMILE    $.5083 \pm .092 $.     $.93\pm .183$     $1.393\pm .0985$     $1.821\pm .15$

DINE    $.429\pm .0969$     $.88 \pm .120$    $1.31\pm 0.121$    $1.73 \pm .110$

Lower bound    .42    .83     1.34     1.78

Upper bound     .99    1.480     1.77     2.22

comparisons of different bounds for optical intensity channel with only average power constraint. results are based on averaging 5 trials.

ytick=,xtick distance=1,
[ybar,bar width=.1cm, mark=no,fill=blue!120,draw=blue,]
table[col sep=comma]./data_m/data_optical_8db_peak.txt;

Computed optimal input distribution by NIT for optical intensity channel with $\epsilon=A=8db$ ( only peak constraint). We have used MINE and a unit variance Gaussian as initialization for our algorithm in this figure. upper bound [28] at 8db=1.19 , lower bound [28] at 8db=.601 , estimated capacity=.922

ytick=,xtick distance=1,

[ybar, bar width=.1cm, mark=no,fill=blue!120,draw=blue,]
table[col sep=comma]./data_m/data-awgn-peak=5,avg=10db.txt;

Computed optimal input distribution by NIT for peak constraint awgn channel. As in [26] simulation is done for $A=\sqrt{P}$.The estimated capacity after 10 round estimation and averaging is .956. The upper bound and optimized lower bound of [27]  for this snr is 1.19 and .901 respectively

[name=plot,xmin=0,xmax=2.5,width=.35,grid=major,scaled y ticks=manual:#1/10000,]]
[ybar, bar width=.1cm, mark=no,fill=blue!120,draw=blue,] table[col sep=comma]./data_m/data-optical-3db.txt ;

Computed optimal input distribution by NIT for optical intensity channel with $\epsilon=A=3db$ ( only peak constraint). This agrees with the points of optimal input distributions for optical intensity channels in [25] in low SNR regime.

§ CONCLUSION

In this paper, we investigated the ability of deep neural network (DNN) to provide an accurate estimate of the capacity of a point-to-point channel and the corresponding optimal input distribution.

As the capacity estimation problem corresponds to a mutual information (MI) maximization problem, we consider four approaches to the MI estimation using neural networks previously proposed in the literature.

We validate the proposed approach to the capacity estimation on the AWGN channel with a power constraint and a peak amplitude constraint.

Numerical evaluations show that all methods perform rather well at low SNR, while only some methods perform well at high SNR. Specifically, we see that MINE method could provide more accurate estimates of achievable rates and optimal input distribution for AWGN and optical intensity channel.

[1]
R. Blahut, “Computation of channel capacity and rate-distortion functions,”
IEEE Transactions on Information Theory, vol. 18, no. 4, pp. 460–473,
July 1972.

[2]
S. Arimoto, “An algorithm for computing the capacity of arbitrary discrete
memoryless channels,” IEEE Transactions on Information Theory,
vol. 18, no. 1, pp. 14–20, 1972.

[3]
R. G. Gallager, Information theory and reliable communication.1em
plus 0.5em minus 0.4emSpringer, 1968, vol. 2.

[4]
P. O. Vontobel, “A generalized Blahut-Arimoto algorithm,” in IEEE
International Symposium on Information Theory, 2003. Proceedings., 2003, pp.

[5]
——, “A generalized blahut-arimoto algorithm for finite state channels,”
in IEEE International Symposium on Information Theory, 2008.
Proceedings., 2008, pp. 94–.

[6]
J. Cao, S. Hranilovic, and J. Chen, “Capacity-achieving distributions for the
discrete-time poisson channel Part I: General properties and numerical
techniques,” IEEE transactions on communications, vol. 62, no. 1, pp.
194–202, 2013.

[7]
I. Naiss and H. H. Permuter, “Extension of the Blahut-Arimoto
algorithm for maximizing directed information,” IEEE Transactions on
Information Theory, vol. 59, no. 1, pp. 204–222, 2013.

[8]
Z. Aharoni, D. Tsur, Z. Goldfeld, and H. H. Permuter, “Capacity of
Continuous Channels with Memory via Directed Information Neural Estimator,”
arXiv e-prints, p. arXiv:2003.04179, Mar. 2020.

[9]
Z. Aharoni, O. Sabag, and H. H. Permuter, “Computing the feedback capacity of
finite state channels using reinforcement learning,” in 2019 IEEE
International Symposium on Information Theory (ISIT).1em plus 0.5em
minus 0.4emIEEE, 2019, pp. 837–841.

[10]
R. Fritschek, R. F. Schaefer, and G. Wunder, “Deep learning for channel coding
via neural mutual information estimation,” in 2019 IEEE 20th
International Workshop on Signal Processing Advances in Wireless
Communications (SPAWC), 2019, pp. 1–5.

[11]
——, “Neural mutual information estimation for channel coding:
State-of-the-art estimators, analysis, and performance comparison,” in
2020 IEEE 21st International Workshop on Signal Processing Advances in
Wireless Communications (SPAWC), 2020, pp. 1–5.

[12]
N. A. Letizia and A. M. Tonello, “Capacity-driven autoencoders for
communications,” IEEE Open Journal of the Communications Society,
vol. 2, pp. 1366–1378, 2021.

[13]
F. A. Aoudia and J. Hoydis, “Joint learning of probabilistic and geometric
shaping for coded modulation systems,” in GLOBECOM 2020 - 2020 IEEE
Global Communications Conference, 2020, pp. 1–6.

[14]
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with
contrastive predictive coding,” arXiv preprint arXiv:1807.03748,

[15]
M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and
R. D. Hjelm, “Mine: Mutual information neural estimation,” 2018.

[16]
B. Poole, S. Ozair, A. v. d. Oord, A. A. Alemi, and G. Tucker, “On variational
bounds of mutual information,” arXiv preprint arXiv:1905.06922, 2019.

[17]
C. Chan, A. Al-Bashabsheh, H. P. Huang, M. Lim, D. S. H. Tam, and C. Zhao,
“Neural entropic estimation: A faster path to mutual information
estimation,” 2019.

[18]
J. Song and S. Ermon, “Understanding the Limitations of Variational
Mutual Information Estimators,” arXiv e-prints, p. arXiv:1910.06222,
Oct. 2019.

[19]
F. Mirkarimi and N. Farsad, “Neural computation of capacity region of
memoryless multiple access channels,” arXiv preprint
arXiv:2105.04453, 2021.

[20]
——, “Neural computation of capacity region of memoryless multiple access
channels,” in 2021 IEEE International Symposium on Information Theory
(ISIT), 2021, pp. 2066–2071.

[21]
H. Ye, G. Y. Li, B.-H. F. Juang, and K. Sivanesan, “Channel agnostic
end-to-end learning based communication systems with conditional gan,” in
2018 IEEE Globecom Workshops (GC Wkshps).1em plus 0.5em minus
0.4emIEEE, 2018, pp. 1–5.

[22]
M. D. Donsker and S. R. S. Varadhan, “Asymptotic evaluation of certain
Markov process expectations for large time. IV,” Communications on
Pure and Applied Mathematics, vol. 36, no. 2, pp. 183–212, 1983.

[23]
A. Lapidoth, S. M. Moser, and M. A. Wigger, “On the capacity of free-space
optical intensity channels,” IEEE Transactions on Information Theory,
vol. 55, no. 10, pp. 4449–4461, 2009.

[24]
A. A. Farid and S. Hranilovic, “Capacity bounds for wireless optical
intensity channels with Gaussian noise,” IEEE Transactions on
Information Theory, vol. 56, no. 12, pp. 6066–6077, 2010.

[25]
N. Sharma and S. Shamai, “Transition points in the capacity-achieving
distribution for the peak-power limited awgn and free-space optical intensity
channels,” Problems of Information Transmission, vol. 46, no. 4, pp.
283–299, 2010.

[26]
A. Thangaraj, G. Kramer, and G. Böcherer, “Capacity bounds for discrete-time,
amplitude-constrained, additive white gaussian noise channels,” IEEE
Transactions on Information Theory, vol. 63, no. 7, pp. 4172–4182, 2017.

[27]
A. McKellips, “Simple tight bounds on capacity for the peak-limited
discrete-time channel,” in International Symposium onInformation
Theory, 2004. ISIT 2004. Proceedings., 2004, pp. 348–348.

[28]
A. Lapidoth, S. M. Moser, and M. A. Wigger, “On the capacity of free-space
optical intensity channels,” IEEE Transactions on Information Theory,
vol. 55, no. 10, pp. 4449–4461, 2009.


11institutetext: University of Hawaii at Manoa, Honolulu HI 96822, USA
22institutetext: 22email<EMAIL_ADDRESS>
# Building the MSR Tool Kaiaulu: Design Principles and Experiences

Carlos Paradis 1122 0000-0002-3062-7547 Rick Kazman 1122 0000-0003-0392-2783

###### Abstract

Background: Since Alitheia Core was proposed and subsequently retired, tools
that support empirical studies of software projects continue to be proposed,
such as Codeface, Codeface4Smells, GrimoireLab and SmartSHARK, but they all
make different design choices and provide overlapping functionality. Aims: We
seek to understand the design decisions adopted by these tools–the good and
the bad–along with their consequences, to understand why their authors
reinvented functionality already present in other tools, and to help inform
the design of future tools. Method: We used action research to evaluate the
tools, and to determine a set of principles and anti-patterns to motivate a
new tool design. Results: We identified 7 major design choices among the
tools: 1) Abstraction Debt, 2) the use of Project Configuration Files, 3) the
choice of Batch or Interactive Mode, 4) Minimal Paths to Data, 5) Familiar
Software Abstractions, 6) Licensing and 7) the Perils of Code Reuse. Building
on the observed good and bad design decisions, we created our own tool
architecture and implemented it as an R package. Conclusions: Tools should not
require onerous setup for users to obtain data. Authors should consider the
conventions and abstractions used by their chosen language and build upon
these instead of redefining them. Tools should encourage best practices in
experiment reproducibility by leveraging self-contained and readable schemas
that are used for tool automation, and reuse must be done with care to avoid
depending on dead code.

###### Keywords:

mining software repositories design choices action research.

##  1 Introduction

Research into quality dimensions of software project requires the analysis of
large quantities of data. For researchers this typically means mining data
from multiple open source software projects. Pre-processing data, calculating
metrics and flaws, and synthesizing composite results from a large corpus of
project artefacts is a tedious and error prone task lacking immediate
scientific value [10]—it is seen merely as a means to an end. This was the
motivation for the Alitheia Core [10], which was made available in 2009 for
the software engineering community. It provided features for data collection,
integration and analysis services and emphasized an easy to use extension
mechanism. Yet, as of today, Alitheia Core is a dormant (read-only) project in
GitHub111https://github.com/istlab/Alitheia-Core and several other tools
replicate at least some of its functionality.

What went wrong? Why have many tools re-implemented the same “tedious and
error prone” tasks the Alitheia Core? And do the current tools live up to the
promise of Alitheia Core? In this work, we revisit lessons learned by the
Alitheia Core authors and the design choices made by the other more recent
tools using an action research [8] approach.

Our contributions in this paper are twofold: first, we present a set of key
design decisions derived from an analysis of the aforementioned tools which
either facilitated or hindered reusability, reproducibility, interoperability
and extension of functionality. Second, we present our tool, Kaiaulu222The
documentation for the tool can be found at https://github.com/sailuh/kaiaulu,
which builds upon the design decisions made from these prior tools, and which
we believe fills a gap in the existing mining software repositories ecosystem.

##  2 Studied Tools and Lessons Learned

The tools that we studied are Codeface [11], Codeface4Smells [23], GrimoireLab
[18, 7] SmartSHARK, [25, 24] and PyDriller [22]. We now present our
observations regarding the strengths and weaknesses of these tools in terms of
their design choices and note, throughout the work, lessons learned by the
authors of Alitheia Core [10] presented in [16]. Many of these lessons are
applicable and worthy of consideration in new tools with similar intents. We
employed an action research methodology in studying these tools, but do not
describe the details of that research here, due to space limitations.

###  2.1 Abstraction Debt

We have observed different levels of abstraction employed in the surveyed
tools, ranging from applications that are built as monoliths to those built
from smaller components. This is consistent with what has been noted in
machine learning systems as abstraction debt [21], i.e. a lack of key
abstractions to support the functions and growth of MSR tools.

Codeface was created as a monolithic application, in which an entire project’s
Git log or mailing list is analyzed. It abstracts a complete end-to-end
pipeline, implemented by a command line interface (CLI), and outputs a
database dump of a project. It is therefore difficult for other applications
to build on some of its unique features, for example, using its Git log parser
that parses at function (rather than file) granularity.

Both GrimoireLab and SmartSHARK define several components, each with its own
CLI, but the component abstractions they employ are not the same. To provide a
point of comparison, Grimoire’s Lab Perceval provides a CLI to obtain data
from many data sources (e.g. GitHub, Git, Bugzilla, Jira, mailing lists, etc),
serving as a single interface for data collection. In contrast, SmartSHARK
defines its abstraction per data source type and, in the case of data
acquisition, at a more fine-grained level than Perceval. For example, consider
issueShark and vcsShark, two components of SmartShark. IssueShark defines
abstractions for different types of issues tracker sources, and vcsShark for
different types of version control systems. SmartSHARK’s abstractions
facilitate defining additional features specific to a data source type, such
as separating static vs. dynamic data in issue trackers (e.g. creation time of
the issue vs. comments), regardless of its underlying implementation (e.g.
Jira or Bugzilla)333https://github.com/smartshark/issueSHARK\\#introduction.

Pydriller is a single component and is smaller in scope as it only abstracts
Git repositories. However it is different from the other tools in that it
provides an API instead of a CLI. Its motivation is also different: it wraps
around PythonGit, which in itself provides a Pythonic API to nearly all
features of Git, to provide an API catered towards mining software
repositories only. In providing just a subset of Git functionality, it exposes
functionality catering specifically to the needs of mining repositories.

The decision between choosing a CLI or API has tradeoffs. An issue with
command line only interfaces occurs when an end-user may be interested in a
different abstraction of the data not preconceived by the authors. However an
API requires the user to be familiar with the programming language the tool
was built on top of, whereas a CLI does not.

From the above we derive the following lessons learned: End-to-end pipelines
such as Codeface’s limit the ability of other researchers to build on top of
them. Defining more specific abstractions per data type, whether via CLI or
API as issueShark and PyDriller do, facilitates building additional
functionality specific to a particular data type, or audience. Moreover, CLIs
can be built on top of a well-defined API, providing the benefit of both
interfaces, as we do in Kaiaulu.

###  2.2 Tool Configuration Files vs Project Configuration Files

In [24], the authors of SmartSHARK noted that one of their goals was to
support replication through the storage of data in a single harmonized schema.
Replication, it is argued, is supported by a common dataset. However, we have
observed that replication is also being done within configuration files in
Codeface.

Codeface uses a concept we named project configuration files. These files
provide a single compact source where parameters associated with the
acquisition and manipulation of a dataset can be stored. Project configuration
file parameters are required for tool execution, and they are a pragmatic,
lightweight and human-readable way to specify reproducible results. Project
configuration files also save time when a project is re-analyzed in other
studies, as some project-specific information may not be obvious from the
dataset alone.

Of all the tools we have reviewed, only Codeface provides users with a means
to specify project configuration files. This led to a large collection of
project configurations that have been versioned in Codeface over time444See
https://github.com/siemens/codeface/tree/master/conf and
https://github.com/maelstromdat/codeface4smells_TR/tree/master/Configurations
for Codeface and Codeface4Smells respectively. This information, which
supports repeatability, may otherwise not have been possible (or at least
easy) to reconstruct if all that was shared was the data.

We note that externalizing parameter choices in data acquisition and
manipulation tasks has been more prominent in machine learning frameworks, for
example to define experiments in configuration
files555https://xnmt.readthedocs.io/en/latest/experiment_config_files.html,
which include machine learning model selection and choice of model hyper-
parameters [19].

From the above, we derive the following lessons: integrating configuration
files that are human-readable and leveraged by the tool can enable
reproducibility, without the hurdles of sharing large quantities of primary
data.

###  2.3 Batch Mode, Interactive Mode, and Literate Programming

As we noted before, with the exception of PyDriller, every tool defines a CLI,
but not an API. This means the only way to interact with these tools is batch
mode. Meanwhile, PyDriller does not offer a CLI, only an API, which confers
its users the ability to leverage Python’s interactive mode to explore the
data. However, it does not include a CLI for batch mode processing, for out-
of-the-box data acquisition, processing or data analysis. What we observe then
is that existing tools decide on either CLI or API, but not both. We believe,
however, that the mining of software repositories requires a tool capable of
both, supporting an iterative process of data exploration, and when concluded,
a way to enact batch processing to scale up.

To illustrate our claim—as no existing tool provides both capabilities—we
provide a few examples: in a recent socio-technical study, we needed to do
identity matching, applying heuristics that have been published by other
authors (e.g. [4, 27]) to assign identities to developers who use different
names and e-mails in version control systems and mailing lists. Consider the
case where we chose the simplest method, where developers whose name or e-mail
match are assigned the same id. At first glance, this seems like a reasonable
assumption. However, it was due to experimenting interactively with the
identity matching API that we discovered that all core developers, due to the
use of an issue tracking system, ended up sharing the same e-mail address. We
noted this case as a unit test until a better heuristic could be found, and
then examined the data for other cases until we were satisfied with the
results. We then saved the observed parameters in a project configuration
file, and used it to deploy a batch process to collect various computationally
intensive architectural metrics.

We have had similar experience in determining and testing heuristics to filter
files in a repository, or determining the method that developers adopt to
annotate issue numbers in commit messages. Because each project may apply its
own conventions, tools that offer an experimentation capability, and then
defer mass data processing to batch more efficiently support the full workflow
of a researcher in mining software repositories.

The described interactive data explorations could certainly have been done in
a Python or R session, but it is better to leverage literate programming
using, for example, Python or R Notebooks, so that the rationale of the design
experiment is not lost. However, care must be taken to not extensively rely on
notebooks without further refactoring functionality into the code base,
leading to dead experimental code paths [21].

Our learned lessons here were: existing tools choose either APIs or CLIs
(supporting batch or interactive modes). However, making both interfaces
available will better support users in their various research efforts in
mining software repositories. The use of Notebooks to illustrate and explain
the API complements the API, provided functionality is not entirely written in
Notebooks. In Kaiaulu, we leverage both APIs and Notebooks, which is a common
practice in R packages, therefore avoiding abstraction debt.

###  2.4 Minimal Paths to Data

According to [16, p.233], the effort required to learn how infrastructure code
works has to be proportional to the gains and account for deprecation. We
agree with this observation. Let us look at how existing tools manage this
concern.

When using GrimoireLab components (in particular Perceval) the minimal path to
data is surprisingly short. Provided with a Git repository URL, or a local
copy, it will output a JSON file to stdout. Likewise, provided with a URL to a
website mbox or local file, it will also provide a JSON file to stdout. A
developer can easily integrate wrappers to its CLI, and users can easily
obtain data for a project of interest. In this ecosystem, a database is
available, but it is optional: users need not to concern themselves with
learning GrimoireLab’s Elastic Search database to obtain data.

This is in contrast to Codeface and SmartSHARK, both of which require user
familiarity with MySQL and MongoDB respectively, along with their data model
schemas to obtain the equivalent version control system and mailing list data.
The minimal path to data in these cases is much longer, including the setup
overhead and integration with other tools.

When data integration is sought in the database, GrimoireLab retains its
approach of keeping the data closest to source, and not harmonizing it in a
schema that facilitates integration [24]. Codeface’s MySQL and SmartSHARK’s
MongoDB provide a harmonized schema, which makes it easier for users to store
the various types of data.

In the case of PyDriller, which provides an API, the minimal path to data
requires familiarity with the Python programming language. This offers the
convenience of reshaping the data to the user’s final need, but adds an
overhead to the user for familiarization with the API, instead of just the raw
data schema from the source of interest (which the user is likely already
familiar with for their research purposes). One researcher [9, p.39] who
extended Codeface4Smells identified a problem of Pipeline Jungles [21], due to
heavy reliance on a folder hierarchy and file name conventions.

Our lessons learned here were: databases need not be a requirement to provide
users with various data sources. This also simplifies component reuse by other
tools and decreases the likelihood of reinventing the wheel. Providing a
minimal path does not exclude providing a database for researchers, as
Perceval shows. However providing a harmonized schema can save researchers
from having to re-implement code to integrate the same kinds of infrastructure
over and over. Lastly, providing an API gives some flexibility to users to
reshape the data with the tool. But user familiarity with the programming
language and API is a kind of overhead and this does not seem ideal, as the
data could be provided directly via a CLI leaving a task for the researcher to
adapt it in their own programming language. As such, we believe having
available a CLI that outputs the data as Perceval does, and a harmonized
schema as in Codeface and SmartSHARK, provides the best combination.

###  2.5 Other Design Decisions

We briefly mention here other (more minor) design decisions that we believe
may cause difficulties in adoption.

Familiar Software Abstractions. Both Perceval and PyDriller leverage a common
interface for end-users. They are both Python libraries, and provide the
expected interactions for CLI and API respectively. In Perceval’s CLI,
provided with a list of parameters and flags, data is output to stdout.
PyDriller exposes an API, an extension to a programmer’s familiar programming
paradigm. This is in contrast to ecosystems that define a different
abstraction, such as SmartSHARK, where detailed instructions must be followed
to extend its functionality
666https://smartshark.github.io/plugin/tutorial/python. Extension instructions
are also not available for Perceval or Codeface.

Licensing. Another important consideration in reusing a code component is how
permissive its license is. For example, stringr, an R package to manipulate
strings used by XGBoost, a popular machine learning algorithm, was replaced by
stringi, another R package to manipulate strings, solely based on the
difference in licenses.777https://github.com/dmlc/xgboost/issues/1338 Similar
reasoning also led an R package that represents data tables efficiently to
adopt a different license because the existing license “could be interpreted
as preventing closed-source products from using
data.table”888https://github.com/Rdatatable/data.table/pull/2456. Lack of
clarity on interactions of open source licenses has been reported by [1].
Among the tools we studied, we have observed the following licenses: Codeface
adopts GPL 2.0, PyDriller Apache 2.0, SmartSHARK Apache 2.0, and Grimoire’s
Lab GPL 3.0 and LGPL 3.0.

Perils of Code Reuse. With the availability of package managers such as CRAN
and PyPi which greatly facilitate code reuse, you can declare dependencies on
others’ code instead of copying it into your own project, taking advantage of
their functionality without assuming the burden of maintenance. However code
interdependence also poses risks [26], such as dependencies going extinct [6].
Hence, care has to be taken to avoid dependencies to non-maintained third-
party code.

An interesting example occurs in
mecoSHARK999https://github.com/smartshark/mecoSHARK through a chain of
dependencies which exemplifies the concern posed here. mecoSHARK is a
component that serves as a wrapper for
OpenStaticAnalyzer101010https://github.com/sed-inf-u-
szeged/OpenStaticAnalyzer, with a last commit date of July 13, 2018. In turn,
OpenStaticAnalyzer also wraps several other dependencies, including FindBugs
111111http://findbugs.sourceforge.net/, last released in March 15, 2015. In
its bug tracker121212https://sourceforge.net/p/findbugs/bugs/1487/, FindBugs
requests for bugs to no longer be reported, noting that
SpotBugs131313https://github.com/spotbugs/spotbugs, FindBugs’ successor,
should be used instead. This confirms that the mecoSHARK wrapper, which
provides OpenStaticAnalyzer functionality to SmartSHARK,is now dependent on
dead code, further increasing the burden of the SmartSHARK ecosystem
maintainers. Nonetheless, SmartSHARK’s approach to wrap black-box packages
into common APIs is considered good practice [21].

As a means to mitigate this risk, relying on and contributing work to open
source communities that more carefully assess the health of projects and try
to maintain them, such as the Apache Software Foundation,
ROpenSci141414https://ropensci.org/about/, and
CHAOSS151515https://chaoss.community/ may be an important consideration. For
example, ROpenSci accepts R packages via a streamlined peer review process
and, for accepted packages, provides community support, package promotion, and
fast-track publication to
journals161616https://devguide.ropensci.org/softwarereviewintro.html\\#whysubmit.

##  3 Design Principles in Kaiaulu

In this section, we discuss how our design principles are translated into
Kaiaulu’s specific design decisions. In the following section, we fully flesh
out Kaiaulu’s modules and features.

Batch Mode, Interactive Mode, and Literate Programming in Kaiaulu. We chose to
use the R language171717https://www.r-project.org/, due to the familiarity of
the authors with the language and a preference for its package architecture.

Minimally, the structure of an R package consists of the package metadata and
its API. In addition, the R ecosystem encourages and promotes best practices
to include documentation packages called vignettes, which leads R users to
expect an API and R Notebooks when installing packages from CRAN (The
Comprehensive R Archive
Network).181818https://cran.r-project.org/web/packages/ CRAN treats R
Notebooks as first class citizens in an R package191919See for example under
Vignettes: https://cran.r-project.org/web/packages/ggplot2/index.html showing
on each package’s website any R Notebooks available. Because of R package
structure, complying with familiar software abstractions (see Section 2.5)
automatically brings the benefits of literate programming (see Section 2.3).

Abstraction Debt in Kaiaulu. R natively supports tables and vectors as data
types, which is a familiar abstraction for data analysts. To capitalize on
this, Kaiaulu’s parse_ functions map most data sources (Git logs, mailing
lists, file dependencies, software vulnerability feeds, metrics, etc.) as
tables with standardized column naming, which allows for quick identification
of what data can be combined. Kaiaulu also offers various
transform_to_network_ functions to represent and interactively visualize these
networks202020https://github.com/sailuh/kaiaulu/blob/master/R/network.R which
in turn enable more complex socio-technical analyses at different
granularities: functions, files, classes, etc.

Tool Configuration Files vs Project Configuration Files in Kaiaulu. Following
the design choice of Codeface (see Section 2.2), and building on best
practices for machine learning configuration files [21] we implemented project
configuration files using YAML. Because we externalize all parameters in
project configuration files, an important concern is that the file does not
grow overly complex, requiring documentation of its own. That is, we do not
wish the minimal path to data to increase as new features are added, as we
discuss next.

Minimal Path to Data in Kaiaulu. As discussed in Section 2.4, it is important
that the path to data remains as simple and short as possible. We again build
upon familiar concepts, specifically with the intent of applying the rule of
least surprise [20, Ch.11]212121Also publicly available at:
http://www.catb.org/~esr/writings/taoup/html/ch11s01.html i.e. ‘do the least
surprising thing’. In an R package, it is expected that R Notebooks provide
examples of how to leverage the API to accomplish a task by combining multiple
functions, while individual functions provide self-contained examples, which
can be obtained in the R environment at any time by preceding a function name
with a question mark, e.g. ‘?parse_gitlog’.

To build upon this we: 1) Do not create any dependency between configuration
files and the API: functions take, as input, parameters which are familiar to
any programmer; 2) Use project configuration files only in the first code
block in R Notebooks to load the variables required to use the functions of
the API, similar to how best practices in static programming languages
encourage variable definitions at the beginning of a program; 3) Create a
dependency between the CLI and the project configuration files, to facilitate
batch processing and reproducibility.

Our intent is that users will first observe the R Notebooks to get a better
understanding of the API for a particular task of interest, and in doing so
will familiarize themselves with both the relevant portion of the API and the
project configuration file. If the interest is only, for example, to
understand how to parse Git logs, using for example the Git log R notebook,
then users should not be concerned with specifying the mailing list. When
comfortable, users can then use their newfound understanding to scale the
analysis to the entire project using the configuration file for the CLI, build
their own analyses as vignettes, or define new CLI interfaces. This design is
consistent with a mining software repositories workflow, in which a researcher
should first explore the data qualitatively to assess threats to validity,
before scaling up data processing in batch mode without clarity of what
assumptions the tool is making using default parameters or arbitrary
thresholds.

Kaiaulu also further decreases the minimal path to data in terms of how it
handles third party dependencies. Users need only concern themselves with
installing dependencies for their task of interest. For example, if the
interest is only to parse Git logs, they need only set up Perceval, and
provide its binary path as a parameter to Kaiaulu’s parse_gitlog to obtain the
parsed data. More generally, the parse_ API minimizes effort to researchers by
transforming various tool-specific data formats, if the researcher so desires,
into tables, and performing minimal processing on potentially inconsistent
fields, such as file paths, to make them internally consistent.

##  4 The Kaiaulu R Package

Based on the above observations and lessons learned, we now describe the
realized modules and features resulting from the design decisions behind the
Kaiaulu R package.

Mining software repositories often requires the handling of multiple data
sources to analyze a project’s ecosystem. Minimally, a researcher is required
to understand the data source in its native form, acquire it (typically using
an API), and parse and save it (e.g. as a table of data). Overheard is
incurred if a tool needs to be purpose-built to accomplish these steps. In the
best case, the acquisition and parsing steps can be accomplished by using an
existing tool. When designing Kaiaulu, we asked ourselves how to emphasize the
minimal path to data (as discussed in Sec. 2.4). To illustrate our rationale,
Figure 1 revisits some of the tools’ design decisions we discussed earlier.

Figure 1: Conceptual diagram of interface, input, and output of tools
showcasing differences in design.

In Figure 1, Perceval (left) provides a single CLI interface for acquisition
of various data sources. For example, a project’s issues can be fetched by
using the ‘jira’ endpoint, while ‘git’ may be used to parse repositories.
Pydriller (center-left), provides functions via a Python API. Users of these
tools gain flexibility in parsing the data, at the cost of a higher learning
curve and familiarity with the language. SmartShark (center-right) provides
similar functionality to Perceval, but endpoints such as ‘jira’ and ‘git’ are
now realized as entirely separate tools, orchestrated by another tool.
Finally, Codeface (right) provides a single CLI interface, like Perceval. But
most of its functionality is executed in batch mode and output into a single
database dump, offering the least flexibility in terms of what analyses to
execute. Unique to these tools, Codeface stores project parameters in reusable
configuration files.

Using Figure 1 as a basis for comparison, Kaiaulu’s design is shown in Figure
2, separated into parts 1) through 4). Kaiaulu borrows from the design of
PyDriller by defining an API and a set of functions (2). The use of
configuration files, inspired by Codeface (3), is done at the R Notebook level
(rather than at the function level). That is, project configuration parameters
are read into an R Notebook, and appropriate parameters are then passed to
functions. This allows us to decouple configurations from function signatures,
to tell best practice stories (interspersed with code) of how parameters are
used in various analyses [14, 5, 4], and offer a reusable end-to-end pipeline
for specific exploratory analysis. For example, the social smells
notebook222222https://github.com/sailuh/kaiaulu/blob/master/vignettes/social_smell_showcase.Rmd
emphasizes care in assessing project’s communication, which are often
fragmented over multiple archives. More importantly, R notebooks enable easy
manual inspection of intermediate data, such as the use of identity match
heuristics. We found this use of ‘reusable data stories’ particularly useful
to familiarize undergraduate and graduate research assistants to common
pitfalls.

We borrowed the use of CLIs (4) from Perceval and Smartshark. The CLI serves
to accommodate users who are unfamiliar with R; it also supports scaling
analyses defined and prototyped in Notebooks, so that they can be run in batch
mode. To build upon (2), the CLI simply utilizes the defined API behind the
scenes, which simplifies code maintenance. As with the Notebooks (3), the CLI
parses project configuration files, which also facilitate server-side reuse.

Kaiaulu was designed by combining these concepts from (1-4). It was
implemented as an R package, building upon familiar software abstractions.
Typically R packages are defined as an API of functions as found in PyDriller
and R Notebooks. By showcasing project configuration files where users are
expected to learn about the package, users can familiarize themselves with
project configuration files and the command line interface, which is less
commonly found in R packages, and entirely optional. In the following
subsections, Kaiaulu’s major processing elements are discussed.

Figure 2: Conceptual diagram of interface, input, and output of tools
showcasing how Kaiaulu coompares to the tools.

###  4.1 Parsers

In the Parsers module, our goal was to minimize a user’s effort, in terms of
acquiring and parsing project source code. These functions were combined into
a single interface with consistent nomenclature.

Each of Kaiaulu’s parsers is defined as a function (e.g. parse_mbox,
parse_gitlog), which are also accessible via a CLI. We wanted parsers in
Kaiaulu to reflect Perceval’s philosophy of minimal paths to data, with a
small learning curve. That is, given a data source, we would like users to
quickly be able to see the data without spending excessive time on setup. As
such, each parser function is given a single responsibility: to display a data
source as a table with a standardized column nomenclature (in case multiple
sources referred to the same data with different names). Unlike Perceval,
since an API option is also available, users can interactively prototype and
analyze the data in the R environment. Having tables as the default output
option minimizes the time spent learning what fields are available in the
source. The standardized nomenclature allows for intuitive joining operations
across the outputs of different parsers.

To account for the perils of code reuse, Kaiaulu limits its interface only to
third party software that have CLI interfaces. Parsers with third party
dependencies simply contain in their signatures an additional parameter for
path to the required binary. This dependency mechanism allows users to bypass
setting up third party tools which they do not directly need to use.
Additionally, users benefit from using the Kaiaulu function to obtain a
tabulated and standardized data input. For example, the
parse_gitlog(git_repo_path,perceval_path) function requires, as input,
Perceval’s binary to tabulate its JSON output. In this way parsers can build
upon third party functionality to implement new features. For example,
parse_gitlog_entity(git_repo_path,utags_path,project_git_log,kinds) implements
a git log parser capable of tabulating developer changes from git logs at the
granularity of functions rather than files (inspired by Joblin et al [12]). In
addition, the assumptions and threats to validity in the cited work are
provided in the Notebook
232323https://github.com/sailuh/kaiaulu/blob/master/vignettes/blamed_line_types_showcase.Rmd.

While simple in concept, we note that existing tools do not offer this
functionality. For instance, Codeface [12] offers an implementation of a
function-based git log parser, but since it has an ‘all-in-all-out’ interface,
this function can not be reused elsewhere. The same is true for SmartShark.
Perceval, while containing a shorter path to data, still requires tabulation
and standardization of the collected results. Lastly, PyDriller does not adopt
the philosophy used here for extending functionality based on third party
software, as it limits its scope to Git.

Kaiaulu currently employs a variety of parsers, providing the ability to parse
git logs, mailing list archives (e.g. pipermail, Apache’s mod_mbox), issue
trackers (e.g. Jira, GitHub), static parsers (file and function dependencies),
evolutionary parsers (file and function changes), commit hashes (e.g. to
identify issue ids from commit messages), and software vulnerability feeds.
Parsers which contain filepaths also contain optional regular expression
filters to whitelist or blacklist files based on their extensions or naming
conventions. For example, we use this to remove test files from analyses as
these files could compromise code metrics.

###  4.2 Transformers, Graphs and Networks

Kaiaulu’s Transformer, Graph, and Network modules are grounded on the
observation that most software and social metrics are graph-based (e.g. co-
change, fan-in, fan-out, communication). These modules transform the data
provided by various kinds of parsers that parse the raw project data.
Transformers reformat the data provided by parsers into lists of nodes and
edges which are then represented as networks, using graph data structures. In
this way we can more easily visualize and explore the networks of
relationships among a software project’s elements.

Kaiaulu represents the socio-technical network for each snapshot as a graph
$G_{st}=(V,E)$, where the set of nodes $V=V_{a}\cup V_{f}\cup V_{t}$ comprises
authors $V_{a}$, source files $V_{f}$ and e-mail threads $V_{t}$. The set of
edges E = $E_{comm}\cup E_{chg}$ models communication and collaboration
between authors, where communication is done via $E_{comm}\subseteq
V_{a}\times V_{t}$, and file changes via $E_{chg}\subseteq V_{a}\times V_{f}$.
Observe by this construction, the socio-technical network is in fact two bi-
modal bipartite networks $G_{st}=G_{chg}\cup G_{comm}$. Both $G_{chg}$ and
$G_{comm}$ are also weighted (representing an author’s count of changes to a
file within a user specified time window (e.g. 3 months), and the number of
replies submitted to an e-mail thread respectively), and undirected (the
direction is irrelevant in this case because it could only go in one direction
in each bipartite network). Likewise, the CVE and File Networks are weighted,
undirected, bipartite graphs. The definition of various transformations are
encapsulated separately in functions, consistent to the overall architecture.

Projection Transformations. Familiar software engineering metrics can be
derived from graph projections. Intuitively, a graph projection operation
eliminates one set of the ‘colored’ nodes in a bipartite graph, and connects
the adjacent black nodes together, where the resulting edge weight is the sum
of the eliminated edges. For instance, in a bipartite network represented by
file and commit nodes, eliminating the commit nodes would result in a file’s
co-change metric, revealing indirect collaboration. For example, if five
authors modified the same file within a given time period, then the projection
operation shows that all five authors indirectly collaborated, irrespective of
the order of their changes (note that the derived uni-modal networks are
undirected). We define this as the projection transformation to go from bi-
modal to uni-modal networks.

Temporal Transformations. Let us now consider a second approach to obtain uni-
modal networks. In [12], the authors define one method to construct uni-modal
networks from the same data by defining indirect collaboration using the
notion of incremental contributions through the timestamps on commits. For
example, if author A modifies a file, and very next change to the same file is
performed by author B, then B is said to have indirectly collaborated with A.
A similar intuition and transformation could be used to categorize e-mail
replies. We define this method, to go from the bipartite network to the uni-
modal network, a temporal transformation (as it relies on the timestamps).
Observe in this case that the derived uni-modal networks will be directed
graphs (which indicate the flow of time). The edge’s weight is defined as the
sum of lines of code added by both developers in their respective file
changes.

Figure 3: Temporal vs Projection Networks, format adapted from [13].

.

Which method to derive uni-modal networks should we choose? This decision is
encapsulated in Kaiaulu by the choice of functions. By swapping projection and
temporal transformation functions, users can experiment with, and visualize,
their various implications.

Figure 4: File vs Sub-file (e.g. Function) Networks, adapted from [13].

.

An example of both projection and temporal networks is shown in figure 5 (the
name of each node’s developer has been blurred). In the projection network,
developers are connected if in a given time window they modified any file in
common. In the temporal network, the direction displays which developers
changed files after which in the given time window. For example, a
bidirectional arrow means two developers change the same file together. A uni-
directional arrow suggests another developer may have “taken over” during that
time window. In both cases, we could derive hypothesis of the nature of
collaboration, and derive hypothesis to be tested in the exploratory analysis.

(a) CVE and File Network.

(b) Projection Network. Nodes indicate developers. Edges represent common
changed files.

Figure 5: Temporal Network. Nodes indicate developers. Edges represent common
changed files. The edge direction indicates the temporal order of change.

File, Functions, and Entities. Another consideration encapsulated in Kaiaulu’s
functions is which entities are analyzed to derive indirect collaboration. For
example, consider Figure 4. As we can see, the choice of granularity will also
affect the number of edges generated, where a file granularity generates more
edges than function granularity. A larger number of edges, in turn, may impact
the social smell metrics, as the existence of connections between developers
in one network, and the absence of edges in another network, may inflate the
count of metrics, such as social smells (which we define in the next section).
The authors in [12] used a combination of temporal transformation and function
granularity. In Kaiaulu, we implemented both the file granularity, and
generalized the function granularity to entities, where an sub-file unit can
be any source code block region of interest (e.g. functions, classes, or
language specific features like structs in C).

###  4.3 Identity

A critical component of conducting socio-technical analysis in open source
communities is assigning a consistent identity to users who may employ
multiple variants of their name and e-mail addresses in their project
interactions. Several approaches to match identities have been proposed (e.g.
[4], [27]). Exact name matching (either names or e-mails) or partial matching
(e.g. based on edit distance) are two commonly used schemes.

Our identity matching was designed as a 3 step pipeline: formatting, name-
email separation and pair-wise matching. Formatting includes the removal of
symbols such as ‘$<$ $>$’, commas or replacing ‘at’ with ‘@’, while avoiding
modifying a name, such as Matt. Name and e-mail separation handles cases where
first or last or both names are not provided, multiple word names, etc.
Finally, pair-wise matching handles comparisons of name and e-mail, or
reversed names. In total, the steps of formatting, name separation, and name
matching amounted to 31 test cases, which were successfully implemented. At
the end of this step, users in the version control system, issue tracker, and
mailing list who matched via the tests we implemented were assigned an
appropriate ID. Thus, given the name, and optionally the e-mail, other
information sources can be matched.

Figure 6: Reply networks combine communication networks. Here dark blue nodes
are issue comments, light blue nodes are mailing list comments, and black
nodes are developers. Red nodes are developers who communicate in both the
mailing list and the issue tracker.

An example of the utility of identity matching is shown in figure 6. Here,
project communication occurs in parallel in both the Jira issue tracker and
the project’s mailing list. We fuse these information sources into a single
“Reply Network”.

In summary, the transformer API provides users with flexibility with respect
to both temporal assumptions and sub-file granularity. Because all networks
are annotated graphs, community detection algorithms in Kaiaulu can be used to
identify important patterns. For example, if applied to a file-commit network
over a fixed period of time, co-changed file clusters can be identified.
Similarly, if the file network is derived from file to file dependencies,
clusters related to modularity measures can be derived. Developer networks can
be used to detect communities.

In figure 7, we apply community detection to a temporal network such as the
one illustrated in figure 5. The result is displayed by re-coloring the black
nodes. Darker blue and lighter blue nodes represent two communities of
developers as determined by the files that they changed in common. Developers
in black represent boundary nodes, which participate of both communities. In
the interactive format, researchers can “zoom in” on these nodes to identify
who are the common developers, and can use this information to draw further
hypotheses.

Figure 7: Community detection applied to temporal projection.

.

###  4.4 Metrics

In the metrics module, we define some commonly used metrics, such as number of
bugs, churn, LOC, as well as the less well-known social metrics. Demographics
are also provided to help contextualize the previously presented social
networks, such as the number of developers modifying files and exchanging
e-mails, number of files, threads, and different timezones242424For a full
analysis with the metrics, see:
http://itm0.shidler.hawaii.edu/kaiaulu/articles/social_smell_showcase.html.
The source code can be found on
https://github.com/sailuh/kaiaulu/blob/master/vignettes/social_smell_showcase.Rmd.

For bug counts, rather than simply interpreting these as metrics, we make it
easy for users to observe their topology. This is evident in figure 8, where a
single issue is associated with multiple files (top right). While some other
files may have a lower count of issues, more complex structure (such as we can
observe at the bottom left of the figure) would be missed if only metrics were
employed. Furthermore, feature issues can be discerned from bugs by examining
the issue labels.

Figure 8: Issue Network. Blue nodes represent issues, and yellow nodes files

.

We devote this section to briefly discuss the social metrics as they leverage
the previously discussed modules. For the social metrics, we adopt the
definitions of social smells defined by [23]. Social smells reflect recurring
sub-optimal organizational structure patterns connected to organizational
behavior patterns, e.g., sub-optimal knowledge sharing, recurrent sharing
delays, misguided collaboration and more. We chose to integrate three of these
smells—Organizational Silo, Missing Link and Radio Silence—and two related
metrics: socio-technical congruence and missing communicability [23]. Here we
explain one of the social smells; additional details about these metrics can
be found in [23].

Figure 9: Missing Link Social Smell [15]

In Figure 9, the collaboration network projection is shown in green to the
left. The communication network is shown in blue to the right. The intent
behind developer networks is to capture developers who modified the same file
in a given snapshot, and also communicated via a common e-mail thread in a
user-specified time window (e.g. 3 months). In this example, we can see to the
left highlighted in red that developers (A,B), (B,E), (B,G), and (D,G)
collaborated (i.e. they have a red edge in the green graph), but do not
communicate (they do not have an edge in the blue graph). Therefore, the
missing link social smell is counted 4 times for this snapshot.

###  4.5 Configuration

Currently, the configuration module is minimal and exists embedded in R
Notebooks. As noted at the start of this section, we borrowed from Codeface
the idea of project configuration files. And we include more parameters in the
configuration file as compared to Codeface. For example, Codeface hardcodes
the set of acceptable file extensions, whereas Kaiaulu defers this choice to
the user in the project configuration file.

More generally, we adopt the philosophy that project configuration files
should serve as a distilled representation of assumptions and analysis
choices. Rather than just serving as a repository of configuration choices for
reproducibility, it is readable as plain text, and so can be easily exchanged
and discussed. In Kaiaulu, a project configuration file is written in YAML. An
example of project configuration file can be found in the public tool
repository ‘conf‘
folder252525https://github.com/sailuh/kaiaulu/tree/master/conf.

As with every design decision in Kaiaulu, the full project configuration file
needs not be specified. Indeed, every R Notebook, at the beginning, clarifies
which parameters are required. In future work, we plan to expand the
Configuration module to tabulate multiple configuration files. For example, it
is often common in software engineering literature to analyze multiple
projects and present a summary statistics table of the projects to assess
generalization of results. Such tables could be generated on the fly from the
files. Ideally, project configuration files should suffice as supplementary
material, alongside Kaiaulu’s version for full reproducibility.

##  5 Conclusions and Future Work

In this paper, through an action research approach, we have determined a set
of key design decisions mined from existing tools. Based on these lessons
learned we iteratively developed Kaiaulu, an R package for mining software
repositories. Our goal in creating Kaiaulu was to simplify most of the boring,
repetitive, and error-prone tasks in mining software repositories, leaving the
user free to focus on the true goals of their research.

In Kaiaulu we have implemented and released a comprehensive set of
capabilities to mine, analyze, and visualize software repositories, including
social smells [23], architecture smells and metrics [17], and bug timelines
based on prior work by other authors. Kaiaulu is licensed under MPL 2.0.

While we have derived the principles for Kaiaulu from our action research, our
future work is to take a more disciplined approach to Kaiaulu’s design, based
on the quality attributes that represent its architectural drivers. Following
the approach outlined in [2] and [3] we can, in the future, attempt to more
systematically collect architectural drivers, make reasoned design decisions,
and support these decisions with well-established design rationale.

## References

  * [1] Almeida, D.A., Murphy, G.C., Wilson, G., Hoye, M.: Do software developers understand open source licenses? In: IEEE/ACM 25th International Conference on Program Comprehension. pp. 1–11 (2017)
  * [2] Bass, L., Clements, P., Kazman, R.: Software architecture in practice. Tech. rep. (2021)
  * [3] Bass, L., Clements, P., Kazman, R.: Software Architecture in Practice. Addison-Wesley, 4 edn. (2021)
  * [4] Bird, C., Gourley, A., Devanbu, P., Gertz, M., Swaminathan, A.: Mining email social networks. In: Proc. International Workshop on Mining Software Repositories. p. 137–143. ACM (2006)
  * [5] Bird, C., Rigby, P.C., Barr, E.T., Hamilton, D.J., German, D.M., Devanbu, P.: The promises and perils of mining git. In: 2009 6th IEEE International Working Conference on Mining Software Repositories. pp. 1–10 (2009). https://doi.org/10.1109/MSR.2009.5069475
  * [6] Coelho, J., Valente, M.: Why modern open source projects fail. In: Proc. 11th Joint Meeting on Foundations of Software Engineering. p. 186–196. ACM (2017)
  * [7] Dueñas, S., Cosentino, V., Robles, G., Gonzalez-Barahona, J.M.: Perceval: software project data at your will. In: Proc. 40th International Conference on Software Engineering: Companion Proceeedings. pp. 1–4. ACM (2018)
  * [8] Easterbrook, S., Singer, J., Storey, M.A., Damian, D.: Selecting Empirical Methods for Software Engineering Research. Springer (2008)
  * [9] Giarola, F.: Detecting code and community smells in open-source: an automated approach. Master’s thesis, Politecnico di Milano (2016)
  * [10] Gousios, G., Spinellis, D.: Alitheia core: An extensible software quality monitoring platform. In: 2009 IEEE 31st International Conference on Software Engineering. pp. 579–582 (2009)
  * [11] Joblin, M., Apel, S., Hunsen, C., Mauerer, W.: Classifying developers into core and peripheral: An empirical study on count and network metrics. In: IEEE/ACM 39th International Conference on Software Engineering. pp. 164–174 (2017)
  * [12] Joblin, M., Mauerer, W., Apel, S., Siegmund, J., Riehle, D.: From developer networks to verified communities: A fine-grained approach. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. vol. 1, pp. 563–573 (2015)
  * [13] Joblin, M.: Structural and Evolutionary Analysis of Developer Networks. Ph.D. thesis, Universitat Passau (2017)
  * [14] Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: The promises and perils of mining github. In: Proceedings of the 11th Working Conference on Mining Software Repositories. p. 92–101. MSR 2014, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2597073.2597074, https://doi.org/10.1145/2597073.2597074
  * [15] Magnoni, S.: An approach to measure Community Smells in software development communities. Ph.D. thesis, Politecnico Milano (2016)
  * [16] Menzies, T., Williams, L., Zimmermann, T.: Perspectives on Data Science for Software Engineering. Morgan Kaufmann Publishers Inc., 1st edn. (2016)
  * [17] Mo, R., Cai, Y., Kazman, R., Xiao, L., Feng, Q.: Architecture anti-patterns: Automatically detectable violations of design principles. IEEE Transactions on Software Engineering (2019)
  * [18] Moreno, D., Dueñas, S., Cosentino, V., Fernandez, M.A., Zerouali, A., Robles, G., Gonzalez-Barahona, J.M.: Sortinghat: Wizardry on software project members. In: IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings. pp. 51–54 (2019)
  * [19] Neubig, G., et al.: XNMT: The extensible neural machine translation toolkit. In: Conference of the Association for Machine Translation in the Americas Open Source Software Showcase (2018)
  * [20] Raymond, E.S.: The Art of UNIX Programming. Pearson Education (2003)
  * [21] Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Proc. 28th International Conference on Neural Information Processing Systems \- Volume 2. p. 2503–2511. NIPS’15, MIT Press, Cambridge, MA, USA (2015)
  * [22] Spadini, D., Aniche, M., Bacchelli, A.: PyDriller: Python framework for mining software repositories. In: Proc. 26th ACM Joint Proceedings of ESEC/FSE. pp. 908–911. ACM Press (2018)
  * [23] Tamburri, D., Palomba, F., Kazman, R.: Exploring community smells in open-source: An automated approach. IEEE Transactions on Software Engineering pp. 1–1 (02 2019)
  * [24] Trautsch, A., Trautsch, F., Herbold, S., Ledel, B., Grabowski, J.: The smartshark ecosystem for software repository mining. arXiv preprint arXiv:2001.01606 (2020)
  * [25] Trautsch, F., Herbold, S., Makedonski, P., Grabowski, J.: Addressing problems with replicability and validity of repository mining studies through a smart data platform. Empirical Software Engineering 23(2) (2018)
  * [26] Valiev, M., Vasilescu, B., Herbsleb, J.: Ecosystem-level determinants of sustained activity in open-source projects: A case study of the pypi ecosystem. In: Proc. 26th ACM Joint Proceedings of ESEC/FSE. p. 644–655. ACM (2018)
  * [27] Zhu, J., Wei, J.: An empirical study of multiple names and email addresses in oss version control repositories. In: IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). pp. 409–420 (2019)