# Fused attention mechanism-based ore sorting network ††thanks: Citation:

Junjiang Zhen,
Department of Mathematical and Information Sciences
Hebei University
baoding,China
<EMAIL_ADDRESS>
&Bojun Xie
Department of Mathematical and Information Sciences
Hebei University
baoding,China
<EMAIL_ADDRESS>

###### Abstract

Deep learning has had a significant impact on the identification and
classification of mineral resources, especially playing a key role in
efficiently and accurately identifying different minerals, which is important
for improving the efficiency and accuracy of mining. However, traditional ore
sorting methods often suffer from inefficiency and lack of accuracy,
especially in complex mineral environments. To address these challenges, this
study proposes a method called OreYOLO, which incorporates an attentional
mechanism and a multi-scale feature fusion strategy, based on ore data from
gold and sulfide ores. By introducing the progressive feature pyramid
structure into YOLOv5 and embedding the attention mechanism in the feature
extraction module, the detection performance and accuracy of the model are
greatly improved. In order to adapt to the diverse ore sorting scenarios and
the deployment requirements of edge devices, the network structure is designed
to be lightweight, which achieves a low number of parameters (3.458M) and
computational complexity (6.3GFLOPs) while maintaining high accuracy (99.3%
and 99.2%, respectively). In the experimental part, a target detection dataset
containing 6000 images of gold and sulfuric iron ore is constructed for gold
and sulfuric iron ore classification training, and several sets of comparison
experiments are set up, including the YOLO series, EfficientDet, Faster-RCNN,
and CenterNet, etc., and the experiments prove that OreYOLO outperforms the
commonly used high-performance object detection of these architectures.

_K_ eywords YOLOv5 $\cdot$ EMA $\cdot$ AFPN $\cdot$ Ore sorting

##  1 Introduction

In the current global context of structural changes in mineral resources, and
facing the challenge of depleting high-grade ore reserves, the mining industry
is inevitably shifting towards the development of more complex and lower-grade
ore resources. This transition not only imposes stricter requirements on
processing technologies but also presents new challenges in environmental
protection and economic cost management. Driven by this trend, the development
of efficient and sustainable ore sorting technologies has become an urgent
task in the mining industry.

The rapid development of deep learning technology has brought new
possibilities for ore classification and identification, especially through
image classification methods based on Convolutional Neural Networks (CNNs).
Due to their superior performance in analyzing and processing color texture
features, CNNs have demonstrated their potential in automated and efficient
ore identification tasks. Su et al. first applied LeNet-5 to a binary
classification task of coal and gangue, training it with 20,000 images to
achieve an accuracy of 95.88% [1]. Wang et al. utilized the Wu-VGG19 transfer
network for binary classification of scheelite ore and surrounding rocks,
addressing the problem of ineffective recognition of scheelite ore and waste
rock in scheelite ore image recognition methods [2]. Using the Inception-v3
network, Zhang et al. established a transfer learning model for mineral
micrograph images, achieving an accuracy of 90.9% in quartz and feldspar
micrograph images [3]. Liu et al. employed ResNet-50 to classify granite,
diorite, and gabbro, achieving excellent results with an accuracy of 95 [4].
These methods mostly focus on classification and subsequent localization
tasks.

With the evolution of technology, the design, deployment, and application of
high-performance visual models have become feasible even in complex working
conditions. Liu et al. proposed a novel lightweight network called LOSN
(Lightweight Ore Sorting Network), specifically designed for the
classification and localization of bituminous coal and anthracite. This method
is integrated with a robotic arm, achieving an end-to-end solution for mineral
sorting. The research team conducted detailed analysis of the LOSN model in
terms of model depth, architecture, and accuracy, providing comprehensive
reference for the application of deep learning in coal classification [5].
Zhao et al. introduced the K-means clustering algorithm and integrated the SE
attention mechanism into YOLOv4, achieving a remarkable average accuracy of
99% in bauxite ore sorting tasks after 7000 training iterations, while
maintaining inference speed faster than 0.05 seconds. This achievement not
only significantly improves the speed and accuracy of bauxite ore sorting, but
also greatly enhances the efficiency and precision of the bauxite ore industry
[6]. Zhang et al. proposed a brightness equalization algorithm to reduce the
impact of illumination intensity on recognition accuracy, and used YOLOv5 to
identify samples, ultimately achieving a 95.6% accuracy rate on 50 common
minerals [7]. Huang et al. utilized the YOLO-v3 model as a base detector,
initialized with pretrained weights on the MS COCO dataset. Subsequently, they
constructed an AL framework model embedded with YOLO-v3 for autonomous
detection of bauxite slurry [8]. Xiao et al. integrated the CECA attention
module into YOLOv4 for foreign object detection in gold ore veins, achieving
90.7% accuracy [9].

Closely following the strides of computing and imaging technologies,
intelligent ore sorting techniques have been widely deployed in the industrial
sector. Leveraging advanced optical systems to capture images and extract
feature information, these systems achieve precise classification under
diverse environmental conditions. They stand out for their advantages of low
cost, high efficiency, absence of radiation hazards, and ease of installation
[10]. Currently, these technologies primarily revolve around machine learning
and deep learning image processing techniques, such as decision trees
[11][12], naive Bayes [13], K-nearest neighbors [14][15], and support vector
machines (SVM) [16], which have found extensive applications in the field of
ore image classification. Nevertheless, machine learning-based methods often
rely on high-definition images and may lack stability in maintaining image
quality in dynamic environments. Additionally, their dependency on domain
expertise for feature extraction increases the practical application
threshold, thereby limiting their scope and deployment speed.

Therefore, this paper proposes a novel method for sorting gold and pyrite ores
by offline ore data collection and training using YOLOv5 network. Considering
the real-time requirements of the actual operation, the network is optimized
for lightweighting in the training process. In addition, to solve the problem
of insufficient feature extraction in the lightweight network, we embed the
scale attention (EMA) mechanism into the backbone network of YOLOv5, which
enables the network to extract richer features. Meanwhile, an efficient
asymptotic feature pyramid network (AFPN) structure is used to avoid losing or
degrading information during the feature fusion process and to retain more
effective features for fusion. In the experimental part, this study compares
OreYOLO with ten common deep learning-based target detection networks (e.g.,
YOLO series, EfficientDet series, Faster-RCNN, and CenterNet) in terms of
accuracy, model size, computational complexity, and inference speed. The
experimental results show that OreYOLO performs well on the gold and pyrite
datasets with an accuracy of 86.6%, which is 3.2% higher than the second place
Faster-RCNN (83.4%). In terms of computational complexity, model size, and
inference speed, OreYOLO has a relatively small number of parameters, lower
GFLOPs, and achieves higher FPS of 3.458M parameters, 6.3G GFLOPs, and
79.07FPS, respectively, which outperforms most target detection networks.

##  2  Details of YOLOv5 Network

As one of the most cutting-edge object detection algorithms currently, YOLOv5
is highly praised for its outstanding speed and accuracy performance. Its core
concept revolves around dividing the entire image into grids, allowing each
grid cell to predict the category and spatial location of objects it contains.
Subsequently, target boxes are filtered based on the Intersection over Union
(IoU) between predicted boxes and ground truth boxes, ultimately outputting
the category and location information for each target. As illustrated in
Figure 1, the network architecture of YOLOv5 can be divided into four key
components: the input layer, the backbone network, the neck network, and the
prediction layer.

At the input stage, the model’s input characteristics are optimized through
Mosaic data augmentation techniques and adaptive anchor box calculation. In
the backbone network section, the Cross Stage Partial Bottleneck (CSP)
structure is employed [17]. This structure divides the input feature map into
two paths: one path for direct transmission, while the other path undergoes a
series of convolution operations. Finally, the two paths are merged, reducing
computational costs and parameter count while ensuring efficient information
flow and feature extraction capability.

Figure 1: (a) The CBS block consists of a convolutional layer, a normalization
layer and a SiLu activation function. (b) The CSP3 block consists of a CBS
block with residual paths and a Bottleneck structure

In the neck section, a combination of Feature Pyramid Network (FPN) [18] and
Path Aggregation Network (PAN) [19] is applied, as illustrated in Figure 2.
The FPN layer passes rich semantic information along the top-down path,
achieving the transmission and fusion of high-level feature information
through upsampling operations, thereby providing enhanced feature maps for the
prediction process. Meanwhile, the PAN layer passes precise localization
features along the bottom-up path, facilitating the classification and
localization accuracy of the detection task.

The prediction layer adopts a multi-level feature fusion strategy. This
strategy first reduces the channel number and scales down the size of the
feature maps output by the backbone network through convolutional operations.
Then, it integrates feature maps from different levels to extract richer
feature information, thereby significantly improving the model’s detection
performance.

Figure 2: The structure of the Path Aggregation Network (PAN), consisting of
Bottom-up feature fusion and Top-down feature fusion,where 20×20, 40×40, and
80×80 are feature map sizes Figure 3: The network structure of YOLOv5 is
divided into four main parts, the Input part for image input, the Backbone
part for extracting ore features, the Neck part for fusing different layers of
features and the YOLO Head part for prediction

##  3  Methodology

This section will delve into three key aspects of constructing the OreYOLO ore
sorting network: implementing attention mechanisms to enhance feature
extraction capabilities, integrating progressive feature pyramid structures to
enhance feature representation capabilities, and defining OreYOLO’s loss
function.

###  3.1 Attention mechanism

In this study, an attention mechanism, Efficient Multi-Scale Attention (EMA
Attention) [20], is introduced to address the problem of the variability of
color texture among different ores, drawing on the human visual physiological
perception mechanism. This mechanism is based on the idea of CA Attention
[21], which adopts a parallel substructure to reduce the sequential processing
requirements and depth of the network, while dividing the input into multiple
sub-feature groups in the channel dimension to learn rich semantic
information, thus effectively improving the generalization ability and
recognition accuracy of the model.Efficient Multi-Scale Attention Structure As
shown in Fig. 3, it follows the parallel substructure in the CA block to avoid
more sequential processing of the network and larger depth and divides the
input into multiple sub-features along the channel dimension direction to
learn different semantics feature.

Figure 4: EMA structure diagram, which consists of three parallel paths to
extract the attention weight descriptors of the feature map group, where X Avg
POOL denotes global pooling in the horizontal direction and Y Avg POOL denotes
global pooling in the vertical direction

####  3.1.1 Feature Grouping

For any given input feature map $X\in\mathbb{R}^{C\times H\times W}$,The EMA
mechanism decomposes $X$ into $G$ sub-feature groups along the channel
dimension $X=\left[X_{0},X_{\boldsymbol{i}},\ldots,X_{G-1}\right]$ ,
$X_{\boldsymbol{i}}\in\mathbb{R}^{C\varparallel G\times H\times W}$. Each
group focuses on capturing different semantic levels, and this grouping method
enables the model to delve deeper into and utilize the intrinsic semantic
features of the input data, further enhancing the expressiveness of the
features.

####  3.1.2 Feature Grouping

EMA adopts three parallel paths to extract attention weight descriptors for
feature map groups, optimizing the overall representation of features. Among
them, two paths apply 1x1 and 3x3 convolutional kernels respectively, aiming
to encode channel information and capture local cross-channel interactions and
feature representations at different spatial scales.In addition, EMA encodes
channel information by performing 2D global average pooling operations in two
spatial directions and aggregates these encoded features. This not only
enhances cross-channel interactions but also achieves efficient feature
sharing without adding additional dimensions. This strategy adjusts the
importance of different channels while accurately preserving spatial
structural information, significantly improving the model’s recognition
accuracy and feature representation capability

####  3.1.3 Cross-spatial learning.

Cross-spatial learning significantly enhances feature representation
capability in computer vision tasks by establishing intrinsic connections
between channels and spatial positions. Specifically, EMA utilizes two
different convolution branches (1x1 and 3x3) to capture multi-scale features
and strengthens inter-channel dependencies through cross-channel interactions.
In this process, 2D global average pooling operations on two spatial
dimensions encode global spatial information from the output of the 1x1
convolution branch. Then, the global spatial information from the output of
the 1x1 branch is encoded, and before the joint activation mechanism of
channel features, the output of the minimal branch is directly transformed
into the corresponding dimensional shape $\mathbb{R}_{1}^{\ 1\times
C\varparallel G}\times\mathbb{R}_{3}^{\ C\varparallel G\times HW}$ the formula
for 2D global pooling operation is:

$\ \text{z}_{c}=\frac{1}{H\times W}\sum_{j}^{H}\sum_{i}^{W}x_{c}(i,j)\ $ (1)

Where, $H,W$ represent the height and width of the feature map, respectively,
$x_{c}\left(i,j\right)$ is the pixel value at position $\left(i,j\right)$on
the feature map at channel $c$

Through this cross-space learning approach, rich feature information can be
aggregated at different scales, generating two spatial attention maps. The
first spatial attention map is obtained by encoding global spatial information
from the output of the 1x1 convolution branch, while the second spatial
attention map is obtained by performing the same process on the output of the
3x3 convolution branch. These two attention maps preserve precise spatial
positional information during the aggregation process.Finally, by aggregating
these two spatial attention values and applying the Sigmoid function, the
model can emphasize pairwise relationships at the pixel level and highlight
the global context of all pixels, effectively enhancing the model’s attention
to higher-level features, thereby improving recognition accuracy and model
performance.

###  3.2 Progressive Feature Fusion

####  3.2.1 Asymptotic Architecture

In object detection tasks, multi-scale features are crucial for encoding
objects with scale variations. A common strategy for multi-scale feature
extraction is to use classical top-down and bottom-up feature pyramid
networks. However, these methods suffer from feature information loss or
degradation, impairing the fusion effect between non-adjacent
levels.Therefore, OreYOLO introduces the Asymptotic Feature Pyramid Network
for Object Detection (AFPN)[22]to support direct interaction between non-
adjacent levels.

The AFPN architecture, as shown in Figure 4, specifically, in the top-down
feature extraction process, first initiates the fusion process by combining
two low-level features with different resolutions, then gradually incorporates
high-level features into the fusion process, and finally merges the top-level
features of the backbone network, namely the most abstract features.This
fusion approach helps to avoid significant semantic gaps between adjacent
levels. In this process, the model learns a method to suppress inconsistency
by assimilating conflicting spatial filtering information, preserving valuable
information before combining it, automatically learning weight parameters. It
progressively integrates semantic and detailed information directly between
lower-level and higher-level features, avoiding information loss or
degradation during multilevel transmission. This enhances scale feature
invariance while incurring minimal computational overhead and ensuring
simplicity of implementation

####  3.2.2 Adaptive spatial fusion

AFPN utilizes ASFF (adaptively spatial feature fusion) [23] to assign
different spatial weights to features at different levels of the multilevel
feature fusion process in order to enhance the importance of the key levels
and to mitigate the effect of contradictory information from different
objects,as shown in Figure 5, $x_{ij}^{n\rightarrow l}$ denotes the feature
vector from level $n$to the location $l$ of level $\left(i,j\right)$,The
resultant feature vector obtained by adaptive spatial fusion of multilayer
features, denoted as $y_{ij}^{l}$, A linear combination of eigenvectors
$x_{ij}^{1\rightarrow l},x_{ij}^{2\rightarrow l}$ and $x_{ij}^{3\rightarrow
l}$ defined as follows:

$\ y_{ij}^{l}=\alpha_{ij}^{l}\cdot x_{ij}^{1\rightarrow l}+\beta_{ij}^{l}\cdot
x_{ij}^{2\rightarrow l}+\gamma\cdot x_{ij}^{3\rightarrow l}\ \ $ (2)

where $y_{ij}^{l}$ r fers to the mapping of $y^{l}$ in the output
$\left(i,j\right)$ feature map between
channels,$\alpha_{ij}^{l},\beta_{ij}^{l}$and $\gamma_{ij}^{l}$ represent the
spatial weights of the three hierarchical features at level $l$. These weights
are obtained by adaptive learning of the network, where it is enforced that
$\alpha_{ij}^{l}+\beta_{ij}^{l}+\gamma_{ij}^{l}=\ 1$ and
$\alpha_{ij}^{l},\beta_{ij}^{l},\gamma_{ij}^{l}\in\left[0,1\right]$, and
defined that

$\
\alpha_{ij}^{l}=\frac{e^{\lambda_{\alpha_{ij}}^{l}}}{e^{\lambda_{\alpha_{ij}}^{l}}+e^{\lambda_{\beta_{ij}}^{l}}+e^{\lambda_{\gamma_{ij}}^{l}}}\
$ (3)

Here $\alpha_{ij}^{l},\beta_{ij}^{l}$ and $\gamma_{ij}^{l}$ are defined by
softmax function where $\lambda_{\alpha_{ij}}^{l},\lambda_{\beta_{ij}}^{l}$
and $\lambda_{\gamma_{ij}}^{l}$ are used as control parameters and a 1×1
convolutional layer is used to compute the weight scalar maps
$\lambda_{\alpha}^{l},\lambda_{\beta}^{l}$ and $\lambda_{\gamma}^{l}$ from
$x^{1\rightarrow l},x^{2\rightarrow l}$ and $x^{3\rightarrow l}$ respectively.

In addition, an adaptive spatial fusion module specific to a particular number
of each stage is implemented in the AFPN. This approach allows the system to
tailor the fusion strategy to the specifics of each fusion stage to more
effectively deal with conflicting information between different layers and
enhance the impact of key features. With this targeted fusion approach, the
AFPN is able to better synthesize feature information from different levels in
order to improve the accuracy and efficiency of target detection.

Figure 5: The architecture of the Asymptotic Feature Pyramid Network
(AFPN).AFPN fuses two low-level features in the initial stage. Subsequent
stages fuse high-level features, while the final stage adds top-level features
during feature fusion. Black arrows indicate convolution and orange arrows
indicate adaptive spatial fusion Figure 6: Description of the adaptive spatial
feature fusion mechanism. The original up-sampling method is replaced with
adaptive fusion, where for each layer the features of all other layers are
adapted to the same shape by convolution and spatially fused based on a map of
learned weights

###  3.3  LOSS function

The loss function of OreYOLO (Equation 4) comprises three components: bounding
box regression loss, classification loss, and objectness loss.

$\mathcal{L}_{\text{OreYOLO}}(t_{p},t_{gt})=\sum_{k=0}^{K\times
K}\left[\alpha_{k}^{\text{balance}}\alpha_{\text{box}}\sum_{i=0}^{S^{2}}\sum_{j=0}^{B}\mathbb{I}_{kij}^{\text{obj}}\mathcal{L}_{\text{MPDIoU}}+\alpha_{\text{obj}}\sum_{i=0}^{S^{2}}\sum_{j=0}^{B}\mathbb{I}_{kij}^{\text{obj}}\mathcal{L}_{\text{obj}}+\alpha_{\text{cls}}\sum_{i=0}^{S^{2}}\sum_{j=0}^{B}\mathbb{I}_{kij}^{\text{obj}}\mathcal{L}_{\text{cls}}\right]$
(4)

The network divides each feature map into several cells, and each cell outputs
a vector of $\left[t_{x},t_{y},t_{w},t_{h},p_{0}\cdots\right]$ , where
$t_{x},t_{y}$ are used to calculate the offsets between the centers of the
prediction box and the corresponding anchor box, $t_{w},t_{h}$ is used to
calculate the width and height of the prediction box, and $p_{0}$ is the
probability that the cell contains the target.$K,S^{2},B$ represents the
output feature map, the number of cells into which the output feature map is
divided, and the number of anchors on each cell, respectively. $\alpha$
represents the weights of different components (such as the weights of
localization loss, confidence loss, and classification loss), which are used
to balance the weights of the output feature maps at each scale. The default
value of $\alpha_{k}^{balance}$ is corresponding to the output feature maps of
80 × 80, 40 × 40, and 20 × 20, respectively.

$\
d_{1}^{2}=\left(x_{1}^{prd}-x_{1}^{gt}\right)^{2}+\left(y_{1}^{prd}-y_{1}^{gt}\right)^{2}\
$ (5)

$\
d_{2}^{2}=\left(x_{2}^{prd}-x_{2}^{gt}\right)^{2}+\left(y_{2}^{prd}-y_{2}^{gt}\right)^{2}\
$ (6)

$\ MPDIoU=IoU-\frac{d_{1}^{2}}{h^{2}+w^{2}}-\frac{d_{2}^{2}}{h^{2}+w^{2}}\ $
(7)

$\ \mathcal{L}_{MPDIoU}=1-MPDIoU\ $ (8)

The bounding box regression loss adopts the MPDIoU [24] loss function
(Equations 5-8), which encompasses all relevant factors considered in existing
loss functions, namely overlap or non-overlap area, center point distance, as
well as deviations in width and height, while simplifying the computation
process. Here, $d_{1}^{2}$ and $d_{2}^{2}$ calculate the squared differences
between the center points of the predicted box ($pred$) and the ground truth
box ($gt$) in the $x$ and $y$ Ddirections, respectively, $h$ and $w$ are the
heights and widths of the bounding boxes, respectively.

$\ \sigma\left(x\right)=Sigmoid\left(x\right)=\frac{1}{1+e^{-x}}\ $ (9)

$\
\mathcal{L}_{cls}=-\frac{1}{N}\sum_{i=1}^{M}{\left[y_{i}\log\left(\hat{y}_{i}\right)+\left(1-y_{y}\right)\log\left(1-\hat{y}_{i}\right)\right]}\
$ (10)

where $C$ is the total number of categories, $y_{i}$ is the true label of the
$i$ category (1 if the target belongs to that category, 0 otherwise), and
$\hat{y}_{i}$ is the probability that the network predicts that the target
belongs to the $i$ category

$\
\mathcal{L}_{obj}=-\frac{1}{M}\sum_{j=1}^{M}{\left[c_{j}\log\left(\hat{c}_{j}\right)+\left(1-c_{j}\right)\log\left(1-\hat{c}_{j}\right)\right]}\
$ (11)

where,$M$ is the total number of predicted bounding boxes,$c_{j}$ is the true
confidence of the $j$ bounding box,$\hat{c}_{j}$ is the confidence that the
network predicts that the $j$ bounding box contains the target.

##  4 EXPERIMENTS

###  4.1 Image dataset preparation

The acquisition of high-quality image data is crucial in the process of
realizing the accurate classification task. In this study, a manual
acquisition strategy was adopted to collect the required images, and the
initial work encompassed the crushing of the construction site ores, which
were cleaned, screened and de-dusted by a conveyor belt, which effectively
minimized the negative impact of the ore surface dust on the image quality.
Two ore types were included in the sample set: sulfide iron ore and gold ore.
In the selection of image acquisition equipment, Hikvision industrial cameras
with high resolution were chosen for this study.The number of ore samples in
the acquisition process was varied, ranging from a single to ten, to truly
reflect the complexity of the industrial site. On this basis, the research
team meticulously maintained the randomness of the type, number and location
of the samples to increase the richness and realism of the dataset. Overall, a
total of 1,913 high-definition ore images were collected and accurately
labeled with the Labelimg tool, providing a solid data foundation for deep
learning model training.

####  4.1.1 Data augmentation methods

In the YOLOv5-based mineral recognition task, a broad and diverse training
dataset is essential to enhance the generalization and accuracy of the model.
Considering this need, this study adopts a series of image enhancement
techniques in the data preprocessing stage to ensure the extensiveness and
complexity of the dataset. Specifically, by randomly combining six strategies
such as noise introduction, image rotation, cropping, panning, reflection, and
brightness adjustment, this study realizes the augmentation and
diversification of the original image data. Each original image was augmented
four times in different combinations, which increased the final image data
volume to 6090 images, thus providing a richer and more varied learning base
for model training. Subsequently, these images are divided into training,
validation and testing sets in a 7:2:1 ratio to support model learning and
evaluation at various stages

###  4.2 Model setting details

The overall architecture of OreYOLO is shown in Fig. 6 and consists of five
parts: the input layer (Input), the backbone network (Backbone), the spatial
pooling pyramid of cross-domain partial connections (SPPFCSPC), the neck
network (Neck) and the prediction layer (YOLO Head). The backbone network
consists of three feature extraction blocks including the basic convolutional
module (CBM), the cross-domain partial module incorporating the attention
mechanism (CSP3 EMA), and the basic cross-domain partially connected
module(CSP3).Specifically, the CBM serves as the basic convolutional block of
the network consisting of a convolutional layer and a Mish activation
function, and we embedded the EMA in the CSPNet attention block, aiming at the
network’s ability to better fuse features of different channel dimensions when
performing feature extraction. In addition, considering that horizontal
feature fusion is also a preferred strategy to enhance the model’s
representational capability, we added the SPPFCSPC [25] module to OreYOLO to
implement horizontal feature fusion, which performs convolutional operations
on the model’s output feature maps through spatial pooling pyramids, and then
passes and fuses the features through the cross-domain partial connectivity as
a way of increasing the model’s receptive field and preserving the model’s the
original feature information of the output. The neck network employs a
progressive feature pyramid structure, in which ASFF weights features at
different levels to effectively fuse them, as described in detail in Section
3.2. The prediction part uses three multi-scale feature maps to detect objects
in the input image.

Figure 7: The structure of the SPPFCSPC modular,consisting of multiple
convolutions, pooling operations and a residual path Figure 8: The
architecture of OreYOLO,including five parts,input, backbone network CSP3 EMA
for feature extraction, SPPFCSPC for horizontal feature fusion, AFPN for
vertical feature fusion, and YOLO Head for prediction

###  4.3 Result analysis

In the result analysis, we build ten commonly used high-performance target
detection networks for comparison experiments. The comparison models include
one-level architectures such as Yolo V3 [26], Yolo V4 [27], YoloV5,
EfficientDet [28] and RetinaNet [29]. Two-tier architectures, such as Faster-
RCNN [30] and unanchored architectures, such as CenterNet [31] and Yolo X
[32], are modeled with the same details as presented in the corresponding
original papers.

The experimental results of OreYOLO and other comparative networks in the gold
and sulfide ore datasets are shown in Table 3. The mAP50-95 is used as one of
the main criteria to measure the model performance in this experiment as a way
to comprehensively evaluate the performance of the model under different IoU
threshold conditions, so as to verify its robustness and accuracy in the
target detection task. Specifically, OreYOLO can achieve 86.6% mAP50-95 in the
gold and sulfide ore datasets, which is 3.2% higher than the third best
Faster-RCNN (83.4%), while OreYOLO still has a clear advantage in detection
performance, e.g., compared with its peer series (YOLO X,YOLO v4,YOLO v3,
etc.). OreYOLO also performs better than the best YOLO X (79.6%), which is 7%
higher. In addition, OreYOLO differs from the best YOLO v5 (Small) by only
0.6% on mAP50-95 while maintaining a low number of parameters and a low
computational effort, which highlights the model’s remarkable ability to
discriminate between these two mineral types with high precision.

In addition, we also compared the model-related influencing factors such as
computational complexity, model size and inference speed in detail in the
comparison experiments, and the differences between OreYOLO and the other
models in terms of model parameters (#Param.), computational volumes (GFLOPs),
etc. are given in Table 4, and the results of the experiments show that
OreYOLO has relatively fewer parameters and lower GFLOPs, respectively, of
3.458M and 6.3G, proving its high performance in terms of model size and
computational complexity, whereas for other single-stage, two-stage, and
unanchored architectures, only EfficientDet-D0 (#Param. of 3.83M and GFLOPs of
4.614G) can compete with OreYOLO.In addition to the inference speed, OreYOLO
achieves the highest frame rate of 79.07, which is 5.77 higher than the best
one-stage architecture (Yolo V5, with a frame rate of 73.26), and 18.19 higher
than the no-anchor architecture (CenterNet, with a frame rate of 60.88).In
addition, when combined with the above results on the detection accuracy, it
is clear that OreYOLO strikes an excellent balance between detection accuracy
and efficiency, while the small number of parameters and low computational
complexity ensure its feasibility for deployment in most edge-complex
environments.

Table 1: Experimental parameter settings Parameter name |  Selected value
---|---
depth_multiple |  0.20
width_multiple |  0.25
Input shape |  640×640
Epoch |  100
Optimizer |  AdamW
learning rate |  1e-3
momentum |  0.937
Nms_iou |  0.45
label-smoothing |  0.005
Confidence |  0.25
Mixup probability |  0.5
Mosaic probability |  0.5
Table 2: Different performance of OreYOLO on gold and sulfide ore datasets) Type |  Precision |  Recall |  mAP50 |  mAP75 |  mAP50-95
---|---|---|---|---|---
Au |  99.3% |  99.3% |  99.4% |  99.4% |  85.7%
S |  99.2% |  99.2% |  99.4% |  99.1% |  87.5%
Table 3: comparison of the performance of OreYOLO with other classical target detection algorithms on the gold and sulfide ore datasets,including accuracy, recall, mAP50,mAP75,mAP50-95 (0.05 intervals) Series |  Type |  Edition |  mAP50 |  mAP75 |  mAP50-95
---|---|---|---|---|---
|  Yolo X |  Tiny |  98.2% |  97.6% |  79.6%
|  Yolo V5 |  Small |  99.3% |  99.1% |  87.4%
|  Yolo V4 |  Darknet 53 |  95.8% |  89.3% |  68.5%
One-stage |  Yolo V3 |  Darknet 53 |  97.1% |  88.4% |  69.1%
|  |  D0 |  97.5% |  95.6% |  77.0%
|  EfficientDet |  D1 |  96.0% |  84.7% |  67.6%
|  |  D2 |  96.3% |  85.2% |  68.0%
|  RetinaNet |  Resnet 50 |  99.2% |  98.0% |  82.2%
Two-stage |  Faster-RCNN |  Resnet 50 |  98.8% |  97.6% |  83.4%
Anchor-free |  Yolo X |  Tiny |  98.2% |  97.6% |  79.6%
|  CenterNet |  Resnet 50 |  98.2% |  95.8% |  79.6%
Proposed |  |  CSP_EMA |  99.3% |  99.2% |  86.6%
Table 4: Comparison between OreYOLO and other classical target detection algorithms on #Param,GFLOPs,FPS,size is the size of the network input image size Series |  Type |  Edition |  #Param |  GFLOPs |  Size |  FPS
---|---|---|---|---|---|---
|  Yolo X |  Tiny |  8.94M |  26.76G |  640 |  47.68
|  Yolo V5 |  Small |  7.074M |  16.411G |  640 |  73.26
|  Yolo V4 |  Darknet 53 |  63.959M |  59.787G |  416 |  17.06
One-stage |  Yolo V3 |  Darknet 53 |  61.545M |  155.549G |  640 |  38.06
|  |  D0 |  3.838M |  4.614G |  512 |  15.70
|  EfficientDet |  D1 |  6.558M |  11.219G |  650 |  14.63
|  |  D2 |  8.010M |  20.132G |  768 |  12.41
|  RetinaNet |  Resnet 50 |  36.413M |  146.311G |  600 |  36.57
Two-stage |  Faster-RCNN |  Resnet 50 |  136.771M |  370.013G |  600 |  21.44
Anchor-free |  Yolo X |  Tiny |  8.94M |  26.76G |  640 |  47.68
|  CenterNet |  Resnet 50 |  32.665M |  69.976G |  512 |  60.88
Proposed |  |  CSP_EMA |  3.458M |  6.3G |  640 |  79.07

###  4.4 Ablation experiment

In the ablation experiments, we first compared in detail the effects of the
EMA attention mechanism and the asymptotic feature pyramid network on OreYOLO
in terms of detection performance, which were mainly evaluated in terms of
mAP50-95 and the number of modeled parameters, as shown in Table 5. The
addition of EMA and AFPN improves the mAP50-95 of OreYOLO by 1.3% and 0.8%,
respectively, compared to the base network, while these improvements only
increase the number of parameters by a small amount (0.039M and 0.138M)
compared to the baseline network.

Table 6 compares the experimental results of OreYOLO with different attention
mechanisms. We compare in detail the differences between CA, ECA, CBAM, and
EMA attention mechanisms in terms of the number of parameters and accuracy.
Specifically, CBAM and EMA improved the mAP50-95 of OreYOLO by 0.3% and 0.8%,
respectively, and only slightly increased the number of parameters, whereas
the performance of CA and ECA decreased slightly (by 0.1% and 0.2%,
respectively). These results show that the EMA attention mechanism has better
feasibility than other attention mechanisms in our network. In addition, we
also compare the effects of different levels of feature fusion strategies on
the performance of OreYOLO, which are detailed in Tables 5 and 7. From the
experimental results in Table 5, it can be seen that SPPCSPC achieves a
similar effect to AFPN in terms of improving the accuracy (mAP50-95 improves
by 0.7%, while AFPN improves by 0.8%). Table 7 further compares the
differences between SPPFCSPC and SPPF modules in terms of accuracy, number of
parameters, and computational effort. The results show that although SPPFCSPC
performs well in improving accuracy (1% improvement in mAP50-95), it brings
more number of parameters and computation compared to SPPF.

Taken together, the results of the ablation experiments show the important
impact of the attention mechanism and feature fusion strategy on the model
performance. These findings provide a useful reference for the optimization of
lightweight target detection networks, helping to select appropriate
improvement strategies to strike a balance between accuracy improvement and
parameter efficiency.

Table 5: Influence of EMA,AFPNM,SPPFCSPC modules on model performance |  EMA |  AFPN |  SPPFCSPC |  mAP50-95 |  #Param
---|---|---|---|---|---
Base |  |  |  |  84.8% |  1.710M
Base |  |  |  ✓ |  85.5% |  3.317M
Base |  |  ✓ |  |  85.6% |  1.848M
Base |  ✓ |  |  |  86.1% |  1.749M
Base |  ✓ |  ✓ |  ✓ |  86.6% |  3.458M
Table 6: Influence of different attention mechanisms on model performance. | CA | ECA | CBAM | EMA | mAP50-95 | #Param
---|---|---|---|---|---|---
Base(AFPN+SPPFCSPC) |  |  |  |  | 85.8% | 3.455M
Base(AFPN+SPPFCSPC) | ✓ |  |  |  | 85.6% | 3.458M
Base(AFPN+SPPFCSPC) |  | ✓ |  |  | 85.7% | 3.455M
Base(AFPN+SPPFCSPC) |  |  | ✓ |  | 86.1% | 3.456M
Base(AFPN+SPPFCSPC) |  |  |  | ✓ | 86.6% | 3.458M
Table 7: Influence of SPPF,SPPFCSPC Modules on Model Performance. |  SPPF |  SPPFCSPC |  mAP50-95 |  #Param |  GFLOPs
---|---|---|---|---|---
Base(AFPN+EMA) |  ✓ |  |  85.6% |  1.851M |  5.0G
Base(AFPN+EMA) |  |  ✓ |  86.6% |  3.458M |  6.3G

##  5 Detection performance visualization

The visualization results demonstrate the actual results of OreYOLO under the
ore collection platform, as shown in Figure 9. In this scenario, OreYOLO is
able to effectively differentiate between gold and sulfide ores, and the
predicted bounding box precisely fits the contours of the ores, demonstrating
its ability to accurately localize the ores. In addition, OreYOLO is able to
accurately distinguish between difficult to distinguish ores (as shown by the
yellow box in Figure 9) due to its excellent detection performance. The
confidence parameter of the predicted object is also shown in the prediction
box. From the visualization results, it can be observed that OreYOLO not only
achieves accurate recognition in the face of difficult-to-distinguish ore
blocks, but also has a high confidence level, fully demonstrating its
effectiveness. Therefore, the above visualization results indicate that the
attention mechanism, progressive feature fusion and horizontal feature fusion
in OreYOLO together significantly enhance the feature extraction capability of
the whole network.

Figure 9: Experimental data captured by an industrial camera, with the input
image source on the left and the visualization results on the right

##  6 Conclusion

While OreYOLO has demonstrated excellent performance on gold and sulphide ore
data and can be successfully deployed to run in our facility environment,
there are still some issues that need to be optimized.

First, deep learning-based ore sorting methods are very dependent on large
amounts of clean data. Although we have performed extensive data augmentation
to increase the diversity of our training samples, a significant amount of
time and effort is still required to collect and prepare this data. This may
involve collecting more samples in the field and ensuring the quality and
diversity of the data to improve the accuracy and generalizability of the
model.Second, the generalizability and robustness of the model needs to be
continuously optimized and improved to ensure reliability and stability in
real-world scenarios. This may require adjusting the model architecture,
optimizing hyperparameters, and exploring more effective training strategies
so that the model can better adapt to different operating conditions and ore
types.Another challenge is model deployment and maintenance. Although OreYOLO
performs well in a device environment, successfully deploying a deep learning
model into a real device is not a simple task. Device resource constraints,
the inference speed of the model, and its stability and reliability in real-
world operation need to be considered. This may require optimizing the
inference efficiency of the model, using lightweight models or model
compression techniques to ensure efficient operation and stability of the
model on the device.

Therefore, we need to continue our efforts to optimize the data collection and
preparation process, as well as to continuously improve the training strategy
and architecture of the models, as well as to enhance the stability and
efficiency of the models during the deployment and operation phases.

## References

  * [1] Lingling Su, Xiangang Cao, Hongwei Ma, and Ying Li.  Research on coal gangue identification by using convolutional neural network.  In 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pages 810–814. IEEE, 2018.
  * [2] Wang Liguan, Chen Sijia, Jia Mingtao, and T Siyu.  Beneficiation method of wolframite image recognition based on deep learning.  Chin. J. Nonferrous Met., 30(5):1192–1201, 2020.
  * [3] Ye Zhang, Mingchao Li, Shuai Han, Qiubing Ren, and Jonathan Shi.  Intelligent identification for rock-mineral microscopic images using ensemble machine learning algorithms.  Sensors, 19(18):3914, 2019.
  * [4] Yang Liu, Zelin Zhang, Xiang Liu, Wang Lei, and Xuhui Xia.  Deep learning based mineral image classification combined with visual attention mechanism.  IEEE Access, 9:98091–98109, 2021.
  * [5] Yang Liu, Xueyi Wang, Zelin Zhang, and Fang Deng.  Losn: lightweight ore sorting networks for edge device environment.  Engineering Applications of Artificial Intelligence, 123:106191, 2023.
  * [6] Pengfei Zhao, Zhengjie Luo, Jiansu Li, Yujun Liu, and Baocheng Zhang.  Machine learning sorting method of bauxite based on se-enhanced network.  Applied Sciences, 12(14):7178, 2022.
  * [7] Junyu Zhang, Qi Gao, Hailin Luo, and Teng Long.  Mineral identification based on deep learning using image luminance equalization.  Applied Sciences, 12(14):7055, 2022.
  * [8] Zhijian Huang, Fangmin Li, Xidao Luan, and Zuowei Cai.  A weakly supervised method for mud detection in ores based on deep active learning.  Mathematical problems in engineering, 2020:1–10, 2020.
  * [9] Dong Xiao, Panpan Liu, Jichun Wang, Zhengmin Gu, and Hang Yu.  Mining belt foreign body detection method based on yolov4_geca model.  Scientific Reports, 13(1):8881, 2023.
  * [10] Xianping Luo, Kunzhong He, Yan Zhang, Pengyu He, and Yongbing Zhang.  A review of intelligent ore sorting technology and equipment development.  International Journal of Minerals, Metallurgy and Materials, 29(9):1647–1655, 2022.
  * [11] Leo Breiman.  Bagging predictors.  Machine learning, 24:123–140, 1996.
  * [12] Leo Breiman.  Classification and regression trees.  Routledge, 2017.
  * [13] Qiong Wang, George M Garrity, James M Tiedje, and James R Cole.  Naive bayesian classifier for rapid assignment of rrna sequences into the new bacterial taxonomy.  Applied and environmental microbiology, 73(16):5261–5267, 2007.
  * [14] Chengzhao Liu, Mingchao Li, Ye Zhang, Shuai Han, and Yueqin Zhu.  An enhanced rock mineral recognition method integrating a deep learning model and clustering algorithm.  Minerals, 9(9):516, 2019.
  * [15] Thomas Cover and Peter Hart.  Nearest neighbor pattern classification.  IEEE transactions on information theory, 13(1):21–27, 1967.
  * [16] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  Machine learning, 20:273–297, 1995.
  * [17] Chien-Yao Wang, Hong-Yuan Mark Liao, Yueh-Hua Wu, Ping-Yang Chen, Jun-Wei Hsieh, and I-Hau Yeh.  Cspnet: A new backbone that can enhance learning capability of cnn.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 390–391, 2020.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [19] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8759–8768, 2018.
  * [20] Daliang Ouyang, Su He, Guozhong Zhang, Mingzhu Luo, Huaiyong Guo, Jian Zhan, and Zhijie Huang.  Efficient multi-scale attention module with cross-spatial learning.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [21] Qibin Hou, Daquan Zhou, and Jiashi Feng.  Coordinate attention for efficient mobile network design.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13713–13722, 2021.
  * [22] Guoyu Yang, Jie Lei, Zhikuan Zhu, Siyu Cheng, Zunlei Feng, and Ronghua Liang.  Afpn: asymptotic feature pyramid network for object detection.  In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2184–2189. IEEE, 2023.
  * [23] Songtao Liu, Di Huang, and Yunhong Wang.  Learning spatial fusion for single-shot object detection.  arXiv preprint arXiv:1911.09516, 2019.
  * [24] Ma Siliang and Xu Yong.  Mpdiou: a loss for efficient and accurate bounding box regression.  arXiv preprint arXiv:2307.07662, 2023.
  * [25] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7464–7475, 2023.
  * [26] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [27] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [29] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [31] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019.
  * [32] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun.  Yolox: Exceeding yolo series in 2021.  arXiv preprint arXiv:2107.08430, 2021.


∎

11institutetext: Abdulkadir Şeker 22institutetext: Sivas Cumhuriyet University
22email<EMAIL_ADDRESS>33institutetext: Banu Diri 44institutetext:
Yıldız Technical University 44email<EMAIL_ADDRESS>55institutetext: Halil
Arslan 66institutetext: Sivas Cumhuriyet University
66email<EMAIL_ADDRESS>
# New developer metrics:

Are comments as crucial as code contributions?

Abdulkadir Şeker  Banu Diri  Halil Arslan

(Received: date / Accepted: date)

###### Abstract

Open-source code development has become widespread in recent years. As a
result, open-source software platforms have also become popular, and millions
of developers from diverse locations are able to contribute to the same
projects. On these platforms, various knowledge about them is obtained from
user activity. This information is used in the form of developer metrics to
solve a variety of challenges. In this study, we proposed new developer
metrics, including commenting and issue-related activity, that require less
information. We concluded that commenting on any feature of a project can be
equally as valuable as code contribution. In addition, besides the
quantitative ones, metrics based on only the existence of the activity have
been shown to offer also considerable results. We saw that issues were crucial
in identifying user contributions. Even if a developer makes a contribution to
only one issue on a project, the relation between the developer and the
project is tight. The hit scores are relatively lower because of the sparsity
problem of our dataset; even so, we believe that we have presented improvable
and remarkable new developer metrics.

###### Keywords:

developer metric OSS GitHub software engineering

##  1 Introduction

Thanks to the increasing capabilities of open source software development
tools, the number of open source users and projects is growing each year.
These platforms include millions of developers, each of whom has a different
character and skill set, as well as a wide variety of projects that offer
solutions to different problems. In environments with such a large amount of
data, it is also difficult for developers to find similar products to their
own, discern projects of interest, and reach projects to which they can
contribute. As developers primarily use search engines or in-platform search
menus to find projects, the constraints of text-based search McMillan2012  and
challenges related to finding the correct keywords also cause some projects to
be missed Hu2015 . While various project recommendation systems are being
developed to overcome this problem, projects must be rated by users for
recommendation models to work properly. In the same way that viewers give
ratings to movies that they have watched, developers need to rate the projects
in which they are interested. However, this is not currently the case on
(open-source) software development platforms. Several software and developer
metrics are used to calculate the score that a user gives projects, which are
extracted from the activity or features of both developers and projects.

Developer metrics which are used in many challenges include the number of
lines of code, developers’ degrees of connection to one another, past
experience, or common features (nationality, location, occupation, gender,
previously used programming languages, etc.). These metrics offer solutions to
different problems within open source software development and distributed
coding, including automatic assignments (task, issue, bug, or reviewer)
DeLima2015 ; Badashian2015 ; Junior2018 , project recommendation systems
Zhang2014 ; Sun2018 , software defect detection OzcanKini2018 , etc.

In this study, new developer metrics are presented to be used for different
problems. We developed a project recommendation system for the evaluation of
metrics and obtained remarkable results. The recommendation system was
developed based on a dataset consisting of data from GitHub. Most GitHub users
are familiar with (i.e., contribute to or interest) relatively few projects
hosted on the platform. Because of this, a critical sparsity problem has
occurred. To address the problem, we selected a dataset with a high users-
projects ratio.

The structure of this paper is as follows. In the background section, we
describe the literature on previously proposed metrics and project
recommendation models for open-source software development platforms. In the
research design section, the dataset used in this study is introduced and the
proposed metrics are detailed. In the final section, the proposed metrics are
assessed in terms of their accuracy rates in the project recommendation
system.

##  2 Background

Pull request (PR) allows users to inform others about changes they have pushed
to a branch in a repository on GitHub. PRs are a key feature in contributing
code by different developers to a single project Gousios2014 . The proposed
metrics related to this feature are used to solve different PR problems. PRs
need to be reviewed by a reviewer to merge projects. If the result of a
revision is positive, the PR is integrated into the master branch. Finding the
correct reviewer is an important parameter for ensuring rapid and fair PR
revisions. In this context, different metrics have been used to address the
problem of automatic PR reviewer assignment. Existing literature has proposed
various metrics to solve this problem, such as PR acceptance rate within a
project, active developers on a project Junior2018 , PR file location
Thongtanunam2015 , pull requesters’ social attributes Tsay2014 , and textual
features of the PR Yu2014 , among others.

Closing a PR with an issue, PR age, and mentioning (@) a user in the PR
comments have all been used to determine the priority of a PR VanDerVeen2015 .
Cosentino proposed three developer metrics (community composition, acceptance
rates and become a collaborator) to investigate project openness and stated
that project owners could evaluate the attractiveness of their projects using
these metrics Cosentino2014 .

Developer metrics are also used in the detection of software defects. In one
study, defects were estimated using different metrics grouped by file and
commit level. The number of files belonging to a commit, the most modified
file of all files in a commit, the time between the first and last commits to
a file, and the experience of a given developer on a committed file were
identified as important metrics OzcanKini2018 .

Reliability metrics are used to quantitatively express the reliability of a
software product Kaur2014 . To measure reliability in open source projects,
metrics such as number of contributors, number of commits, number of rows in
commits, and certain metrics derived from them are used. Tiwari et al.
proposed two important metrics for reliability: contributors and number of
commits per 1000 code lines Tiwari2012 .

Code ownership metrics are also important for open source software. One study
used the modified (touched) number of files to rank developer ownership
according to code contributions Bird2011 . In another study, the number of
changed lines in a file (churn) was used to address this problem Munson1998 .
Foucault confirmed the relationship among these code ownership metrics and
software quality Foucault2015 .

Recommender systems are an important research topic in software engineering
Happel2008 ; Robillard2010 . In ordinary recommendation models, previously
known user-item matrices are used; in other words, the rating given by a user
for an item is known. In this state, the essential research topic is on
estimating with different algorithms and models the rating that the user has
already given Sharma2013 . However, the point in question is different on open
source software platforms. Considering the developer as the user and the
project (repository) as the item, the rating given by a developer to a project
is unknown. In this context, the first problem that must be solved is how to
create an accurate developer-project matrix. At this point, different
developer metrics come into play.

GitHub is the world’s largest code server, hosting more than 40 million
repositories to which over 100 million developers have
contributed111https://en.wikipedia.org/wiki/GitHub. As such, GitHub is a
reasonable choice for developing a project recommendation model. Sun et al.
relied on basic user activity to develop a model using GitHub data.
Specifically, when rating a project for a developer, they used like-star-
create activities related to projects Sun2018 . We also used this scoring
method to compare with our metrics in this paper. In another study, a
developer’s social connections, programming with a common language, and
contributions to the same projects or files were used as metrics
Casalnuovo2015 . Liu designed a neural network–based recommendation system
which used metrics such as working at the same company, previous collaboration
with the project owner, and different time-related features of a project
Liu2018 . In a study aiming to predict whether a user would join a project in
the future, the metrics used included a developer’s GitHub age (i.e., when
their account was opened), the number of projects that they had joined, the
programming languages of their commits, how many times a project was starred,
the number of developers that joined a project, and the number of commits to a
project Nielek2017 .

##  3 Research Design

###  3.1 Dataset

One of the most serious challenges in developing a recommender system is
sparsity Niu2016 , a problem that occurs when most users rate only a few items
Guo2012 . This issue is present on GitHub because it is not possible for
developers to be aware of the millions of repositories on the platform. In the
studies mentioned in the previous section, we observed that limited (less
sparse) data were used, which is contrary to the nature of the GitHub
environment. Although the results obtained in these studies appear successful,
the question remains how successful the proposed algorithms will be on real
platform data. In light of this, a sub-dataset reflective of the sparsity
problem inherent to GitHub was used in this study. The dataset contained all
data related to 100 developers and 41,280 projects Seker2020 . The creators of
the dataset indicated that they selected the most active users on the
platform. They then extracted all related data for these users from GitHub
(commits, issues, pull requests, comments about these activities, watchers,
etc.). Thus, we anticipated that the recommender system we developed would
produce results parallel to those for the larger dataset of the platform as a
whole. In this regard, although our evaluation results seem weak compared to
similar studies, we believe that the proposed metrics are worthy of
consideration.

###  3.2 Project Recommendation System

Designing a project recommender system for open source software development
platforms includes two stages. First, the project-developer rating matrix is
generated using specific metrics. Second, the top-k projects are recommended
to each developer. Finally, the accuracy of the suggestions is evaluated. In
this study, the recommendation model was designed as follows:

  1. 1.

Different developer metrics were used to obtain the score matrix. The values
of the features (metrics) were scaled from 0 to 10. As in the movie-user
model, each developer will thus have given a rating (0–10) to each project.

  2. 2.

The similarity between unknown projects222We assumed that “unknown” projects
were those to which developers had no relationship and had made no
contributions and rated projects was used to calculate the rating of unknown
projects Sun2018 . The similarity value between the projects was calculated
using cosine similarity. When calculating the rating of an unknown one, the
dot product of the similarity values between the projects that the user rated
and the unknown project was used (Equation 1). An example scenario involving
this calculation is presented in Figure 1.

$unknown_{\\_}{rating}=\sum_{\\_}{n=0}^{i}known_{\\_}{rating}^{i}*similarity_{\\_}{known^{i},unknown}$
(1)

  3. 3.

The top 5 highest-rated projects among the unknown projects were recommended
to each developer.

  4. 4.

The accuracy of the recommendations was evaluated.

Figure 1: Calculating unrated project with the help of similarity rated
projects.

###  3.3 Evaluation Techniques

While recommending projects to developers, there should be a ground truth for
evaluating the proposed projects. Unlike ordinary recommender systems, there
is an unsupervised model. The evaluation criteria for some studies related to
this subject are set forth below.

  * •

A user-project-rating matrix was split randomly into test and training
subsets. Accuracy or recall scores were then calculated from the intersection
between the top-n scores of the test and training subsets Sun2018 . However,
another study stated that this method should not be used on platforms like
GitHub where time is an important parameter, pointing out that the problem of
predicting past activity with future data will occur when using k-fold cross-
validation by dividing the data randomly Junior2018 .

  * •

In another study, the recommended projects accuracy was evaluated using the
developer’s past commits to the related project. A recommendation was assumed
to be correct if the number of commits belonging to a certain developer on the
project was over a certain value. The average number of commits per project
was set as the threshold value in the dataset Liu2018 .

  * •

In a study predicting whether a developer would join a project in the future,
the dataset was split into two different sets by time. In this way, the
predicted result was verified with actual future data Nielek2017 .

In this study, GitHub’s “watching” feature was used as the ground truth.
GitHub users can follow, or “watch,” projects whose developments they want to
monitor Sheoran2014 . If a developer is watching a project, this indicates
that he/she is interested in the project. Thus, “watching” can be considered
as a real evaluation criterion. In our model, the top-n projects were
recommended to each developer. If the recommended projects were among the
developer’s watched projects, the projects were considered a hit333In other
words, it is a correct recommendation. The case of a developer watching fewer
than n projects was taken into account in the score equation 2.

$Hit_{\\_}{score}=\begin{cases}100*\frac{hit_{\\_}{fullname}+(hit_{\\_}{owner}*0.5)}{n},&\text{if
}num_{\\_}{watched}\geq n\\\ \\\
100*\frac{hit_{\\_}{fullname}+(hit_{\\_}{owner}*0.5)}{num_{\\_}{watched}},&\text{otherwise}\end{cases}$
(2)

The full name of a GitHub repository (project) is created by concatenating the
owner’s username with the repository name. In analyzing our results, we
noticed that the model recommended a project to a developer that only hit the
owner’s name—that is, the model found an incorrect project by the correct
owner. We evaluated this proposal has half the correct score, as recommending
the correct owner to a developer will allow the developer to access the
owner’s other projects.

An example scenario demonstrating this situation is given in Table 1. The
projects recommended for Alice are listed in the first column. Since four of
them are among the projects that Alice watches, the initial score is 4. In
addition, there are two projects by a developer named ”fengmk2” among Alice’s
watched projects (“fengmk2/parameter” and “fengmk2/cnpmjs.org”). For the
fourth proposed project “fengmk2/emoji”, the owner’s name was guessed
correctly, but the repository name was missed. In this case, Alice will be
aware of other projects by ”fengmk2”. Thus a half-point is added to the
initial score and 4.5 is the final score.

Table 1: A sample that is recommended correctly of the only the project owner  Top-5 Recommendation | Full_name Matchs | Owner matchs
---|---|---
iojs/io.js | iojs/io.js |
juliangruber/co-read | juliangruber/co-read |
koajs/compose | koajs/compose |
fengmk2/emoji |  | fengmk2
visionmedia/co | visionmedia/co |

In this way, a project recommendation model has been created for open source
platforms. The algorithm of the recommendation model is presented in Figure 2,
starting with selecting a feature as a metric and end with calculating hit
scores.

Figure 2: Project recommending model flowchart

##  4 Empirical Results

In this section, different developer metrics are given. These metrics provide
information about a developer’s past activity on a project. All metrics used
were scaled from 0 to 10 using the min-max normalization technique. The
developer-project relationship was thus rated in the range 0–10 (as with a
viewer’s rating of a movie). The results were calculated for the top 5
recommendation hit scores. We experimented with several metrics, including
those based on coding language; obtained from the ratio of how many times a
developer performed an activity to the total number of the activity; and
created using different normalization methods. However, only metrics that
obtained hit scores greater than 5% are shown in this study (except single
metrics).

###  4.1 Single Metrics

Developer activity on projects was handled as a metric. Activity includes all
kinds of comments, code contributions, revisions, and so on. In this section,
all metrics were used individually in order to evaluate the significance of
each. These metrics refer to the number of activities per project for a given
developer (Table 2).

Table 2: Single developer metrics | Metric | Definition
---|---|---
1 | issue_opened | Number of issue opened
2 | issue_commented | Number of comments to issues
3 | issue_closed | Number of issue closed
4 | issue_closedwithPR | Number of issues closed with a PR
5 | issue_assigned | Number of issues assigned
6 | commit_commented | Number of comments to commits
7 | commit_authored | Number of authorship in commits
8 | commit_committed | Number of commits
9 | pr_opened | Number of PR opened
10 | pr_merged | Number of PR merged
11 | pr_assigned | Number of PR assigned
12 | pr_commented | Number of comments to PRs

All single metrics were given to the model individually, and the scores
presented in Table 3 were obtained according to the evaluation technique
outlined above. In addition to our metrics, another metric that extracted from
the study of Sun et al. was added to make a comparison. They scored developers
and projects using like-star-create activities. They used text data extracted
from projects’ README and source code files to find project similarities
Sun2018 . There were approximately 22,000 repositories and 1,700 developers in
their dataset, which was created using data from four groups of projects.

We planned to use this less sparsed dataset to make a fair comparison but
could not because the dataset was unshared. Owing to we could not communicate
with them, we applied their rating algorithms to our dataset.

Table 3: Single developer metrics scores Metric | Hit Score (%)
---|---
issue_commented | 15.3
issue_closedwithPR | 15
issue_opened | 14
pr_opened | 13.7
commit_commented | 11.9
pr_commented | 10.9
pr_merged | 9.3
Sun’s metric | 7.7
commit_authored | 6
commit_committed | 5.7
issue_closed | 3
issue_assigned | 2.8
pr_assigned | 2.5

When the results are analyzed, it is clear that the issue-related metrics are
crucial even by themselves. Closing an issue with a PR means that the PR fixed
a bug or issue in the project VanDerVeen2015 . As our results indicate,
issue_closedwithPR is a remarkable metric. Opening an issue or PR is also an
important metric. The most interesting conclusion that can be drawn from these
results is that comments have higher hit scores than direct code
contributions.

###  4.2 Fusion Metrics

In these results, we observed that some metric groups came to the forefront.
New metrics can be proposed by grouping comments, code contributions, or other
common featured metrics. In this context, fusion metrics were created from
single metrics.

  1. 1.

count: is created from the sum of all metrics.

  2. 2.

contribution: is created from the sum of all code contribution-related
metrics.

$contribution=pr\\_opened+issue\\_opened+issue\\_closedwithPR+pr\\_merged+commit\\_committed$

  3. 3.

comment: is created from the sum of all comment-related metrics.

$comment=issue\\_commented+commit\\_commented+pr\\_commented$

  4. 4.

issue_related: is created from the sum of all issue-related metrics.

$issue\\_related=issue\\_opened+issue\\_closedwithPR+issue\\_commented+issue\\_assigned$

  5. 5.

pr_related is created from the sum of all PR-related metrics.

$pr\\_related=pr\\_opened+pr\\_merged+pr\\_closed+pr\\_assigned$

  6. 6.

commit_related: is created from the sum of all commit-related metrics.

$commit\\_related=commit\\_commented+commit\\_authored+commit\\_committed$

  7. 7.

commit2comment is created from the (commit_committed divided by
commit_commented)

$commit2comment=\frac{commit\\_committed}{commit\\_commented}$

  8. 8.

issue2comment is created from the (issue_opened divided by issue_commented)

$issue2comment=\frac{issue\\_opened}{issue\\_commented}$

  9. 9.

pr2comment is created from the (pr_opened divided by pr_commented)

$pr2comment=\frac{pr\\_opened}{pr\\_commented}$

  10. 10.

code2comment is created from the ratio of two fusion metrics (contribution
divided by comment)

$code2comment=\frac{contribution}{comment}$

The results of the fusion metrics are presented in Table 4.a. Most fusion
metrics had a positive impact on hit score. comment in particular was a
remarkable metric which showed that commenting is as important as code
contributions. In addition, the issue_related also drew our attention. The
results of the ratio-based metrics were not as successful as the others.

Table 4: Fusion developer metrics scores

(a)
---
Fusion Metric | Hit Score (%)
comment | 17
issue_related | 16
contribution | 15.6
count | 15.2
pr_related | 14.9
issue2comment | 14.2
code2comment | 13
commit_related | 11
pr2comment | 11
Sun’s metric | 7.7
commit2comment | 6.9

& (b) Binary Fusion Metric Hit Score (%) issue_related 20 count 19
contribution 18.8 comment 18.4 pr_related 15.5 commit_related 11.6 Sun’s
metric 7.7 issue2comment x pr2comment x commit2comment x code2comment x

###  4.3 Binary Metrics

We tried different metrics for the project recommendation in the previous
section. It is interesting that the metric consisting only of comments
achieved a higher success rate than the metric created by collecting all
single metrics. We continued to study different metrics to analyze whether
success could be increased with less information.

In the above single metrics, there is information about how many activities
were made (For instance, John opened 18 issues in the projectX, the rating of
John-projectX is 18). Alternatively, a set of metrics was created simply
showing whether that activity existed (For instance, even so, John opened an
issue on the projectX, the rating of John-projectX is 1 or there is no opened
issue by John in the projectY, the rating of John-projectY is 0). In this
context, the binary metrics were created using the equation 3.

$BinaryMetrics=\begin{cases}1,&\text{if }SingleMetric>0\\\
0,&\text{otherwise}\end{cases}$ (3)

####  4.3.1 Binary Fusion Metrics

Binary metrics and ratio-based fusion metrics consist only of 0s and 1s. As
such, using them directly will not generate logical results. Therefore, binary
fusion metrics were generated from these metrics, and only the results of
these binary fusion metrics are given (Table 4.b).

The results improved, and the top success metrics rankings swapped places. The
best five metrics were the same as in the previous section. These results show
that the number of comments is crucial in using comments as a metric. So, the
presence of comments alone is not sufficient to use it properly. On the other
hand, the issue_related metric is leading among the binary metrics, indicating
that issues are the most important feature for the developer-project
relationship. Apart from our findings regarding the importance of comments,
obtaining better results from issues than commits was one of the most
surprising results of this study.

##  5 Conclusion

The main purpose of this study was to propose new developer metrics that could
be used to solve different software engineering challenges. In this study, the
features extracted from user activity on an open source platform (GitHub) were
used. The study focused on finding metrics that would enable greater success
with less knowledge. Some fusion metrics gave successfull results.

It is clearly seen that the comment made gave impressive results. For this
metric, quantity is an important parameter. It means the more a developer
writes comments, the more related to the project. On the contrary, it can
deduce from the results of binary fusion metrics that the issue_related  is a
quantity-free metric. It also means, to use this metric is adequate to know
it’s presence is whether or not. On this regard, It is revealing that issue is
a significant feature for open-source platforms. In addition, metrics that
used features based on existence (binary metrics) were highly successful,
showing that for some activities, there is no need for quantities in order to
extract knowledge.

We presented these new developer metrics, but we are curious why some of them
became prominent. In light of this, we are planning another study involving a
survey for junior and senior developers whom we can contact to understand the
ground truth of our metrics’ success (especially the comment metric).

Because of the sparsity problem, our hit scores may not higher enough when
comparing the other similar studies. Even so, we think that we offered some
new improvable developer metrics. Moreover, to compare our results, we added a
very similar metric used in Sun et al.’s study Sun2018 , which revealed that
most of our metrics surpassed that metric. In this context, we plan to apply
the obtained metrics to different datasets for validity.

In addition, we anticipate that these metrics will be useful for solving
various problems. Many developers besides owners and collaborators have
contributed to projects due to the open source nature of GitHub. On some
projects, external developers even made more contributions than the core team.
These metrics can reveal developers’ contribution rankings on a project. To
implement this, we plan to cooperate with a software company.

## References

  * (1) Badashian, A.S., Hindle, A., Stroulia, E.: Crowdsourced bug triaging.  In: 2015 IEEE 31st International Conference on Software Maintenance and Evolution, ICSME 2015 - Proceedings, pp. 506–510. Institute of Electrical and Electronics Engineers Inc. (2015).  DOI 10.1109/ICSM.2015.7332503
  * (2) Bird, C., Nagappan, N., Murphy, B., Gall, H., Devanbu, P.: Don’t touch my code! Examining the effects of ownership on software quality.  In: Proceedings of the 19th ACM SIGSOFT Symposium on Foundations of Software Engineering, pp. 4–14. ACM Press, New York, New York, USA (2011).  DOI 10.1145/2025113.2025119.  URL http://dl.acm.org/citation.cfm?doid=2025113.2025119
  * (3) Casalnuovo, C., Vasilescu, B., Devanbu, P., Filkov, V.: Developer On boarding in GitHub: The role of prior social links and language experience.  In: 2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings, pp. 817–828. Association for Computing Machinery, Inc, New York, New York, USA (2015).  DOI 10.1145/2786805.2786854.  URL http://dl.acm.org/citation.cfm?doid=2786805.2786854
  * (4) Cosentino, V., Izquierdo, J.L.C., Cabot, J.: Three Metrics to Explore the Openness of GitHub projects.  arXiv preprint arXiv:1409.4253 (2014).  URL http://arxiv.org/abs/1409.4253
  * (5) De Lima, M.L., Soares, D.M., Plastino, A., Murta, L.: Developers assignment for analyzing pull requests.  In: Proceedings of the ACM Symposium on Applied Computing, vol. 13-17-Apri, pp. 1567–1572. Association for Computing Machinery, New York, New York, USA (2015).  DOI 10.1145/2695664.2695884.  URL http://dl.acm.org/citation.cfm?doid=2695664.2695884
  * (6) Foucault, M., Teyton, C., Lo, D., Blanc, X., Falleri, J.R.: On the usefulness of ownership metrics in open-source software projects.  In: Information and Software Technology, vol. 64, pp. 102–112. Elsevier (2015).  DOI 10.1016/j.infsof.2015.01.013
  * (7) Gousios, G., Pinzger, M., Deursen, A.V.: An exploratory study of the pull-based software development model.  In: Proceedings - International Conference on Software Engineering, 1, pp. 345–355. IEEE Computer Society, New York, New York, USA (2014).  DOI 10.1145/2568225.2568260.  URL http://dl.acm.org/citation.cfm?doid=2568225.2568260
  * (8) Guo, G.: Resolving data sparsity and cold start in recommender systems.  In: Proceedings of the 20th international conference on User Modeling, Adaptation, and Personalization, vol. 7379 LNCS, pp. 361–364 (2012).  DOI 10.1007/978-3-642-31454-436
  * (9) Happel, H.J., Maalej, W.: Potentials and challenges of recommendation systems for software development.  In: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp. 11–15. ACM Press, New York, New York, USA (2008).  DOI 10.1145/1454247.1454251.  URL http://portal.acm.org/citation.cfm?doid=1454247.1454251
  * (10) Hu, J., Sun, X., Lo, D., Li, B.: Modeling the evolution of development topics using Dynamic Topic Models.  In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015 - Proceedings, pp. 3–12. Institute of Electrical and Electronics Engineers Inc. (2015).  DOI 10.1109/SANER.2015.7081810
  * (11) Júnior, M.L.d.L., Soares, D.M., Plastino, A., Murta, L.: Automatic assignment of integrators to pull requests: The importance of selecting appropriate attributes.  Journal of Systems and Software 144, 181–196 (2018).  DOI 10.1016/j.jss.2018.05.065
  * (12) Kaur, G., Bahl, K.: Software Reliability, Metrics, Reliability Improvement Using Agile Process.  International Journal of Innovative Science, Engineering & Technology 1(3), 143–147 (2014).  URL www.ijiset.com
  * (13) Liu, C., Yang, D., Zhang, X., Ray, B., Rahman, M.M.: Recommending GitHub Projects for Developer Onboarding.  IEEE Access 6, 52082–52094 (2018).  DOI 10.1109/ACCESS.2018.2869207
  * (14) McMillan, C., Grechanik, M., Poshyvanyk, D.: Detecting similar software applications.  In: Proceedings - International Conference on Software Engineering, pp. 364–374 (2012).  DOI 10.1109/ICSE.2012.6227178
  * (15) Munson, J.C., Elbaum, S.G.: Code churn: a measure for estimating the impact of code change.  In: International Conference on Software Maintenance, pp. 24–31. IEEE (1998).  DOI 10.1109/icsm.1998.738486
  * (16) Nielek, R., Jarczyk, O., Pawlak, K., Bukowski, L., Bartusiak, R., Wierzbicki, A.: Choose a Job You Love: Predicting Choices of GitHub Developers.  In: IEEE/WIC/ACM International Conference on Web Intelligence, WI 2016, pp. 200–207. Institute of Electrical and Electronics Engineers Inc. (2017).  DOI 10.1109/WI.2016.0037
  * (17) Niu, J., Wang, L., Liu, X., Yu, S.: FUIR: Fusing user and item information to deal with data sparsity by using side information in recommendation systems.  Journal of Network and Computer Applications 70, 41–50 (2016).  DOI 10.1016/j.jnca.2016.05.006
  * (18) Ozcan Kini, S., Tosun, A.: Periodic developer metrics in software defect prediction.  In: Proceedings - 18th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2018, pp. 72–81. Institute of Electrical and Electronics Engineers Inc. (2018).  DOI 10.1109/SCAM.2018.00016
  * (19) Robillard, M., Walker, R., Zimmermann, T.: Recommendation systems for software engineering.  IEEE Software 27(4), 80–86 (2010).  DOI 10.1109/MS.2009.161
  * (20) Sharma, L., Gera, A.: A Survey of Recommendation System: Research Challenges.  International Journal of Engineering Trends and Technology 4(5), 1989–1992 (2013)
  * (21) Sheoran, J., Blincoe, K., Kalliamvakou, E., Damian, D., Ell, J.: Understanding ”watchers” on GitHub.  In: 11th Working Conference on Mining Software Repositories, MSR 2014 \- Proceedings, pp. 336–339. Association for Computing Machinery, Inc, New York, New York, USA (2014).  DOI 10.1145/2597073.2597114.  URL http://dl.acm.org/citation.cfm?doid=2597073.2597114
  * (22) Sun, X., Xu, W., Xia, X., Chen, X., Li, B.: Personalized project recommendation on GitHub.  Science China Information Sciences 61(5), 1–14 (2018).  DOI 10.1007/s11432-017-9419-x
  * (23) Thongtanunam, P., Tantithamthavorn, C., Kula, R.G., Yoshida, N., Iida, H., Matsumoto, K.I.: Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review.  In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015 - Proceedings, pp. 141–150. Institute of Electrical and Electronics Engineers Inc. (2015).  DOI 10.1109/SANER.2015.7081824
  * (24) Tiwari, V., Pandey, R.: Open source software and reliability metrics.  In: International Journal of Advanced Research in Computer and Communication Engineering, pp. 808–815 (2012).  URL https://pdfs.semanticscholar.org/ce3f/2129c3c735ce4e0d4ac048ea36f990903022.pdf
  * (25) Tsay, J., Dabbish, L., Herbsleb, J.: Influence of social and technical factors for evaluating contribution in GitHub.  In: Proceedings - International Conference on Software Engineering, 1, pp. 356–366. IEEE Computer Society, New York, New York, USA (2014).  DOI 10.1145/2568225.2568315.  URL http://dl.acm.org/citation.cfm?doid=2568225.2568315
  * (26) Van Der Veen, E., Gousios, G., Zaidman, A.: Automatically prioritizing pull requests.  In: IEEE International Working Conference on Mining Software Repositories, vol. 2015-Augus, pp. 357–361. IEEE Computer Society (2015).  DOI 10.1109/MSR.2015.40
  * (27) Yu, Y., Wang, H., Yin, G., Ling, C.X.: Reviewer recommender of pull-requests in GitHub.  In: Proceedings - 30th International Conference on Software Maintenance and Evolution, ICSME 2014, pp. 609–612. Institute of Electrical and Electronics Engineers Inc. (2014).  DOI 10.1109/ICSME.2014.107
  * (28) Zhang, L., Zou, Y., Xie, B., Zhu, Z.: Recommending relevant projects via user behaviour: An exploratory study on Github.  In: 1st International Workshop on Crowd-Based Software Development Methods and Technologies, CrowdSoft 2014 - Proceedings, pp. 25–30. Association for Computing Machinery, Inc, New York, New York, USA (2014).  DOI 10.1145/2666539.2666570.  URL http://dl.acm.org/citation.cfm?doid=2666539.2666570
  * (29) Şeker, A., Diri, B., Arslan, H.: Summarising Big Data: Common GitHub Dataset for Software Engineering Challenges (2020).  URL http://arxiv.org/abs/2006.04967