Proceedings of the CTD/WIT 2019 PROC-CTD19-029 ATL-PHYS-PROC-2019-044

Primary Vertex Selection in VBF Higgs to Invisibles at the HL-LHC with the
ATLAS Experiment

Murtaza Safdari

On behalf of the ATLAS Experiment,
SLAC National Accelerator Laboratory,
Department of Physics,
Stanford University, U.S.A.

> ABSTRACT
>
>
>
>
> ATLAS has developed a new approach for primary vertex selection in VBF Higgs
> invisible events under HL-LHC conditions, exploiting its new forward
> tracking capabilities, integrating calorimeter and tracking information to
> mitigate the impact of pileup vertex merging, and introducing a new way to
> apply pile-up jet suppression methods for the selection of VBF jets. The new
> algorithm is insensitive to pileup density and improves the average vertex
> selection efficiency from 86% to 95% under tight VBF Higgs event selection
> cuts.

> PRESENTED AT
>
>
>
>
> Connecting the Dots and Workshop on Intelligent Trackers (CTD/WIT 2019)
>
> Instituto de Física Corpuscular (IFIC), Valencia, Spain
>
> April 2-5, 2019

Copyright 2019 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0
license.

##  1 Introduction

Vertex selection algorithms used to identify the hard-scatter primary vertex
among the multiple reconstructed vertices in each bunch crossing at the LHC
have thus far relied primarily upon the hardness of the hard-scatter (HS)
vertex relative to vertices from pileup (PU). The high PU environment at the
HL-LHC [1] will, however, introduce major experimental challenges for the
correct selection of the HS vertex. In particular, the expected average PU
vertex density of around 2 vtx/mm can often lead to the merging of nearby PU
vertices resulting in PU vertices with very large summed transverse momentum
($p_{\mathrm{T}}$ ) that can be incorrectly identified as HS by the standard
algorithm (Section 3).

This is especially relevant for vector boson fusion (VBF) Higgs invisible
final state topologies where the HS process does not have very high visible
$p_{\mathrm{T}}$ activity, resulting in a low selection efficiency as a
function of PU density when using the standard approach.

A new approach for primary vertex selection in VBF Higgs invisible events
under HL-LHC conditions is developed, exploiting the new forward tracking
capabilities of the planned ATLAS detector upgrade [1]. This new technique
integrates tracking information with calorimeter information to mitigate the
impact of PU vertex merging, and introduces a new way to apply PU jet
suppression for the selection of VBF Higgs events. The new algorithm is
insensitive to PU density and improves the average vertex selection efficiency
from 86% to 95% under tight and unbiased VBF Higgs event selection cuts.

##  2 Data Used and Event Selection

Run 4 simulation data was used ($\langle\mu\rangle=200$, $\sqrt{s}=14$ TeV)
for [VBF $H125\rightarrow ZZ\rightarrow 4\nu$] events, with the new Inner
Tracker (ITk) geometry layout and reconstruction performance as described in
Ref. [1] assumed.

Physics event selection ensures that every event has a vertex of interest,
setting the theoretical maximum vertex selection efficiency at 100%. A key
challenge in studying primary vertex identification is the fact that physics
event selection depends on the choice of the primary vertex. For example, in
order to suppress PU, jets are required to have a large fraction of their
track $p_{\mathrm{T}}$ pointing to the selected HS primary vertex. This is
typically achieved by requiring jet $R_{p_{\mathrm{T}}}>0.1$, where
$R_{p_{\mathrm{T}}}$ [2] is defined as:

$R_{p_{\mathrm{T}}}=\sum^{\text{tracks}}_{\text{in
vertex}}\frac{p_{\mathrm{T}}}{p_{\mathrm{T}(\text{jet})}}\mathds{1}(\Delta
R<0.4).$ (1)

Here $\Delta R$ is the distance between the track and jet in $\eta-\phi$
space. Since the $R_{p_{\mathrm{T}}}$ condition requires the definition of an
HS vertex, it biases the measurement of the vertex selection efficiency. This
is because VBF Higgs events in which the selected vertex is not the true HS
primary vertex will not pass the jet selection due to low $R_{p_{\mathrm{T}}}$
with respect to the incorrect vertex, and will not be considered in the
calculation of the efficiency, leading to an overestimation of the vertex
selection efficiency.

To overcome this, a new event selection is defined that is unbiased with
respect to the choice of the vertex. The idea is to first associate all
physics objects (jets in the case of VBF Higgs invisibles) to vertices, and
then apply the VBF Higgs event selection to each vertex separately; if at
least one vertex contains 2 jets passing the VBF jet criteria, the event is
considered for analysis. This ensures that all suitable VBF Higgs events will
be considered for the calculation of the vertex selection efficiency,
regardless of which vertex was selected by the event selection algorithm,
leading to a vertex-unbiased event selection.

Jets are associated to the vertices that maximize $R_{p_{\mathrm{T}}}$, after
which the jets attached to individual vertices are tested for the VBF jet
criteria.

##  3 Current Approach and its Limitations

The current approach to finding the HS vertex in an event is to find the
vertex with the largest scalar summed squared track transverse momentum,

$\text{sumPT}=\sum_{\text{tracks}}p^{2}_{\mathrm{T}}.$ (2)

One limitation of this method is that, under high luminosity conditions, it
becomes vulnerable to selecting high $p_{\mathrm{T}}$ vertices resulting from
the merging of nearby PU vertices. Nearby PU vertices can be merged when the
typical distance between pileup vertices is comparable or smaller to the
vertex resolution. An example where the sumPT algorithm incorrectly selects
one of such merged vertices is shown in Figure 1. This effect is further
illustrated in Figure 2, where the sumPT vertex selection efficiency shows a
downward trend with increasing average local PU vertex density. This
consequence of merged PU vertices is exacerbated for VBF Higgs to invisibles
vertices with their modest visible $p_{\mathrm{T}}$ .

Figure 1: PU vertices with many tracks, as shown in the left r-z event
display, dominate the softer reconstructed HS Primary Vertex (in the same
Event), on the right, when using sumPT. The left plot is centered (red
ellipse) on the vertex incorrectly chosen by sumPT, the right plot is centered
on the reconstructed HS Primary Vertex. All the jets with $p_{\mathrm{T}}>30$
GeV are drawn in the r-z plane on top of the vertex of interest. Also shown in
the lower half of the plots is the truth Z axis with the truth HS and PU
vertex positions marked.

##  4 New Technique

The key difference between the selected PU vertex and the correct HS vertex in
Figure 1 is that in the case of the HS vertex, the tracks are correlated with
the high $p_{\mathrm{T}}$ jets in the event whereas the tracks within the PU
merged vertex are not strongly correlated in angles, because they originate
from independent, close-by, interactions. Therefore there is room for
improvement by resolving the substructure associated with vertices, and
grooming the tracks attached to the vertices such that tracks correlated to HS
processes are selected preferentially.

Based on this insight, we proposed a new algorithm for vertex selection in VBF
Higgs invisible events at HL-LHC that makes use of the track-jet correlations
within each vertex, based on maximizing an augmented version of the standard
sumPT algorithm that projects tracks onto hard jets,

$\text{sumPTw}=\sum_{\text{tracks}}p_{\mathrm{T}_{w}}=\sum_{\text{tracks}}p_{\mathrm{T}}^{2}\,p_{\mathrm{T}(\text{closest
jet})}^{2}\frac{1}{\Delta R}\mathds{1}(\Delta
R<0.8)\mathds{1}(p_{\mathrm{T}(\text{jet})}>p_{\mathrm{T}(\text{threshold})}).$
(3)

Here $p_{\mathrm{T}(\text{threshold})}=30$ GeV is used, and $\Delta R$ is the
distance between a track and the closest jet with sufficient $p_{\mathrm{T}}$
in $\eta-\phi$ space. This projects tracks onto jets and adds the weighted
$p_{\mathrm{T}_{w}}$ as defined above. Thus sumPTw is a bolstered measure of
the total vertex sumPT in jets. Since merged PU vertices are made of 2 or more
independent interactions, their tracks are not correlated in $\eta-\phi$ space
with the HS interaction. Therefore by projecting their tracks onto hard jet
axis directions, their large sumPT is significantly reduced.

Figure 2: Vertex selection efficiency as a function of the Average local PU
vertex density at the reconstructed HS Primary Vertex in the event. Default
sumPT selection method (circles) is compared with proposed sumPTw selection
(squares). sumPTw modestly flattens the PU vertex density dependence of vertex
selection efficiency. Error bars are statistical errors.

Figure 2 shows the new sumPTw algorithm (square) performance versus the
standard sumPT algorithm (circle). sumPTw modestly flattens the PU vertex
density dependence of vertex selection efficiency. This demonstrates the
robustness of sumPTw in the face of merged PU vertices at high PU vertex
densities. The new algorithm improves the average Vertex Selection Efficiency
from 86% to 95%, a 10% relative improvement on the current approach under the
tight event selection criteria as described in Section 2.

##  5 Experimental Challenges and Future Improvements

The new vertex selection algorithm effectively mitigates the effects of PU
vertex merging that would limit the performance of selection in VBF Higgs
invisible events. Three experimental challenges are identified that keep the
vertex selection efficiency of the new selection technique below 100% (Figure
2); vertex splitting, jets outside detector acceptance, and presence of Hard
QCD PU interactions in the event.

Vertex splitting: In events where too many HS tracks have been mis-associated
to nearby PU vertices, one finds that the hard PU vertices are selected
instead of the HS Primary Vertex. This could manifest in two ways; since the
HS vertex has reduced sumPTw, either a random hard PU vertex with large enough
sumPTw gets selected, or a PU vertex near the HS Primary Vertex acquires too
many tracks pointing along hard jets and gets selected.

Jets outside acceptance: In events where HS jets are outside the detector
acceptance the HS Primary Vertex loses out to PU vertices. This is because
there are no reconstructed tracks associated to these jets, and thus sumPTw
can do nothing to help the Primary Vertices in these events.

One can quantify the effect of these two cases, by mitigating vertex splitting
by reattaching all the HS tracks to the Primary Vertex using truth
information, and by limiting to events with HS jets that are within detector
acceptance. The performance with these cases mitigated is shown in Figure 3.

Figure 3: Vertex selection efficiency as a function of the Average local PU
vertex density at the reconstructed HS Primary Vertex in the event. The
baseline sumPTw performance (circle) is compared with the performance of the
sumPTw selection on events where, (square) HS tracks have been reattached to
the reconstructed HS Primary Vertex using truth information, and (triangle)
Only events with $>2$ truth jets with (truth $p_{\mathrm{T}}$)$>30$ GeV &
(truth $|\eta|$)$<4$ are considered. Error bars are statistical errors.

Hard QCD PU: The remaining cases where the new algorithm runs short is
illustrated in Figure 4. This is where the event has Hard QCD PU interactions,
along with the VBF Higgs event of interest. As a result, the PU vertices have
very hard PU tracks pointing along very hard PU jets and thus get selected by
sumPTw.

Figure 4: r-z Event display highlighting presence of a Hard QCD PU vertex that
gets chosen by sumPTw, instead of the reconstructed HS Primary Vertex. The
left plot is centered on the vertex incorrectly chosen by sumPTw
(sumPTw=1.2e8), the right plot is centered on the reconstructed HS Primary
Vertex (sumPTw=6.7e6). All the jets with $p_{\mathrm{T}}>30$ GeV are drawn in
the r-z plane on top of the vertex of interest. Lower half of the plots show
the truth Z axis with the truth HS and PU vertex positions marked.

This final limitation is a fundamental shortcoming of topology agnostic vertex
selection algorithms; They can’t distinguish between different topologies. For
this reason sumPTw can pick a Hard QCD PU vertex instead of the VBF Higgs HS
vertex, even though there was no malfunction in the vertex selection
algorithm. Consequently this paves the way for a more sophisticated vertex
selection algorithm, one which extends the innovations presented here and
combines them with topology discriminating properties. Such an algorithm would
then be able to tell QCD PU apart from HS vertices, despite competing summed
scalar transverse momentum, etc.

##  6 Conclusions

ATLAS has developed a new approach for primary vertex selection that
integrates calorimeter and tracking information to mitigate the impact of PU
vertex merging. The new algorithm is insensitive to PU density and improves
the average Vertex Selection Efficiency from 86% to 95%.

A possible avenue for improvement upon the new algorithm proposed here
involves developing a topology exploiting selection algorithm which can handle
Hard QCD PU interactions in the event.

ACKNOWLEDGEMENTS

We are grateful to Graham Richard Lee, Valentina Cairo, Matthias Danninger,
and Nora Emilia Pettersson for fruitful discussions and help with this
project.

## References

  * [1] The ATLAS collaboration, “Technical Design Report for the ATLAS Inner Tracker Strip Detector”, [ATLAS-TDR-025], [https://cds.cern.ch/record/2257755].
  * [2] The ATLAS collaboration, “Tagging and suppression of pileup jets”, [ATL-PHYS-PUB-2014-001], [https://cds.cern.ch/record/1643929].


# Domain-Specific Evaluation Strategies for AI in Journalism

Sachita Nishal<EMAIL_ADDRESS>Northwestern UniversityUSA ,
Charlotte Li<EMAIL_ADDRESS>Northwestern UniversityUSA and
Nicholas Diakopoulos<EMAIL_ADDRESS>Northwestern UniversityUSA

(2024)

††copyright: rightsretained††journalyear: 2024††conference: Make sure to enter
the correct conference title from your rights confirmation emai; May 12, 2024;
Honolulu, HI††booktitle: Workshop on Evaluating AI at the CHI conference on
Human Factors in Computing Systems, May 12, 2024, Honolulu, HI

##  1\. Motivation

News organizations today rely on AI tools to increase efficiency and
productivity across various tasks in news production and distribution. These
tools are oriented towards stakeholders such as reporters, editors, and
readers. However, practitioners also express reservations around adopting AI
technologies into the newsroom, due to the technical and ethical challenges
involved in evaluating AI technology and its return on investments (Beckett
and Yaseen, 2023). This is to some extent a result of the lack of domain-
specific strategies to evaluate AI models and applications. In this paper, we
consider different aspects of AI evaluation (model outputs, interaction, and
ethics) that can benefit from domain-specific tailoring, and suggest examples
of how journalistic considerations can lead to specialized metrics or
strategies. In doing so, we lay out a potential framework to guide AI
evaluation in journalism, such as seen in other disciplines (Reddy et al.,
2021; Guha et al., 2023). We also consider directions for future work, as well
as how our approach might generalize to other domains.

##  2\. Existing AI Evaluation Approaches

###  2.1. From Benchmarking to Human-Centered Evaluation

Present strategies of evaluating AI models and applications range from
generalized quantitative evaluation on benchmark datasets, to scoped
qualitative or mixed-methods evaluation in specific contexts. In the former
approach, model performance is measured on human-validated metrics, over
particular benchmark datasets e.g., for image captioning (Chen et al., 2015),
code synthesis (Chen et al., 2021), action detection (Sevilla-Lara et al.,
2019), and so on. While these evaluations can be conducted rapidly and at
scale, the metrics and benchmark datasets themselves capture model performance
over generalized notions of ”quality” and decomposed tasks, which limits the
their capability for measuring models performance in real-world scenarios
(Raji et al., 2021).

On the other end of this spectrum, human-centered and HCI-based approaches
rely on situated and contextual evaluation of AI models and applications, such
as via highly-scoped user studies (e.g. (Xu et al., 2023)), longitudinal
studies (e.g. (Long et al., 2024)), and human-grounded metrics (e.g. (Hoffman
et al., 2019)). Recent work illustrates the importance of such scoped
approaches, showing how AI models that rank lower on benchmark datasets still
perform well in user studies within interactive applications (Lee et al.,
2023). While these processes allow for more nuanced evaluation in light of a
particular application or context, they can be difficult to conduct in a
manner that is continuous, iterative, and at scale, which would help to keep
pace with model releases, and ensuing novel interactional affordances or
ethical issues (Etzioni and Etzioni, 2016).

Frameworks for evaluating efficacy of AI models and applications within a
specific domain (e.g. journalism, medicine, law) can help strike a balance
between these approaches. For instance, domain-specific frameworks can guide
crafting and validation of domain-specific task benchmarks (e.g. measuring not
just ”coherence” or ”readability” (Howcroft et al., 2020) but ”newsworthiness”
of LLM-generated news summaries) and draw on domain-specific ethics and values
for conducting ethics-based evaluations and audits (e.g. operationalizing and
auditing for professional values like ”immediacy” in journalism (Deuze,
2005)). To this end, frameworks must identify domain-specific aspects, such as
tasks, values, and stakeholder needs, that benchmarking must be scoped toward.
They must also provide actionable guidance for continuous evaluation in real-
world settings (e.g. newsrooms, hospitals). The next section highlights how
this has been approached in prior work, and how journalism could benefit from
such domain-specific AI evaluation frameworks.

###  2.2. Domain-specific Frameworks for Evaluation

Prior work on developing domain-specific evaluations of AI mainly exists in
the context of healthcare (Nsoesie, 2018; Reddy et al., 2021; Scott et al.,
2021) and law (Guha et al., 2023). Similar to journalism, medical and law
practitioners’ concerns for the deployment of AI into real-world use cases
stem from the lack of metrics for evaluating domain-specific quality and
ethical alignment. In the medical domain, researchers tackle this issue by
proposing a framework that can be incorporated in various stages of model
development and deployment, with an emphasis on assessing ethical dimensions
of models including privacy, non-maleficence, and explainability (Reddy et
al., 2021). While parts of this framework can be applicable to journalism,
use-cases of AI and ethical concerns in journalism differ from medicine given
the public nature and the scope of potential harm caused, calling for the
development of journalism-specific frameworks. Such frameworks for
benchmarking can supplement qualitative evaluation methods, since they can
capture certain aspects of real-world usage scenarios while allowing for
iteration and suggesting potential directions for re-design.

The Partnership on AI (PAI) has offered resources to guide AI procurement and
use in newsroom, which organize and categorize useful AI tools, and suggest
different ways of measuring the outcomes they produce in deployment (PAI
Staff, 2023). Our work builds in this direction, but offers more specific
guidelines on evaluating AI for journalism for both researchers and
practitioners, and with additional focus on human-AI interaction as a site of
evaluation.

##  3\. Blueprints for AI Evaluation in Journalism

This section presents three considerations that evaluations of AI in
journalism can include and operationalize: (1) quality of model outputs, based
on editorial goals and news values (2) quality of interaction with AI
applications, based on needs and work processes of stakeholders (3) ethical
alignment, based on professional values and newsroom standards. A useful
framework would support practitioners in evaluating AI models and applications
along these dimensions, in a manner that is flexible, iterative, and provides
feedback for future designs. To actualize these domain-specific metrics, we
believe methodologies that invite the collaboration between practitioners and
researchers, such as co-design and participatory design, are necessary.

###  3.1. Quality of Model Outputs

To evaluate the quality of AI model outputs, a suite of automatic evaluation
metrics and human assessments have been proposed (Maslej et al., 2023). While
some of these automatic evaluation metrics can be applied to assess output
quality for specific journalistic tasks such as text summarization (Lin, 2004;
Krishna et al., 2023), machine translation (Papineni et al., 2002; Goyal et
al., 2022), and object detection (Lin et al., 2014), many other journalistic
tasks would benefit from evaluation strategies that center more on human
assessments. These human assessment evaluation methods are often generalized
based on the modality of generation but are not domain-specific. For example,
for generated text, human assessments tend to focus on clarity, fluency,
accuracy, and coherence of the output (Howcroft et al., 2020). While these
criteria translate well into the domain of journalism, they overlook nuances
specific to journalistic writing practice or context, such as specificity or
the provision of adequate context. Similarly, for image generation, existing
metrics focus mainly on image fidelity, alignment, and counting (Petsiuk et
al., 2022; Dinh et al., 2022; Saharia et al., 2022), with a few exceptions
that look at social biases, robustness, and generalization (Cho et al., 2023;
Bakr et al., 2023) but lack consideration of editorial judgments around e.g.
image framing.

To tailor evaluation strategies towards journalistic uses of AI, researchers
and practitioners can draw from news values that guide editorial decision-
making. The definitions of news values remain fluid and subjective (Parks,
2019), but some of these can be evaluated in model outputs, for instance, in
the case of AI-generated news summaries. These elements include controversy,
surprise, timeliness, negative or positive overtones, and news organization’s
agenda (Harcup and O’Neill, 2017). Additionally, the ability of AI tools to
support creativity is also important for reporters. Examples of creativity
support includes producing varied but relevant outputs, and maintaining
professional tonality.

###  3.2. Interactions with AI Systems

Recent work has called attention to different aspects of in-situ interactions
with AI systems that may lend themselves well to automated evaluation metrics.
These aspects include the ease, enjoyment, and feelings of ownership that
users experience when engaged in tasks like question-answering, solving
crossword puzzles, and generating metaphors with AI (Lee et al., 2023). Human-
AI interaction can also be evaluated by the long-term goals that a system
facilitates, such as promoting personal growth and emotional resilience for
users (Cai et al., 2024); making connections outside users’ comfort zone
(Knijnenburg et al., 2016); and allowing customization of applications or
appropriation for novel use-cases (Amershi et al., 2019; Stevens et al.,
2010).

Within journalism, the in-situ interactions and long-term processes that a
system could optimize will vary by stakeholders and the tasks facilitated by
AI. For instance, AI systems that provide writing feedback to reporters may be
evaluated based on the new perspectives or angles they add to a reporter’s
writing (Motta et al., 2020) (e.g. by measuring semantic similarity between
initial and post-feedback writing). How interactions with AI are configured
also impacts the choice of metrics: minor errors in AI outputs (i.e. low
accuracy) may be more acceptable to reporters engaged in brainstorming and
ideation with text or image models (Nishal et al., 2023), than to readers who
only view a final product. Recent work also suggests a desire for long-term
skill development among reporters who use AI systems (Nishal et al., 2023),
which could be evaluated based on periodic user surveys and AI usage analytics
over time. A shared criterion of interaction quality across stakeholders could
be the enjoyment they experience when engaged with an AI system, which applies
both to reporters who use specific tools in news production (e.g. (Thurman et
al., 2017; Retegui, 2021), or to readers who rely on AI (e.g., when solving
crossword puzzles (Lee et al., 2023)).

Datasets to evaluate these metrics will derive from users’ interactions with
AI systems, rather than solely model outputs like in Section 3.1.
Understanding the short and long-term goals of different stakeholders, as well
as the configurations of their interactions with AI, can support the design of
such interaction metrics.

###  3.3. Ethical Alignment of AI Systems

Different professions and institutions adhere to different ethical principles
and codes of conduct (Kovach and Rosenstiel, 2021; Deuze, 2005). Recent work
suggests that subjective and multivalent principles of journalistic practice
(e.g. truth, independence, accountability) can be translated to AI systems via
value-sensitive design approaches and ethics-based audits (Diakopoulos et al.,
2024). Evaluation practices can also measure adherence to codes of conduct and
style guides of different newsrooms to be more sensitive to institutional
needs (e.g. (The New York Times, 2018; Reuters News Agency, 2021; Froke et
al., 2022)). This translational exercise can also provide practitioners with
an opportunity to reflect on the inherent limitations of their existing codes
and guides (Bien-Aimé, 2016; Bessette et al., 2022).

Challenges to general-purpose evaluations of ethics and values hold for
domain-specific evaluations as well: operationalizing and evaluating ethics
can be difficult because generative AI models cannot produce consistent,
causal explanations for their outputs (Liao and Vaughan, 2023). Fine-tuned
versions of the same model can also exhibit drastic variations in their
adherence to ethical values. Journalism’s orientation toward timeliness of
communication also exacerbates these challenges. This further indicates the
need for iterative and continuous evaluations of AI models and applications
(Mökander et al., 2021). It also indicates a need for technical innovation to
support operationalization and of journalism-specific ethics and values in AI
systems.

##  4\. Future Directions and Conclusion

In this position paper, we summarized current approaches for AI evaluation,
and made recommendations for additional journalism-specific evaluation
criteria across three different aspects of AI evaluation. The primary goal of
this paper is to call for researchers and practitioners to come together and
develop domain-specific frameworks for evaluating AI systems, so the
adaptation of AI tools into newsrooms become easier and more equitable. Such
frameworks can enable the development of procurement guidelines for AI tools
in newsrooms, as seen in prior work (PAI Staff, 2023). We further hope this
approach can empower stakeholders to create newsroom-specific custom
evaluation datasets, for both short-term and longitudinal assessments of the
technology.

## References

  * (1)
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, Glasgow Scotland Uk, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Bakr et al. (2023) Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. 2023.  HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ 00 (2023), 19984–19996.  https://doi.org/10.1109/iccv51070.2023.01834
  * Beckett and Yaseen (2023) Charlie Beckett and Mira Yaseen. 2023.  _Generating Change: A Global Survey of What News Organisations Are Doing with AI_.  Technical Report. JournalismAI, London School of Economics.
  * Bessette et al. (2022) Lily G. Bessette, Sacha C. Hauc, Heidi Danckers, Agata Atayde, and Richard Saitz. 2022.  The Associated Press Stylebook Changes and the Use of Addiction-Related Stigmatizing Terms in News Media.  _Substance Abuse_ 43, 1 (Dec. 2022), 127–130.  https://doi.org/10.1080/08897077.2020.1748167
  * Bien-Aimé (2016) Steve Bien-Aimé. 2016.  AP Stylebook Normalizes Sports as a Male Space.  _Newspaper Research Journal_ 37, 1 (March 2016), 44–57.  https://doi.org/10.1177/0739532916634640
  * Cai et al. (2024) Alice Cai, Ian Arawjo, and Elena L. Glassman. 2024.  Antagonistic AI.  arXiv:2402.07350 [cs]
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating Large Language Models Trained on Code.  arXiv:2107.03374 [cs]
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015.  Microsoft COCO Captions: Data Collection and Evaluation Server.  https://doi.org/10.48550/arXiv.1504.00325 arXiv:1504.00325 [cs]
  * Cho et al. (2023) Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023.  DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 3043–3054.
  * Deuze (2005) Mark Deuze. 2005.  What Is Journalism?: Professional Identity and Ideology of Journalists Reconsidered.  _Journalism_ 6, 4 (Nov. 2005), 442–464.  https://doi.org/10.1177/1464884905056815
  * Diakopoulos et al. (2024) N. Diakopoulos, C. Trattner, D. Jannach, I. Costera Meijer, and E. Motta. 2024.  Leveraging Professional Ethics for Responsible AI.  _Commun. ACM_ 67, 2 (Jan. 2024), 19–21.  https://doi.org/10.1145/3625252
  * Dinh et al. (2022) Tan M Dinh, Rang Nguyen, and Binh-Son Hua. 2022.  TISE: Bag of metrics for text-to-image synthesis evaluation. In _European Conference on Computer Vision_. Springer, 594–609.
  * Etzioni and Etzioni (2016) Amitai Etzioni and Oren Etzioni. 2016.  AI Assisted Ethics.  _Ethics and Information Technology_ 18, 2 (June 2016), 149–156.  https://doi.org/10.1007/s10676-016-9400-6
  * Froke et al. (2022) Paula Froke, Anna Jo Bratton, Andale Gross, Jeff McMillan, Pia Sarkar, Jerry Schwartz, and Raghuram Vadarevu (Eds.). 2022.  _The Associated Press Stylebook, 2022-2024_ (56th edition ed.).  Basic Books, New York, NY.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 522–538.  https://doi.org/10.1162/tacl_a_00474
  * Guha et al. (2023) Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John J Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023.  LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.  https://openreview.net/forum?id=WqSPQFxFRC
  * Harcup and O’Neill (2017) Tony Harcup and Deirdre O’Neill. 2017.  What is News?  _Journalism Studies_ 18, 12 (2017), 1470–1488.  https://doi.org/10.1080/1461670x.2016.1150193
  * Hoffman et al. (2019) Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2019.  Metrics for Explainable AI: Challenges and Prospects.  https://doi.org/10.48550/arXiv.1812.04608 arXiv:1812.04608 [cs]
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions. In _Proceedings of the 13th International Conference on Natural Language Generation_. Association for Computational Linguistics, Dublin, Ireland, 169–182.
  * Knijnenburg et al. (2016) Bart P. Knijnenburg, Saadhika Sivakumar, and Daricia Wilkinson. 2016.  Recommender Systems for Self-Actualization. In _Proceedings of the 10th ACM Conference on Recommender Systems_. ACM, Boston Massachusetts USA, 11–14.  https://doi.org/10.1145/2959100.2959189
  * Kovach and Rosenstiel (2021) Bill Kovach and Tom Rosenstiel. 2021.  _The Elements of Journalism: What Newspeoble Should Know and the Public Should Expect_ (revised and updated 4th edition ed.).  Crown, New York.
  * Krishna et al. (2023) Kundan Krishna, Prakhar Gupta, Sanjana Ramprasad, Byron Wallace, Jeffrey Bigham, and Zachary Lipton. 2023.  USB: A Unified Summarization Benchmark Across Tasks and Domains. In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 8826–8845.  https://doi.org/10.18653/v1/2023.findings-emnlp.592
  * Lee et al. (2023) Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E. Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. 2023.  Evaluating Human-Language Model Interaction.  _Transactions on Machine Learning Research_ (July 2023).
  * Liao and Vaughan (2023) Q. Vera Liao and Jennifer Wortman Vaughan. 2023.  AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap.  arXiv:2306.01941 [cs]
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context.  _CoRR_ abs/1405.0312 (2014).  arXiv:1405.0312 http://arxiv.org/abs/1405.0312
  * Long et al. (2024) Tao Long, Katy Ilonka Gero, and Lydia B. Chilton. 2024.  Not Just Novelty: A Longitudinal Study on Utility and Customization of AI Workflows.  arXiv:2402.09894 [cs]
  * Maslej et al. (2023) Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark, and Raymond Perrault. 2023.  _The AI Index 2023 Annual Report_.  Technical Report. AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA.
  * Mökander et al. (2021) Jakob Mökander, Jessica Morley, Mariarosaria Taddeo, and Luciano Floridi. 2021.  Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations.  _Science and Engineering Ethics_ 27, 4 (July 2021), 44.  https://doi.org/10.1007/s11948-021-00319-4
  * Motta et al. (2020) Enrico Motta, Enrico Daga, Andreas L. Opdahl, and Bjornar Tessem. 2020.  Analysis and Design of Computational News Angles.  _IEEE Access_ 8 (2020), 120613–120626.  https://doi.org/10.1109/ACCESS.2020.3005513
  * Nishal et al. (2023) Sachita Nishal, Jasmine Sinchai, and Nicholas Diakopoulos. 2023.  Understanding Practices around Computational News Discovery Tools in the Domain of Science Journalism.  https://doi.org/10.48550/arXiv.2311.06864 arXiv:2311.06864 [cs]
  * Nsoesie (2018) Elaine O Nsoesie. 2018.  Evaluating Artificial Intelligence Applications in Clinical Settings.  _JAMA Network Open_ 1, 5 (2018), e182658.  https://doi.org/10.1001/jamanetworkopen.2018.2658
  * PAI Staff (2023) PAI Staff. 2023.  _PAI Seeks Public Comment on the AI Procurement and Use Guidebook for Newsrooms_.  Technical Report. Partnership on AI.  https://partnershiponai.org/pai-seeks-public-comment-on-the-ai-procurement-guidebook-for-newsrooms/
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02_ (2002), 311–318.  https://doi.org/10.3115/1073083.1073135
  * Parks (2019) Perry Parks. 2019.  Textbook News Values: Stable Concepts, Changing Choices.  _Journalism & Mass Communication Quarterly_ 96 (09 2019), 784–810.  https://doi.org/10.1177/1077699018805212
  * Petsiuk et al. (2022) Vitali Petsiuk, Alexander E Siemenn, Saisamrit Surbehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hicke, Bryan A Plummer, Ori Kerret, Tonio Buonassisi, Kate Saenko, Armando Solar-Lezama, and Iddo Drori. 2022.  Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark.  _arXiv_ (2022).  https://doi.org/10.48550/arxiv.2211.12112 arXiv:2211.12112
  * Raji et al. (2021) Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021.  AI and the Everything in the Whole Wide World Benchmark.  https://doi.org/10.48550/arXiv.2111.15366 arXiv:2111.15366 [cs]
  * Reddy et al. (2021) Sandeep Reddy, Wendy Rogers, Ville-Petteri Makinen, Enrico Coiera, Pieta Brown, Markus Wenzel, Eva Weicken, Saba Ansari, Piyush Mathur, Aaron Casey, and Blair Kelly. 2021.  Evaluation framework to guide implementation of AI systems into healthcare settings.  _BMJ Health & Care Informatics_ 28, 1 (2021), e100444.  https://doi.org/10.1136/bmjhci-2021-100444
  * Retegui (2021) L.M. Retegui. 2021.  Metrics at Work: A Case Study about the Tensions in the Media Industry.  _Estudios Sobre el Mensaje Periodistico_ 27, 4 (2021), 1205–1214.  https://doi.org/10.5209/ESMP.71296
  * Reuters News Agency (2021) Reuters News Agency. 2021.  Standards, Values & Style Guide.  _Reuters News Agency_ (2021).
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ 35 (2022), 36479–36494.
  * Scott et al. (2021) Ian Scott, Stacey Carter, and Enrico Coiera. 2021.  Clinician checklist for assessing suitability of machine learning applications in healthcare.  _BMJ Health & Care Informatics_ 28, 1 (2021), e100251.  https://doi.org/10.1136/bmjhci-2020-100251
  * Sevilla-Lara et al. (2019) Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. 2019.  Only Time Can Tell: Discovering Temporal Data for Temporal Modeling.  https://doi.org/10.48550/arXiv.1907.08340 arXiv:1907.08340 [cs]
  * Stevens et al. (2010) Gunnar Stevens, Volkmar Pipek, and Volker Wulf. 2010.  Appropriation Infrastructure: Mediating Appropriation and Production Work.  _Journal of Organizational and End User Computing_ 22, 2 (April 2010), 58–81.  https://doi.org/10.4018/joeuc.2010040104
  * The New York Times (2018) The New York Times. 2018.  Ethical Journalism.  _The New York Times_ (Jan. 2018).  https://www.nytimes.com/editorial-standards/ethical-journalism.html
  * Thurman et al. (2017) N. Thurman, K. Dörr, and J. Kunert. 2017.  When Reporters Get Hands-on with Robo-Writing: Professionals Consider Automated Journalism’s Capabilities and Consequences.  _Digital Journalism_ 5, 10 (2017), 1240–1259.  https://doi.org/10.1080/21670811.2017.1289819
  * Xu et al. (2023) Ruiyun Xu, Yue (Katherine) Feng, and Hailiang Chen. 2023.  ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience.  _SSRN Electronic Journal_ (2023).  https://doi.org/10.2139/ssrn.4498671


11affiliationtext: Complexity Science Hub Vienna, A-1080 Vienna,
Austria22affiliationtext: Vienna University of Economics and Business, A-1020
Vienna, Austria33affiliationtext: Medical University Vienna, A-1090 Vienna,
Austria44affiliationtext: Santa Fe Institute, Santa Fe, USA55affiliationtext:
Sony Computer Science Laboratories, 75005 Paris, France66affiliationtext:
Physics Department, Sapienza University of Rome, 00185 Rome,
Italy**affiliationtext: Corresponding author:
<EMAIL_ADDRESS>These
authors contributed equally to this work

# Evolving Collaboration, Dependencies, and Use in the Rust Open Source
Software Ecosystem

William Schueller  Johannes Wachs  Vito D. P. Servedio  Stefan Thurner
Vittorio Loreto

###### Abstract

Open-source software (OSS) is widely spread in industry, research, and
government. OSS represents an effective development model because it harnesses
the decentralized efforts of many developers in a way that scales. As OSS
developers work independently on interdependent modules, they create a larger
cohesive whole in the form of an ecosystem, leaving traces of their
contributions and collaborations. Data harvested from these traces enable the
study of large-scale decentralized collaborative work. We present curated data
on the activity of tens of thousands of developers in the Rust ecosystem and
the evolving dependencies between their libraries. The data covers seven years
of developer contributions to Rust libraries and can be used to reconstruct
the ecosystem’s development history, such as growing developer collaboration
networks or dependency networks. These are complemented by statistics on
downloads and popularity, tracking the dynamics of use and success over time.
Altogether the data give a comprehensive view of several dimensions of the
ecosystem.

## Background & Summary

Open Source Software (OSS) has recently been described as the “infrastructure”
of the digital society [1]. OSS is an excellent example of open collaboration
among many individuals that has a significant impact on the economy [2, 3, 4,
5]. Within specific OSS ecosystems \- collections of software programs or
libraries usually delineated by the use of a particular programming language
like Rust, Python, or PHP - developers contribute software that depends on
software already in the ecosystem, often created by strangers. For instance, a
library that generates data from probability distributions may use a random
number generator from another library rather than writing a new one. The
outsourcing of core functions leads to a rich structure of technical
dependencies, often represented as a network [6]. These libraries are usually
hosted on coding platforms like GitHub or Gitlab.

The nature of OSS contributions is such that the traces of activity of
individuals are observable, i.e., what they contributed to which libraries and
when. The cumulative efforts of thousands of developers can reveal a great
deal about the nature of collaborative projects and work [7]. Information on
the use and popular success of individual libraries can be tracked over time
[8], along with the co-evolution of technical dependencies and social
collaboration [9]. Such data can give insight into the dynamics of massive and
decentralized collaborations [6] and how these digital ecosystems evolve.

Here, we present a comprehensive dataset on one such ecosystem built around
the Rust programming language. Rust, a relatively young language, has recently
seen a sharp increase in popularity. Besides its significant connections with
Mozilla [10], it is, as of December 2021, the second approved language of the
Linux kernel besides C. For several years now, it has been voted the “most
loved” language in the Stack Overflow Developer
Survey111https://insights.stackoverflow.com/survey/2021. We have collected and
curated temporal data on the technical dependencies, developer contributions,
and the use and success of individual libraries. Specifically, we can observe
when a developer made an elemental contribution of code to a specific library,
what other libraries that library depends on, and how widely used and popular
the library is. We record over three million distinct contributions of over 34
thousand developers, contributing to over 31 thousand libraries over seven
years.

Our data processing pipeline, available as open-source software, combines data
from Cargo (the Rust ecosystem library manager) and the code hosting platforms
GitHub and Gitlab. It considers and handles a host issues common to the study
of collaborative software development data [11]: contributor disambiguation
[12, 13], bot detection [14], and the identification of nested projects and
merged work. The result is a database tracking the evolution of a large,
interconnected software ecosystem at a fine scale.

In contrast to other data sources on collaborative software development, our
dataset contains more accurate and complete data for the Rust software
ecosystem. Focusing on Rust allows us to integrate developer contributions
with data on software dependencies and usage. In this way, our data is richer
and more focused than what can be found in more extensive databases such as
GHTorrent [15], GHArchive222https://www.gharchive.org/, Software Heritage
[16], or World of Code [17]. Moreover, as we highlight in the Technical
Validation section, we achieve a broader coverage by focusing on the Rust
ecosystem: 15% of the packages in our dataset are not in the GHTorrent
database. Our dataset also requires significantly less storage space than the
sources mentioned above and can be directly analysed by researchers with
minimal computing infrastructure requirements. At the same time, Rust is a
large ecosystem that has evolved in a decentralized manner with contributions
from thousands of developers hosted on multiple platforms, differentiating it
from data sourced from single projects like the Linux Kernel or Apache
projects [18].

We plan to update the dataset annually provided that the primary upstream
sources (crates.io, GitHub & Gitlab) remain stable. Researchers can also use
our pipeline to reconstruct the dataset, a process that requires some data
storage space (around 150 GB, though this volume will likely increase over
time as the dataset is updated) and several days to query the data sources and
process the results. Our code can also be adapted to collect data from other
ecosystems, such as the Julia programming language’s ecosystem. However, we
note that not all ecosystems offer the same scope of data as Rust.

We proceed as follows: first, we describe our data collection and wrangling
process and the resulting database. We compare our data coverage against
GHTorrent, a widely used database of OSS contributions, finding that our data
is more complete. We then outline usage notes for researchers interested in
topics such as online cooperation [7, 19] and collaborative innovation [20],
success [21], and supply chain networks [22, 23] in software. Our data can
easily be represented as, for example, dynamic networks of collaborating
developers, time series of usage statistics, growing networks of
interdependent libraries, or combinations thereof.

Figure 1: Data processing pipeline. We collect data from Cargo, the package
registry of the Rust programming language, and complement it with data from
the code hosting platforms GitHub and Gitlab. The processed result integrates
information on package dependencies, use (downloads and stars), and authors.

## Methods

We describe the data sources, and how we combine and curate data from various
sources to create a comprehensive overview of the Rust ecosystem. We provide a
visual overview of the established data processing pipeline in Figure 1.

### Data Sources and Collection

#### Cargo: Libraries and Dependencies

Our first source of data is the Cargo package (which are called crates in the
Rust community) registry. Registries, often called package managers, play an
important role in nearly all OSS ecosystems. They allow users to download and
update different libraries while resolving dependencies and managing
conflicts. Other examples of registries around different programming languages
include PyPI for Python, CRAN for R, Rubygems for Ruby, and NPM for Node. We
use Cargo as a source of technical dependencies and downloads for Rust. These
are available as part of a daily dump from crates.io333https://crates.io/data-
access.

The data can be directly imported in a PostgreSQL database, and contains
package names and creation dates, their versions, a list of dependencies for
each version with the semantic versioning (semver) syntax associated to them,
and the daily downloads per version of each package. For a relevant discussion
of the importance of semantic versioning in OSS ecosystems, see recent work by
Decan and Mens [24]. Packages are also often associated to a repository URL
and a documentation URL, but those are not always provided and depend on
maintainer input.

#### Code Repositories: GitHub and Gitlab

To understand who contributes to which library, we turn to the social coding
platforms on which these packages are hosted. In the case of Rust, nearly all
packages in Cargo are hosted on either GitHub or Gitlab. Specifically, 63,578
packages had links to either platform. Of these links, 44,379 were unique and
40,927 of them could be cloned from either GitHub (39,536) or Gitlab (1,391).
The inclusion of data from Gitlab represents an important extension over the
most widely used databases in OSS research GHTorrent and GH Archive, which
only use data from GitHub. Both of these platforms use Git version control,
making projects hosted on either alternative comparable.

The elemental code contributions to OSS projects using the Git version control
system are called _commits_ and are associated to an email address belonging
to the contributor. GitHub and Gitlab both host Git projects (called
repositories or _repos_ , for short), which we downloaded and used to extract
information about activity and collaboration. The mapping between repos and
the libraries hosted on Cargo is not one-to-one and requires additional
processing, described below. The Git version control history of a project
allows us to examine in detail the contribution histories of all developers
working on a project. Indeed, previous work has shown how this highly granular
data can be exploited to study collaboration and interactions among developers
[25, 26]. To do so, we “clone” (download) each repo locally. We also make use
of the GitHub and Gitlab GraphQL APIs to disambiguate contributors.

#### Measuring Use and Success: Downloads and Stars

We quantify two dimensions of the use of libraries: the number of times they
are downloaded and the number of times they received positive social feedback
(stars) on GitHub. The two metrics highlight different aspects of use.
Downloads, sourced directly from Cargo, present a more technical measurement
of use. GitHub’s stars are more suggestive of visibility. For example, a
highly technical library that provides background functions may be downloaded
many times but have relatively few stars. The social aspect of GitHub and
platforms is known to play an important role in collaborative software
engineering [27, 28]. GitHub stars and other forms of social feedback
including followers and sponsors matter as information signals in the software
development community for both libraries and contributors [29, 30], for
example on the labour market [31]. We collected GitHub stars and forks, a
measure of code reuse, for repos with timestamps directly from the GitHub
GraphQL API. At the developer (GitHub/Gitlab user) level, we also collected
data on the number of followers and sponsors each user has using the same API.
GitHub implemented sponsorships for developers in 2019 [32], enabling
developers to crowdfund from users who appreciate their work directly on their
GitHub pages.

### Data Processing

Several additional curation steps need to be taken to insure the data
collected is useful for the purposes of research into digital collaboration.
We describe three specific steps here.

#### Repositories, Projects, Forks

One would easily assume individual packages of the Rust ecosystem tracked by
Cargo correspond one-to-one with repositories on GitHub or Gitlab, and this is
generally the case. However, some repositories host several packages. For
example modules or plugins that extend a core package are often hosted
together in the same repo, but are distinguished in Cargo. This presents a
data challenge: dependencies are recorded between packages, while
contributions are recorded at the repo level.

For each package listed in the registry, one or several URLs are typically
provided. They correspond to a link to the code and/or the documentation.
Here, we take the URL corresponding to the code, and if empty, coalesce it
with the documentation URL. We parse the individual URLs by recognizing the
prefixes synonyms of github.com and gitlab.com and the pattern
`github.com/<owner>/<name>` corresponding to a repository.

Repositories are sometimes renamed, and both URLs can be present in distinct
packages. The old URL typically redirects to the new one. To resolve this and
be able to merge repositories under one entity, we use the `<owner>/<name>`
returned by the GraphQL API when querying about the repository, for example
when collecting information about forks.

After downloading the repositories (also called cloning), we analyse the
commit data and retrieve commit hash, author email and name, and commit
parents. We also compute the number of lines added and deleted for each
commit. We analyse all available branches, and the unique hash ensures that we
do not count commits twice per repository. However, commits can appear in
several repositories, when one repository is a fork of another. We keep
attribution of commits to each repository where they appear, but we also
attribute each commit to a main repository, supposed to be its origin. For
this, we retrieve the information about forks from the GitHub or Gitlab
GraphQL API, and take as origin the highest repository in the fork tree
containing the commit. When this method is unsuccessful (e.g., undeclared
forks, or forks between different Git platforms), we take the repository
having the oldest package, using its creation date from the package manager.

#### Dependencies

When analysing the dependencies between packages sourced from Cargo, and
aggregating the network to dependencies between repositories, we noted the
presence of cycles. In this context a cycle represents a pattern like: Package
A depends on Package B, Package B depends on Package C, and Package C depends
on Package A. Though there were only few of such examples, we decided to prune
dependencies to remove such cycles for two reasons. The first is that they
represent a logical inconsistency in what dependency should mean. Second,
without cycles, the resulting dependency network is a directed acyclic graph
(DAG). DAGs are themselves interesting data structures appearing in a variety
of data science contexts [33]. Given the small number of packages involved in
cycles, we manually inspected them. Dependency cycles in package space
correspond mainly to unnecessary dependency links or even fake packages for
the sake of testing dependency declarations. Repository space cycles are more
complex to prune. We adopted the heuristic to remove the dependencies (in both
repo and package space) in cycles of length 2 by pruning the dependency of the
oldest node to the newest (by creation date or earliest date of the repo’s
corresponding packages), and naturally removing dependency cycles of length 1.
The remaining cycles were inspected manually, and the cycles were broken by
removing the dependency links where it made more sense, in most cases from a
repository having one of the highest download counts to one having one of the
lowest. One remaining repository, although corresponding to numerous
downloads, has been pruned from all dependencies to it because of the high
number of cycles of the dependency network involving it. We included these
pruned dependencies in the dataset for the sake of completeness, but we
flagged them for easy removal, or for letting the possibility to investigate
other link removal policies. We guarantee absence of cycles for the state of
the dependency network at the end of the dataset (2022-03-14) and at the end
of the preceding year (2021-12-31) in both spaces, but not at any arbitrary
timestamp.

#### Merging of Developer Identities

There are many potential ways to disambiguate the identities of contributors
[12, 13], each presenting tradeoffs. In general, Git commits are signed by an
email address, not a platform-specific username. Developers often commit code
from different computers or environments with different email addresses in
their configuration files, and this can result in a significant disambiguation
problem. Rather than attempting to infer which email addresses potentially
refer to the same person, we query the GitHub API for the GitHub account
linked to each commit [34]. While this ignores the potential that developers
use multiple accounts, we argue that it makes a larger amount of highly
justifiable merges among commit author identities than an email address based
approach. Email address-based author identity disambiguation would scale
better in larger systems at the cost of accuracy.

For each email address, we carry out this process for the most recent commit
registered for that email, and if this fails to return an account, we try
again with a randomly sampled commit among all those corresponding to this
email. After doing the same for Gitlab, we also merge matching GitHub and
Gitlab logins 84. An additional step is to parse the emails and discard
obviously belonging to GitHub, following the patterns
<EMAIL_ADDRESS>or
<EMAIL_ADDRESS>
#### Bot Accounts

Bots play an important role in modern software development [35, 36], but need
to be handled with care in any study of software systems, as they can make
orders of magnitude more contributions than any human developer. Ignoring them
would skew any analysis of cooperation between human developers [11, 14].
While bots have interesting effects on project evolution [35], we chose to
detect and mark bots and more generally invalid accounts in our dataset with a
view to excluding them as we are primarily interested in the patterns of
contributions of developers. To identify bots, we used a two-step filtering
process. First, we extracted all bots on a curated list used as ground truth
for bot detection in the software engineering community [14]. We then filtered
remaining Github accounts with the substrings “*[bot]”,
“*-bot”,“*-bors”,“bors-*”,“dependabot-*” in their usernames, and finally
inspecting manually each individual account with pattern “*bot*”. After
filtering out bot accounts labelled this way, a manual inspection of the 100
most active remaining accounts was conducted, as well as of all accounts
containing the substring “bot”. The manual inspection took the following
steps: checking the GitHub webpage of a user for a clear name, looking at the
description, looking at the commit/PR comments. The 100 most active accounts
by number of commits across the full timespan of the dataset were considered,
as well the 100 most active accounts in the year 2020.

For users that could not be associated to a GitHub account, their emails are
filtered when the last part of their prefix (separated by ., - or +) is equal
to “bot”, “ghbot”, “bors”, “travis” or “bot”. A few remaining email strings
without “@” were discarded, like “localhost”, “N/A” or empty string. Manual
inspection of the most active 100 emails without a GitHub account revealed a
few more bots. The list of manually discarded bots is available in the file
`botlist.csv`. Our dataset includes the bots among the users, flagged with a
Boolean “is_bot” attribute to enable filtering.

## Data Records

We host our data on Figshare and the code used to collect and process the data
from our sources on GitHub. Both platforms track the history of the data and
code, allowing researchers to use any version they prefer as we continue to
update and extend both. We share data in several formats, noting that in no
format does the data exceed 5 GB when compressed.

Table Name | Description
---|---
`commit_parents` | Parenthood relationships between commits. Typically one parent/commit.
`commit_repos` | The repos to which commits belong. At least one, but can be several (via forks).
`commits` | Metadata about specific commits
`filtered_deps_package` | Packages which are filtered when appearing as a dependency to avoid cycles
`filtered_deps_packageedges` | Edges between packages directly discarded in the dependency graph
`filtered_deps_repo` | Repositories which are filtered when appearing as a dependency to avoid cycles
`filtered_deps_repoedges` | Edges between repositories directly discarded in the dependency graph
`followers` | Followers of GitHub accounts
`forks` | Forks declared on GitHub
`identities` | Individual identities of developers (email, GitHub account, Gitlab account; hashed)
`identity_types` | Identity types (email, GitHub account, Gitlab account)
`issues` | Issues per repository
`merged_identities` | Identities that have been merged
`merged_repositories` | Repositories that have been merged (after identifying renaming or typo in URL)
`package_dependencies` | Package dependencies (version to package with semver)
`package_version_downloads` | Daily downloads of package versions
`package_versions` | Versions of packages
`packages` | Packages
`repo_languages` | Language composition of repositories, as reported by GitHub
`repositories` | Repositories
`sources` | Data sources
`sponsors_user` | Sponsorships of developers
`stars` | Starring events of repositories
`urls` | Retrieved URLs, and their parsed equivalent
`user_languages` | Language composition of GitHub user contributions (beyond the dataset)
`users` | Users (who can have several identities: email, GitHub, Gitlab)

Table 1: Table of tables in the database. A full database schema is available
on Figshare: https://figshare.com/s/93158d03416765444650.

Before we describe the data, we discuss data pseudoanymisation. To preserve
developer privacy, we provide data that is scrubbed of information that can
directly be used to identify individuals. We do so in the following way: we
discard name attributes and hash (via MD5 with a random salt) email address
prefixes and GitHub/Gitlab logins. Researchers interested in studying social
or demographic characteristics of developers, such as gender [37, 38],
geography [39, 40, 41], or both [42], could adapt our approach to data
collection and analyse these attributes. However, they should consider
potential ethical issues that arise when associating such information to users
[43].

We now proceed to describe the tables in the database. In Table Data Records,
we list the tables in the database along with a description of their content
and purpose. For the sake of brevity, we refer the reader to the accompanying
materials on Figshare for a schema of the database and a description of the
variables included.

## Technical Validation

In this section, we report statistics on the completeness of our data. An
advantage of defining the Rust ecosystem as all those packages hosted on
Cargo, is that we can precisely measure how many packages we can successfully
integrate into our database. In particular, we can report the share of
packages that we can connect to repos on the social coding platforms GitHub
and Gitlab. As we will see below, we have a very high rate of linkage.
Moreover, the packages that we could not integrate are typically those with
very few downloads. This suggests that most projects on Cargo that do not
appear on GitHub or Gitlab are small personal projects or preliminary work.

### Package Coverage among Repositories

Some Rust packages hosted on Cargo could not be linked to repos on GitHub or
Gitlab. They either are on a different platform, for example on Bitbucket,
Google Cloud, Sourceforge, or on personal websites. Still others had a link to
a GitHub or Gitlab domain (i.e. a repo) but could not be cloned. This can
happen if a link was incorrectly transcribed, if the repo was deleted, or if
the package is listed only for name squatting or test purposes and does not
correspond to any repository. Specifically, out of 78,935 packages, 63,578
were linked to a repository on GitHub or Gitlab and 59,106 of these were
successfully cloned, although only 44,379 packages pointed to distinct URLs
(and 40,927 were cloned). Hypothesizing that the most important packages are
the most downloaded ones, we can see in Figure 2 that our coverage increases
among the most important packages, measured by use (downloads).

Across the cloned repos, we gathered 4,921,359 commits, the elemental units of
contribution in the git version control system. Excluding 162 bots identified
among the GitHub accounts, these contributions were made by 51,663 GitHub
users and 440 Gitlab users. The raw data contains 78,197 identifying email
addresses, highlighting the significant amount of disambiguation of author
identities our pipeline implements. 12,051 of them could not be associated to
a GitHub or Gitlab account.

Figure 2: Data coverage: we check the number of Rust packages for which we
could identify and download a corresponding Git repo (from GitHub or Gitlab)
in terms of their use, measured in downloads. We could link a large majority
of packages to repos, and have a significantly higher success rate if we
consider packages that have been downloaded more often.

### Comparison with data from GHTorrent

We compared our data with data collected in the GHTorrent project [15]. The
GHTorrent project aims to collect all activity on GitHub for use in research.
As we have already noted, most activity in the Rust ecosystem takes place on
GitHub, with a small but significant share taking place on Gitlab. Besides the
inclusion of Gitlab data, we observed that our data contains a significant
amount of activity hosted on GitHub that is missing from the GHTorrent
database (the SQL version), when comparing the data in our dataset tagged as
happening before the last date of user creation in the GHTorrent database –
May 31st 2019, just before midnight – and corresponding to the repositories
that could be cloned. Specifically, we found only 14,596 unique users (vs.
22,066 GitHub users in our database), 13,68616,608 unique repos (vs 15,992
identified GitHub repos in our database), and 1,315,396 unique commits (vs
1,934,310 GitHub hosted commits in our database before the last date of
GHTorrent). The GHTorrent project uses the GitHub REST API and collects data
from the public event timeline using user-donated API keys. Outages on either
the GHTorrent side or on the GitHub REST API, or rate limited API keys may
explain missing data. While GHTorrent remains an excellent source of dataset
for all of GitHub, these comparisons suggests that for a focused look at a
single ecosystem, a customised pipeline can significantly increase data
coverage. More detailed statistics concerning the comparison can be found on
Figshare in the file `ghtorrent_comparison.yml`.

## Usage Notes

Figure 3: Illustrative plots from demonstration notebooks indicating potential
data-processing workflows. A) Average number of dependencies per package at
the beginning of each year. B) Evolution of the number of transitive
dependencies per package. C) The dependency network of the 100 most downloaded
Rust packages in early 2022. D) Time series of monthly downloads of three
successful Rust packages. E) Bipartite adjacency matrix of users/developers
and the repos they work on in the year 2021, lightly filtered. F) Developer-
developer collaboration network in 2021. G) Developer-developer collaboration
network, filtered for developers collaborating on at least three repos.

To demonstrate how to read in and analyse the database, we provide short
Jupyter notebooks that extract data and carry out elementary data
manipulations, with the data aggregated at the monthly level. These notebooks
are included with the other software in our materials. In one, we create the
dependency network of the Rust ecosystem at different times and measure its
growth. This data can be used to study ecosystem health, as errors and issues
are known to spread through these networks [6]. In another, we plot the time
series of stars and downloads of specific repositories over time. Such time
series can be used to study the dynamics of success at a fine-grained level
[8]. In a third, we show how to load in the data of developers and packages as
a rectangular matrix, which can be analysed as a bipartite network or, after a
projection, as a developer-developer collaboration network. The bipartite
network could be used to study the overall complexity of the ecosystem [44,
45], while the developer network reveals patterns of collaborations between
projects [46]. In Figure 3 we present several illustrative examples of
descriptive analyses resulting from the demonstration notebooks.

These vignettes together indicate how this dataset can be used to explore the
interactions between social collaboration, technical dependencies, and the
success and usage of components of a large software system. The dynamic
interactions between these layers of the data offer significant potential for
research relating to massive decentralized cooperation (similar to Wikipedia
[47, 48]), the dynamics of teams and their success in digital communities
[21], and the evolution of software systems [49].

## Code availability

Code to recreate the database is included in our Figshare upload:
https://figshare.com/s/93158d03416765444650. The software is written in the
Python programming language. The database can be created as either PostgreSQL
or SQLite database. Version requirements are recorded in the project’s Readme
file.

## Acknowledgements

The extraction and construction of the database has been partially funded by
Sony Computer Science Laboratories Paris under a dedicated contract with the
Complexity Science Hub Vienna. This work has also been partially supported by
the Austrian Research Promotion Agency FFG project # 882184: CSH-Fortsetzung.
The authors thank Amélie Desvars-Larrive for advice regarding the manuscript.

## Author contributions statement

WS, JW, VDPS, VL, ST coordinated the production of the dataset. WS designed
the database schema, created the tables, and carried out the dataset
validation checks. JW implemented the exploratory analyses and demonstrations.
VL and ST supervised and mentored the team. All authors contributed to writing
the data descriptor.

## Competing interests

The authors declare no competing interests.

## References

  * [1] Eghbal, N.  _Working in public: The making and maintenance of open source software_ (Stripe Press, 2020).
  * [2] Lerner, J. & Tirole, J.  Some simple economics of open source.  _The Journal of Industrial Economics_ 50, 197–234 (2002).
  * [3] Greenstein, S. & Nagle, F.  Digital dark matter and the economic contribution of Apache.  _Research Policy_ 43, 623–631 (2014).
  * [4] Nagle, F.  Learning by contributing: Gaining competitive advantage through contribution to crowdsourced public goods.  _Organization Science_ 29, 569–587 (2018).
  * [5] Nagle, F.  Open source software and firm productivity.  _Management Science_ 65, 1191–1215 (2019).
  * [6] Decan, A., Mens, T. & Grosjean, P.  An empirical comparison of dependency network evolution in seven software packaging ecosystems.  _Empirical Software Engineering_ 24, 381–416 (2019).
  * [7] Zöller, N., Morgan, J. H. & Schröder, T.  A topology of groups: What github can tell us about online collaboration.  _Technological Forecasting and Social Change_ 161, 120291 (2020).
  * [8] Sinatra, R., Wang, D., Deville, P., Song, C. & Barabási, A.-L.  Quantifying the evolution of individual scientific impact.  _Science_ 354, aaf5239 (2016).
  * [9] Cataldo, M., Herbsleb, J. D. & Carley, K. M.  Socio-technical congruence: a framework for assessing the impact of technical and work dependencies on software development productivity.  In _Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement_ , 2–11 (2008).
  * [10] Jung, R., Jourdan, J.-H., Krebbers, R. & Dreyer, D.  Safe systems programming in rust.  _Communications of the ACM_ 64, 144–152 (2021).
  * [11] Kalliamvakou, E. _et al._ An in-depth study of the promises and perils of mining GitHub.  _Empirical Software Engineering_ 21, 2035–2071 (2016).
  * [12] Fry, T., Dey, T., Karnauch, A. & Mockus, A.  A dataset and an approach for identity resolution of 38 million author ids extracted from 2b git commits.  In _Proceedings of the 17th international conference on mining software repositories_ , 518–522 (2020).
  * [13] Gote, C. & Zingg, C.  gambit–An Open Source Name Disambiguation Tool for Version Control Systems.  In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_ , 80–84 (IEEE, 2021).
  * [14] Golzadeh, M., Decan, A., Legay, D. & Mens, T.  A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments.  _Journal of Systems and Software_ 175, 110911 (2021).
  * [15] Gousios, G. & Spinellis, D.  Ghtorrent: Github’s data from a firehose.  In _2012 9th IEEE Working Conference on Mining Software Repositories (MSR)_ , 12–21 (IEEE, 2012).
  * [16] Pietri, A., Spinellis, D. & Zacchiroli, S.  The software heritage graph dataset: public software development under one roof.  In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 138–142 (IEEE, 2019).
  * [17] Ma, Y., Bogart, C., Amreen, S., Zaretzki, R. & Mockus, A.  World of code: an infrastructure for mining the universe of open source vcs data.  In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 143–154 (IEEE, 2019).
  * [18] Roberts, J. A., Hann, I.-H. & Slaughter, S. A.  Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects.  _Management science_ 52, 984–999 (2006).
  * [19] Szell, M. & Thurner, S.  Measuring social dynamics in a massive multiplayer online game.  _Social networks_ 32, 313–329 (2010).
  * [20] Monechi, B., Pullano, G. & Loreto, V.  Efficient team structures in an open-ended cooperative creativity experiment.  _Proceedings of the National Academy of Sciences_ 116, 22088–22093 (2019).
  * [21] Klug, M. & Bagrow, J. P.  Understanding the group dynamics and success of teams.  _Royal Society open science_ 3, 160007 (2016).
  * [22] Ma, Y.  Constructing supply chains in open source software.  In _2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion)_ , 458–459 (IEEE, 2018).
  * [23] Ohm, M., Plate, H., Sykosch, A. & Meier, M.  Backstabber’s knife collection: A review of open source software supply chain attacks.  In _International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_ , 23–43 (Springer, 2020).
  * [24] Decan, A. & Mens, T.  What do package dependencies tell us about semantic versioning?  _IEEE Transactions on Software Engineering_ 47, 1226–1240 (2019).
  * [25] Scholtes, I., Mavrodiev, P. & Schweitzer, F.  From Aristotle to Ringelmann: a large-scale analysis of team productivity and coordination in Open Source Software projects.  _Empirical Software Engineering_ 21, 642–683 (2016).
  * [26] Gote, C., Scholtes, I. & Schweitzer, F.  git2net-mining time-stamped co-editing networks from large git repositories.  In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 433–444 (IEEE, 2019).
  * [27] Dabbish, L., Stuart, C., Tsay, J. & Herbsleb, J.  Social coding in GitHub: transparency and collaboration in an open software repository.  In _Proceedings of the ACM 2012 conference on computer supported cooperative work_ , 1277–1286 (2012).
  * [28] Marlow, J., Dabbish, L. & Herbsleb, J.  Impression formation in online peer production: activity traces and personal profiles in GitHub.  In _Proceedings of the 2013 conference on Computer supported cooperative work_ , 117–128 (2013).
  * [29] Borges, H. & Valente, M. T.  What’s in a GitHub star? Understanding repository starring practices in a social coding platform.  _Journal of Systems and Software_ 146, 112–129 (2018).
  * [30] Moldon, L., Strohmaier, M. & Wachs, J.  How gamification affects software developers: Cautionary evidence from a natural experiment on GitHub.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , 549–561 (IEEE, 2021).
  * [31] Papoutsoglou, M., Ampatzoglou, A., Mittas, N. & Angelis, L.  Extracting knowledge from on-line sources for software engineering labor market: A mapping study.  _IEEE Access_ 7, 157595–157613 (2019).
  * [32] Shimada, N., Xiao, T., Hata, H., Treude, C. & Matsumoto, K.  Github sponsors: Exploring a new way to contribute to open source.  _arXiv preprint arXiv:2202.05751_ (2022).
  * [33] Corominas-Murtra, B., Goñi, J., Solé, R. V. & Rodríguez-Caso, C.  On the origins of hierarchy in complex networks.  _Proceedings of the National Academy of Sciences_ 110, 13316–13321 (2013).
  * [34] Montandon, J. E., Silva, L. L. & Valente, M. T.  Identifying experts in software libraries and frameworks among GitHub users.  In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 276–287 (IEEE, 2019).
  * [35] Wessel, M. _et al._ The power of bots: Characterizing and understanding bots in OSS projects.  _Proceedings of the ACM on Human-Computer Interaction_ 2, 1–19 (2018).
  * [36] Wessel, M. _et al._ Bots for pull requests: The good, the bad, and the promising.  In _Proceedings of the 44th ACM/IEEE International Conference on Software Engineering (ICSE’22)_ , vol. 26, 16 (ACM/IEEE, 2022).
  * [37] Vasilescu, B. _et al._ Gender and tenure diversity in github teams.  In _Proceedings of the 33rd annual ACM conference on human factors in computing systems_ , 3789–3798 (2015).
  * [38] Rossi, D. & Zacchiroli, S.  Worldwide gender differences in public code contributions: and how they have been affected by the covid-19 pandemic.  In _2021 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ (2022).
  * [39] Rastogi, A., Nagappan, N., Gousios, G. & van der Hoek, A.  Relationship between geographical location and evaluation of developer contributions in GitHub.  In _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 1–8 (2018).
  * [40] Braesemann, F., Stoehr, N. & Graham, M.  Global networks in collaborative programming.  _Regional Studies, Regional Science_ 6, 371–373 (2019).
  * [41] Wachs, J., Nitecki, M., Schueller, W. & Polleres, A.  The Geography of Open Source Software: Evidence from GitHub.  _Technological Forecasting and Social Change_ 176, 121478 (2022).
  * [42] Prana, G. A. A. _et al._ Including everyone, everywhere: Understanding opportunities and challenges of geographic gender-inclusion in OSS.  _IEEE Transactions on Software Engineering_ (2021).
  * [43] Gousios, G. & Spinellis, D.  Mining software engineering data from GitHub.  In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_ , 501–502 (IEEE, 2017).
  * [44] Hidalgo, C. A. & Hausmann, R.  The building blocks of economic complexity.  _Proceedings of the national academy of sciences_ 106, 10570–10575 (2009).
  * [45] Servedio, V. D. P., Buttà, P., Mazzilli, D., Tacchella, A. & Pietronero, L.  A new and stable estimation method of country economic fitness and product complexity.  _Entropy_ 20, 783 (2018).
  * [46] Singh, P. V.  The small-world effect: The influence of macro-level properties of developer collaboration networks on open-source project success.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 20, 1–27 (2010).
  * [47] Brandes, U., Kenis, P., Lerner, J. & Van Raaij, D.  Network analysis of collaboration structure in wikipedia.  In _Proceedings of the 18th international conference on World wide web_ , 731–740 (2009).
  * [48] Mestyán, M., Yasseri, T. & Kertész, J.  Early prediction of movie box office success based on wikipedia activity big data.  _PloS one_ 8, e71226 (2013).
  * [49] Solé, R. & Valverde, S.  Evolving complexity: how tinkering shapes cells, software and ecological networks.  _Philosophical Transactions of the Royal Society B_ 375, 20190325 (2020).