# Search for nearly degenerate higgsinos via photon fusion with hadronic
channel at the LHC

Hang Zhou  School of Microelectronics and Control Engineering, Changzhou
University, Changzhou, 213164, China
Email<EMAIL_ADDRESS>Ning Liu  Physics Department and Institute of
Theoretical Physics, Nanjing Normal University, Nanjing, 210023, China
Email<EMAIL_ADDRESS>
###### Abstract

Electroweak scale higgsinos with a nearly degenerate spectrum in
supersymmetric models are well-motivated, but generally less constrained at
collider experiments as the decay products are often too soft to detect.
Initial photon fusions alongside the collision of protons at the Large Hadron
Collider (LHC) have drawn attention recently as a way to search for new
physics with such kind of spectra. In this paper, we demonstrate a search
strategy for chargino pair production from photon fusion $pp\to
p(\gamma\gamma\to\tilde{\chi}^{+}_{1}\tilde{\chi}^{-}_{1})p$ at the 13 TeV LHC
via the hadronic decay channel, as a probe for the compressed spectra of
higgsinos. Forward detectors make it possible to detect the outgoing protons
after emitting the initial photons in these processes. We here provide simple
event selections on missing energy and transverse momentum of jets, which are
effective to reach significant sensitivity. The chargino mass can be excluded
at 95% C.L. up to about 235 GeV with the mass difference $\Delta
m(\tilde{\chi}^{\pm}_{1},\tilde{\chi}^{0}_{1})$ being a only few GeV with the
integrated luminosity of 3 ab-1. With a relatively small luminosity of 100
fb-1, the $2\sigma$ exclusion bounds can also exceed current experimental
limits in the range of $\Delta m=1\sim 2$ GeV, reaching over 130 GeV for
chargino mass.

##  I Introduction

Supersymmetric models have long been perceived as one of the natural
extensions of the Standard Model (SM) to explain the naturalness problem as
well as other beyond SM phenomena, including the nature of the dark matter.
Tremendous effort has been made for decades to search for the supersymmetric
(SUSY) particles through a variety of experiments, especially at the Large
Hadron Collider (LHC), yet with null results to date. Consequently, SUSY
particles have been cornered in an ever narrower parametric space and the low-
energy SUSY models are severely restricted. Based on full set of data during
Run 2 at the LHC (with an integrated luminosity of 139 fb-1 at a 13 TeV
center-of-mass energy), recent analyses on the search for pair production of
gluinos or squarks have excluded their masses up to 2.2 TeV and 1.7 TeV,
respectively, assuming either R-parity conservation or violation ATLAS:2023afl
. And the chargino mass can be excluded in the range from 260 to 520 GeV for a
mass splitting between chargino and neutralino $\Delta
m(\tilde{\chi}^{\pm}_{1},\tilde{\chi}^{0}_{1}$) around the electroweak scale,
with a massless neutralino ATLAS:2023act . As the mass splitting gets smaller
below 100 GeV, the mass bound for chargino ranges from 200 to 300 GeV within a
simplified SUSY model ATLAS:2021moa . For an even more compressed spectra of
the electroweakinos, the most constraint lower limit on chargino mass still
comes from the LEP2 experiment around 90 GeV LEPbound . Rough patterns appear
that the restrictions on the electroweakino masses get weaker with a smaller
mass splitting.

This less restrictive corner of parameter space survives partly because of the
low cross sections for charginos and neutralinos at the LHC as they cannot be
produced through strong interaction. On the other hand, the compressed mass
splitting between them often leads to soft leptons or jets in the final state
that can hardly be detected and requires additional photons or jets from the
initial state radiation to serve as a trigger for detection Han:2013usa ;
Schwaller:2013baa ; Baer:2014kya . Last but not the least, difficulties in
accurate reconstruction for neutralinos as a missing momentum also make the
compressed scenario less constrained. Strategies have been proposed taking
advantage of the disappearing tracks to probe such spectra with the mass
splitting down to several hundreds of MeV Fukuda:2017jmk ; Mahbubani:2017gjh .
For the spectra not squeezed that much, it has been suggested that utilizing
forward detectors such as CMS-TOTEM Precision Proton Spectrometer CTPPS2014
or ATLAS Forward Proton AFP2015 , the LHC can be used as a photon collider to
probe light SUSY particles characteristic of nearly degeneracy
Beresford:2018pbt ; Harland-Lang:2018hmi ; Godunov:2019jib ; Zhou:2022jgj .
Unlike the deep inelastic collisions studied in the traditional searches at
the hadron collider, these proposals target the unsuccessful ones termed as
the ultraperipheral collision, throughout which the colliding protons remain
intact and can be detected by the above-mentioned forward detectors. What
matters is that electromagnetic field around the high-velocity protons can be
approximated to equivalent on-shell photons Budnev:1975poe , which may also
collide with each other giving rise to photon fusions at the LHC

$\displaystyle\gamma\gamma\to X^{+Q}X^{-Q},$ (1)

where $X^{\pm Q}$ refer to particles carrying $Q$ units of electric charge.
Detection of the outgoing unscathed protons can then help determine the
3-dimensional missing momentum in the final states, rather than the transverse
missing one, realizing a more sensitive search. Further, the cross sections of
such ultraperipheral collisions have been demonstrated to be large enough for
promising search for new physics beyond the SM Ohnemus:1993qw ; Schul:2008sr ;
Fichet:2013gsa ; Khoze:2017igg .

As a complementary research to our previous work Zhou:2022jgj  searching for
the nearly degenerate higgsinos via photon fusion at the LHC utilizing the
leptonic decay channel of the $W$ boson, we will demonstrate in this paper
that the hadronic channel can also reach significant sensitivity for such a
compressed scenario of SUSY models. In the next section, we introduce the
general parameter setting for the investigated scenario and relevant processes
as our signal and background. Simulations and search strategies are presented
in Section III. Section IV is the conclusion.

##  II The higgsino world scenario and signal at the LHC

We consider the aforementioned compressed scenario within the framework of the
Minimal Supersymmetric Standard Model (also known as the MSSM), which realizes
SUSY in a way as simple as possible and at the same time, can be satisfactory
with regard to the naturalness problem. In a quantitative way, minimizing the
Higgs potential relates the mass of the SM $Z$ boson to the two Higgs doublets
in the MSSM and the superpartner higgsino at tree level Arnowitt:1992qp

$\displaystyle\frac{m^{2}_{Z}}{2}=\frac{m^{2}_{H_{u}}\tan^{2}\beta-m^{2}_{H_{d}}}{1-\tan^{2}\beta}-\mu^{2},$
(2)

in which $\mu$ is the mass parameter of higgsino, $m_{H_{u}(H_{d})}$ is the
soft SUSY breaking mass of the Higgs doublet $H_{u}$($H_{d}$). The ratio of
their vacuum expected values is defined as $\tan\beta\equiv v_{u}/v_{d}$. A
natural argument then arises for the terms of Eq.(2) to be in a similar order
of magnitudes, if we require an observed $Z$ mass and no significant fine-
tuning in parameters Barbieri:1987fn . In consideration of the 125 GeV SM
Higgs boson and current lower limits on top squark mass and gluino mass around
2 TeV, one of the commonly accepted schemes is called the higgsino world where
superscalars are of order a few TeV while the higgsino masses are relatively
low at sub-TeV Kane:1997fa . Such schemes include the minimal supergravity
Feng:1999mn  and the radiative nSUSY Baer:2012up , and the spectra of light
higgsinos also lead to rich phenomenology related to dark matter search, muon
$g-2$ and collider experiments Hikasa:2015lma ; Han:2020exx ; Lv:2022pme ;
Abdughani:2017dqs . Assuming a wide gap between $\mu$ and $M_{1,2,3}$, the
masses for bino, wino and gluino respectively, the lightest mass eigenstates
chargino and neutralino can then acquire a relatively tiny mass splitting of a
few GeV with a relatively large $\tan\beta>10$ Giudice:1995np ,

$\displaystyle\Delta
m({\tilde{\chi}^{\pm}_{1},\tilde{\chi}^{0}_{1}})=\frac{m^{2}_{W}}{2M_{1}}\left(1-\sin
2\beta-\frac{2\mu}{M_{2}}\right)+\frac{m^{2}_{W}}{2M_{2}}(1+\sin
2\beta)\tan^{2}\theta_{W},$ (3)

in which $\theta_{W}$ is the Weinberg angle and $m_{W}$ the SM $W$ mass. As
discussed in Section I, this light and nearly degenerate spectrum of
electroweakinos is less constrained by collider experiments and we use SUSYHIT
package Djouadi:2006bz  to generate such mass spectra which will then be used
for event generation and further analysis.

(a)

(b)

(c)

(d)

Figure 1: Feynman diagrams of the signal and the SM background of hadronic
channel from chargino decay via equivalent photon fusion at the LHC.

In the pair production of charginos from photon fusion through the above-
mentioned UPC (Eq.(1)), virtual $W$ bosons can mediate the subsequent decays
to neutralinos with either leptonic or hadronic final states, as the mass
difference between the parent chargino and neutralino is tiny and real $W$
bosons cannot be produced in this compressed spectrum. We focus on the latter
case in the present paper as a complementary study to the former Zhou:2022jgj
, the Feynman diagram of which as our signal is displayed in FIG. 1(a):

$\displaystyle pp\to
p\left(\gamma\gamma\to\tilde{\chi}^{+}_{1}\tilde{\chi}^{-}_{1}\to\tilde{\chi}^{0}_{1}W^{+}\tilde{\chi}^{0}_{1}W^{-}\to\tilde{\chi}^{0}_{1}\tilde{\chi}^{0}_{1}+\text{jets}\right)p\,.$
(4)

Feynman diagrams in the present paper are drawn using the TikZ-Feynman package
Ellis:2016jkw . Final states consist of a large amount of missing momentum and
multiple soft jets, as well as two intact outgoing protons which will be seen
by the forward detectors. Similar signatures can be produced within the SM
through the following processes,

$\displaystyle pp\to{}$ $\displaystyle p\left(\gamma\gamma\to
W^{+}W^{-}\to\ell^{\pm}\nu+\text{jets}\right)p\,,$ (5) $\displaystyle pp\to{}$
$\displaystyle p\left(\gamma\gamma\to
q\overline{q}\to\ell^{\pm}\nu+\text{jets}\right)p\,,$ (6) $\displaystyle
pp\to{}$ $\displaystyle
p\left(\gamma\gamma\to\ell^{+}\ell^{-}\to\overline{\nu}\nu+\text{jets}\right)p\,,$
(7)

as shown in FIG. 1 (b)(c)(d). In the above processes, final leptons from
mediating $W$ bosons may be misidentified and fake jets, leading to a
signature of multiple jets with missing energy made up of neutrinos, mimicking
the final states of the signal Eq. (4). Note that the corresponding Hermitian
conjugate processes are not illustrated in the Feynman diagrams for
simplicity, which we will include in the event generation and further
analysis. And we did not differentiate the directions of final quarks or anti-
quarks in these diagrams and simply show them as jets. In Fig. 1 (b) and (c),
the final quarks from $W$ decay can go through strong interaction radiating
gluons and form more jets. Mediating leptons in Fig. 1 (d) can be muons and
tauons $\ell=\mu,\tau$. Although the third one Eq. (7) (Fig. 1 (d)) in the
background likens the signal most in the aspect of topological structure, the
first one Eq. (5) makes more contribution because its cross section is much
larger than the other two. To illustrate the huge gap among the three
background processes, the cross section for the $W$-mediated process (Eq. (5))
at 13 TeV LHC is of order 0.01 pb, while for the fermion-mediated ones (Eq.
(6) and Eq. (7)), the cross sections approximate to $10^{-6}$ pb and $10^{-7}$
pb, respectively. Apart from the listed SM processes, $WWZ$ and $t\bar{t}$
from photon fusion can also mimic the signal. But the production of three
bosons is estimated to be rather subdominant even if it can be irreducible.
And the $t\bar{t}$ events can be well suppressed by b-veto. In consideration
of these facts, we consider only Eq. (5) (Fig. 1 (b)) as our background.

(a)

(b)

(c)

(d)

Figure 2: Kinematic distributions of signal $pp\to
p\left(\gamma\gamma\to\tilde{\chi}^{+}_{1}\tilde{\chi}^{-}_{1}\to\tilde{\chi}^{0}_{1}W^{+}\tilde{\chi}^{0}_{1}W^{-}\to\tilde{\chi}^{0}_{1}\tilde{\chi}^{0}_{1}+\text{jets}\right)p$
versus background $pp\to p\left(\gamma\gamma\to
W^{+}W^{-}\to\ell^{\pm}\nu+\text{jets}\right)p$, for benchmarks of chargino
mass $m_{\tilde{\chi}^{\pm}_{1}}=125$ and 245 GeV, with the mass difference
between chargino and neutralino $\Delta m$=1, 10 and 20 GeV.

##  III Simulation and search strategies

To arrive at the sensitivity of searching for the hadronic decay channel from
chargino Eq. (4) at the LHC, we perform simulations for both of the signal and
the SM background. Parton-level events are generated via MG5. Parton showering
and detector simulation are performed using pythia 8.2 Sjostrand:2014zea  and
Delphes 3.5.0 deFavereau:2013fsa . checkmate2 (version 2.0) Dercks2017  is
also used to connect the workflow of the above processes and for further
analysis of the final-state kinematics and cutflow.

The signal and background differ in the most distinctive way in regard to the
magnitude of missing momentum and the softness of final jets. In the signal
process, missing momentum comes from massive neutralinos while the jets,
coming from an off-shell $W$ boson, keep only a small fraction of the total
energy, leading to multiple rather soft jets. While in the background process,
the missing momentum is consisted of massless neutrinos (it is quite safe to
assume the SM neutrinos to be massless in our simulation) and the final jets
are decayed from much more energetic $W$ bosons. Distributions of missing
transverse energy $\not{E}_{T}$ and transverse momentum of the leading jet
$p_{T}(j_{1})$ are presented in FIG. 2 to show the kinematic difference
between signal and background. The benchmarks are chosen as
$m_{\tilde{\chi}^{\pm}_{1}}=$125 and 245 GeV, with $\Delta m=$1, 10 and 20
GeV. Simple patterns appear in the kinematic distributions, as expected for
the above reasons on the features of signal and background. The histograms of
missing transverse energy tend to center around smaller values for the SM
background than that for the signal (FIG. 2(a)(c)). And harder jets directly
decayed from $W$ bosons lead to peaks at larger values for the background than
that for the signal (FIG. 2(b)(d)).

$E_{\gamma}$ (GeV) |  (0,100] |  (100,120] |  (120,150] |  (150,400] |  (400,$+\infty$)
---|---|---|---|---|---
Eff. | 0 | 50% | 70% | 90% | 80%
Table 1: Acceptance rates for initial EPA photons for difference ranges of
energies, which are equivalent to tagging efficiencies for the outgoing
protons corresponding to their energy losses CTPPS2014 ; AFP2015 .

According to these kinematic distributions, event selection will then be
performed to suppress the background and to achieve better significance. But
before that, note that the outgoing protons are not to be detected at a 100%
efficiency by the forward detectors, but rather at certain acceptance rates
($\xi$), which mainly depend on the energy losses of the protons. For the
range of $\xi\in(0.015,0.15)$, acceptance rates for outgoing protons can be
close to 100% CTPPS2014 ; AFP2015 , corresponding to energies of equivalent
photons with $E_{\gamma}\in(100,1000)$ GeV at the 13 TeV LHC. Since
phenomenological studies generally indicate relatively lower acceptances $\sim
90\%$ for outgoing protons when they emit photons Khoze2002 , hence we in the
present work adopt conservative values for the proton tagging efficiencies
varied with their energy losses, as shown in TABLE. 1. And we then apply these
rates to every generated event, serving as a procedure of pre-selection to
realize the effect of forward detectors tagging the outgoing protons with
these energy-dependant efficiencies. This is done by checking the energies of
the initial fusing photons emitted by the protons going through
ultraperipheral collision, since the protons energy losses actually correspond
to energies of the initial photons exactly. Four-momenta of protons are also
smeared in advance using a 5%-width Gaussian function to simulate the
resolution of the forward detectors $\simeq 5$ GeV AFP2015 . PYLHE packages
pylhe  are used when processing and smearing photon momenta from parton-level
events.

cuts SR |  $\Delta m=1\sim 5$ GeV |  $\Delta m=10$ GeV |  $\Delta m=15$ GeV |  $\Delta m=20$ GeV
---|---|---|---|---
Cut-1 |  $\not{E}_{T}>10$ GeV |  $\not{E}_{T}>10$ GeV |  $\not{E}_{T}>10$ GeV |  $\not{E}_{T}>10$ GeV
Cut-2 |  $P_{T}(j_{1})<17$ GeV |  $P_{T}(j_{1})<18$ GeV |  $P_{T}(j_{1})<21$ GeV |  $P_{T}(j_{1})<24$ GeV
Table 2: Cuts applied to proton-tagged events in four different signal regions
divided according to mass splitting $\Delta m$ between chargino and
neutralino. Cut-1 is on missing transverse energy $\not{E}_{T}$ and cut-2 on
transverse momentum $P_{T}$ of leading jet ($j_{1}$).

After the pre-selection, the surviving events are ones that are detected by
the forward detectors, which will then go through two further cuts on missing
transverse energy and on transverse momentum of the leading jet. These cuts
are applied based on their distributions for benchmarks with nearly degenerate
chargino and neutralino of hundreds GeV scale, that is, chargino mass
$m_{\tilde{\chi}^{\pm}_{1}}\in[105,275]$ GeV and mass splitting with
neutralino $\Delta m=1,2,5,10,15,20$ GeV (Some of these benchmark points are
shown for illustration in FIG. 2). To realize better significance in this
parametric space of interest, we perform the cutflow in four signal regions
according to different ranges of the mass splitting, as presented in TABLE. 2.
At each step of event selection, the effective cross sections are also
calculated for both signal and background, shown in TABLE. 3 for one of the
above benchmarks ($\Delta m$=10 GeV and $m_{\tilde{\chi}^{\pm}_{1}}=195$ GeV)
as an example. The cutflow includes as well the pre-selection as tagging two
outgoing protons according to tagging rates in TABLE. 1. It can be seen from
the cutflow that the most effective cut on the background events is requiring
a relatively small $P_{T}$ for the leading jets, since the energies of the
final jets in the signal come from a tiny mass splitting between chargino and
neutralino. But for the background, there is no such limitation and the curves
for distributions of $P_{T}(j_{1})$ slope more gently than that of the signal.

|  $\Delta m$ = 10 GeV, $m_{\tilde{\chi}^{\pm}_{1}}=195$ GeV |  $\gamma\gamma\to W^{+}W^{-}\to jj\ell\nu$
---|---|---
No cuts applied |  $8.670\times 10^{-5}$ pb |  $2.227\times 10^{-2}$ pb
Outgoing protons tagged |  $3.967\times 10^{-5}$ pb |  $4.468\times 10^{-3}$ pb
Cut-1: $\not{E}_{T}>10$ GeV |  $3.631\times 10^{-5}$ pb |  $4.346\times 10^{-3}$ pb
Cut-2: $P_{T}(j_{1})<18$ GeV |  $8.626\times 10^{-6}$ pb |  $4.469\times 10^{-5}$ pb
Table 3: Effective cross sections after each step of cutflow listed in TABLE.
2 for signal in the signal region of $\Delta m$=10 GeV and
$m_{\tilde{\chi}^{\pm}_{1}}=195$ GeV, and for the SM background $W^{+}W^{-}$
events from photon fusion.  Figure 3: $2\sigma$ exclusion on chargino mass and
the mass difference $\Delta m$ with neutralino, based on a simulated search
for hadronic channel via initial photon fusion at the LHC.

Finally, statistical significance is calculated according to the formula
$\alpha=S/\sqrt{B+(\beta B)^{2}}$, where $S$ ($B$) is the events number of
signal (background) after the cutflow in TABLE. 2. Integrated luminosity is
adopted as 100 fb-1 and 3 ab-1, and the systematic uncertainty is taken as
$\beta=5\%$. $2\sigma$ exclusion limits are then obtained corresponding to
luminosities of $\mathcal{L}=100\,\text{fb}^{-1}$ (black solid line) and
$\mathcal{L}=3\,\text{ab}^{-1}$ (red solid line) in FIG. 3. The results of the
authors’ previous work through leptonic channel $pp\to
p\left(\gamma\gamma\to\tilde{\chi}^{+}_{1}\tilde{\chi}^{-}_{1}\to\ell^{+}\ell^{-}+\not{E}_{T}\right)p$
are also displayed in the figure. As comparison, we present the current bounds
from collider experiments for this space of parameter, including the blue
region of the ATLAS search for soft final leptons at
$\mathcal{L}=139\,\text{fb}^{-1}$ Aaboud2018  and the gray part from LEP2
experiment Heister2002 . Although the leptonic channel exhibits better
sensitivity for obvious reasons that leptonic final states can be detected
much more effectively than the hadronic ones at a hadronic collider, the
present work demonstrates that hadronic channel can also exceed limits of
current experiments, reaching about $m_{\tilde{\chi}^{\pm}_{1}}~{}130$ GeV
with the mass splitting of only $\sim 1$ GeV for
$\mathcal{L}=100\,\text{fb}^{-1}$. If the integrated luminosity is increased
to 3 ab-1, chargino mass can be excluded over 200 GeV, at most 235 GeV, for
$\Delta m$ of a few GeV. In the range of a slightly large mass difference of
up to 20 GeV, the $2\sigma$ exclusion limits on chargino mass can also reach
$\sim 190$ GeV.

##  IV Conclusion

We demonstrate in this paper a search strategy for photon fusion to chargino
pair at the 13 TeV LHC via the hadronic decay channel, as a probe for the
nearly degenerate spectra of higgsinos. Colliding protons remain intact in
these processes and can be detected by forward detectors like the AFP and CT-
PPS at CERN. Simple event selections on missing energy and transverse momentum
of jets are effective enough to reach significant sensitivity. The chargino
mass can be excluded at 95%C.L. up to around 235 GeV with the mass difference
$\Delta m(\tilde{\chi}^{\pm}_{1},\tilde{\chi}^{0}_{1})$ being a only few GeV
under the integrated luminosity of 3 ab-1. Even with a relatively small
luminosity of 100 fb-1, the $2\sigma$ exclusion bounds can exceed current
experimental limits in the range of $\Delta m=1\sim 2$ GeV, reaching over 130
GeV for chargino mass.

##  V Acknowledgments

This work is supported by the Natural Science Foundation of Jiangsu Province
under Grant No. BK20230623, and the Natural Science Foundation of the Jiangsu
Higher Education Institutions of China under Grant No. 22KJB140007.

## References

  * (1) G. Aad et al. [ATLAS], JHEP 02, 107 (2024)
  * (2) G. Aad et al. [ATLAS], JHEP 12, 167 (2023)
  * (3) G. Aad et al. [ATLAS], Eur. Phys. J. C 81, no.12, 1118 (2021)
  * (4) http://lepsusy.web.cern.ch/lepsusy/
  * (5) C. Han, A. Kobakhidze, N. Liu, A. Saavedra, L. Wu and J. M. Yang, JHEP 02, 049 (2014)
  * (6) P. Schwaller and J. Zurita, JHEP 03, 060 (2014)
  * (7) H. Baer, A. Mustafayev and X. Tata, Phys. Rev. D 90, no.11, 115007 (2014)
  * (8) H. Fukuda, N. Nagata, H. Otono and S. Shirai, Phys. Lett. B 781, 306-311 (2018)
  * (9) R. Mahbubani, P. Schwaller and J. Zurita, JHEP 06, 119 (2017) [erratum: JHEP 10, 061 (2017)]
  * (10) M. Albrow et al. [CMS and TOTEM], CERN-LHCC-2014-021
  * (11) L. Adamczyk, E. Banaś, A. Brandt, M. Bruschi, S. Grinstein, J. Lange, M. Rijssenbeek, P. Sicho, R. Staszewski and T. Sykora, et al. CERN-LHCC-2015-009.
  * (12) L. Beresford and J. Liu, Phys. Rev. Lett. 123, no.14, 141801 (2019)
  * (13) L. A. Harland-Lang, V. A. Khoze, M. G. Ryskin and M. Tasevsky, JHEP 04, 010 (2019)
  * (14) S. I. Godunov, V. A. Novikov, A. N. Rozanov, M. I. Vysotsky and E. V. Zhemchugov, JHEP 01, 143 (2020)
  * (15) H. Zhou and N. Liu, JHEP 10, 092 (2022)
  * (16) V. M. Budnev, I. F. Ginzburg, G. V. Meledin and V. G. Serbo, Phys. Rept. 15, 181-281 (1975)
  * (17) J. Ohnemus, T. F. Walsh and P. M. Zerwas, Phys. Lett. B 328, 369-373 (1994)
  * (18) N. Schul and K. Piotrzkowski, Nucl. Phys. B Proc. Suppl. 179-180, 289-297 (2008)
  * (19) S. Fichet, G. von Gersdorff, O. Kepka, B. Lenzi, C. Royon and M. Saimpert, Phys. Rev. D 89, 114004 (2014)
  * (20) V. A. Khoze, A. D. Martin and M. G. Ryskin, J. Phys. G 44, no.5, 055002 (2017)
  * (21) R. L. Arnowitt and P. Nath, Phys. Rev. D 46, 3981-3986 (1992)
  * (22) R. Barbieri and G. F. Giudice, Nucl. Phys. B 306, 63-76 (1988)
  * (23) G. L. Kane, Nucl. Phys. B Proc. Suppl. 62, 144-151 (1998)
  * (24) J. L. Feng, K. T. Matchev and T. Moroi, Phys. Rev. Lett. 84, 2322-2325 (2000)
  * (25) H. Baer, V. Barger, P. Huang, A. Mustafayev and X. Tata, Phys. Rev. Lett. 109, 161802 (2012)
  * (26) K. i. Hikasa, J. Li, L. Wu and J. M. Yang, Phys. Rev. D 93, no.3, 035003 (2016)
  * (27) C. Han, M. L. López-Ibáñez, A. Melis, Ó. Vives, L. Wu and J. M. Yang, JHEP 05, 102 (2020)
  * (28) H. Lv, D. Wang and L. Wu, Phys. Rev. D 106, no.5, 055008 (2022)
  * (29) M. Abdughani, L. Wu and J. M. Yang, Eur. Phys. J. C 78, no.1, 4 (2018)
  * (30) G. F. Giudice and A. Pomarol, Phys. Lett. B 372, 253-258 (1996)
  * (31) A. Djouadi, M. M. Muhlleitner and M. Spira, Acta Phys. Polon. B 38, 635-644 (2007)
  * (32) J. Ellis, Comput. Phys. Commun. 210, 103-123 (2017)
  * (33) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen and P. Z. Skands, Comput. Phys. Commun. 191, 159-177 (2015)
  * (34) J. de Favereau et al. [DELPHES 3], JHEP 02, 057 (2014)
  * (35) D. Dercks, N. Desai, J. S. Kim, K. Rolbiecki, J. Tattersall and T. Weber, Comput. Phys. Commun. 221, 383-418 (2017)
  * (36) V. A. Khoze, A. D. Martin and M. G. Ryskin, Eur. Phys. J. C 23, 311-327 (2002)
  * (37) Lukas Heinrich, Matthew Feickert, and Eduardo Rodrigues.  pylhe: v0.3.0
https://github.com/scikit-hep/pylhe/tree/v0.3.0

  * (38) M. Aaboud et al. [ATLAS], Phys. Rev. D 97, no.5, 052010 (2018)
  * (39) A. Heister et al. [ALEPH], Phys. Lett. B 533, 223-236 (2002)


# Nalanda: A Socio-technical Graph Platform for Building Software Analytics
Tools at Enterprise Scale

Chandra Maddila<EMAIL_ADDRESS>Microsoft ResearchUSA ,  Suhas
Shanbhogue<EMAIL_ADDRESS>Microsoft ResearchIndia ,  Apoorva Agrawal
<EMAIL_ADDRESS>Microsoft ResearchIndia ,  Thomas Zimmermann
<EMAIL_ADDRESS>Microsoft ResearchUSA ,  Chetan Bansal
<EMAIL_ADDRESS>Microsoft ResearchUSA ,  Nicole Forsgren
<EMAIL_ADDRESS>Microsoft ResearchUSA ,  Divyanshu Agrawal
<EMAIL_ADDRESS>Microsoft ResearchIndia ,  Kim Herzig
<EMAIL_ADDRESS>MicrosoftUSA and  Arie van Deursen
<EMAIL_ADDRESS>Delft University of TechnologyThe Netherlands

(2022)

###### Abstract.

Software development is information-dense knowledge work that requires
collaboration with other developers and awareness of artifacts such as work
items, pull requests, and file changes. With the speed of development
increasing, information overload, and information discovery are challenges for
people developing and maintaining these systems. Finding information about
_similar_ code changes and _experts_ is difficult for software engineers,
especially when they work in large software systems or have just recently
joined a project. In this paper, we build a large-scale data platform named
Nalanda platform to address the challenges of information overload and
discovery. Nalanda contains two subsystems: (1) a large-scale socio-technical
graph system, named _Nalanda graph system_ , and (2) a large-scale index
system, named _Nalanda index system_ that aims at satisfying the information
needs of software developers.

To show the versatility of the Nalanda platform, we built two applications:
(1) a software analytics application with a news feed named MyNalanda that has
Daily Active Users (DAU) of 290 and Monthly Active Users (MAU) of 590, and (2)
a recommendation system for related work items and pull requests that
accomplished similar tasks (_artifact recommendation_) and a recommendation
system for subject matter experts (_expert recommendation_), augmented by the
Nalanda socio-technical graph. Initial studies of the two applications found
that developers and engineering managers are favorable toward continued use of
the news feed application for information discovery. The studies also found
that developers agreed that a system like Nalanda artifact and expert
recommendation application could reduce the time spent and the number of
places needed to visit to find information.

Collaborative software development, Socio-Technical Graphs, Recommender
Systems for Software Engineering, Empirical study

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM Joint European Software Engineering Conference and Symposium on the
Foundations of Software Engineering; November 14–18, 2022; Singapore,
Singapore††booktitle: Proceedings of the 30th ACM Joint European Software
Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE ’22), November 14–18, 2022, Singapore, Singapore††price:
15.00††doi: 10.1145/3540250.3558949††isbn: 978-1-4503-9413-0/22/11††ccs:
Software and its engineering Programming teams

##  1\. Introduction

Building software is a highly collaborative process that requires awareness of
the activities by many different stakeholders and interaction with many
different artifacts such as files, pull requests, and work items. At the same
time, large-scale software development creates lots of data about how people
work with each other and with software artifacts. As a consequence, finding
information can be hard, especially when software engineers work on large
software projects with thousands of files and team members. A lot of times
this knowledge about software development activity and expertise is hidden in
the form of software development process data and the interaction map between
stakeholders and artifacts. This data is hard to mine and represent in a form
that allows practitioners to build applications on top of this data. This is
primarily due to the scale at which this data is generated and the fact that
this data is scattered across disparate data sources and systems. Therefore,
it is hard to take full advantage of this data and extract hidden knowledge,
without employing a plethora of tailored tools, customized for each source
control system and the software development environment.

Socio-technical data that captures social and technical aspects of software
development (Sarma et al., 2009b) is often captured in graph structures. For
example, the Hipikat tool builds a project memory from past activities to
support newcomers with software modification tasks (Cubranic et al., 2005). In
2010, the Codebook framework was introduced with a focus on discovering and
exploiting relationships in software organizations to support inter-team
coordination (Begel et al., 2010). Codebook provided a graph and a query
language to support a wide range of applications: find the most relevant
engineers, find out why a recent change was made, and general awareness of
engineering activity (Begel and Zimmermann, 2010). Codebook was built for a
single team with 420 developers only.

With the advent of cloud services, the scale at which software development
happens and the volume of data generated during the software development
process increased significantly (Potvin and Levenberg, 2016; Ma et al., 2019).
To address the challenges that come with scale, in this paper, we present a
large-scale software analytics data platform named Nalanda111Nalanda is named
after an ancient university and knowledge center located in India. It is
famous for its huge corpus of scriptures, books, and knowledge repositories.
which is built on top of the software development activity data and the
artifacts. Nalanda builds a socio-technical graph at _enterprise scale_ , with
thousands of repositories. Additionally, the Nalanda index system, helps with
the search and recommendation of software development artifacts and the
experts. Nalanda stores its graph in a native graph database and optimizes
heavily to query complex relationships so that software analytics applications
can operate directly on the graph via cloud services and a high degree of
performance.

Figure 1. Nalanda’s Graph Schema

The Nalanda platform, which is a generic and _enterprise scale_ software
analytics data platform consists of two subsystems: the Nalanda graph system,
which provides a _large scale_ socio-technical graph of software data, and the
Nalanda index system, which is an _enterprise scale_ index system that can be
used to support a wide range of software engineering tasks such as
recommendation and search. Nalanda scales to enterprise-scale data from 6,500
repositories. The socio-technical graph has 37,410,706 nodes and 128,745,590
edges (The schema of the Nalanda graph is shown in Figure 1.) The index system
contains 8,079,748 documents.

To show the _versatility_ of the Nalanda platform, we describe two tools that
have been built on top of the Nalanda platform and _deployed_ at Microsoft: A
software analytics news feed application built on top of the Nalanda graph
system named MyNalanda, and a novel recommendation system (Nalanda artifact
and expert recommendation application) leveraging the socio-technical graph
for ranking the recommendations.

The goal of this paper is to describe the design, implementation, and
deployment of the Nalanda graph system, the index system, and two successful
applications (MyNalanda and the Nalanda artifact and expert recommendation
application) built and deployed at Microsoft. We also share details about the
extensive analyses and user studies that we conducted to evaluate the
perceived usefulness of MyNalanda and Nalanda artifact and expert
recommendation application from our deployments at Microsoft. Additionally, we
share insights from building the Nalanda platform, MyNalanda, and the Nalanda
artifact and expert recommendation application.

To that end, we explain the construction of the Nalanda graph system in
Section 2 and the index system in Section 3. We explain the applications built
leveraging these two systems i.e., MyNalanda in Section 4 and the Nalanda
artifact and expert recommendation application in Section 5.

##  2\. Building The Nalanda Graph

Key challenges in the construction of the Nalanda Graph are scale and
consistency. In this section, we lay out what content we store in the Nalanda
graph, from which sources we collect the data, and how we ensure that the
graph is kept up to date and consistent as hundreds of thousands of events
from thousands of repositories arrive on a daily basis.

###  2.1. Nalanda’s Graph Schema

Nodes in the Nalanda graph represent the actors or entities involved in the
software development life cycle, while the edges represent the relationships
that exist between them.

Each node in the Nalanda graph has a type associated with it and attributes
specific to that node type, as listed in Figure 1. The central node is the
Pull Request, which has incoming edges from Author, Reviewer, Work item, and
Repository nodes, and has a outgoing edges to File nodes changed by the pull
request. A developer takes the role of an author when they make source code
changes and submit pull requests and they assume the role of a reviewer when
they perform code reviews. These are represented as user nodes in the Nalanda
graph with different edge types but are listed here as two different nodes in
Figure 1 for clarity. For File nodes, different types are distinguished,
including source code, configuration, and project files. Files are edited by
the authors via pull requests. Files are represented as nodes in the Nalanda
graph with a second-order relationship established between the user and file
nodes via a Pull request node.

Edges in the Nalanda graph represent the relationships between various actors
and entities. Like nodes, edges can be of different types and can have
properties associated with them. An edge is created between an author node and
a pull request node when a developer creates a pull request. Similarly, an
edge is established between the reviewer and the pull request nodes when a
developer is assigned a code review. A linked to edge is created when
developers link a pull request to a work item, commonly done in Azure DevOps
(AzD, 2021) to connect earlier, related, pull requests to new issues.
Likewise, a parent of edge is created between two work items if they are
linked by the developers with a parent-child relationship in Azure DevOps.
Finally, a reports to edge is created between two user nodes if one of them is
the reporting manager of the other.

###  2.2. Data Collection and Graph Construction

The Nalanda platform architecture is shown in Figure 2. The primary source of
data for the Nalanda graph is Azure DevOps. Instead of directly crawling the
Azure DevOps system for data, we leverage an intermediate data source called
CloudMine (Czerwonka et al., 2013a). The Nalanda platform takes the raw event
data from CloudMine and processes it to create the nodes and edges of the
Nalanda graph. The graph can be queried using the APIs we provide, or directly
by means of the graph query language Gremlin (Gre, 2021).

Figure 2. Nalanda’s data collection and graph construction architecture

The platform builds upon Azure (AzS, 2021): key services used include Azure
batch, Azure CosmosDB, Azure SQL Server, Azure Blob Storage, and Lens explorer
(Len, 2021). As shown in Figure 2, the Nalanda platform is built using two
independently operated pipelines: a data aggregation pipeline and a graph
construction pipeline, as explained below.

####  2.2.1. Nalanda’s Data Aggregation Pipeline

As indicated as step 1 in Figure 2, the data aggregation pipeline is
responsible for fetching data from different data sources (most notably
CloudMine) and making it available for the graph construction pipeline to
process.

We use Lens Orchestrator (Len, 2021) for orchestration and scheduling
purposes. Lens has the ability to connect to multiple data sources and systems
and move data around. In the aggregator pipeline, Lens first connects to the
CloudMine data store (which is hosted on Cosmos (AzS, 2021)) and executes
Scope scripts to gather data from various data streams, such as the pull
request stream, the work item stream, the code review stream, etc. Lens saves
the aggregated data in the form of CSV files in Cosmos. Then, Lens connects to
Azure Blob Storage to temporarily store these CSV files for further
processing. This intermediate store is required as CloudMine does not allow
any other service (except Lens) to connect to and access the data files for
security and compliance reasons.

Additionally, we use the Lens job scheduling utilities to configure a job in
Lens to run once every eight hours to pull the latest data from CloudMine and
save it to the Azure Blob Storage.

####  2.2.2. Nalanda’s Graph Construction Pipeline

Once the data is available in Azure Blob Storage, we process it using an Azure
batch job to construct the Nalanda graph (Steps 2 and 3 in Figure 2). We use
Azure CosmosDB as our graph data store.

When the batch job discovers new data files as generated by the aggregation
pipeline, it updates the pipeline registry with new file information. This
includes file names, size, timestamp, whether the file contains data from the
bootstrap or the incremental stream, file processing status, processing
duration, etc. The pipeline registry is a SQL database whose purpose is to
serve as a transactional store for the graph construction pipeline. We create
one row in the pipeline registry database for every file discovered. After the
new data is downloaded, for each data file the corresponding node and edges
are added to, deleted from, or updated in the Nalanda graph. Once all data for
a file is read, its registry status is set to “completed”.

The graph construction pipeline operates in two modes: bootstrap mode and
incremental mode.

Bootstrap mode. This mode helps ingest all of a repository’s data, from
repository creation time until when it is run. Typically, we run this mode for
a repository when it is being onboarded onto the Nalanda graph platform for
the first time.

Incremental mode. This mode helps keep the data in the Nalanda graph updated
without needing to read the massive original streams of CloudMine (whose size
is in the orders of hundreds of Terabytes), using the incremental streams
offered instead (with sizes in the order of tens of Gigabytes). We run the
incremental pipeline once every eight hours.

Table 1. Comparison of pipeline run time and number of records with the increase in number of repositories Mode | Run time |  # Records to process
---|---|---
| 350 repos | 6500 repos | 350 repos | 6500 repos
Bootstrap | 9 hrs | 28 hrs | 1.05M | 7.41M
Incremental | 10 min | 20 min | 10K | 57K

These separate modes offer the flexibility to bootstrap any new or existing
repository data in an independent and asynchronous manner. Furthermore, the
separation of bootstrap and incremental pipelines offers substantial
performance improvement in terms of run time and resource utilization, as
illustrated in Table 1.

Refreshing the data by querying the original streams of CloudMine each time
the pipeline is run, takes 28 hours for 6,500 repositories. As the incremental
streams are substantially smaller, each incremental job finishes in 20
minutes, yielding an improvement of 98.8% in pipeline run time, for each
pipeline run. Note that for an increase of the number of repositories by a
factor of 20, the run time for the bootstrap mode was increased by a factor of
3 only. This is an effect of the careful design and implementation of the data
pipeline by massively parallelizing the data processing code and enabling
distributed processing on multiple Azure batch nodes.

###  2.3. Data consistency and Self-healing

The Nalanda graph platform is a distributed system that works with multiple
external data sources and large-scale data processing systems, which are prone
to introduce data inconsistencies. Data gaps can manifest due to various
factors related to infrastructure and availability of the CloudMine crawlers.

Detecting and remediating data gaps in such a massive distributed system is
not a trivial task. We devised a novel self-healing system that detects data
gaps and consistency issues proactively and performs self-healing. This helps
the pipeline to guarantee data consistency irrespective of the failures
manifested in external data sources such as CloudMine. The self-healing system
uses the pipeline registry to monitor pipeline states and can switch from
incremental to bootstrap mode if this is warranted.

When an incremental pipeline is run, the timestamp of the oldest record to be
processed is compared with the timestamp of the last successful pipeline run.
If the difference between these timestamps is bigger than three days (the
period for which incremental streams hold their data), this means a data gap
has occurred. To address this, the bootstrap mode is triggered for the
repositories involved, and ongoing incremental pipelines are halted.
Furthermore, the pipeline registry is updated to indicate that bootstrapping
is taking place, thereby locking new incremental jobs.

###  2.4. Scale

The Nalanda graph has been designed to accommodate thousands of repositories.
At the time of writing, it holds the software development activity data from
6,500 repositories at Microsoft. We ingest data starting from January 1, 2019,
or from more recent repositories when their first pull request is created.

To keep its graph up to date, the Nalanda platform processes 500,000 events
per day. These events include new pull requests, updates or commits on those
pull requests, pull request state changes, code review assignments, and code
review comments. At the time of writing, the Nalanda graph contains 37 million
nodes and 128 million edges as detailed in Table 2.

Table 2. Nalanda node and edge types and their prevalence

Node type | Count
---|---
file | 14,537,998
text | 12,104,427
pull request | 7,568,949
work item | 3,067,754
user | 131,578
repository | 6,500
Total nodes | 37,410,706

Edge type | Count
---|---
changes | 65,706,621
reviews | 39,447,635
creates | 7,569,086
contains | 7,337,036
linked to | 7,094,597
parent of | 843,728
comments on | 746,887
Total edges | 128,745,590

##  3\. Indexing Nalanda for Information Retrieval

Many nodes in the Nalanda Graph contain text. To facilitate _search_ over such
text at the Nalanda scale, we need to create appropriate indexes. The actual
indices needed may depend on the specific applications built on top of the
Nalanda graph. In this section, we discuss the indices we create and how we
ensure they remain up to date at scale.

For every search, we use the BM25 algorithm (Robertson et al., 1992) for
determining text similarity between a query and documents (pull requests, work
items, …) and ranking the results. BM25 is a bag-of-words model developed
based on the probabilistic retrieval framework (art, 2009). For a given query
Q containing keywords q1,…,qn, the BM25 score for a document D is:

(1)
$\text{score}(D,Q)=\sum_{i=1}^{n}\text{IDF}(q_{i})\cdot\frac{f(q_{i},D)\cdot(k_{1}+1)}{f(q_{i},D)+k_{1}\cdot(1-b+b\cdot\frac{|D|}{\text{avgdl}})}$

where $f(q_{i},D)$ is $q_{i}$’s term frequency in the document $D$,
IDF($q_{i}$) is $q_{i}$’s inverse document frequency, $|D|$ is the length of
the document $D$ in words, and avgdl is the average document length in the
text collection from which documents are drawn. $k_{1}$ and $b$ are free
parameters. We use standard recommended values ($b=0.75$, $k_{1}=1.2$) for
these constants (Schütze et al., 2008).

###  3.1. The Nalanda Artifact Index

The Nalanda artifact index facilitates search through pull requests and work
items. We index the metadata, titles, and descriptions of the artifacts (pull
requests and work items). The metadata consists of elementary properties,
namely project name, repository name, and organization name.

The Nalanda artifact index can be used to find relevant pull requests given a
work item, feature, technical, or functional concept. Furthermore, with this
index completed pull requests and work items can be used as a template and
inspiration to solve similar problems. They provide code samples, expose code
review comments, and help as informal documentation to learn the best
practices. Additionally, they help understand the team or project-specific
processes involved in getting such pull requests completed.

###  3.2. The Nalanda Expert Index

The Nalanda expert index is built to map subject matter experts (SMEs) to
technical and functional skills. Experts can be of two types: functional and
technical. Functional experts have expertise in specific functionality of a
software product or service, such as the query optimizer in an RDBMS product
or the ranker in a search engine product. Technical experts have expertise
with a technology concept such as socket programming in Java.

The Nalanda expert index relies on a collaborative software development
platform like Azure DevOps to mine and associate expertise with people. We
match pull requests and work items (as also used for the artifact index) to
their authors and contributors. The process of building the Nalanda expert
index consists of the following steps:

  1. (1)

Find all pull requests and work items completed by a developer and extract key
phrases from them. The key phrases include tokens in pull request minus
English stopwords (Sto, 2021).

  2. (2)

Create a document using the data from step 1, which is the representation of a
developer’s skills.

  3. (3)

Repeat steps 1-2 for every developer and builds the corpus for the expert
index

Similar to the Nalanda artifact index, we use the BM25 algorithm for querying
the expert index corpus. The intuition is that if a developer makes frequent
code changes related to a topic (functional or technical), they must be
knowledgeable in that topic area. We represent the frequency of a topic in a
developer’s activity as term frequency in the BM25 index. Therefore, the more
a topic appears in the document corpus constructed for that developer, the
more weight that topic is given.

We leverage the Nalanda socio-technical graph for re-ranking the search
results returned by the artifact and expert indices. A detailed analysis of
the impact of employing the Nalanda graph in refining the search results is
discussed in Section 5.3.

###  3.3. Scale

Building indices such as the artifact and expert index at scale is an
expensive operation. We carefully crafted the system design to make the
Nalanda index creation and refresh pipelines robust and tolerant to failures
(details about implementation are explained in Section 5.2). The Nalanda
search system has been built as a cloud-native service. This enables us to
scale out the system horizontally with the increase in data and query volume.
This also helps us in meeting high uptime Service Level Agreements (SLAs)
requirements to move to production. Currently, the artifact index contains
8,018,320 documents and the expert index contains 61,428 documents.

We ingest data from 6,500 repositories. Our index data refresh pipeline, which
runs once every week, completes in 65 minutes on average. The graph data
refresh pipeline, which runs every 8 hours and finishes in 18 minutes. We
optimized the API service to return the response in 1.7 seconds in accordance
to the SLA requirements.

##  4\. Nalanda Applications (I): The MyNalanda Portal

The Nalanda graph and indexing platform can be used to build many applications
to support software development teams and organizations in their daily work.
The first application we discuss is MyNalanda, an online news feed in
production at Microsoft, in which developers and managers alike can monitor
ongoing software development activities.

###  4.1. MyNalanda Motivation

The motivation behind MyNalanda is that it is common practice for developers
to work on multiple work items or pull requests at once. It is also common
practice for developers at Microsoft to work on multiple source code
repositories simultaneously. Microsoft does not have many large mono-
repositories, but a lot of small or medium sized repositories. Keeping track
of one’s work items in their repository or across multiple repositories is a
difficult and time consuming task. Moreover, these tools operate in a
workitem-centric fashion, i.e., the primary goal of the tool is to search for
and find a work item one is interested in.

By contrast, MyNalanda is a _developer-centric_ news feed. Upon login,
MyNalanda shows the activity (pull request, work item, code review) of a
developer, from multiple repositories, in their homepage. Additionally,
MyNalanda enables developers to discover what their teammates and other
collaborators are working on without the hassle of going to different Azure
DevOps repositories.

Figure 3. The MyNalanda homepage

###  4.2. The MyNalanda Homepage

For a user, the central hub in MyNalanda is their _homepage_. An example of
such a MyNalanda homepage is shown in Figure 3. Its information is organized
in the following sections:

News feed: The centralized news feed (‘F’ in Figure 3) is located in the
middle of the page. The news feed shows events such as updates in pull
requests, code review comments, and pull request status changes from all the
repositories a developer works in. For managers, the news feed provides
updates from their reports’ activity.

User details: This section (‘A’ in Figure 3) provides details about
developers, such as name, email address, job title, and their expertise
(extracted from their software development activity data). This helps in
facilitating easy discovery of developers’ skills and their current projects.

Active items: There are separate sections for active repositories, pull
requests, work items, and code review requests (‘B’-‘E’ in Figure 3). Users of
MyNalanda can prioritize the discovery of updates from these items by
_following_ or _unfollowing_ them.

Related people: This section (‘G’ in Figure 3) visualizes who a developer
collaborates with and how local software development communities are formed. A
developer’s collaborators include others who work together with the developer
on a coding task or work item, or who are either being reviewed by that
developer, or who are involved in reviewing a developers’ pull request.

Search box: The search box (‘H’ in Figure 3) can be used to find developers
and discover their activity. It also can help with searching for technical and
functional concepts by leveraging the Nalanda artifact and expert indices (as
explained in Section 3).

All elements in MyNalanda, such as pull requests, work items, people, and
repositories have embedded URLs which take them to the corresponding item in
Azure DevOps. This makes it easy for developers to navigate between MyNalanda
and Azure DevOps.

Additionally, MyNalanda facilitates integration of other machine learning
recommenders due to its extensible architecture. For example, overdue pull
request are indicated with a subtle warning icon in the active pull requests
section. This is powered by the Nudge machine learning models (Maddila et al.,
2021b).

###  4.3. MyNalanda Usage

MyNalanda leverages the graph representation of the data (the Nalanda graph)
and its schema design to navigate efficiently through complex relationships
and find the content presented in various sections. As a result, the MyNalanda
homepage including the news feed and the other sections loads in less than a
second. As quoted by one of the MyNalanda users “It is simple, blazing fast to
load, adapts to screen size”.

Based on organic growth alone, MyNalanda reached 290 Daily Active Users (DAU),
and 590 MonthlyActive Users (MAU), in the first six months of deployment of
the Beta version at Microsoft.

###  4.4. MyNalanda Evaluation: Perceived Usefulness

To evaluate how developers and engineering managers perceive the usefulness of
MyNalanda, we follow a mixed method research design involving interviews and
surveys. With increasing sources of development-related information available,
there remains an open question about how they want to access and integrate
information about their own development activities and the development work
done by their peers, and if current platforms are adequate. Through this
evaluation we assess how MyNalanda matches the corresponding information
needs.

####  4.4.1. Evaluation Setup

##### Semi-structured Interviews.

We conducted interviews to investigate information discovery and overload, and
if users might use an interface like MyNalanda. Participants included five
developers and two engineering managers; seven participants were men and zero
were women. Semi-structured interviews were conducted remotely, and ranged
from 30-45 minutes. Interview topics included interest in accessing
information about their own and peers’ development activity, information
overload and how they typically get information (in both in-office and work-
from-home contexts). We then showed a deployed version of MyNalanda and asked
for reactions including if they would like it, what information they would
find useful, and where they would want to see it. If the interviewee was an
engineering manager, they were also asked what information they would be
interested in seeing related to their team’s work.

Immediately following the interviews, notes were taken by the interviewer to
augment the transcription and interviews were coded for emergent themes.
Following each subsequent interview, themes were revisited to see if any codes
should be combined or separated. Once no new themes emerged (i.e., theoretical
saturation), we concluded our interview phase. After seven interviews, themes
remained consistent. Finally, we reviewed notable excerpts from all interviews
and organized the themes by topic.

##### Surveys.

Following interviews, we conducted surveys to validate and quantify the themes
that emerged. Survey participants included full-time employees who were
developers or engineering managers.

We designed our survey based on themes that emerged during our interviews,
resulting in a 19-item instrument that took a median of 8.5 minutes to
complete. Participants were shown their MyNalanda newsfeed (with their own
development activity) and given the survey. Topics included demographics,
usefulness of information included in their MyNalanda feed, information pain
points and privacy concerns, and current and anticipated work location (e.g.,
office or work-from-home). We included items asking about usefulness of
MyNalanda information, and preferences for possible features (based on jobs-
to-be-done). We asked where respondents would like to see MyNalanda integrated
(if at all), and their comfort level in sharing their development activity
through something like MyNalanda. The full survey instrument is available
online at research.microsoft.com (Maddila et al., 2021a).

The survey was sent to 2,000 people in total (1,400 developers and 600
engineering managers) with 144 responses (92 developers and 44 engineering
managers), resulting in an 8% response rate after considering the 10% out of
office responses. Our low response rate could be due to the fact that the
survey is not a trivial one to fill in (cognitively and time it takes to
complete the survey) and/or because we did not offer any incentives for
participation. Of those respondents, 92 (67.65%) were developers (including
software development engineers, senior software engineers, etc.) and 44
(32.35%) were engineering managers (including software engineering manager,
software engineering lead, etc.). Our respondents included five women (6.17%),
73 men (90.12%), one who preferred to self-describe (1.32%), and two who
preferred not to answer (2.63%). They reported an average of 10.15 years
working at Microsoft, ranging from 0.6 to 29.9 years (standard deviation
7.13).

####  4.4.2. Results of the Study

We first present findings from our semi structured interviews, with each noted
as developer (D) or engineering manager (EM). We then present our survey
results.

##### Interviews.

Participants expressed two themes related to information needs: integration
and overload. Information integration was echoed by many interviewees; we
define this as having development-related information for self and others
integrated into a single, easy-to-access interface. P1 (EM) and P2 (D)
discussed easily linking design docs and associated artifacts like pull
requests. P1 (EM) discussed the usefulness of graphic summaries for their
teams’ development activity across time periods; this reflects consolidation
via visualization. P2 (D) called out that the ability to see detailed
information about peers’ work is helpful when coordinating work, and is not
readily available.

P5 (D) spoke about viewing the information in different ways to deal with
information overload and ensure they were not missing things: using a ‘Most
recent’ view for chronologically-ordered information, an algorithmic
‘Relevance’ view to combine information across teams, and team-only view to
further filter. This speaks to strategies they use to ensure keep up on
development-related information both within and across teams they collaborate
with. Similar sentiments were expressed by other interviewees.

##### Surveys.

We asked participants to rate how useful each feature of MyNalanda was on a
five-point Likert-type scale ranging from 1= Not at all useful to 5= Extremely
useful. All items were optional; of those who took the survey, 85 answered
questions about MyNalanda features (63 developers and 22 engineering
managers). This allowed us to capture participant reactions to a real
integrated information platform instead of a hypothetical one.

Table 3 lists the accumulated percentages of “Extremely useful” (Likert=5) and
“Very useful” (Likert = 4) for each feature. Here, we see that Active pull
requests (55.6% and 54.5% among developers and engineering managers,
respectively) and Active code review requests (49.2% and 45.5% among
developers and engineering managers, respectively) are the highest rated
features, with User details (27.0% and 38.2% among developers and engineering
managers, respectively) and Related People (23.8% and 9.1% among developers
and engineering managers, respectively) rated the lowest.

Based on this, we conclude that the active items (pull requests, code review
requests, repositories, and work items) are the most valued features of
MyNalanda, and that user details are primarily of interest to engineering
managers.

Table 3. MyNalanda survey Feedback Feature |  Cumulative (n=85) |  Developers (n=63) |  Managers (n=22)
---|---|---|---
Active pull requests (C) |  55.3% | 55.6% | 54.5%
Active code review requests (E) |  48.2% | 49.2% | 45.5%
Active repositories (B) |  44.7% | 46.0% | 40.9%
Active work items (D) |  34.1% | 34.9% | 31.8%
Feed (F) |  28.2% | 34.9% | 29.1%
User Details (A) |  24.7% | 27% | 38.2%
Related people (G) |  20% | 23.8% | 9.1%

##  5\. Nalanda Applications (II): Artifact and Expert Recommender

Figure 4. Nalanda recommendations in Azure DevOps

When a developer is working on a work item or feature, finding the relevant
pull requests and code samples is one of the biggest pain points for
developers (Maddila et al., 2021a). The intensity of the problem multiplies in
large organizations with teams working on multiple source code repositories.
Similarly, finding functional and technical experts in large organizations is
a difficult task. A lot of times, this involves the developers to mine git
history, going through wiki pages, design documents, etc.

The Nalanda artifact and expert recommendation application implements a
recommendation plug-in for Azure DevOps (AzDO) (AzD, 2021). When a new work
item is assigned to a developer, the plug-in triggers an API call to the
Nalanda search API. The client passes the necessary input parameters, such as
the search query (work item title and description), the work item owner, and
the repository metadata. Upon receiving the results (artifact and expert
recommendations), the client add recommendations to the work item page (in
Azure DevOps). A sample work item recommendation page from live deployment
(Beta version) is shown in Figure 4. Besides, through its easy-to-use APIs,
the Nalanda artifact and expert recommendation application system powers other
applications such as the _search box_ in MyNalanda (as explained in Section
4.2).

When a work item or an issue is assigned to a developer, the Nalanda artifact
and expert recommendation application uses a combination of title and
description of the assigned work item as an input search query and provides
recommendations about work items or pull requests that accomplished similar
tasks. Additionally, the system also provides a list of subject matter experts
whom a developer can reach out to seek help while working on that work item.

###  5.1. The Nalanda Ranking Algorithm

To construct the Nalanda Expert and Artifact recommenders, we devised a
ranking algorithm consisting of three steps: 1) querying the artifact index
(see Section 3.1) to get the relevant pull requests and work items, 2)
querying the expert index (see Section 3.2) to get a list of relevant experts,
and 3) re-ranking the results using the Nalanda graph. To that end, we take
the following steps:

Step 1: We first construct a query as a combination of the title and
description of the work item. Then, we tokenize the query using heuristics
that we built for the software engineering domain, such as splitting strings
into camel-cased or pascal-cased tokens. We also create n-gram based tokens
since we found that bi-grams and tri-grams, such as ImapTransfer and
MailboxSyncEngine, capture important information. Next, we filter out stop
words (Luhn, 1960).

We employ the BM25 algorithm, which takes care of prioritizing important
tokens using Inverse Document Frequency (IDF) scores. It assigns more weight
to the documents with a higher overlap with the search query tokens (term
frequency), and calculates a relevance score as shown in Equation 1.

Step 2: We query the expert index, which contains one document per person.
These documents contain the tokens mined from a developer’s pull requests and
work items history. We perform the same pre-processing explained in Step 1 on
the query. The BM25 index returns a ranked list of experts.

Step 3: A heuristics-based filtering scheme is used to filter out results with
relevance scores below a threshold. We determine the threshold empirically to
set it at the 75th percentile of the relevance score distribution. We
determined this value based on a series of experiments to optimize the
accuracy of the results while reducing the size of the results set that is
passed to the next step.

Step 4: We use the Nalanda socio-technical graph to assign proximity scores
based on the edge distance between the person performing the search and the
results returned by the BM25 index (obtained from Step-3). The proximity score
is the length of the shortest path between two nodes. We use the proximity
score to re-rank the results. For example, the proximity score is 1 if a pull
request, work item, or people node is 1-edge away from a developer node, in
their shortest path.

Step 5: We then pick the top-$k$ results from the results set and return them
to the user. Items that have a higher BM25 relevance score and are in close
proximity to a developer are ranked higher. Furthermore, we use proximity
score to break the tie between results where relevance scores are the same.

###  5.2. Implementation

We implemented the Nalanda search system on the Azure platform, with an
emphasis on scalability to thousands of repositories. The underlying
architecture is visualized in Figure 5.

Figure 5. Nalanda artifact and expert recommendation application architecture

####  5.2.1. Nalanda Indexers

The backbone of the recommendation system is the batch jobs creating and
continuously updating the indices and the socio-technical graph, as discussed
in Section 3 and shown at the bottom of Figure 5. We leverage CodeMine
(Czerwonka et al., 2013b) to help us with aggregating source control system
data from thousands of repositories.

We rely on Azure Cognitive Search (ACS) (ACS, 2021), which provides BM25 as a
service, to store and access the indices. This helps us in alleviating the
problems associated with service maintenance, uptime, and scale-out.

####  5.2.2. Search API

Given the indices and the socio-technical graph, the Nalanda Search API (shown
in the middle of Figure 5) implements the search algorithm from Section 5.1.
Each search query (typically initiated by the users in MyNalanda or through
Azure DevOps) passed to the Nalanda search API is processed in real-time. The
mean response time for the API call is 1.7 seconds.

The Nalanda API service asynchronously saves telemetry to an Azure SQL
database without impacting the query performance. The telemetry includes the
search query, user metadata, search results, click logs, and the API response
time. We use this information to evaluate engagement and to improve the
performance of the Nalanda search system and the API service.

Table 4. Evaluation Data Summary |  Artifact Recommendation |  Expert Recommendation
---|---|---
Index data |  Pull request title and description, socio-technical graph |  Pull request title and description, socio-technical graph
Test set |  Work Item title and description |  Internal StackOverflow questions
Ground truth |  Pull requests linked to the work item |  People who answered the post and people who have answered at least five other questions with the same tag as the post
Data set |  80K randomly sampled work items from the 6,500 repositories |  10K randomly sampled questions and answers from StackOverflow

###  5.3. Quantitative Evaluation

To understand the efficacy and usefulness of the Nalanda artifact and expert
recommendation application, we conduct a large-scale offline evaluation and a
user study.

####  5.3.1. Experiment Setup

We randomly sample 80,000 work items from the 6,500 repositories such that
there are at least 10 work items selected from each repository. Subsequently,
we use the title and description of each of these work items as the input to
the Nalanda search API. We expect the right pull requests and people to be
returned from the search API.

Since a recommendation system like ours does not exist in the company, we do
not have a ground truth to conduct a large-scale evaluation. Therefore, we
rely on the pull requests manually tagged by developers to the work items in
Azure DevOps to build the evaluation dataset. To create the ground truth
dataset for the expert recommendations, we leverage the private instance of
StackOverflow deployed at Microsoft. Details about the index, ground truth,
and test sets used for these experiments are shown in Table 4.

####  5.3.2. Results

We use two commonly used metrics for recommender systems: 1. Top K accuracy,
which measures the number of times the correct item is found in the top K
recommendations 2. Mean Reciprocal Rank (MRR), which calculates the reciprocal
of the rank at which the first relevant document was retrieved (Craswell,
2009).

Table 5. Evaluation and Comparative study for Artifact Recommendation | K = 3 | K = 5 | K = 10
---|---|---|---
Indexed Properties | Accuracy | MRR | Acc | MRR | Acc | MRR
PR metadata | 0.26 | 0.23 | 0.29 | 0.28 | 0.32 | 0.30
PR attributes |  |  |  |  |  |
\+ PR title | 0.38 | 0.36 | 0.43 | 0.41 | 0.51 | 0.48
\+ PR description | 0.49 | 0.47 | 0.53 | 0.51 | 0.60 | 0.59
| \+ socio-technical graph
---
0.71 | 0.71 | 0.74 | 0.73 | 0.78 | 0.77
Table 6. Evaluation and Comparative study for Expert Recommendation | K = 3 | K = 5 | K = 10
---|---|---|---
Indexed Properties | Accuracy | MRR | Acc | MRR | Acc | MRR
PR metadata | 0.35 | 0.30 | 0.39 | 0.33 | 0.43 | 0.38
PR attributes |  |  |  |  |  |
\+ PR title | 0.51 | 0.46 | 0.54 | 0.49 | 0.59 | 0.53
\+ PR description | 0.60 | 0.54 | 0.64 | 0.59 | 0.69 | 0.61
\+ socio-technical graph | 0.63 | 0.60 | 0.69 | 0.63 | 0.75 | 0.67

Table 5 shows the results from the evaluation for different values of K. We
can see that incorporating more attributes of the pull request, such as its
title and description, improves both the MRR and accuracy considerably.
Furthermore, re-ranking the results using the Nalanda graph also substantially
improves the recommendations. Similar improvements can be noticed for the
expert recommendation task too (Table 6).

###  5.4. User Perception

We conducted a user study among developers regarding the usefulness of the
recommendations. We selected ten participants (identified as P1–P10) from
Microsoft to evaluate the recommendations on their recently completed work
items.

####  5.4.1. Participants and Protocol

We conducted semi-structured interviews, which were conducted remotely, and
ranged from 15-30 minutes. The average experience of the subjects is 7.7 years
in the company and ranged from 10 months to 21 years.

We employ a one-group pretest-posttest pre-experimental design (Campbell and
Stanley, 1963). We used the Likert scale (Robinson, 2014) for rating the
responses. The respondents can provide their responses on a 1 to 5 scale,
ranging from ‘strongly disagree’ to ‘strongly agree’. We posed them the
questions listed below.

_When you are working on a work item, how useful would the recommendations be
in completing the work item (on a scale of 1 to 5):_

  1. (1)

You are going to refer to the work items and pull requests recommended as
inspiration and informal documentation on accomplishing the work item.

  2. (2)

You would likely reach out to the recommended people for consultation on
accomplishing the work item.

Figure 6. Participants’ evaluation of the Nalanda system

####  5.4.2. Results

Do the users feel work item, pull request, and expert recommendations are
useful? The participants expressed that the Nalanda recommendation system can
be a great value addition to the software development process. 60% of the
participants rated ‘agree’ or ‘strongly agree’ when asked whether they find
the artifact recommendations useful and 40% responded favorably to the expert
recommendations.

Through question 2 we measure the difference between the expectations the
participants had of a hypothetical recommendation system with the Nalanda
artifact and expert recommendation application. This question measures the
dependent variable for the user study (introduction of the Nalanda system). In
Figure 6, the radar chart shows some differences between the participants’
original expectations and their perception of the Nalanda recommendations.

These differences can be observed better in Figure 7, in which the averages of
the rating are shown. The difference in expectation versus perception was more
apparent with the expert recommendations compared to the artifact
recommendations.

To offer an impression, we list some typical quotes (positive and negative)
that we received from the developers.

> “A tool like this will help greatly to understand the processes involved in
> pushing my changes through.”

> “It is great to see the recommendations about people to talk to. My team is
> large, mostly remote, and I am new. So this is very helpful.”

> “Finding people to talk to has never been a problem for me as I have been
> working in the same org for a while.”

Figure 7. Expectations and perceptions of the Nalanda system

##  6\. Discussion

###  6.1. Outlook

In the future, we anticipate both the Nalanda graph system and MyNalanda to be
scaled out significantly inside Microsoft in terms of the number of
repositories and users. Furthermore, we see opportunities for implementing the
Nalanda graph platform as a service on top of the open-source software
development data mined from systems like GitHub.

We also expect the Nalanda artifact and expert recommendation application to
be employed at substantially more software development systems at Microsoft.
Future research could entail including other types of useful recommendations
such as internal and external documentation, tutorials, and recommendations
from question-answer forums.

The rich socio-technical data in the Nalanda graph in combination with
advances in deep learning and Graph Convolutional Networks (GCNs), hold
promise for applications such as neural reviewer recommendations by leveraging
the socio-technical structures. The Nalanda graph system lays the foundation
to bring various techniques from the graph representation world (Gra, 2021),
such as link prediction, social network analysis, etc. into the software
engineering and analytics domain.

###  6.2. Threats and Limitations

####  6.2.1. Internal Validity

Conducting trustworthy experiments on data collected from thousands of
repositories is challenging especially due to the problems of avoiding data
leakage and obtaining credible ground truth. In our experiments, we addressed
this (see Table 4). While our results are highly promising and an important
first step, more experiments are needed to better understand the true nature
of the graph’s contribution to expert and artifact recommendation.

The risk of response bias is minimal in all our studies because all the
participants of the user study are organizationally distant from the people
involved in building this system. However, there remains a small chance that
people in the user study may be positive about the system because they want to
make the developers who are from the same company motivated and happy.

####  6.2.2. External Validity

The context of our evaluation and user studies is a software development
company with a large number of developers. These developers work on a
portfolio of products across many contexts and domains. By conducting a user
study within one single company, we were able to control for factors like
culture, tooling, frameworks, and programming languages. However, our results
may not be generalizable across all developers in all contexts. Hence, our
results are not verified in the context of other organizations or the open-
source community. Therefore, our findings may be limited and warrant further
research. Future work could investigate user interfaces, integrating our
findings with design guidelines that span usability and technical and
organizational complexity (Jaferian et al., 2008).

##  7\. Related work

##### Graph Representations

Hipikat (Cubranic et al., 2005) is one of the earliest works to build a graph
for software development entities like tasks, file versions, and documents,
using a fixed schema. It was built for onboarding new hires quickly by
providing easy access to relevant artifacts. Codebook (Begel et al., 2010)
builds a prototype graph consisting of various software development entities.
The data was mined from software repositories for a single team at Microsoft.
Bhattacharya et al. (Bhattacharya et al., 2012) use graph representation of
source code and bug tracking information to construct predictors for software
engineering metrics like bug severity and maintenance efforts. Other
applications of graph representations of software artifacts include
visualizing relationships among project entities (Sarma et al., 2009a) and
extracting changes in variability models (Dintzner et al., 2018). Compared to
all this work, the scale of the Nalanda graph is significantly larger with 37M
nodes and 128M edges, and Nalanda has been designed to be actively used in a
production setting.

##### Source Control Dashboards

GitHub offers a dashboard for developers on its homepage (Git, 2021). It
displays the repositories, active pull requests, and issues (work items). A
key limitation of the GitHub dashboard is there exists no notion of a “Team
feed” in Github unlike MyNalanda. Team feed helps the discovery of the items
worked on by other team members.

##### Information Needs

Ko et al. (Ko et al., 2007) studied information needs in colocated development
teams. Fritz and Murphy (Fritz and Murphy, 2010) provide a list of questions
developers ask for the most frequently sought-after information within a
project. Information needs have also been studied in the context of change
tasks (Sillito et al., 2008), inter-team coordination (Begel et al., 2010) and
software analytics (Buse and Zimmermann, 2012; Begel and Zimmermann, 2014;
Huijgens et al., 2020) Through its applications, Nalanda can efficiently
address most information needs related to people, code, and work items. For
example, the answer to questions such as “Who is working on what” and “What
are coworkers working on right now” is easily available in the MyNalanda
application.

##### Artifact Recommendation systems for software developers

Recommendation systems for software engineering aim at assisting developers
with activities such as code reusability, writing effective bug reports, etc.
(Robillard et al., 2010). Tools like CodeBroker (Ye and Fischer, 2002) help in
finding the relevant code samples extracted from the standard Java
documentation generated by Javadoc from Java source programs and deliver the
suggestion to the Emacs editor. Anvik et al. (Anvik et al., 2006) proposed a
semi-automated method to assign bug reports to reporters based on their
expertise using a machine learning algorithm. Mockus and Herbsleb(Mockus and
Herbsleb, 2002) used quantity as a measure of expertise. Fu et al.(Fu et al.,
2017) used the node2vec algorithm to convert file entities within projects
into knowledge mappings. They proposed four features to capture the social
relationships between developers. Devrec(Zhang et al., 2017), a developer
recommendation system, mines the development activities of developers in
GitHub and StackOverflow to recommend collaborators for a given project.
Hammad et al (Hammad et al., 2020) use keywords from the textual content of
commits. On the other hand, Canfora et al (Canfora et al., 2012) use mailing
lists and versioning systems to recommend experts for newcomers joining a
software project. Compared to these approaches, the Nalanda artifact and
expert recommendation application is designed to be highly scalable and
provide responses in real-time.

##  8\. Conclusion

In this paper, we seek to build a _large scale_ software analytics data
platform named Nalanda with two subsystems (the Nalanda graph systemand the
Nalanda index system). The Nalanda graph system consists of a socio-technical
graph encompassing the entities, people, and relationships involved in the
software development life cycle. The Nalanda index system is an _enterprise
scale_ index system that can be used to support a wide range of software
engineering tasks such as recommendation, and search.

We built the Nalanda graph system using software development activity data
from 6,500 source code repositories. The graph consists of 37,410,706 nodes
and 128,745,590 edges. To the best of our knowledge, it is the largest socio-
technical graph built to date using private software development data.
Similarly, the Nalanda index system contains 8,018,320 documents in its
artifact index and 61,428 documents in its expert index with data ingested
from 6,500 repositories at Microsoft.

The Nalanda platform and its applications (MyNalanda and Nalanda artifact and
expert recommendation application) help in developing awareness of each
other’s work, and building connections between developers across repositories,
while offering mechanisms to discover information while managing information
overload. We also seek to address the problems of information discovery by
finding related work items and experts for software developers.

Based on organic growth alone, MyNalanda has Daily Active Users (DAU) of 290
and Monthly Active Users (MAU) of 590. A preliminary user study shows that 74%
of developers and engineering managers surveyed are favorable toward continued
use of MyNalanda for information discovery. The Nalanda artifact and expert
recommendation application, with the help of the socio-technical graph for
customization, lifted the accuracy of artifact recommendations by 30.45
percentage points to 0.78. In a study with ten professional software
developers, participants agreed that a system like Nalanda artifact and expert
recommendation application could reduce the time spent and the number of
places needed to visit to find information.

In the future, we anticipate both the Nalanda graph system and MyNalanda to be
scaled out significantly inside Microsoft in terms of the number of
repositories and users. We believe the systems and the techniques have
applicability beyond Microsoft. Furthermore, we see opportunities for
implementing the Nalanda graph platform as a service on top of the open-source
data mined from platforms like GitHub.

## References

  * (1)
  * art (2009) 2009\.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Foundations and Trends in Information Retrieval_ 3 (01 2009), 333–389.  https://doi.org/10.1561/1500000019
  * ACS (2021) Accessed 2021.  Azure Cognitive Search.  https://azure.microsoft.com/en-us/services/search/
  * AzD (2021) Accessed 2021.  Azure DevOps.  https://azure.microsoft.com/en-us/services/devops/.
  * AzS (2021) Accessed 2021.  Azure services.  https://azure.microsoft.com/en-us/services/
  * Sto (2021) Accessed 2021.  English stop words.  https://gist.github.com/sebleier/554280
  * Git (2021) Accessed 2021.  GitHub personal dashboard.  https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-github-user-account/managing-user-account-settings/about-your-personal-dashboard
  * Gra (2021) Accessed 2021.  Graph Neural Network Techniques.  https://neptune.ai/blog/graph-neural-network-and-some-of-gnn-applications
  * Gre (2021) Accessed 2021.  Gremlin query language.  https://docs.janusgraph.org/getting-started/gremlin/
  * Len (2021) Accessed 2021.  Lens Explorer.  https://docs.microsoft.com/en-us/system-center/orchestrator/learn-about-orchestrator
  * Anvik et al. (2006) John Anvik, Lyndon Hiew, and Gail Murphy. 2006.  Who should fix this bug?  _28th International Conference on Software Engineering (ICSE)_ 2006, 361–370.  https://doi.org/10.1145/1134285.1134336
  * Begel et al. (2010) Andrew Begel, Khoo Yit Phang, and Thomas Zimmermann. 2010\.  Codebook: Discovering and Exploiting Relationships in Software Repositories. In _Proceedings of the ACM/IEEE 32nd International Conference on Software Engineering_. Association for Computing Machinery, Inc.  https://doi.org/10.1145/1806799.1806821
  * Begel and Zimmermann (2010) Andrew Begel and Thomas Zimmermann. 2010.  Keeping up with your friends: Function foo, library bar. dll, and work item 24. In _Proceedings of the 1st Workshop on Web 2.0 for Software Engineering_. 20–23.  https://doi.org/10.1145/1809198.1809205
  * Begel and Zimmermann (2014) Andrew Begel and Thomas Zimmermann. 2014.  Analyze this! 145 questions for data scientists in software engineering. In _Proceedings of the 36th International Conference on Software Engineering_. 12–23.  https://doi.org/10.1145/2568225.2568233
  * Bhattacharya et al. (2012) Pamela Bhattacharya, Marios Iliofotou, Iulian Neamtiu, and Michalis Faloutsos. 2012.  Graph-based analysis and prediction for software evolution. In _2012 34th International Conference on Software Engineering (ICSE)_. 419–429.  https://doi.org/10.1109/ICSE.2012.6227173
  * Buse and Zimmermann (2012) Raymond PL Buse and Thomas Zimmermann. 2012.  Information needs for software development analytics. In _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, 987–996.  https://doi.org/10.1109/ICSE.2012.6227122
  * Campbell and Stanley (1963) Donald T. Campbell and Julian C. Stanley. 1963.  Experimental and Quasi-Experi-mental Designs for Research.
  * Canfora et al. (2012) Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebastiano Panichella. 2012\.  Who is Going to Mentor Newcomers in Open Source Projects?. In _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_ (Cary, North Carolina) _(FSE ’12)_. Association for Computing Machinery, New York, NY, USA, Article 44, 11 pages.  https://doi.org/10.1145/2393596.2393647
  * Craswell (2009) Nick Craswell. 2009\.  _Mean Reciprocal Rank_.  Springer US, Boston, MA, 1703–1703.  https://doi.org/10.1007/978-0-387-39940-9_488
  * Cubranic et al. (2005) Davor Cubranic, Gail Murphy, Janice Singer, and K.S. Booth. 2005.  Hipikat: A project memory for software development.  _Software Engineering, IEEE Transactions on_ 31 (07 2005), 446 – 465.  https://doi.org/10.1109/TSE.2005.71
  * Czerwonka et al. (2013a) Jacek Czerwonka, Nachi Nagappan, Wolfram Schulte, and Brendan Murphy. 2013a.  CODEMINE: Building a Software Development Data Analytics Platform at Microsoft.  _IEEE Software_ (July 2013).  https://doi.org/10.1109/MS.2013.68
  * Czerwonka et al. (2013b) Jacek Czerwonka, Nachiappan Nagappan, Wolfram Schulte, and Brendan Murphy. 2013b.  CODEMINE: Building a Software Development Data Analytics Platform at Microsoft.  _IEEE Software_ 30, 4 (2013), 64–71.  https://doi.org/10.1109/MS.2013.68
  * Dintzner et al. (2018) Nicolas Dintzner, Arie Deursen, and Martin Pinzger. 2018\.  FEVER: An approach to analyze feature-oriented changes and artefact co-evolution in highly configurable systems.  _Empirical Software Engineering_ 23 (04 2018).  https://doi.org/10.1007/s10664-017-9557-6
  * Fritz and Murphy (2010) Thomas Fritz and Gail C. Murphy. 2010.  Using Information Fragments to Answer the Questions Developers Ask. In _Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1_ (Cape Town, South Africa) _(ICSE ’10)_. Association for Computing Machinery, New York, NY, USA, 175–184.  https://doi.org/10.1145/1806799.1806828
  * Fu et al. (2017) Chenbo Fu, Mingming Zhou, Qi Xuan, and Hong-Xiang Hu. 2017\.  Expert recommendation in oss projects based on knowledge embedding. In _2017 International Workshop on Complex Systems and Networks (IWCSN)_. 149–155.  https://doi.org/10.1109/IWCSN.2017.8276520
  * Hammad et al. (2020) Maen Hammad, Haneen Hijazi, Mustafa Hammad, and Ahmed Otoom. 2020.  Mining expertise of developers from software repositories.  _International Journal of Computer Applications in Technology_ 62 (01 2020), 227.  https://doi.org/10.1504/ijcat.2020.106581
  * Huijgens et al. (2020) Hennie Huijgens, Ayushi Rastogi, Ernst Mulders, Georgios Gousios, and Arie van Deursen. 2020\.  Questions for data scientists in software engineering: a replication. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 568–579.  https://doi.org/10.1145/3368089.3409717
  * Jaferian et al. (2008) Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and Konstantin Beznosov. 2008\.  Guidelines for designing IT security management tools. In _Proceedings of the 2nd ACM Symposium on Computer Human interaction For Management of information Technology_. 1–10.  https://doi.org/10.1145/1477973.1477983
  * Ko et al. (2007) Amy J Ko, Robert DeLine, and Gina Venolia. 2007.  Information needs in collocated software development teams. In _29th International Conference on Software Engineering (ICSE’07)_. IEEE, 344–353.  https://doi.org/10.1109/ICSE.2007.45
  * Luhn (1960) Hans Peter Luhn. 1960\.  Key word‐in‐context index for technical literature (kwic index).  _American Documentation_ 11 (1960), 288–295.
  * Ma et al. (2019) Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Audris Mockus. 2019.  World of code: an infrastructure for mining the universe of open source VCS data. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 143–154.  https://doi.org/10.1109/MSR.2019.00031
  * Maddila et al. (2021a) Chandra Maddila, Apoorva Agrawal, Tom Zimmermann, Nicole Forsgren, Kim Herzig, Arie van Deursen, Chandra Maddila, Nicole Forsgren, and Kim Herzig. 2021a.  _Appendix to Nalanda: A Large-Scale Socio-Technical Graph of Entities and Relationships in Software Development Environment_.  Technical Report MSR-TR-2021-28. Microsoft.  https://aka.ms/MSR-TR-2021-28
  * Maddila et al. (2021b) Chandra Shekhar Maddila, Sai Surya Upadrasta, Chetan Bansal, Nachiappan Nagappan, Georgios Gousios, and Arie van Deursen. 2021b.  Nudge: Accelerating Overdue Pull Requests Towards Completion.  _CoRR_ abs/2011.12468 (2021).  arXiv:2011.12468 https://arxiv.org/abs/2011.12468
  * Mockus and Herbsleb (2002) Audris Mockus and James D Herbsleb. 2002.  Expertise browser: a quantitative approach to identifying expertise. In _Proceedings of the 24th International Conference on Software Engineering (ICSE 2002)_. IEEE, 503–512.  https://doi.org/10.1109/ICSE.2002.1007994
  * Potvin and Levenberg (2016) Rachel Potvin and Josh Levenberg. 2016.  Why Google stores billions of lines of code in a single repository.  _Commun. ACM_ 59, 7 (2016), 78–87.  https://doi.org/10.1145/2854146
  * Robertson et al. (1992) Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Aarron Gull, and Marianna Lau. 1992.  Okapi at TREC. In _Proceedings of The First Text REtrieval Conference, TREC 1992, Gaithersburg, Maryland, USA, November 4-6, 1992_ _(NIST Special Publication, Vol. 500-207)_ , Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST), 21–30.  http://trec.nist.gov/pubs/trec1/papers/02.txt
  * Robillard et al. (2010) Martin Robillard, Robert Walker, and Thomas Zimmermann. 2010\.  Recommendation Systems for Software Engineering.  _IEEE Software_ 27, 4 (2010), 80–86.  https://doi.org/10.1109/MS.2009.161
  * Robinson (2014) John Robinson. 2014\.  _Likert Scale_.  Springer Netherlands, Dordrecht, 3620–3621.  https://doi.org/10.1007/978-94-007-0753-5_1654
  * Sarma et al. (2009a) Anita Sarma, Larry Maccherone, Patrick Wagstrom, and James Herbsleb. 2009a.  Tesseract: Interactive visual exploration of socio-technical relationships in software development. In _2009 IEEE 31st International Conference on Software Engineering_. 23–33.  https://doi.org/10.1109/ICSE.2009.5070505
  * Sarma et al. (2009b) Anita Sarma, Larry Maccherone, Patrick Wagstrom, and James D. Herbsleb. 2009b.  Tesseract: Interactive visual exploration of socio-technical relationships in software development.  _2009 IEEE 31st International Conference on Software Engineering_ (2009), 23–33.  https://doi.org/10.1109/ICSE.2009.5070505
  * Schütze et al. (2008) Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_. Vol. 39.  Cambridge University Press Cambridge.
  * Sillito et al. (2008) Jonathan Sillito, Gail C Murphy, and Kris De Volder. 2008\.  Asking and answering questions during a programming change task.  _IEEE Transactions on Software Engineering_ 34, 4 (2008), 434–451.  https://doi.org/10.1109/TSE.2008.26
  * Ye and Fischer (2002) Yunwen Ye and Gerhard Fischer. 2002.  Supporting Reuse by Delivering Task-Relevant and Personalized Information. In _Proceedings of the 24th International Conference on Software Engineering_ (Orlando, Florida) _(ICSE ’02)_. Association for Computing Machinery, New York, NY, USA, 513–523.  https://doi.org/10.1145/581339.581402
  * Zhang et al. (2017) Xunhui Zhang, Tao Wang, Gang Yin, Cheng Yang, Yue Yu, and Huaimin Wang. 2017\.  DevRec: A Developer Recommendation System for Open Source Repositories. 3–11.  https://doi.org/10.1007/978-3-319-56856-0_1