# Physics-Informed Neural Networks for Predicting the Asymptotic Outcome of
Fast Neutrino Flavor Conversions

Sajad Abbar  0000-0001-8276-997X Max-Planck-Institut für Physik (Werner-
Heisenberg-Institut), Boltzmannstr. 8, 85748 Garching, Germany Meng-Ru Wu
0000-0003-4960-8706 Institute of Physics, Academia Sinica, Taipei, 11529,
Taiwan  Institute of Astronomy and Astrophysics, Academia Sinica, Taipei,
10617, Taiwan  Physics Division, National Center for Theoretical Sciences,
Taipei 10617, Taiwan  Zewei Xiong  0000-0002-2385-6771 GSI Helmholtzzentrum
für Schwerionenforschung, Planckstraße 1, D-64291 Darmstadt, Germany

###### Abstract

In the most extreme astrophysical environments, such as core-collapse
supernovae (CCSNe) and neutron star mergers (NSMs), neutrinos can undergo fast
flavor conversions (FFCs) on exceedingly short scales. Intensive simulations
have demonstrated that FFCs can attain equilibrium states in certain models.
In this study, we utilize physics-informed neural networks (PINNs) to predict
the asymptotic outcomes of FFCs, by specifically targeting the first two
moments of neutrino angular distributions. This makes our approach suitable
for state-of-the-art CCSN and NSM simulations. Through effective feature
engineering and the incorporation of customized loss functions that penalize
discrepancies in the predicted total number of $\nu_{e}$ and $\bar{\nu}_{e}$,
our PINNs demonstrate remarkable accuracies, with an error margin of $\lesssim
3\%$. Our study represents a substantial leap forward in the potential
incorporation of FFCs into simulations of CCSNe and NSMs, thereby enhancing
our understanding of these extraordinary astrophysical events.

††preprint: MPP-2023-270

##  I Introduction

Core-collapse supernovae (CCSNe) and neutron star mergers (NSMs) are among the
most extreme astrophysical phenomena, representing the death of massive stars’
life cycles and the collision of extremely dense remnants, respectively. These
events not only signify the demise of massive stars and dense objects but also
reveal some of the universe’s most energetic and mysterious phenomena.

At the heart of these extraordinary settings lies a captivating process: the
emission of neutrinos, released in copious amounts during both CCSNe and NSMs
Burrows:2020qrp ; Janka:2012wk ; Foucart:2022bth ; Kyutoku:2021icp ;
Colgate:1966ax ; Lattimer:1974slx . During their propagation through the
extreme conditions within these events, neutrinos undergo a fascinating
phenomenon known as _collective_ neutrino oscillations pantaleone:1992eq ;
sigl1993general ; Pastor:2002we ; duan:2006an ; duan:2006jv ; duan:2010bg ;
Mirizzi:2015eza  (for a recent review see Ref. Volpe:2023met ). This
intriguing behavior emerges from the intricate interplay between the
propagating neutrinos and the dense background neutrino gas, where coherent
forward scatterings play a crucial role. This phenomenon behaves in a
nonlinear and collective manner, leading to a complex tapestry of flavor
conversions.

Of particular interest are the so-called _fast_ flavor conversions (FFCs),
which occur on scales characterized by $\sim(G_{\rm{F}}n_{\nu})^{-1}$ (see,
e.g., Refs. Sawyer:2005jk ; Sawyer:2015dsa ; Chakraborty:2016lct ;
Izaguirre:2016gsx ; Capozzi:2017gqd ; Wu:2017qpc ; Wu:2017drk ; Abbar:2017pkh
; Abbar:2018beu ; Capozzi:2018clo ; Martin:2019gxb ; Abbar:2018shq ;
Abbar:2019zoq ; Capozzi:2019lso ; George:2020veu ; Johns:2019izj ;
Martin:2021xyl ; Tamborra:2020cul ; Sigl:2021tmj ; Morinaga:2021vmc ;
Nagakura:2021hyb ; Sasaki:2021zld ; Padilla-Gay:2021haz ; Xiong:2021dex ;
Capozzi:2020kge ; Abbar:2020qpi ; Capozzi:2020syn ; DelfanAzari:2019epo ;
Harada:2021ata ; Abbar:2021lmm ; Just:2022flt ; Padilla-Gay:2022wck ;
Capozzi:2022dtr ; Zaizen:2022cik ; Shalgar:2022rjj ; Kato:2022vsu ;
Zaizen:2022cik ; Bhattacharyya:2020jpj ; Wu:2021uvt ; Richers:2021nbx ;
Richers:2021xtf ; Dasgupta:2021gfs ; Capozzi:2022dtr ; Nagakura:2022kic ;
Ehring:2023lcd ; Ehring:2023abs ; Xiong:2023upa ; Fiorillo:2023hlk ;
Nagakura:2023wbf ; Martin:2023gbo ; Fiorillo:2023mze ; Grohs:2023pgq ). Here,
$G_{\rm{F}}$ represents the Fermi coupling constant, and $n_{\nu}$ denotes the
neutrino number density. These FFCs can take place on scales much shorter than
what would be expected in the vacuum. FFCs occur _iff_ the angular
distribution of the neutrino lepton number, defined as,

$\begin{split}G(\mathbf{v})=\sqrt{2}G_{\mathrm{F}}\int_{0}^{\infty}\frac{E_{\nu}^{2}\mathrm{d}E_{\nu}}{(2\pi)^{3}}&[\big{(}f_{\nu_{e}}(\mathbf{p})-f_{\nu_{x}}(\mathbf{p})\big{)}\\\
&-\big{(}f_{\bar{\nu}_{e}}(\mathbf{p})-f_{\bar{\nu}_{x}}(\mathbf{p})\big{)}],\end{split}$
(1)

crosses zero at some $\mathbf{v}=\mathbf{v}(\mu,\phi_{\nu})$, with
$\mu=\cos\theta_{\nu}$ Morinaga:2021vmc . Here, $E_{\nu}$, $\theta_{\nu}$, and
$\phi_{\nu}$ are the neutrino energy, the zenith, and azimuthal angles of the
neutrino velocity, respectively. The $f_{\nu}$’s are the neutrino occupation
numbers of different flavors, with $\nu_{x}$ and $\bar{\nu}_{x}$ denoting the
heavy-lepton flavor of neutrinos and antineutrinos. When $\nu_{x}$ and
$\bar{\nu}_{x}$ have similar angular distributions, a scenario commonly
observed in state-of-the-art CCSN simulations, this expression transforms into
the conventional definition of the neutrino electron lepton number, $\nu$ELN.

FFCs tend to occur on spatial and temporal scales which are expected to be
significantly shorter than those typically addressed in hydrodynamical
simulations of CCSNe and NSMs. As a consequence, integrating FFC into these
simulations presents a notable challenge. A potential strategy to address this
challenge involves breaking down the problem into two scale hierarchies. This
consideration motivates conducting local dynamical simulations at shorter
scales and subsequently integrating the findings into some practical
prescriptions. Such prescriptions can then be efficiently applied to the
broader astrophysical modelings and hydrodynamic simulations
xiong2020potential ; George:2020veu ; Li2021a ; Just:2022flt ;
Fernandez:2022yyv ; Ehring:2023abs ; Ehring:2023lcd .

The assessment of the outcome of FFCs has undergone thorough examinations of
local dynamical simulations conducted within confined spaces employing
periodic boundary conditions Bhattacharyya:2020dhu ; Bhattacharyya:2020jpj ;
Wu:2021uvt ; Richers:2021nbx ; Zaizen:2021wwl ; Richers:2021xtf ;
Bhattacharyya:2022eed ; Grohs:2022fyq ; Abbar:2021lmm ; Richers:2022bkd ;
Zaizen:2022cik ; Xiong:2023vcm  (see also Ref. Zaizen:2023ihz  for the
possible impact of the choice of boundary conditions). Insights from these
investigations indicate a tendency toward kinematic decoherence in flavor
conversions, generally resulting in quasistationary states. These stationary
states can be characterized by survival probabilities, which are governed by
the conservation of neutrino lepton number and have been demonstrated to be
possibly modeled by analytical formulation to a good accuracy Xiong:2023vcm .

Despite the existence of such analytical formulae for the angular distribution
of survival probabilities, implementing FFCs in CCSN and NSM simulations
remains a challenge. The obstacle lies in the requirement of having access to
complete angular distributions of neutrinos to determine FFC outcomes through
these analytical expressions. However, acquiring such detailed angular
information proves challenging in most cutting-edge CCSN and NSM simulations
due to their computationally intensive nature.

As a practical alternative to considering the full neutrino angular
distributions, many state-of-the-art simulations opt for a more feasible
approach by simplifying neutrino transport through a limited set of angular
distribution moments Shibata:2011kx ; Cardall:2012at ; thorne1981relativistic
. In our specific investigation, we concentrate on radial moments, defined as,

$I_{n}=\int_{-1}^{1}\mathrm{d}\mu\ \mu^{n}\
\int_{0}^{\infty}\int_{0}^{2\pi}\frac{E_{\nu}^{2}\mathrm{d}E_{\nu}\mathrm{d}\phi_{\nu}}{(2\pi)^{3}}\
f_{\nu}(\mathbf{p}).$ (2)

These moments effectively capture crucial aspects of the neutrino angular
distribution while facilitating a computationally more manageable
treatment111Our primary focus is presently on axisymmetric crossings,
particularly emphasizing radial moments where the angular distribution
integrates over $\phi_{\nu}$. It’s crucial to highlight that our current study
excludes non-axisymmetric crossings. Exploring these aspects is a subject
reserved for future investigations..

In practical scenarios, one often encounters a situation where simulations
directly provide only the first two moments, $I_{0}$ and $I_{1}$, for
(anti)neutrinos. The problem then becomes determining the ultimate values of
$I_{0}$ and $I_{1}$ following FFCs, based on their initial states. Note that
despite the availability of analytical formulae for the angular distribution
of the neutrino survival probabilities, determining these final values is
inherently complex.

This paper represents a pioneering effort in predicting the asymptotic
outcomes of FFCs in the moments scenario, using artificial neural network
(NNs). NNs imitate closely the brain’s network of connected neurons.
Specifically, they have layers of artificial neurons that handle information.
NNs have been proven to be useful in solving tricky problems due their strong
learning capacities, resulting from adjusting the connections between neurons
during the training phase. Their ability to learn from data without explicit
programming sets NNs apart, making them attracting tools across various
domains. In particular, NNs have been extensively used in the field of
astrophysics and high energy physics Guest:2018yhq ; smith2023astronomia .

Our approach involves the utilization of a NN, which takes the essential
information extracted from the initial (anti)neutrino zeroth and first moments
and then outputs the corresponding moments regarding the asymptotic outcome of
FFCs. In particular, we employ physics-informed neural networks (PINNs), where
the learning and performance of the NN can be enhanced with the utilization of
the domain knowledge (specialized information specific to the problem that can
be integrated into the NN) karniadakis2021physics ; cuomo2022scientific ;
raissi2019physics . Our findings demonstrate the efficacy of a single hidden
layer PINN, achieving a remarkable accuracy for the asymptotic values of
$I_{0}$ and $I_{1}$.

The paper is structured as follows. In Sec. II, we initiate by detailing our
simulations of FFCs and elucidating the assumptions to deriving the outcomes
of FFCs. Moving forward in Sec. III, we delve into the architecture of our
NNs, shedding light on the requisite feature engineering and the deployment of
customized loss functions. The ensuing discussion encompasses the results
gleaned from both two- and three-flavor scenarios. Finally, we conclude in
Sec. IV.

##  II FFCs simulations

To effectively train our NN, we require a substantial number of training
samples containing initial values of the (anti)neutrino moments, $I_{0}$ and
$I_{1}$, within the neutrino gas. These samples should also encompass their
corresponding final values, reflecting the asymptotic outcomes of FFCs.

Our chosen physical model involves the evolution of FFCs within a one-
dimensional (1D) box, mirroring the setup outlined in Ref. Wu:2021uvt . This
model assumes translation symmetry along the $x$ and $y$ axes, axial symmetry
around the $z$ axis, and periodic boundary conditions in the $z$ direction.
Notably, we omit considerations of vacuum mixing and neutrino-matter forward
scattering in this model.

In this study, we prime the neutrino gas for tracking its flavor evolution by
employing two widely utilized parametric neutrino angular distributions
documented in existing literature. The first one is the maximum entropy
distribution defined as,

$f^{\rm{max-ent}}_{\nu}(\mu)=\exp[\eta+a\mu],$ (3)

where we here consider the $\phi_{\nu}$-integrated distribution, i.e.,

$f_{\nu}(\mu)=\int_{0}^{\infty}\int_{0}^{2\pi}\frac{E_{\nu}^{2}\mathrm{d}E_{\nu}\mathrm{d}\phi_{\nu}}{(2\pi)^{3}}f_{\nu}(\mathbf{p}).$
(4)

This is a very natural choice for the neutrino angular distribution since the
maximum entropy closure Cernohorsky:1994yg  is currently very popular in the
moment-based neutrino transport methods. This parametric distribution has been
also used to detect $\nu$ELN crossings using fitting and machine learning
techniques Richers:2022dqa ; abbar2023applications ; Abbar:2023zkm . Another
angular distribution considered in the literature of FFCs (see, e.g., Refs.
Wu:2021uvt ; Yi:2019hrp ) is the Gaussian distribution defined as,

$f^{\rm{Gauss}}_{\nu}(\mu)=A\exp[-\frac{(1-\mu)^{2}}{\xi}].$ (5)

Note that both of these distributions have a parameter which determines the
overall neutrino number density, namely $\eta$ and $A$, and the other
parameters determining the shape of the distribution, i.e., $a$ and $\xi$.
Allowing for two distinct forms of angular distributions takes into
consideration potential deviations in the shape of neutrino angular
distributions in realistic simulations, which can occur, e.g., due to the use
of different closure relations.

To ready our datasets, we begin with the initial angular distributions of
neutrinos, which can either follow a maximum entropy distribution or a
Gaussian distribution. Subsequently, we utilize analytical neutrino survival
probabilities to determine the asymptotic outcome of FFCs. By performing
integration over the neutrino angular distributions, we can obtain the initial
and final values of $I_{0}$ and $I_{1}$.

In our analytical treatment of the survival probability, we follow closely our
recent work in Ref. Xiong:2023vcm . We assume that $G(\mu)\
(=\int_{0}^{2\pi}\mathrm{d}\phi_{\nu}G(\mathbf{v}))$ has only one zero
crossing $\mu_{c}$. This helps us to define,

$\begin{split}\Gamma_{+}&=\bigg{|}\int_{-1}^{1}\mathrm{d}\mu\
G(\mu)\Theta[G(\mu)]\bigg{|},\\\
\Gamma_{-}&=\bigg{|}\int_{-1}^{1}\mathrm{d}\mu\
G(\mu)\Theta[-G(\mu)]\bigg{|},\end{split}$ (6)

as the integration of positive and negative parts of $G(\mu)$. Here $\Theta$
is the Heaviside theta function. In the following, we specify the $\mu$ range
over which the above integral is smaller (larger) by $\mu^{<}(\mu^{>})$. For
the survival probability in the two-flavor scenario, we use the analytical
formula:

$P^{\mathrm{2f}}_{\mathrm{sur}}(\mu)=\begin{cases}\frac{1}{2}&{\rm
for\leavevmode\nobreak\ }\mu^{<},\\\ \mathcal{S}(\mu)&{\rm
for\leavevmode\nobreak\ }\mu^{>},\end{cases}$ (7)

where the distribution over $\mu^{>}$ is formulated as,

$\mathcal{S}(\mu)=1-\frac{1}{2}h(|\mu-\mu_{c}|/\zeta).$ (8)

Here, $h(x)$ is a $\mu$-dependent function that monotonically decreases from 1
to 0 when $x$ increases from 0 to infinity. To be specific, we here assume
$h(x)$ to have a power-1/2 form, i.e., $h(x)=(x^{2}+1)^{-1/2}$. In addition,
the parameter $\zeta$ can be found such that the survival probability function
is continuous.

In the three-flavor case where $\nu_{\mu}$ and $\nu_{\tau}$ are
indistinguishable, one can simply find the survival probabilities by the
expression
$P^{\mathrm{3f}}_{\mathrm{sur}}(\mu)=1-4[1-P^{\mathrm{2f}}_{\mathrm{sur}}(\mu)]/3$.

As illustrated in Table. 1 of Ref. Xiong:2023vcm , adopting a power-1/2 form
for the survival probability proves to yield a comparatively low error in
computing $I_{0}$ and $I_{1}$ analytically. This explains the rationale behind
opting for this analytical survival probability in this work. In the
concluding part of Sec. III, we also explore the scenario where the outcomes
derived from actual simulations of FFCs are applied.

##  III Applications of Neural Networks

Before unveiling our findings, it’s crucial to emphasize that to ensure high
performance of our NN models in the test set, it is necessary to divide the
dataset into three distinct sets. These sets are defined as follows: i)
Training Set: This set serves as the foundation for training the NN, allowing
it to learn and adapt based on the provided data. ii) Development Set: Also
known as the validation set, this subset plays a pivotal role in determining
the optimal hyper-parameters of the algorithm. It serves as a testing ground
to fine-tune the model for optimal performance. iii) Test Set: To assess the
NN efficacy on novel, unseen data, the test set is utilized. This set provides
a critical evaluation of the model’s generalization capabilities beyond the
training data.

###  III.1 The architecture of NNs

For a given arbitrary neutrino gas, one is provided with the initial values of
$I_{0}$’s and $I_{1}$’s of $\nu_{e}$, $\bar{\nu}_{e}$, and $\nu_{x}$. In this
context, we make the assumption that the initial distributions of
$\bar{\nu}_{x}$ and $\nu_{x}$ are identical (though their final ones following
FFCs could be different), a simplification that aligns with the majority of
state-of-the-art CCSN and NSM simulations. In order to enhance the performance
of our NNs, we introduce a layer of feature engineering, employing the
following features as pertinent inputs in our NNs:

$\begin{split}&\alpha,\quad\alpha_{\nu_{x}},\quad F_{\nu_{e}},\quad
F_{\bar{\nu}_{e}},\quad\mathrm{and}\quad F_{\nu_{x}},\\\
\mathrm{with:}\quad&\alpha=\frac{n_{\bar{\nu}_{e}}}{n_{\nu_{e}}},\quad\alpha_{\nu_{x}}=\frac{n_{\nu_{x}}}{n_{\nu_{e}}},\quad\mathrm{and}\quad
F_{\nu}=\bigg{(}\frac{I_{1}}{I_{0}}\bigg{)}_{\nu}.\end{split}$ (9)

Note that the selection of these features offers explicit insights into the
configuration of neutrino angular distributions, which plays a crucial role in
understanding the asymptotic outcome of FFCs. Furthermore, it is worth
highlighting that all quantities in this context are normalized by the initial
$\nu_{e}$ number density, allowing the convenient choice of setting it to
$n^{\rm{initial}}_{\nu_{e}}=1$. This simplification reduces the number of
inputs to our NNs, and notably, there is no input parameter related to
$n_{\nu_{e}}$.

Though the aforementioned features serve as a necessary foundation for
developing a NN, there remains room for further enhancement through more
advanced feature engineering to optimize the performance of our NNs. This
optimization can be achieved by gaining insights from the neutrino survival
probability’s shape, as expressed in Eq. (7). Substantial information
pertaining to the distribution of the survival probability can be derived by
learning the position of $\mu_{c}$. Another valuable piece of information,
given $\mu_{c}$, is determining the specific side of $\mu_{c}$ on which
equipartition occurs, while the behavior of the survival probability on the
other side is governed by conservation laws. Information regarding the side on
which equipartition occurs is provided in the quantity $E_{RL}$, a binary
number which is 1 if the equipartition occurs for $\mu_{c}\leq\mu$, and 0
otherwise.

Figure 1:  Schematic architecture of our NNs. The green zone shows the
implementation of the extra features, $\mu_{c}$, and $E_{RL}$, which are
obtained through an extra layer of regression, using linear and logistic
regressions, respectively. Here, $\mu_{c}$ is the crossing direction and
$E_{RL}$ is a binary, which is 1 if the equilibrium occurs for
$\mu_{c}\leq\mu$, and 0 otherwise. In our basic NN, referred to as the NN with
no extra features, the NN only takes the inputs highlighted in Eq. (9).
However and in our PINN, we provide our NN with the extra features $\mu_{c}$
and $E_{RL}$.

In our NN framework, we explore two distinct architectures, as illustrated in
Fig. 1. In the foundational architecture, we integrate only $\alpha$,
$\alpha_{\nu_{x}}$, $F_{\nu_{e}}$, $F_{\bar{\nu}_{e}}$, and $F_{\nu_{x}}$ into
our NN. An alternative NN that we examine involves an additional layer of
feature engineering, as discussed in the preceding paragraph, encompassing
information about $\mu_{c}$ and $E_{RL}$.

Practically, this augmentation is accomplished by constructing a _separate_
regression model trained on our dataset, from which information regarding
$\mu_{c}$ and $E_{RL}$ can be readily extracted. We have confirmed that the
computation of $E_{RL}$ and $\mu_{c}$ can be performed with small errors.

As illustrated in Fig. 1, our feedforward NN has a single hidden layer
containing 50 neurons, unless stated otherwise. The rationale for this choice
is illustrated in Fig. 3 and the text around it. Also regarding the output
layer, our NNs provide the values of $I_{0}$ and $I_{1}$ for both $\nu_{e}$
and $\bar{\nu}_{e}$, effectively utilizing a total of 4 neurons. The
determination of $I_{0}$ and $I_{1}$ for $\nu_{x}$ and $\bar{\nu}_{x}$ can be
deduced by applying the principles of neutrino and antineutrino number density
and momentum conservation. In simpler terms, our NN’s ensures that the
fundamental laws governing neutrino conservation are respected without any
exceptions.

###  III.2 Loss functions

The loss function, $\mathcal{L}$, is a crucial component in training NNs,
serving as a measure of the model’s predictive performance. It quantifies the
disparity between predicted values and actual target values, providing a guide
for the model to adjust its parameters during the optimization process.

When it comes to neutrino flavor conversions in CCSNe and NSMs, a critical
parameter of utmost significance is the number of neutrinos in the electron
channel, i.e., $N_{{\nu_{e}}+{\bar{\nu}_{e}}}=n_{\nu_{e}}+n_{\bar{\nu}_{e}}$,
as opposed to the number of neutrinos in the heavy-lepton channel,
$N_{{\nu_{x}}+{\bar{\nu}_{x}}}=n_{\nu_{x}}+n_{\bar{\nu}_{x}}$. Leveraging this
crucial physical insight to enhance the performance of our NN, we incorporate
an additional loss term in the optimization of the NN model with the extra
features. This loss term is designed to penalize discrepancies in
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$, and is defined as,

$\mathcal{L}_{\rm{extra}}=\frac{1}{N_{\rm{sample}}}\Sigma_{i}(\Delta
N_{{\nu_{e}}+{\bar{\nu}_{e}},i})^{2},$ (10)

where $\Delta$, $N_{\rm{sample}}$, and $\Sigma_{i}$ denote the difference
between the true and predicted values, the number of samples in the training
set, and the summation over the training samples, respectively. This loss term
provides an additional constraint for the model. The integration of the domain
knowledge characterizes this particular NN architecture as a PINN, given that
its distinctive nature is shaped by our insights into the underlying physics
of the problem karniadakis2021physics ; cuomo2022scientific ;
raissi2019physics . The PINN should be compared with our basic NN, referred to
as NN with _no extra features_ , for which the loss term only includes the
ordinary mean squared errors of the output parameters. Note that the
incorporation of the domain knowledge in our PINN includes both architectural
aspects, utilizing additional features, and learning-based enhancements via
the loss function. It’s also important to highlight that unlike what typically
observed in PINNs, our approach does not involve a loss term associated with
some partial differential equations.

###  III.3 Three-flavor scenario

Figure 2:  Performance evaluation of various NN models. In the upper panels,
we present the absolute error in the output parameters of the NN models, along
with the relative error in the total number of neutrinos within the electron
channel, $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. Moving to the lower panels, one can
find the distribution in the error of $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. The
results are showcased for our PINN, our basic NN with no extra features, as
well as a NN model that doesn’t enforce the conservation laws for neutrinos,
respectively. In the upper right panel, the black and green curves represent
the relative error in the total number and its first moments for
(anti)neutrinos.

In this section, we discuss the evaluation of our NNs concerning their
predictions for the asymptotic outcome of FFCs in the three-flavor scenario.
To train and assess our model, we utilize a dataset comprising a well-balanced
combination of maximum entropy and Gaussian initial neutrino angular
distributions. The ultimate outcome of FFCs is determined through a three-
flavor survival probability, as detailed in Eq. (7) and the surrounding text.
To better encapsulate realistic conditions regarding the values of $n_{\nu}$’s
and the hierarchy among $F$’s, we prepare each sample by randomly selecting
the inputs for our NNs. Specifically, we set $\alpha\in(0,2.5)$,
$\alpha_{\nu_{x}}\in(0,3)$, $F_{\nu_{x}}\in(0,1)$,
$F_{\bar{\nu}_{e}}\in(0.4F_{\nu_{x}},F_{\nu_{x}})$, and
$F_{\nu_{e}}\in(0.4F_{\bar{\nu}_{e}},F_{\bar{\nu}_{e}})$. This selection
process ensures consistency with the expected hierarchy $F_{\nu_{e}}\lesssim
F_{\bar{\nu}_{e}}\lesssim F_{\nu_{x}}$, characteristic of CCSN environment.
Given these quantities, one can then determine the initial angular
distributions of neutrinos. Using the analytical survival probability, the
final $I$’s can be derived.

In the left panels of Fig. 2, we present the performance results of our PINN
model. Here, an epoch refers to a single pass through the entire training
dataset during the training phase. Notably, the relative error in the electron
neutrino number density, defined as
$|\Delta(n_{\nu_{e}}+n_{\bar{\nu}_{e}})|/(n_{\nu_{e}}+n_{\bar{\nu}_{e}})$, can
attain values as low as 2.5%. Furthermore, we observe that the mean absolute
error in the output quantities, defined as $(|\Delta I^{\nu_{e}}_{0}|+|\Delta
I^{\nu_{e}}_{1}|+|\Delta I^{\bar{\nu}_{e}}_{0}|+|\Delta
I^{\bar{\nu}_{e}}_{1}|)/4$, can reach values $\sim 3\%$. Additionally, from
the lower panel, it is evident that almost 90% of the predictions exhibit
errors of $\lesssim 5\%$ in $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$.

In the middle panels of Fig. 2, we present the performance of our basic NN
with no extra feature and no loss term for enhancing the accuracy of
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. It is evident that the errors in this case
are a bit greater than those of the PINN model. It’s also worth noting that
while the error in $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$ is smaller than the
absolute error in the output quantities, the gap between them has been
reduced. This could be attributed to the lack of a specific loss term
targeting the reduction of errors in $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. In
addition and by examining the lower panel, we can observe that almost 85% of
the predictions still exhibit errors of $\lesssim 5\%$ in
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$.

Figure 3:  Performance evaluation of various NN models (on the validation set)
as a function of the number of neurons in the hidden layer. It is evident that
the NNs perform best on the validation set once $n_{h}\gtrsim 50$. The labels
and NN models are the same as those in Fig. 2.

Instead of rigidly enforcing the conservation of neutrino quantities, an
alternative approach involves a more flexible NN model for which strict
conservation laws are not forcefully respected.  Instead, one can aim to
derive all the neutrino moments ($I_{0}$’s and $I_{1}$’s for all flavors) as
outputs, while introducing an additional loss term, which _effectively_
enforces the conservation of (anti)neutrino number densities and momenta. This
increased flexibility, combined with the loss term addressing conservation,
has the potential to enhance the training of the NN, resulting in reduced
errors in its predictions. Consider that this NN architecture could also be
referred to as a PINN due to its loss term encompassing domain-specific
knowledge, accounting for the conservation laws governing neutrino number
density and momentum.

However, our findings emphasize the potential significant risks associated
with applying such an informed NN to our specific problem. This concern is
evident in the right panel of Fig. 2. While the absolute error and the error
observed in $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$ are comparable to what one
observes in the left and middle panels, there is a new type of error which
appears in the total number of neutrinos and its first moment, soaring to
values as high as 1-3%. Such errors, whether in total neutrino number density
or momentum, have the potential to distort the physics of CCSNe and NSMs. Note
that such errors can generate greater risks in scenarios where a disparity
might exist between the training and test datasets.

All calculations presented so far employ a feedforward NN with a single hidden
layer containing $n_{h}=50$ neurons. The rationale for this choice of the
number of neurons is illustrated in Fig. 3, where different errors are shown
for different NN architectures. It is evident that the NNs perform best on the
validation set once $n_{h}\gtrsim 50$. It is also illuminating to note that
if, for any reason such as computational constraints, a simpler NN with a
smaller $n_{h}$ is used, the performance of the PINN surpasses notably that of
the model without additional features. However, this performance gap
diminishes as larger $n_{h}$ values are utilized. Furthermore, it is evident
that even the model without explicitly enforced conservation laws achieves its
optimal performance when $n_{h}\gtrsim 50$.

Figure 4:  Absolute error in the output of our PINN (red curve) vs the
relative error in the number of neutrinos in electron channel, i.e.,
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$ (blue curve). Note that having a few thousand
data points in the training set already leads to an error $\lesssim 5\%$, and
also the disappearance of error variations between the validation and training
sets.

In Fig. 4, we present an analysis of the performance of our PINN as a function
of the size of the training set. The red curve represents the absolute error
in the PINN’s output, while the blue curve illustrates the relative error in
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. It is noteworthy that as the training dataset
expands to incorporate several thousand data points, the error rapidly
diminishes to values below 5% and the difference between the error in the
validation and training set disappears. This establishes the absolute minimum
number of data points essential for conducting dependable calculations using
NN’s. However, it’s important to bear in mind that this requisite number is
expected to inherently grow as one explores increasingly intricate models
involving more inputs and outputs. It is also interesting to observe that the
performance of our NN remains satisfactory even when trained on relatively
small datasets, comprising just a few hundred data points.

###  III.4 Two-flavor scenario

Figure 5:  The performance of our NNs in the two-flavor scenario for the PINN
and the NN with no extra features. The left, middle, and right panels present
the performance of the NN model trained using artificial data (with analytical
prescription) and tested on simulation data, the one trained and tested on the
artificial data, and the model trained and tested on the simulation data,
respectively. Note that the PINN continues to outperform especially when it
comes to the error in the total number of neutrinos in the electron channel.

In this section, we conduct an evaluation of the performance of our NNs within
a two-flavor scenario. The architectural configuration of the NNs and the
training process closely mirror those detailed in the preceding section. A
notable departure from the prior section lies in the fact that, in this case,
we employ the two-flavor version of the survival probability (Eq. (7)).
Additionally, we have considered the results derived from our 1D box
simulations of FFCs. These simulations encompass 10,000 data points
characterized by the Gaussian angular distributions. It is crucial to
emphasize that when working with the simulation results, no analytical
prescription is employed for the survival probability. Instead, we derive the
outcomes of FFCs directly from the simulations. This distinctive approach
offers the advantage of enabling us to assess the reliability of the NN models
trained on artificial data, when tested on outcomes derived from actual
simulations of FFCs.

The left panels of Fig. 5 present the performance of our NNs trained using the
artificial data, when tested on the simulation data. Notably, the PINN
consistently outperforms the basic NN with no extra features, particularly in
the error associated with $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. It is crucial to
observe that the errors exhibit more pronounced variations compared to
previous cases. This increased variability might be attributed to inherent
systematic errors in the analytical prescription (for an estimation of the
error associated with the analytical prescription, see Table.1 of Ref.
Xiong:2023vcm ).

During the evaluation of our NNs’ performance on simulation data, we noted the
critical importance of ensuring the equivalence between the input parameter
space covered in the training set and the test set. This holds particularly
true for the ranges of $\alpha$ and $\alpha_{\nu_{x}}$, as well as for the
hierarchical relationship among $F_{\nu}$’s. To elaborate further, if the test
set encounters regions within the input parameter space that were completely
unseen during the training phase, it could significantly degrade the
performance of the NN, potentially resulting in very poor performance.

The middle and right panels of Fig. 5 display errors encountered during the
computations in which both training and testing were performed on identical
datasets—either artificial data generated using analytical formulas or data
obtained from simulations. As discussed before, we have considered here a
single form of neutrino angular distributions, namely the Gaussian one. This
analysis provides insights into the inherent errors presented in each of the
training sets. Then by comparing them with the left panels, one gets an idea
of the error existing in the analytical formula.

The results depicted in Fig. 5 demonstrate the notable enhancement achieved by
utilizing our PINN method in the case of two-flavor scenario. Specifically, a
noticeable disparity is evident between the performance of PINN and that of
the NN with no extra features, surpassing the distinctions observed in Fig. 2
for the three-flavor scenario. This fundamental discrepancy between two- and
three-flavor scenarios highlights a greater degeneracy in the former,
ultimately resulting in a more overall performance improvement when additional
information, such as the implementation of PINN, is incorporated.

It is also worth noting a shift in the hierarchy between the absolute error
and the error in $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$, in the two- and three-flavor
scenarios (compare Fig. 5 with the left and middle panels of Fig. 2). Although
this observation is intriguing, it is crucial to recognize that comparing an
absolute error with a relative error may not be entirely equitable. Such a
hierarchy is anticipated to be sensitive to changes in the data, and thus, its
intrinsic merit is limited. To address this concern, we investigated the
hierarchy between the _relative_ absolute error and the relative error in
$N_{{\nu_{e}}+{\bar{\nu}_{e}}}$. This led us to find that the hierarchy
remains consistent when considering these two types of errors, providing a
fair basis for comparison.

Despite what discussed above, we opted to utilize absolute error instead of
relative absolute error throughout this study for two primary reasons.
Firstly, in our calculations, we have already normalized all quantities by
$n_{\nu_{e}}$, resulting in having relative values for each quantity. This
implies that any absolute error could be interpreted already to be relative in
spirit. Secondly, to avoid excessive sensitivity to the small values
associated with some of the neutrino quantities, we found it more appropriate
to employ absolute error as a metric in our study. This decision ensures a
balanced and meaningful evaluation of our results.

##  IV DISCUSSION AND OUTLOOK

Intensive simulations have demonstrated that FFCs can achieve equilibrium
states in some models. In this study, we have employed neural networks (NNs)
to predict the asymptotic outcome of FFCs in a three-flavor neutrino gas
within a 1D box with periodic boundary conditions. Specifically, our focus was
on the first two moments of neutrino angular distributions as inputs/outputs,
making our NN models applicable to cutting-edge CCSNe and NSM simulations. We
have shown that our NNs can predict the asymptotic outcomes of the
(anti)neutrino $I_{0}$’s and $I_{1}$’s with a notable accuracy, corresponding
to an error of $\lesssim 3\%$.

In order to enhance the performance of our NNs, we implement some novel
features aiming at capturing the characteristics of the expected neutrino
survival probability distributions. Firstly, we incorporate a new feature
related to the position of the zero crossing in the distribution of $\nu$ELN,
$\mu_{c}$. Additionally, we introduce another feature indicating on which side
of $\mu_{c}$ the expected equipartition between different neutrino flavors
occurs. Both of these features are derived through a layer of regression
applied to the initial inputs of the NN (see Fig. 1).

In the context of neutrino flavor conversions in CCSNe and NSMs, a critical
parameter is the quantity of neutrinos and antineutrinos in the electron
channel. To further optimize our NNs, we incorporate a supplementary loss term
penalizing any discrepancies in predicting $N_{{\nu_{e}}+{\bar{\nu}_{e}}}$.
The results demonstrate a relative improvement in our customized physics-
informed neural network (PINN) due to the incorporation of extra features and
a tailored loss function, outperforming a basic neural network that uses a
standard mean squared error loss function and lacks these extra features.

We have also conducted a comprehensive evaluation of the performance of our
NNs focusing on the variance between the training and validation sets (Fig.
4). Our findings reveal that the observed variance almost disappears when
considering a minimum of a few thousand data points. This establishes an
absolute minimum number of data points essential for developing a dependable
NN for predicting the outcome of FFCs in our model.

An intriguing observation arising from our study is that, even with the
utilization of relatively small datasets, the variance remains modest, with an
associated error limited to $\lesssim 15\%$. This insight further emphasizes
the potential of NNs in scenarios where obtaining extensive datasets may be
challenging or resource-intensive.

Instead of rigidly adhering to the strict conservation of neutrino quantities,
we have also assessed the performance of a NN with a more flexible approach,
where all the neutrino moments ($I_{0}$’s and $I_{1}$’s for all flavors) are
treated as outputs. Here, we introduced an additional loss term effectively
ensuring the conservation of (anti)neutrino number densities and momenta.
However, our research has underscored noteworthy concerns associated with the
application of such an informed NN to our specific problem. Indeed, we have
shown that there could exist unignorable errors in the total number of
neutrinos and their first moments, indicating a capacity to distort the
physics of CCSNe and NSMs.

In our research, our NN models were predominantly trained on artificial data
derived from two initial parametric angular distributions: the maximum entropy
and Gaussian distributions. Additionally, we assessed the performance of our
NN models using simulation data in a two-flavor scenario. Our findings
indicate that the observed errors can be rationalized by accounting for the
anticipated discrepancies between analytical and numerical results, as well as
the inherent errors present in the training set.

In summary, our findings underscore the viability of NNs in forecasting the
asymptotic outcomes of FFCs, once only the initial two moments of neutrino
angular distributions are taken into account. This marks a significant
advancement in the potential integration of FFCs into simulations of CCSNe and
NSMs. However, there are still vital avenues for further exploration. Firstly,
our NN models were notably constrained to scenarios where neutrino
distributions were assumed to be axisymmetric. Additionally, we operated under
the assumption that $\nu_{x}$ and $\bar{\nu}_{x}$ exhibit similar
distributions. Relaxing these assumptions necessitates access to training
datasets derived from actual simulations of FFC evolution in models without
imposed axisymmetry, and where $\nu_{x}$ and $\bar{\nu}_{x}$ distributions may
differ. Furthermore, our results are founded on a single-energy neutrino gas,
prompting a crucial question regarding expectations in a multi-energy neutrino
environment. This consideration is especially relevant, given that almost all
practical applications involve predicting FFC outcome regarding the neutrino
energy spectrum. Given the efficacy of NNs in this domain, taking these
crucial steps enhances remarkably the feasibility of incorporating FFCs into
CCSN and NSM simulations.

## Acknowledgments

We are deeply grateful to Georg Raffelt for insightful conversations and
reading our manuscript. We also thank Gabriel Martínez-Pinedo and Oliver Just
for fruitful discussions. S.A. was supported by the German Research Foundation
(DFG) through the Collaborative Research Centre “Neutrinos and Dark Matter in
Astro- and Particle Physics (NDM),” Grant SFB-1258, and under Germany’s
Excellence Strategy through the Cluster of Excellence ORIGINS
EXC-2094-390783311. M.-R. W. acknowledges supports from the National Science
and Technology Council under Grant No. 111-2628-M-001-003-MY4, the Academia
Sinica under Project No. AS-CDA-109-M11, and the Physics Division, National
Center for Theoretical Sciences, as well as the resource of the Academia
Sinica Grid-computing Center (ASGC). Z.X. was supported by the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (ERC Advanced Grant KILONOVA No. 885281). We would also
like to acknowledge the use of the following softwares: Scikit-learn
pedregosa2011scikit , Keras chollet2015keras , Matplotlib Matplotlib , Numpy
Numpy , SciPy SciPy , and IPython IPython .

## References

  * (1) A. Burrows, D. Vartanyan, Core-Collapse Supernova Explosion Theory, Nature 589 (7840) (2021) 29–39.  arXiv:2009.14157, doi:10.1038/s41586-020-03059-w.
  * (2) H.-T. Janka, Explosion Mechanisms of Core-Collapse Supernovae, Ann. Rev. Nucl. Part. Sci. 62 (2012) 407–451.  arXiv:1206.2503, doi:10.1146/annurev-nucl-102711-094901.
  * (3) F. Foucart, Neutrino transport in general relativistic neutron star merger simulations (9 2022).  arXiv:2209.02538, doi:10.1007/s41115-023-00016-y.
  * (4) K. Kyutoku, M. Shibata, K. Taniguchi, Coalescence of black hole–neutron star binaries, Living Rev. Rel. 24 (1) (2021) 5.  arXiv:2110.06218, doi:10.1007/s41114-021-00033-4.
  * (5) S. A. Colgate, R. H. White, The Hydrodynamic Behavior of Supernovae Explosions, Astrophys. J. 143 (1966) 626.  doi:10.1086/148549.
  * (6) J. M. Lattimer, D. N. Schramm, Black-hole-neutron-star collisions, Astrophys. J. Lett. 192 (1974) L145.  doi:10.1086/181612.
  * (7) J. T. Pantaleone, Neutrino oscillations at high densities, Phys. Lett. B287 (1992) 128–132.  doi:10.1016/0370-2693(92)91887-F.
  * (8) G. Sigl, G. Raffelt, General kinetic description of relativistic mixed neutrinos, Nuclear Physics B 406 (1) (1993) 423–451.  doi:10.1016/0550-3213(93)90175-O.
  * (9) S. Pastor, G. Raffelt, Flavor oscillations in the supernova hot bubble region: Nonlinear effects of neutrino background, Phys. Rev. Lett. 89 (2002) 191101.  arXiv:astro-ph/0207281, doi:10.1103/PhysRevLett.89.191101.
  * (10) H. Duan, G. M. Fuller, J. Carlson, Y.-Z. Qian, Simulation of Coherent Non-Linear Neutrino Flavor Transformation in the Supernova Environment. 1. Correlated Neutrino Trajectories, Phys. Rev. D 74 (2006) 105014.  arXiv:astro-ph/0606616, doi:10.1103/PhysRevD.74.105014.
  * (11) H. Duan, G. M. Fuller, J. Carlson, Y.-Z. Qian, Coherent Development of Neutrino Flavor in the Supernova Environment, Phys. Rev. Lett. 97 (2006) 241101\.  arXiv:astro-ph/0608050, doi:10.1103/PhysRevLett.97.241101.
  * (12) H. Duan, G. M. Fuller, Y.-Z. Qian, Collective Neutrino Oscillations, Ann. Rev. Nucl. Part. Sci. 60 (2010) 569–594.  arXiv:1001.2799, doi:10.1146/annurev.nucl.012809.104524.
  * (13) A. Mirizzi, I. Tamborra, H.-T. Janka, N. Saviano, K. Scholberg, R. Bollig, L. Hüdepohl, S. Chakraborty, Supernova neutrinos: Production, oscillations and detection, Riv. Nuovo Cim. 39 (1-2) (2016) 1–112.  arXiv:1508.00785, doi:10.1393/ncr/i2016-10120-8.
  * (14) M. C. Volpe, Neutrinos from dense: flavor mechanisms, theoretical approaches, observations, new directions (1 2023).  arXiv:2301.11814.
  * (15) R. F. Sawyer, Speed-up of neutrino transformations in a supernova environment, Phys. Rev. D 72 (2005) 045003.  arXiv:hep-ph/0503013, doi:10.1103/PhysRevD.72.045003.
  * (16) R. F. Sawyer, Neutrino cloud instabilities just above the neutrino sphere of a supernova, Phys. Rev. Lett. 116 (8) (2016) 081101.  arXiv:1509.03323, doi:10.1103/PhysRevLett.116.081101.
  * (17) S. Chakraborty, R. S. Hansen, I. Izaguirre, G. Raffelt, Self-induced neutrino flavor conversion without flavor mixing, JCAP 03 (2016) 042.  arXiv:1602.00698, doi:10.1088/1475-7516/2016/03/042.
  * (18) I. Izaguirre, G. Raffelt, I. Tamborra, Fast Pairwise Conversion of Supernova Neutrinos: A Dispersion-Relation Approach, Phys. Rev. Lett. 118 (2) (2017) 021101\.  arXiv:1610.01612, doi:10.1103/PhysRevLett.118.021101.
  * (19) F. Capozzi, B. Dasgupta, E. Lisi, A. Marrone, A. Mirizzi, Fast flavor conversions of supernova neutrinos: Classifying instabilities via dispersion relations, Phys. Rev. D 96 (4) (2017) 043016.  arXiv:1706.03360, doi:10.1103/PhysRevD.96.043016.
  * (20) M.-R. Wu, I. Tamborra, Fast neutrino conversions: Ubiquitous in compact binary merger remnants, Phys. Rev. D95 (10) (2017) 103007.  arXiv:1701.06580, doi:10.1103/PhysRevD.95.103007.
  * (21) M.-R. Wu, I. Tamborra, O. Just, H.-T. Janka, Imprints of neutrino-pair flavor conversions on nucleosynthesis in ejecta from neutron-star merger remnants, Phys. Rev. D96 (12) (2017) 123015.  arXiv:1711.00477, doi:10.1103/PhysRevD.96.123015.
  * (22) S. Abbar, H. Duan, Fast neutrino flavor conversion: roles of dense matter and spectrum crossing, Phys. Rev. D98 (4) (2018) 043014.  arXiv:1712.07013, doi:10.1103/PhysRevD.98.043014.
  * (23) S. Abbar, M. C. Volpe, On Fast Neutrino Flavor Conversion Modes in the Nonlinear Regime, Phys. Lett. B 790 (2019) 545–550.  arXiv:1811.04215, doi:10.1016/j.physletb.2019.02.002.
  * (24) F. Capozzi, B. Dasgupta, A. Mirizzi, M. Sen, G. Sigl, Collisional triggering of fast flavor conversions of supernova neutrinos, Phys. Rev. Lett. 122 (9) (2019) 091101.  arXiv:1808.06618, doi:10.1103/PhysRevLett.122.091101.
  * (25) J. D. Martin, C. Yi, H. Duan, Dynamic fast flavor oscillation waves in dense neutrino gases, Phys. Lett. B 800 (2020) 135088.  arXiv:1909.05225, doi:10.1016/j.physletb.2019.135088.
  * (26) S. Abbar, H. Duan, K. Sumiyoshi, T. Takiwaki, M. C. Volpe, On the occurrence of fast neutrino flavor conversions in multidimensional supernova models, Phys. Rev. D100 (4) (2019) 043004.  arXiv:1812.06883, doi:10.1103/PhysRevD.100.043004.
  * (27) S. Abbar, H. Duan, K. Sumiyoshi, T. Takiwaki, M. C. Volpe, Fast Neutrino Flavor Conversion Modes in Multidimensional Core-collapse Supernova Models: the Role of the Asymmetric Neutrino Distributions, Phys. Rev. D 101 (4) (2020) 043016.  arXiv:1911.01983, doi:10.1103/PhysRevD.101.043016.
  * (28) F. Capozzi, G. Raffelt, T. Stirner, Fast Neutrino Flavor Conversion: Collective Motion vs. Decoherence, JCAP 09 (2019) 002.  arXiv:1906.08794, doi:10.1088/1475-7516/2019/09/002.
  * (29) M. George, M.-R. Wu, I. Tamborra, R. Ardevol-Pulpillo, H.-T. Janka, Fast neutrino flavor conversion, ejecta properties, and nucleosynthesis in newly-formed hypermassive remnants of neutron-star mergers, Phys. Rev. D 102 (10) (2020) 103015.  arXiv:2009.04046, doi:10.1103/PhysRevD.102.103015.
  * (30) L. Johns, H. Nagakura, G. M. Fuller, A. Burrows, Neutrino oscillations in supernovae: angular moments and fast instabilities, Phys. Rev. D 101 (4) (2020) 043009.  arXiv:1910.05682, doi:10.1103/PhysRevD.101.043009.
  * (31) J. D. Martin, J. Carlson, V. Cirigliano, H. Duan, Fast flavor oscillations in dense neutrino media with collisions, Phys. Rev. D 103 (2021) 063001.  arXiv:2101.01278, doi:10.1103/PhysRevD.103.063001.
  * (32) I. Tamborra, S. Shalgar, New Developments in Flavor Evolution of a Dense Neutrino Gas, Ann. Rev. Nucl. Part. Sci. 71 (2021) 165–188.  arXiv:2011.01948, doi:10.1146/annurev-nucl-102920-050505.
  * (33) G. Sigl, Simulations of fast neutrino flavor conversions with interactions in inhomogeneous media, Phys. Rev. D 105 (4) (2022) 043005.  arXiv:2109.00091, doi:10.1103/PhysRevD.105.043005.
  * (34) T. Morinaga, Fast neutrino flavor instability and neutrino flavor lepton number crossings, Phys. Rev. D 105 (10) (2022) L101301.  arXiv:2103.15267, doi:10.1103/PhysRevD.105.L101301.
  * (35) H. Nagakura, L. Johns, A. Burrows, G. M. Fuller, Where, when, and why: Occurrence of fast-pairwise collective neutrino oscillation in three-dimensional core-collapse supernova models, Phys. Rev. D 104 (8) (2021) 083025.  arXiv:2108.07281, doi:10.1103/PhysRevD.104.083025.
  * (36) H. Sasaki, T. Takiwaki, A detailed analysis of the dynamics of fast neutrino flavor conversions with scattering effects, PTEP 2022 (7) (2022) 073E01.  arXiv:2109.14011, doi:10.1093/ptep/ptac082.
  * (37) I. Padilla-Gay, I. Tamborra, G. G. Raffelt, Neutrino Flavor Pendulum Reloaded: The Case of Fast Pairwise Conversion, Phys. Rev. Lett. 128 (12) (2022) 121102\.  arXiv:2109.14627, doi:10.1103/PhysRevLett.128.121102.
  * (38) Z. Xiong, Y.-Z. Qian, Stationary solutions for fast flavor oscillations of a homogeneous dense neutrino gas, Phys. Lett. B 820 (2021) 136550.  arXiv:2104.05618, doi:10.1016/j.physletb.2021.136550.
  * (39) F. Capozzi, M. Chakraborty, S. Chakraborty, M. Sen, Fast flavor conversions in supernovae: the rise of mu-tau neutrinos, Phys. Rev. Lett. 125 (2020) 251801\.  arXiv:2005.14204, doi:10.1103/PhysRevLett.125.251801.
  * (40) S. Abbar, F. Capozzi, R. Glas, H.-T. Janka, I. Tamborra, On the characteristics of fast neutrino flavor instabilities in three-dimensional core-collapse supernova models, Phys. Rev. D 103 (6) (2021) 063033.  arXiv:2012.06594, doi:10.1103/PhysRevD.103.063033.
  * (41) F. Capozzi, S. Abbar, R. Bollig, H. T. Janka, Fast neutrino flavor conversions in one-dimensional core-collapse supernova models with and without muon creation, Phys. Rev. D 103 (6) (2021) 063013.  arXiv:2012.08525, doi:10.1103/PhysRevD.103.063013.
  * (42) M. Delfan Azari, S. Yamada, T. Morinaga, W. Iwakami, H. Okawa, H. Nagakura, K. Sumiyoshi, Linear Analysis of Fast-Pairwise Collective Neutrino Oscillations in Core-Collapse Supernovae based on the Results of Boltzmann Simulations, Phys. Rev. D 99 (10) (2019) 103011.  arXiv:1902.07467, doi:10.1103/PhysRevD.99.103011.
  * (43) A. Harada, H. Nagakura, Prospects of Fast Flavor Neutrino Conversion in Rotating Core-collapse Supernovae, Astrophys. J. 924 (2) (2022) 109.  arXiv:2110.08291, doi:10.3847/1538-4357/ac38a0.
  * (44) S. Abbar, F. Capozzi, Suppression of fast neutrino flavor conversions occurring at large distances in core-collapse supernovae, JCAP 03 (03) (2022) 051.  arXiv:2111.14880, doi:10.1088/1475-7516/2022/03/051.
  * (45) O. Just, S. Abbar, M.-R. Wu, I. Tamborra, H.-T. Janka, F. Capozzi, Fast neutrino conversion in hydrodynamic simulations of neutrino-cooled accretion disks, Phys. Rev. D 105 (8) (2022) 083024.  arXiv:2203.16559, doi:10.1103/PhysRevD.105.083024.
  * (46) I. Padilla-Gay, I. Tamborra, G. G. Raffelt, Neutrino fast flavor pendulum. II. Collisional damping, Phys. Rev. D 106 (10) (2022) 103031.  arXiv:2209.11235, doi:10.1103/PhysRevD.106.103031.
  * (47) F. Capozzi, M. Chakraborty, S. Chakraborty, M. Sen, Supernova fast flavor conversions in 1+1D: Influence of mu-tau neutrinos, Phys. Rev. D 106 (8) (2022) 083011.  arXiv:2205.06272, doi:10.1103/PhysRevD.106.083011.
  * (48) M. Zaizen, H. Nagakura, Simple method for determining asymptotic states of fast neutrino-flavor conversion, Phys. Rev. D 107 (10) (2023) 103022.  arXiv:2211.09343, doi:10.1103/PhysRevD.107.103022.
  * (49) S. Shalgar, I. Tamborra, Neutrino decoupling is altered by flavor conversion, Phys. Rev. D 108 (4) (2023) 043006.  arXiv:2206.00676, doi:10.1103/PhysRevD.108.043006.
  * (50) C. Kato, H. Nagakura, Effects of energy-dependent scatterings on fast neutrino flavor conversions, Phys. Rev. D 106 (12) (2022) 123013.  arXiv:2207.09496, doi:10.1103/PhysRevD.106.123013.
  * (51) S. Bhattacharyya, B. Dasgupta, Fast Flavor Depolarization of Supernova Neutrinos, Phys. Rev. Lett. 126 (6) (2021) 061302.  arXiv:2009.03337, doi:10.1103/PhysRevLett.126.061302.
  * (52) M.-R. Wu, M. George, C.-Y. Lin, Z. Xiong, Collective fast neutrino flavor conversions in a 1D box: Initial conditions and long-term evolution, Phys. Rev. D 104 (10) (2021) 103003.  arXiv:2108.09886, doi:10.1103/PhysRevD.104.103003.
  * (53) S. Richers, D. E. Willcox, N. M. Ford, A. Myers, Particle-in-cell Simulation of the Neutrino Fast Flavor Instability, Phys. Rev. D 103 (8) (2021) 083013.  arXiv:2101.02745, doi:10.1103/PhysRevD.103.083013.
  * (54) S. Richers, D. Willcox, N. Ford, Neutrino fast flavor instability in three dimensions, Phys. Rev. D 104 (10) (2021) 103023.  arXiv:2109.08631, doi:10.1103/PhysRevD.104.103023.
  * (55) B. Dasgupta, Collective Neutrino Flavor Instability Requires a Crossing, Phys. Rev. Lett. 128 (8) (2022) 081102.  arXiv:2110.00192, doi:10.1103/PhysRevLett.128.081102.
  * (56) H. Nagakura, M. Zaizen, Time-Dependent and Quasisteady Features of Fast Neutrino-Flavor Conversion, Phys. Rev. Lett. 129 (26) (2022) 261101.  arXiv:2206.04097, doi:10.1103/PhysRevLett.129.261101.
  * (57) J. Ehring, S. Abbar, H.-T. Janka, G. Raffelt, I. Tamborra, Fast neutrino flavor conversion in core-collapse supernovae: A parametric study in 1D models, Phys. Rev. D 107 (10) (2023) 103034.  arXiv:2301.11938, doi:10.1103/PhysRevD.107.103034.
  * (58) J. Ehring, S. Abbar, H.-T. Janka, G. Raffelt, I. Tamborra, Fast Neutrino Flavor Conversions Can Help and Hinder Neutrino-Driven Explosions, Phys. Rev. Lett. 131 (6) (2023) 061401.  arXiv:2305.11207, doi:10.1103/PhysRevLett.131.061401.
  * (59) Z. Xiong, M.-R. Wu, Y.-Z. Qian, Symmetry and bipolar motion in collective neutrino flavor oscillations, Phys. Rev. D 108 (4) (2023) 043007.  arXiv:2303.05906, doi:10.1103/PhysRevD.108.043007.
  * (60) D. F. G. Fiorillo, G. G. Raffelt, Flavor solitons in dense neutrino gases, Phys. Rev. D 107 (12) (2023) 123024.  arXiv:2303.12143, doi:10.1103/PhysRevD.107.123024.
  * (61) H. Nagakura, Global features of fast neutrino-flavor conversion in binary neutron star mergers, Phys. Rev. D 108 (10) (2023) 103014.  arXiv:2306.10108, doi:10.1103/PhysRevD.108.103014.
  * (62) J. D. Martin, D. Neill, A. Roggero, H. Duan, J. Carlson, Equilibration of quantum many-body fast neutrino flavor oscillations (7 2023).  arXiv:2307.16793.
  * (63) D. F. G. Fiorillo, G. G. Raffelt, Slow and fast collective neutrino oscillations: Invariants and reciprocity, Phys. Rev. D 107 (4) (2023) 043024\.  arXiv:2301.09650, doi:10.1103/PhysRevD.107.043024.
  * (64) E. Grohs, S. Richers, S. M. Couch, F. Foucart, J. Froustey, J. Kneller, G. McLaughlin, Two-Moment Neutrino Flavor Transformation with applications to the Fast Flavor Instability in Neutron Star Mergers (9 2023).  arXiv:2309.00972.
  * (65) Z. Xiong, A. Sieverding, M. Sen, Y.-Z. Qian, Potential Impact of Fast Flavor Oscillations on Neutrino-driven Winds and Their Nucleosynthesis, Astrophys. J. 900 (2) (2020) 144.  arXiv:2006.11414, doi:10.3847/1538-4357/abac5e.
  * (66) X. Li, D. M. Siegel, Neutrino Fast Flavor Conversions in Neutron-Star Postmerger Accretion Disks, Phys. Rev. Lett. 126 (25) (2021) 251101.  arXiv:2103.02616, doi:10.1103/PhysRevLett.126.251101.
  * (67) R. Fernández, S. Richers, N. Mulyk, S. Fahlman, Fast flavor instability in hypermassive neutron star disk outflows, Phys. Rev. D 106 (10) (2022) 103003\.  arXiv:2207.10680, doi:10.1103/PhysRevD.106.103003.
  * (68) S. Bhattacharyya, B. Dasgupta, Late-time behavior of fast neutrino oscillations, Phys. Rev. D 102 (6) (2020) 063018.  arXiv:2005.00459, doi:10.1103/PhysRevD.102.063018.
  * (69) M. Zaizen, T. Morinaga, Nonlinear evolution of fast neutrino flavor conversion in the preshock region of core-collapse supernovae, Phys. Rev. D 104 (8) (2021) 083035.  arXiv:2104.10532, doi:10.1103/PhysRevD.104.083035.
  * (70) S. Bhattacharyya, B. Dasgupta, Elaborating the ultimate fate of fast collective neutrino flavor oscillations, Phys. Rev. D 106 (10) (2022) 103039\.  arXiv:2205.05129, doi:10.1103/PhysRevD.106.103039.
  * (71) E. Grohs, S. Richers, S. M. Couch, F. Foucart, J. P. Kneller, G. C. McLaughlin, Neutrino fast flavor instability in three dimensions for a neutron star merger, Phys. Lett. B 846 (2023) 138210.  arXiv:2207.02214, doi:10.1016/j.physletb.2023.138210.
  * (72) S. Richers, H. Duan, M.-R. Wu, S. Bhattacharyya, M. Zaizen, M. George, C.-Y. Lin, Z. Xiong, Code comparison for fast flavor instability simulations, Phys. Rev. D 106 (4) (2022) 043011.  arXiv:2205.06282, doi:10.1103/PhysRevD.106.043011.
  * (73) Z. Xiong, M.-R. Wu, S. Abbar, S. Bhattacharyya, M. George, C.-Y. Lin, Evaluating approximate asymptotic distributions for fast neutrino flavor conversions in a periodic 1D box, Phys. Rev. D 108 (6) (2023) 063003.  arXiv:2307.11129, doi:10.1103/PhysRevD.108.063003.
  * (74) M. Zaizen, H. Nagakura, Characterizing quasisteady states of fast neutrino-flavor conversion by stability and conservation laws, Phys. Rev. D 107 (12) (2023) 123021.  arXiv:2304.05044, doi:10.1103/PhysRevD.107.123021.
  * (75) M. Shibata, K. Kiuchi, Y.-i. Sekiguchi, Y. Suwa, Truncated Moment Formalism for Radiation Hydrodynamics in Numerical Relativity, Prog. Theor. Phys. 125 (2011) 1255–1287.  arXiv:1104.3937, doi:10.1143/PTP.125.1255.
  * (76) C. Y. Cardall, E. Endeve, A. Mezzacappa, Conservative 3+1 General Relativistic Variable Eddington Tensor Radiation Transport Equations, Phys. Rev. D 87 (10) (2013) 103004.  arXiv:1209.2151, doi:10.1103/PhysRevD.87.103004.
  * (77) K. S. Thorne, Relativistic radiative transfer: moment formalisms, Monthly Notices of the Royal Astronomical Society 194 (2) (1981) 439–473.
  * (78) D. Guest, K. Cranmer, D. Whiteson, Deep Learning and its Application to LHC Physics, Ann. Rev. Nucl. Part. Sci. 68 (2018) 161–181.  arXiv:1806.11484, doi:10.1146/annurev-nucl-101917-021019.
  * (79) M. J. Smith, J. E. Geach, Astronomia ex machina: a history, primer and outlook on neural networks in astronomy, Royal Society Open Science 10 (5) (2023) 221454\.
  * (80) G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, L. Yang, Physics-informed machine learning, Nature Reviews Physics 3 (6) (2021) 422–440.
  * (81) S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, F. Piccialli, Scientific machine learning through physics–informed neural networks: Where we are and what’s next, Journal of Scientific Computing 92 (3) (2022) 88.
  * (82) M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational physics 378 (2019) 686–707.
  * (83) J. Cernohorsky, S. A. Bludman, Maximum entropy distribution and closure for Bose-Einstein and Fermi-Dirac radiation transport (3 1994).
  * (84) S. Richers, Evaluating approximate flavor instability metrics in neutron star mergers, Phys. Rev. D 106 (8) (2022) 083005.  arXiv:2206.08444, doi:10.1103/PhysRevD.106.083005.
  * (85) S. Abbar, Applications of Machine Learning to Detecting Fast Neutrino Flavor Instabilities in Core-Collapse Supernova and Neutron Star Merger Models, Phys. Rev. D 107 (10) (2023) 103006.  arXiv:2303.05560, doi:10.1103/PhysRevD.107.103006.
  * (86) S. Abbar, H. Nagakura, Detecting Fast Neutrino Flavor Conversions with Machine Learning (10 2023).  arXiv:2310.03807.
  * (87) C. Yi, L. Ma, J. D. Martin, H. Duan, Dispersion relation of the fast neutrino oscillation wave, Phys. Rev. D 99 (6) (2019) 063005.  arXiv:1901.01546, doi:10.1103/PhysRevD.99.063005.
  * (88) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, Journal of machine learning research 12 (Oct) (2011) 2825–2830.
  * (89) F. Chollet, et al., Keras, https://keras.io (2015).
  * (90) J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9 (3) (2007) 90–95.  doi:10.1109/MCSE.2007.55.
  * (91) S. van der Walt, S. C. Colbert, G. Varoquaux, The numpy array: A structure for efficient numerical computation, Computing in Science & Engineering 13 (2) (2011) 22–30.  doi:10.1109/MCSE.2011.37.
  * (92) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17 (2020) 261–272.  doi:10.1038/s41592-019-0686-2.
  * (93) F. Pérez, B. E. Granger, IPython: a system for interactive scientific computing, Computing in Science and Engineering 9 (3) (2007) 21–29.  doi:10.1109/MCSE.2007.53.
URL https://ipython.org


# An Insight into the Pull Requests of GitHub

Mohammad Masudur Rahman Chanchal K. Roy University of Saskatchewan, Canada
{mor543<EMAIL_ADDRESS>
###### Abstract

Given the increasing number of unsuccessful pull requests in GitHub projects,
insights into the success and failure of these requests are essential for the
developers. In this paper, we provide a comparative study between successful
and unsuccessful pull requests made to 78 GitHub base projects by 20,142
developers from 103,192 forked projects. In the study, we analyze pull request
discussion texts, project specific information (e.g., domain, maturity), and
developer specific information (e.g., experience) in order to report useful
insights, and use them to contrast between successful and unsuccessful pull
requests. We believe our study will help developers overcome the issues with
pull requests in GitHub, and project administrators with informed decision
making.

###### category:

H.4 Information Systems Applications Miscellaneous

###### category:

D.2.8 Software Engineering Analysis

###### keywords:

maintenance, open source development

††conference: MSR ’2014, Hyderabad, India

##  1 Introduction

GitHub, a popular web-based source code hosting service, provides a convenient
way for the software developers to collaborate on open source software
development with one another. In order to contribute, a developer either
creates her own repository or forks from a _base repository_ , and continues
her work. GitHub maintains the source code and associated content (e.g.,
committed code, commit comments) for both base and forked repositories
separately. The idea is to allow the developer to continue her work without
reporting every single commit instantly to the _base repository_. The approach
helps her to avoid the _frequent merge conflicts_ with other developers of the
project, and also provides flexibility in the development. Once the developer
completes a milestone (e.g., module) involving several commits to the forked
repository, she makes a _pull request_ to the owner (i.e., administrator) of
the _base repository_ , and attempts to get her commits merged. Then other
members of the project analyze the posted commits, review the code, and the
streams of discussion among them are captured in terms of _pull request commit
comments_. The posted commits are generally accepted if both the merge
operation succeeds without conflicts and the identified concerns by other
developers are properly addressed. Unfortunately, not all the pull requests
succeed and there are growing concerns of how to make successful pull requests
in GitHub [4]. In this research, we perform a comparative study between
successful (i.e., merged with base repository) and unsuccessful (i.e., failed
to merge with base repository) pull requests by analyzing different related
artifacts such as the pull request discussion texts (i.e., code review
comments), pull request history, and project and developer specific
statistics. The study can provide important insights into the success and the
failure of a pull request at GitHub repositories.

A number of existing studies focus on the analysis of email messages [3], bug
reports [8], MSR papers [2], and commit messages of source code repositories
[5, 6] for various software maintenace activities. Our work is closely related
with the study by Hindle et al. [6], where they extract the hidden topics from
the commit comments of a code repository, and then map to different cross-
project non-functional requirements in order to analyze the software
maintenance activities. In this paper, we examine the pull request discussion
texts along with project and developer specific information using a machine
learning technique and then report the frequent technical issues and
inefficiencies in the source code hosted at GitHub. We use MSR dataset [4],
and collect information about 78,517 pull requests made to 78 base projects by
20,142 developers from 103,192 forked projects. We extract 100 underlying
topics that the reported issues of 9,421 pull requests (containing pull
request discussion) are based on. In order to extract the topics, Latent
Dirichlet Allocation (LDA) with Gibbs sampling is used, and we manually label
64 topics. We identify eight frequently discussed technical topics, and
manually analyze the pull request discussion texts for useful insights. From
the analysis of project and developer specific information, our study reports
that programming language and domain specific factors can influence the
success and failure rates of the pull requests. More importantly, it finds out
that success rate of pull requests for a project degrades comparatively with a
large number developers (e.g., more than 4,000) or a large number of forked
projects (e.g., more than 3,000). While the extracted frequent technical
topics and language or domain specific insights can help the developers with
successful pull requests, project and developer specific insights can aid the
GitHub project administrators with informed decision making in the management
of pull requests, projects and developers involved.

Figure 1: Pull Requests vs. Topics

##  2 Dataset

MSR challenge dataset [4] contains 78,955 pull requests (33,910 merged and
45,045 merge failed) made to 88 _base projects_ by 20,142 developers. Among
them, 9,601 pull requests (4,091 merged and 5,510 merge failed) made to 78
base projects contain _pull request commit comments_. Pull request commit
comments are generally short comments containing code reviews by other
developers of the project. Each pull request contains a series of
conversations (e.g., comments), and discusses a few topics concerning the
committed code. We use the challenge datatset, and consider the whole sequence
of conversations associated with a successful or a failed pull request as a
document, and collect 9,601 conversation documents for the experiment. It
should be noted that we limit our study to the 78 _base projects_ and their
forked projects, and also use other 68,916 pull requests (i.e., not containing
discussion) for the experiment. We collect the details of each project (e.g.,
domain, programming language, age and maturity) and the corresponding
developers (e.g., number, experience) for the comparative analysis. The hosted
projects belong to different application domains such as _reusable frameworks_
and _libraries, networking, database management, IDE, statistics_ and so on.
The projects are written in 13 different programming languages such as _Scala,
Python, Java, C#_ and so on.

##  3 Tools and Methodology

We apply Latent Dirichlet Allocation (LDA) [8], a popular topic modeling
technique, on the document collection to find out the underlying topics
discussed in each document (e.g., pull request conversation). In order to
apply topic modeling on the corpus, we normalize the content of each document
given that they contain natural language texts. We remove stop words from them
using a word list111https://code.google.com/p/stop-words/ provided by Google,
and perform stemming to extract the root form of each word. Then we represent
the content of each document as a collection of stemmed tokens, and we use
9,421 such documents. It should be noted that after stemming and removal of
stop words, we find the content of 180 documents insignificant (e.g., empty,
contains single word), and they are discarded from the corpus.

Topic Modeling: We use _JGibbLDA_ 222http://jgibblda.sourceforge.net/, a LDA
implementation that uses Gibbs sampling, in order to determine the
probabilistic topic model. We use 3,000 iterations of sampling, and collect
100 topics discussed in the document collection, where each topic is described
using ten relevant words. We also collect the probabilistic measures of the
extent to which a corpus document discusses each of the 100 topics, which we
use for comparative analysis in the later phase.

Topic Labeling: The tool returns a list of ten relevant words along with their
probabilistic expressiveness for each topic. However, the extracted topics
should be more comprehensive for effective analysis which demands topic
labeling. In order to label a topic, we analyze the corresponding word list
and choose the top four words representing the topic. These words are not
necessarily the words with the topmost probabilities. The labeling approach is
partially motivated by the approach of Lau et al. [7], where they use the
article titles extracted from _Wikipedia_ search for automatic topic labeling.
In our research, we use the most representative words from the list and a few
programming or technical jargons extracted from various online
sources333http://www.webopedia.com/Programming, and manually label each topic.
We successfully label 64 out of 100 topics, and the labeled topics are hosted
elsewhere [1] due to space limitation. We show the top four representative
words for each of the topics in the table, and the complete word list for each
of the topics can also be found online [1].

Figure 2: Pull Requests vs. Languages

##  4 Comparative Analysis

In this study, we analyze different artifacts and aspects associated with the
pull requests in order to extract useful information that can provide insights
into the success and failure of the pull requests.

Technical Topics Discussed: The discussion texts associated with a pull
request often contain useful information about the technical concerns
identified in the code. In our research, we identify and use them to contrast
between successful and failed pull requests. From _JGibbLDA_ tool, we collect
the probabilistic alignments of a document (e.g., pull request discussion) to
each of the 100 extracted topics. We sort the measures in descending order and
collect the top five dominant topics (i.e., due to short volume of each
discussion) discussed in the document. Then for each topic, we collect the
frequency of the pull requests (i.e., discussion documents) and the
information regarding their successes and failures. Fig. 1 shows the
percentage of the 9,421 pull requests that involve each topic in the
discussion. We note that eight topics are widely discussed (i.e., each topic
on average in 17.54% documents), and six of them can be labeled– _Recursion
and Refactoring_ (in 18.35% documents), _Database Query Execution_ (in 16.16%
documents), _Arrays and functions_ (in 11.12% documents), _Actor Model_ (in
12.22% documents), _OOP Paradigm_ (in 16.29% documents) and _Space and
Indents_ (in 10.98% documents). The two unlabeled dominant topics are
discussed in 28.60% documents on average. The rest 92 topics are less
discussed (i.e., each topic in less than 4% documents on average). We also
note that each topic is _more prevalent_ in the discussion of the unsuccessful
pull requests than that of the successful pull requests except a dominant
topic– _Actor Model_. Thus, the study reports that a few technical problems
(i.e., topics) are frequently faced by the developers in the pull requests;
however, most of the time, they fail to address them, and therefore, the pull
requests do not succeed.

Programming Language: Programming language of a project is an important aspect
to take into account when we are interested in comparative analysis of pull
requests. We find 13 programming languages used in the 78 GitHub projects, and
we find nine of them having 8-10 projects each. However, we also select _R_
language containing three projects, and discard _CSS_ , _Go_ and _TypeScript_
from the experiment due to their insignificant number of projects. We consider
the average number of successful and failed pull requests for a project from
each of the programming languages. We also note that age of the project can be
an influencing factor in this case, and therefore, we determine the average
number of successful and unsuccessful pull requests made per month. Fig. 2
shows the average number of pull requests made per month to any single base
project of each programming language by its forked projects. We note that
projects using _Scala, C, C#, R_ and _PHP_ programming languages made more
successful pull requests on average than failed ones, whereas projects of
_JavaScript(JS), Java, Python, Ruby_ and _C++_ did the opposite. We
investigate into the forks and the developer pool associated with those
projects, and find out that the first group of projects except the _PHP_
-based ones have less forks but more developers involved than those of the
later group. We also note that on average, _PHP_ and _Ruby_ -based projects
made the highest number of pull requests per month; however, Ruby-based
projects are often found with increasing number of failed pull requests per
month. Although we speculate, this is due to the maximum number of forks in
_Ruby_ projects, the finding can encourage the research on the language
specific factors on pull requests.

Application Domain: Domain specific concerns can be introduced in the pull
request discussions, and they can affect the chance of merging for a pull
request. In our research, we consider the domain of the project, and identify
seven major domains– _Networking, Database, IDE, Statistics, Framework,
Library_ and _Client Apps_. We manually categorize each of the 78 base
projects into different domains consulting their documentations provided
online. We also determine the number of pull requests received each month by a
_base project_ from a particular domain. Fig. 3 shows the pull request
statistics for each domain. We note that projects from _Framework_ and _IDE_
domains made the maximum number of pull requests each month, and their success
rates are relatively higher than that of the projects from other domains.
Projects from _Networking, Library_ and _Client apps_ domains showed
comparatively similar success and failure rates in the merging of pull
requests.

Figure 3: Pull Requests vs. Domains Figure 4: Pull Requests vs. Durations

Project Age & Maturity: Over time, a project either may get lost or get
matured through the collaboration of a number of open source developers
online. We believe that _age_ (i.e., the time interval between project
creation date and latest recorded date in the dataset, October 5, 2013) and
_number of forked projects_ are two important factors that may influence the
success rate of the pull requests of a project. We consider a timeline of five
plus years from February, 2008 to October, 2013 with one year interval, and
determine the average number of pull requests made to any single base project
each month during each interval. Fig. 4 shows the results of the experiment.
We note that up to September, 2009, no pull request are made, and from
October, 2009 to onward, the pull requests (i.e., both successful and failed)
increase almost exponentially. It should be noted that throughout the
intervals there is an increasing trend on age and number of projects, and
developer participation, which actually help the higher rate of pull requests
for the projects.

We consider the number of forked projects as an estimate of the maturity of a
base project. We find at most 103,192 forks for 78 base projects, and we
choose certain ranges. Then, for each fork count range, we determine the
average number of pull requests made to a corresponding base project each
month. Fig. 5 shows the results of the experiment. We note that with increase
in forked projects, the average number of pull requests per month increases;
however, it does not show a regular pattern. Moreover, with the increase in
forked projects, the failure rate of pull requests increases especially for
the projects with more than 2,000 forks.

Figure 5: Pull Requests vs. Project Maturity Figure 6: Pull Requests vs. No.
of Developers Figure 7: Pull Requests vs. Experience

Project Developers & Experience: Number of developers involved into a project
along with their working experience with the project are also two contributing
factors that can influence the success and failure rate of the pull requests.
We collect the information of 20,142 developers involved into 78 base
projects, whose working experience varies from five months to 68 months. We
sort the projects according to their developer range, and Fig. 6 shows the
average number of pull requests made to a base project each month with a
certain range of developers. We note that the average number of pull requests
does not increase comparatively with higher participation; however, the
projects with a large developer crowd may make an excessive number of
unsuccessful pull requests.

We choose a set of ranges for the developer experience, and Fig. 7 shows the
average number of pull requests made each month to a base project having
developers in the forked projects with certain range of experience. We
interestingly note that developers with 20 months to 50 months of experience
are found the most productive, and they made the maximum number of pull
requests each month. However, developers with further experiences are found
either less productive or making a number of unsuccessful pull requests each
month. We can speculate that the experienced developers might be involved in
management activities rather development; however, more insights could be
extracted if the role information is provided, which the dataset does not
provide.

##  5 Discussion and Conclusion

In this study, we conduct a comparative study between successful and
unsuccessful pull requests made to 78 GitHub base projects by 20,142
developers from 103,192 forked projects. We analyze the pull request
discussion texts that contain useful information about the frequent technical
issues encountered. We also analyze the pull request history along with
project and developer specific information in order to extract useful insights
into the success and the failure of pull requests. This section reports our
findings in brief as follows:

Eight _topics_ (i.e., technical issues), six of which can be labeled–
_Recursion and Refactoring_ , _Database Query Execution_ , _Arrays and
functions_ , _Actor Model_ , _OOP Paradigm_ and _Space and Indents_ , are
widely discussed in the texts of pull request discussion. Each of those eight
technical issues is faced by 17.54% of the 9,421 pull requests on average, and
7.76% of the requests succeed to merge. The rest 92 topics are less discussed,
and each of them is covered only in 3.90% of the discussion.

In case of 24 GitHub _base projects_ using three _programming languages_ –
_Ruby, Java_ and _JavaScript_ , average number of unsuccessful pull requests
per month is exceptionally higher than that of successful pull requests. While
the study points out _excessive forking_ as a possible reason, future research
can investigate into the language specific factors affecting the success and
failure of the pull requests.

Most of the projects under study belong to seven major _application domains_ ,
and three of them– _Framework, Library_ and _Client Apps_ , contain 39
projects. Projects from _IDE, Framework_ and _Client Apps_ domains demonstrate
a comparatively higher success rate of pull requests, whereas projects from
_Database_ and _Statistics_ domains show very limited activity in terms of
pull requests. The study suggests the _limited number of projects_ in the
target domains as a possible explanation of low activity; however, it also
encourages the future research on the domain specific concerns affecting pull
requests.

The _age_ and _maturity_ (i.e., number of forks) of a GitHub project clearly
affects the success and failure rates of pull requests. As time goes by and
more forks are created from the base project, the number of pull requests
increases and so do their success and failure rates. More importantly, our
study finds that the failure rate of pull requests increases rapidly (for
seven projects) when more than 3000 forks are created, which can be an
important piece of information for the administrators of the _base projects_.

The _number of developers_ involved into a project and their _experience_ can
affect the success and failure rates of pull request for a project. Our study
finds that the average number of pull requests per month for a project
increases almost regularly against increased participation of the developers;
however, the rate of unsuccessful pull requests increases exponentially (for
one project) with more than 4000 developers involved. The study also suggests
that developers with 20 to 50 months experience are found the most productive
in terms of submitting and getting pull requests accepted. While we cannot
provide a suitable explanation for this due to lack of information in the
dataset, the findings can help the project administrator to attract the
appropriate audience, and manage the existing developer pool involved into the
project.

## References

  * [1] Experiment Data.  URL http://www.usask.ca/~mor543/msr2014.
  * Demeyer et al. [2013] S. Demeyer, A. Murgia, K. Wyckmans, and A. Lamkanfi.  Happy Birthday! A Trend Analysis on Past MSR Papers.  In _Proc. MSR_ , pages 353–362, 2013.
  * Dredze et al. [2008] M. Dredze, H.M. Wallach, D. Puller, and F. Pereira.  Generating Summary Keywords for Emails Using Topics.  In _Proc. IUI_ , pages 199–206, 2008.
  * Gousios [2013] G. Gousios.  The GHTorrent Dataset and Tool Suite.  In _Proc. MSR_ , pages 233–236, 2013.
  * Hindle et al. [2009] A. Hindle, M.W. Godfrey, and R.C. Holt.  What’s Hot and What’s Not: Windowed Developer Topic Analysis.  In _Proc. ICSM_ , pages 339–348, 2009.
  * Hindle et al. [2011] A. Hindle, N.A. Ernst, M.W. Godfrey, and J. Mylopoulos.  Automated Topic Naming to Support Cross-Project Analysis of Software Maintenance Activities.  In _Proc. MSR_ , pages 163–172, 2011.
  * Lau et al. [2011] J.H. Lau, K. Grieser, D. Newman, and T. Baldwin.  Automatic Labelling of Topic Models.  In _Proc.HLT_ , pages 1536–1545, 2011.
  * Martie et al. [2012] L. Martie, V.K. Palepu, H. Sajnani, and C. Lopes.  Trendy Bugs: Topic Trends in the Android Bug Reports.  In _Proc. MSR_ , pages 120–123, 2012.