# Growing Excesses of New Scalars at the Electroweak Scale

Srimoy Bhattacharya ID<EMAIL_ADDRESS>School of Physics and
Institute for Collider Particle Physics, University of the Witwatersrand,
Johannesburg, Wits 2050, South Africa  Guglielmo Coloretti ID
<EMAIL_ADDRESS>Physik-Institut, Universität Zürich,
Winterthurerstrasse 190, CH–8057 Zürich, Switzerland  Paul Scherrer Institut,
CH–5232 Villigen PSI, Switzerland  Andreas Crivellin ID
<EMAIL_ADDRESS>Physik-Institut, Universität Zürich,
Winterthurerstrasse 190, CH–8057 Zürich, Switzerland  Paul Scherrer Institut,
CH–5232 Villigen PSI, Switzerland  Salah-Eddine Dahbi ID salah-
<EMAIL_ADDRESS>School of Physics and Institute for Collider Particle
Physics, University of the Witwatersrand, Johannesburg, Wits 2050, South
Africa  Yaquan Fang<EMAIL_ADDRESS>Institute of High Energy Physics, 19B,
Yuquan Road, Shijingshan District, Beijing, 100049, China  University of
Chinese Academy of Sciences (CAS), 19A Yuquan Road, Shijingshan District,
Beijing, 100049, China  Mukesh Kumar ID<EMAIL_ADDRESS>School of Physics
and Institute for Collider Particle Physics, University of the Witwatersrand,
Johannesburg, Wits 2050, South Africa  Bruce Mellado<EMAIL_ADDRESS>School of Physics and Institute for Collider Particle Physics, University of
the Witwatersrand, Johannesburg, Wits 2050, South Africa  iThemba LABS,
National Research Foundation, PO Box 722, Somerset West 7129, South Africa

###### Abstract

We combine searches for scalar resonances at the electroweak scale performed
by the Large Hadron Collider experiments ATLAS and CMS where persisted
excesses have been observed in recent years. Using both the side-bands of
Standard Model Higgs analyses as well as dedicated beyond the Standard Model
analyses, we find significant hints for new scalars at $\approx 95\,$GeV
($S^{\prime}$) and $\approx 152\,$GeV ($S$). The presence of a $95\,$GeV
scalar is preferred over the Standard Model hypothesis by $3.8\sigma$, while
interpreting the $152\,$GeV excesses in a simplified model with resonant pair
production of $S$ via a new heavier scalar $H(270)$, a global significance of
$\approx 5\sigma$ is obtained. While the production mechanism of the
$S^{\prime}$ cannot yet be determined, data strongly favours the associated
production of $S$, i.e. via the decay of a heavier boson $H$ ($pp\to H\to
SS^{*}$). A possible alternative or complementary decay chain is $H\rightarrow
SS^{\prime}$, where $S\to WW^{*}$ ($S^{\prime}$) would be the source of the
leptons ($b$-quarks) necessary to explain the multi-lepton anomalies found in
Large Hadron Collider data.

††preprint: PSI-PR-23-21, ZU-TH 31/23, ICPP-70

##  I Introduction

The Standard Model (SM) of particle physics is the mathematical description of
the fundamental constituents of matter and their interactions at microscopic
scales. It has been extensively and successfully tested [1, 2, 3], with the
discovery of the Brout-Englert-Higgs boson ($h$) [4, 5, 6, 7] in 2012 at the
Large Hadron Collider (LHC) [8, 9] at CERN providing the last missing
particle. Furthermore, measurements of the properties of this $125\,$GeV boson
agree with the SM predictions [10, 11, 12, 13].

Despite the overwhelming success of the SM, the existence of additional scalar
bosons is not excluded as long as their role in electroweak symmetry breaking
is sufficiently small. In fact, it is clear that the SM cannot be the ultimate
fundamental theory of nature. It can neither account for the observed non-
vanishing neutrino masses nor the existence of Dark Matter (DM) established at
astrophysical scales. Moreover, the minimality of the SM Higgs sector, i.e.
the presence of a single $SU(2)_{L}$ doublet scalar that simultaneously gives
mass to the electroweak (EW) gauge bosons and all fermions, is not guaranteed
by any symmetry or principle. In fact most New Physics (NP) models in the
literature contain new scalars, such as $SU(2)_{L}$ singlets [14, 15, 16],
doublets [17, 18, 19, 20, 21] and triplets [22, 23, 24, 25, 26, 27].
Furthermore, also models with an even more complex scalar sector, which are
especially relevant for the excesses discussed in this work, such as the next-
to-minimal two-Higgs-doublet model (N2HDM) [28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49], are frequently
studied.

On the experimental side, LHC searches for new particles in general, and new
scalars in particular, have not led to any discovery yet. However, the
searches for additional Higgses have been mostly performed inclusively or with
a limited number of topologies, such that significant regions of the phase-
space remain unexplored. In particular, associated production received
relatively little attention. In this context, in recent years the so-called
“multi-lepton anomalies” emerged, which are constituted by several tensions in
channels with multiple electrons and/or muons in the final states. These
discrepancies are statistically significant and point towards associated
production of EW scale new scalars [43, 50, 51, 52, 53, 54]. In particular,
the multi-lepton anomalies are compatible with the direct production of a
scalar $H$, with a mass of $\approx$270 GeV, that decays dominantly into a
pair of lighter scalars, $S$. A sub-set of these anomalies contains non-
resonant opposite sign, different flavour di-leptons final states (with and
without the presence of $b$-quark jets), pointing towards the decay
$S\rightarrow W^{+}W^{-}\rightarrow\ell^{+}\ell^{-},\ell=e,\mu$ with
$m_{S}=150\,\pm 5$ GeV [50].

In Ref. [55], we showed that the side-bands of the SM Higgs boson analyses of
ATLAS [56, 57, 58] and CMS [59, 60, 61, 62] in fact suggest the presence of a
narrow scalar resonance with a mass of $\approx\\!151\,$GeV, produced in
association with leptons and ($b-$)jets. Furthermore, several hints for the
existence of a new neutral scalar $S^{\prime}$ with a mass of
$\approx\\!95\,$GeV were presented by the CMS experiment [63, 64, 65]. While
previous ATLAS analyses did not exclude this potential signal [66, 67], the
latest result [68] shows a weaker-than-expected limit at this mass.
Furthermore, an old LEP measurement suggests $e^{+}e^{-}\to Z^{*}\to
S^{\prime}Z$ with $Z\to bb$ [69] and CMS finds a hint for resonant $\tau$ pair
production at a similar mass [65].

In this article, we combine, for the first time, the hints for a 95 GeV scalar
and extend and update the fit of Ref. [55] for the $\approx\\!151\,$GeV one by
including the side-bands of the recently released analysis of associate
production of the SM Higgs [70] as well as the $WW$ channel analyzed in Ref.
[71], in order to obtain combined evidence for new scalar resonances at the EW
scale.

##  II Channels

###  II.1 Low mass range: $S^{\prime}$ ($\approx 95$ GeV)

$\bm{(S^{\prime}\to bb)+Z}$: LEP reported an excess with a local significance
of 2.3$\sigma$ at $\approx 98$ GeV in Higgsstrahlung, i.e. $e^{+}e^{-}\to
ZS^{\prime}$ with $S^{\prime}\to bb$ [69]. We will not include this excess
directly in the combination, but rather use it to obtain a more narrow mass
window of $93\,{\rm GeV}<m_{S^{\prime}}<103\,$GeV to reduce the look-
elsewhere-effect (i.e. the trail factor).

$\bm{S^{\prime}\to\gamma\gamma}$: We use the $p$-value graph in Fig. 7 of the
CMS analysis [64] and Figure 7 (a) of the ATLAS analysis [68].

$\bm{{S^{\prime}\to\tau\tau}}$: The relevant $p$-value graph is provided in
the supplemental material of Ref. [65] and shows an excess with a local
significance of $3.1\sigma$. While ATLAS did not perform an explicit search
for new scalars in this final state, Ref. [67] see no excess in the side-band
of the SM Higgs boson analysis. We therefore treat this as a null result which
reduces the significance of the CMS excess by a factor of $\sqrt{2}$, assuming
that the ATLAS and CMS analyses have similar sensitivity.

$\bm{{S^{\prime}\to WW^{*}}}$: We use the combined transverse mass
distributions of lower graphs of Fig. 2 in Ref. [72] (CMS) and the upper-left
graph of Fig. 11 in Ref. [73] (ATLAS). The details of the combination and the
simulation are described in Ref. [71] where an excess with a local
significance of $\approx 2.6\sigma$ was found.

###  II.2 High mass range: $S$ ($\approx 152$ GeV)

We utilise CMS and ATLAS studies of SM Higgses, which essentially encompass
the search for other resonances in their side-bands. Depending on the channel,
they range up to 180 GeV. However, because some analyses stop at 160 GeV.
Since we want to avoid to be too close to the SM Higgs resonance, we will
utilise the region between 140 GeV and 155-160 GeV, when
appropriate.111Importantly, this mass range is suggested by, and compatible
with, the multi-lepton anomalies.

Figure 1: The $p$-value as a function of the mass of $S^{\prime}$ for the low-
mass channels (see main text for details).

The combination will be performed in two steps. The first combination includes
data reported up until 2021.

$\bm{{(S\to Z(\to\ell^{+}\ell^{-})\gamma)+\ell}}$: The invariant mass of the
$Z\gamma$ pair is used to reconstruct $m_{S}$ and, in addition, the presence
of an extra lepton (apart from the leptons produced by the $Z$ decay) is
required (Fig. 5 in Ref. [74]).

$\bm{{(S\to\gamma\gamma)+E^{T}_{\rm miss}}}$: $m_{S}$ is reconstructed from
the invariant mass of the photon pair and moderate additional missing energy
of $\approx 100$ GeV is required222The range of missing energy is dictated by
a predefined simplified model of $H\rightarrow SS^{*}$ and it is not
determined by the experimental analysis. (see Fig. 6 in Ref. [75] and Fig. 3
in Ref. [61]).

$\bm{{(S\to b\bar{b})+E^{T}_{\rm miss}}}$: $m_{S}$ is reconstructed from the
invariant mass of the bottom quark pair and missing transverse energy is
required (Fig. 14 (a) in Ref. [76]). Here, some overlap of this category and
$t\bar{t}h$ production in the SM exists. However, the contamination of the
former by the latter due to the production of $E^{T}_{\rm miss}$ from the
decay of $W$s is quite minimal (approximately 1$\%$).

$\bm{{(S\to\gamma\gamma)+b}}$-jet: In this channel, $S$ decays to two photons
and is produced in association with at least a $b$-jet. The experimental data
is extracted from Fig. 2 (top-right) in Ref. [77] and Fig. 2 in Ref. [78].

$\bm{{(S\to\gamma\gamma)+W,Z}}$: In this study, $S$ decays to two photons and
is produced in association with a weak gauge boson ($W$ or $Z$). The relevant
data is obtained from Fig. 15 (bottom-left) in Ref. [79] and Fig. 9 c) and d)
in Ref. [56].

$\bm{{S\to\gamma\gamma}}$ (inclusive): Also in the (quasi-)inclusive case,
$m_{S}$ is reconstructed from the invariant mass of the photon pair. However,
vector boson fusion, as well as the presence of additional $W$ and $Z$ bosons
and top quark-associated production are not included. Note that even though
there is no veto on missing energy, the $S(\to\gamma\gamma)+E^{T}_{\rm miss}$
channel only covers a small portion of phase space of the quasi-inclusive
final search. Here we use Fig. 15 (top-left) of Ref. [79] and Fig. 9 a) of
Ref. [56].

This determines a very narrow mass range of interest, thus avoiding further
scanning, and removing look-else-where effects when including new data. In the
second step, data from the following recent CMS and ATLAS analysis will be
added to the first combination:

$\bm{{(S\to\gamma\gamma)}+\geq 4j}$: Here $m_{S}$ corresponds to invariant
mass of di-photon pair which is produced in association with at least 4 jets
(Fig. 2 a) in Ref. [70]).

$\bm{{(S\to WW^{(*)}})+E^{T}_{\rm miss}}$: The CMS and ATLAS analyses of the
SM Higgs boson decaying to a pair of $W$ bosons are recast and combined. Here
we use the 0-jet category for which the dominant contribution from the
simplified model described above arises from $H\rightarrow S(\rightarrow
WW)S^{*}(\rightarrow E^{T}_{\rm miss})$. Other final states from associated
production have very small jet veto survival probability. For ATLAS, we have
used the data from Fig. 11 of Ref. [73] and for CMS the $m_{T}$ distributions
($p_{T2}<20\,$GeV and $p_{T2}>20\,$GeV) of Fig. 1 of Ref. [72].

$\bm{{S(\to\gamma\gamma)}+}\bm{\geq 1\ell+b-}$jet: $S$ decays to two photons
and is produced in association with at least one electron or muon ($\ell$) and
at least one tagged $b$-jet. The relevant experimental data are taken from
Fig. 5 a) in Ref. [70].

$\bm{S(\to\gamma\gamma)+\gamma}$: $S$, whose mass is obtained from the
invariant mass of the leading and sub-leading photon, is produced in
association with at least one additional photon moderate by $p_{T}\geq
25\,$GeV (see Fig. 6 a in Ref. [70]).

$\bm{{S\to e\mu}}$: Here we combine the data from ATLAS and CMS using the
right-hand panel of Fig. 1 in Ref. [80] and Fig. 8 of Ref. [81]. However,
since this signal is exotic and not easy to account for in a UV complete model
[82, 83], we will show both the combinations with and without including this
channel.

##  III Combinations

###  III.1 Low mass range (95 GeV)

First, we combine the $\gamma\gamma$ results from ATLAS and CMS by assuming
that both experiments have the same sensitivity to the signal see blue line in
Fig. 1. We then add to this the $\tau\tau$ and $S\to WW^{*}$ signals, using
Fisher’s combined probability test [84] with three degrees of freedom (DOF):

$\chi_{2n}^{2}=-2\sum_{i=1}^{n}\log\left(p_{i}\right),$ (1)

where $p_{i}$ is the $p$-value of each channel in the combination and
$\chi_{2n}^{2}$ represents the chi-squared distribution with $2n$ degrees of
freedom, where $n$ is the number of channels being combined.

The resulting $\chi^{2}$ distribution is used to calculate the combined
$p$-value shown in Fig. 1. The highest local significance of $4.1\sigma$ is
obtained at $m_{S^{\prime}}\approx 95$ GeV. Taking into account the LEP excess
which narrows the mass range, the look-elsewhere effect, for a trial factor
[85, 86] of $5/1.5\approx 3.3$ (obtained from dividing half the mass range by
the resolution of 1.5 GeV) [85], results in a global significance of
$3.8\sigma$.

Figure 2: The $p$-values of the individual high mass channels as well as their
combination, both including and excluding the $\mu e$ signal.

###  III.2 High mass range (152 GeV)

The required main production mechanism here is clearly associated production.
Therefore, we assume a simplified model with a scalar $H$, with a mass of
$270\,$GeV and directly produced via gluon fusion (as motivated by the multi-
lepton anomalies), which decays into two lighter ones where one of them is
off-shell ($SS^{*}$).333Refs. [43, 87] considered $H\rightarrow Sh$. While
this has similar signatures than $H\to SS^{*}$, it leads to problems with SM
Higgs signal strength measurements [88, 89, 90, 91, 92, 93] if it is the
dominant decay mode. We assume $S$ to be SM-like, i.e. to have the branching
ratios of a hypothetical SM Higgs with the same mass [94, 95, 96, 97, 98, 99,
100, 101, 102, 103, 104, 105, 106].

In the first step, we assume an additional branching ratio to invisible final
states. This means that the combination is thus performed with two DOFs.
Considering only $\gamma\gamma$ and $Z\gamma$ channels for the on-shell $S$,
we combine that data reported up to 2021.444The observed yields of the
$\gamma\gamma$ and $Z\gamma$ are consistent with those predicted by the
simplified model. Other channels are considered in the second combination
pass. The blue line in Fig. 2 is obtained, corresponding to a maximal local
significance of 4.0$\sigma$ at 152 GeV.555The largest local significance for
this first combination is $4.1\sigma$ at 151.5 GeV. However, the largest
significance in the global combination is obtained for 152 GeV. As such, the
significance is reported for this mass. The trials factor is computed taking
into account the different signal resolutions and the mass range (140-155
GeV). This reduces the significance to 3.8$\sigma$. This result is combined
with the results of the second combination for 152 GeV.

In order to verify the consistency between the observed signal yields of the
data released after 2021 within this simplified model, we simulated the
processes, $pp\to H\to SS^{*}$, where $S$ decays to
$S\to\gamma\gamma(Z\gamma)$ and $S^{*}$ decays to $b\bar{b}$,
$\tau^{+}\tau^{-}$, $WW^{*}$ and missing energy. Again, we assumed that the
ratios of the branching fractions of $b\bar{b}$ vs $\tau^{+}\tau^{-}$ and
$WW^{*}$ are SM-like. Afterwards, we applied the event selection criteria
detailed in Ref. [70] to extract the signal efficiency of
${{S(\to\gamma\gamma)}}$ \+ ${\geq 4}$ jets. The resulting expected yields
from the simulation are found to be in agreement within $1\sigma$ with the
observed yields from ATLAS.

A similar procedure is followed to analyze the cross-section for ${{S(\to
WW^{(*)}})+E^{T}_{\rm miss}}$. The ratio of the extracted cross-sections of
${{S(\to\gamma\gamma)+E^{T}_{\rm miss}}}$ to that of ${{S(\to
WW^{(*)}})+E^{T}_{\rm miss}}$ is also consistent with the prediction of the
simplified model. However, for purely SM-like branching ratios, the simplified
model predicts an excess in $S\rightarrow ZZ^{*}\rightarrow 4\ell$, which is
not observed. As such, the significance of the ${{S(\to WW^{(*)}})+E^{T}_{\rm
miss}}$ is included in the combination with an additional DOF.

Because ${(S\to\gamma\gamma)\geq 1\ell}+b$-jet and $(S\to\gamma\gamma)+\gamma$
are not predicted by the simplified model, they are added using two additional
DOF. The invariant mass spectra of the channels being combined are fitted with
the sum of background and signal functions described in Eq. (2) and Eq. (3).
The parameterization of each function takes into account the signal resolution
of the channel, while the background corresponds to the SM hypothesis.Figure 2
shows the local $p$-value for the considered channels separately, where the
significance for $(S\to\gamma\gamma)+\gamma$, $(S\to\gamma\gamma)+\geq 4$ jets
and $(S\to\gamma\gamma)+\geq\ell+b$-jet are calculated individually using the
formula 4 with the weighted signal efficiency $\epsilon$ is equal to one.

We proceed similarly with the $S\to e\mu$ channel, which is not present in the
simplified model. The combined results from ATLAS and CMS of $S\to e\mu$
channel are detailed in Appendix 3. Finally, all channels are combined using
Fisher’s combined probability Eq. 1 with five DOFs. Figure 2 displays the
results, where a global significance of $5.0\sigma$ is found for $m_{S}=152$
GeV. Since the $S\to e\mu$ signal is exotic, we also show the combination
without this channel, leading to a global significance of $4.9\sigma$.

##  IV Conclusions

The multi-lepton anomalies signify the current statistically most significant
deviation of LHC data from the SM predictions. They can be consistently
explained by assuming a simplified model in which a heavy scalar $H$ decays
into two lighter scalars $S$ with EW scale masses. Assuming a sizable decay
width for $S\rightarrow W^{+}W^{-}\to\ell^{+}\ell^{-},\ell=e,\mu$, the mass of
the scalar was determined to be $m_{S}=150\,\pm 5$ GeV.

Motivated by these anomalies and their possible explanations, we searched for
narrow resonances in the side-bands of SM Higgs analysis and found a hint for
a $\approx 151$ GeV scalar [55]. In this article, we added to this combination
the recent ATLAS and CMS analyses released after 2021 and found that the
significance is further strengthened: assuming a simplified model with 5 DOF,
we found $\approx\\!5\sigma$ for $m_{S}\approx 152\,$GeV.

Furthermore, we combined the hints for the presence of a $\approx\\!95\,$GeV
scalar $S^{\prime}$, finding a preference of $3.8\sigma$ over the Standard
Model hypothesis. This opens the possibility of a decay chain $H\rightarrow
SS^{\prime}$ explaining the multi-lepton anomalies. In this case, the decay of
the $S$ would be the source of leptons, while $S^{\prime}$ would be the origin
of $b$-quarks. Finally, the absence of a $S\to ZZ^{*}\to 4\ell$ signal
suggests that $S$ could be the neutral component of an $SU(2)$ triplet [107,
108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 119], as motivated
by average [120] of the $W$ mass measurement [121, 122, 123, 124].

Figure 3: Individual and combined $p$-values for $S^{\prime}\to e\mu$ as a
function of the mass.

###### Acknowledgements.

The work of A.C. is supported by a professorship grant from the Swiss National
Science Foundation (No. PP00P21_76884). B.M. gratefully acknowledges the South
African Department of Science and Innovation through the SA-CERN program, the
National Research Foundation, and the Research Office of the University of the
Witwatersrand for various forms of support.

##  Appendix A Details of the analysis

###  A.1 $S$ $\rightarrow$ $\gamma\gamma$

As outlined in the introduction, we use CMS and ATLAS analyses to search for
new scalars in the mass range between 90 GeV and 110 GeV as well as between
140 GeV and $155\,$GeV. For the side-band searches, for each category, we
model the background via

$f(m;b,\\{a\\})=(1-m)^{b}(m)^{a_{0}+a_{1}\log(m)}\,,$ (2)

where $a_{0,1}$ and $b$ are free parameters (different for each category) and
$m$ is the invariant mass of the distribution, e.g. the di-photon mass. The
choice of the functional form to model the background is not important for our
study, as shown in Ref. [55] to which the reader is refereed to for more
technical details.

We add to the background parametrized by Eq. (2) a double-sided-crystal-ball
function:

$N\cdot\begin{cases}e^{-t^{2}/2}&\mbox{if $-\alpha_{\rm Low}\leq
t\leq\alpha_{\rm High}$}\\\ \frac{e^{-0.5\alpha_{\rm
Low}^{2}}}{\left[\frac{\alpha_{\rm Low}}{n_{\rm Low}}\left(\frac{n_{\rm
Low}}{\alpha_{\rm Low}}-\alpha_{\rm Low}-t\right)\right]^{n_{\rm
Low}}}&\mbox{if $t<-\alpha_{\rm Low}$}\\\ \frac{e^{-0.5\alpha_{\rm
High}^{2}}}{\left[\frac{\alpha_{High}}{n_{\rm High}}\left(\frac{n_{\rm
High}}{\alpha_{\rm High}}-\alpha_{\rm High}+t\right)\right]^{n_{\rm
High}}}&\mbox{if $t>\alpha_{\rm High}$}.\end{cases}$ (3)

Here $N$ is a normalization parameter, $t=(m-m_{S})/\sigma_{CB}$ where
$\sigma_{CB}$ is the width of the Gaussian part of the function, $m$ is the
invariant mass of the distribution and $m_{S}$ the mass of the new scalar.

###  A.2 $S\rightarrow WW$

We have simulated the process $pp\rightarrow H\rightarrow SS^{*},S\rightarrow
W^{+}W^{-(*)}\rightarrow\ell^{+}\ell^{-}\nu\bar{\nu}$ and $S^{*}$ going to
missing energy. Note that this is dominant compared to $S^{*}\to WW$ and $S$
going to missing energy if the mass of the invisible particle is small. To
validate and improve our fast simulation we simulated the SM Higgs boson
signal, i.e. $gg\rightarrow h\rightarrow
WW^{(*)}\rightarrow\ell^{+}\ell^{-}\nu\bar{\nu}$ and compared to the ATLAS one
for the SM Higgs boson signal given as a function of the transverse mass
$m_{T}$ in Fig. 11 in Ref. [73]. In addition, smearing etc. was applied to
correct for our fast simulation as explained in Ref. [71].

###  A.3 Combination of ATLAS and CMS data of the $S\rightarrow e\mu$ channel

The combination of ATLAS [80] and CMS [81] data are obtained from a
simultaneous fit to the invariant mass of the electron–muon pair, in the mass
range between $140$ GeV and $157$ GeV. The backgrounds in both ATLAS and CMS
are parameterized using functional form in Eq. (2) with different
normalization factor to take in the account the difference in background
contamination from each experiment. Afterward, we scanned the invariant mass
spectrum of ATLAS and CMS simultaneously by adding a double-sided-crystal-ball
function described in Eq. (3), the parameter $N$ of the added DSCB functions
is considered as a common and free parameter, while the remaining parameters
are fixed to the values extracted from the signal fit to Higgs-like scalar
decaying to one electron and one muon in ATLAS and CMS detectors. Figure 3
shows the combination of ATLAS and CMS data in the $S\to e\mu$ decay channel,
where the individual significance is estimated using the median significance
formula [86]

$S=\sqrt{2}\sqrt{\left(\epsilon\cdot S+B\right)\log\left(1+{\epsilon\cdot
S}/{B}\right)-\epsilon\cdot S}.$ (4)

Here, $B$ and $S$ are the continuum background and signal yields respectively,
which are extracted from the simultaneous fit. While, the weighted signal
efficiencies $\epsilon$ are found to be $55\%$ and $45\%$, accordingly with
the individual efficiency of the $S\to e\mu$ channel in the ATLAS and CMS
analyses, respectively. Finally, the combined significance is calculated by
summing the two individual significance in quadrature. After the combination,
the ATLAS significance of $3.8\sigma$ at $m_{S}\approx 146$ GeV is reduced to
$3.1\sigma$ by included the CMS result. Note that a significance of $\approx
2\sigma$ around $m_{S}=151$ GeV is found.

## References

  * Workman  _et al._ [2022] R. L. Workman _et al._ (Particle Data Group), PTEP 2022, 083C01 (2022).
  * Amhis  _et al._ [2023] Y. S. Amhis _et al._ (Heavy Flavor Averaging Group, HFLAV), Phys. Rev. D 107, 052008 (2023), arXiv:2206.07501 [hep-ex] .
  * Schael  _et al._ [2006] S. Schael _et al._ (ALEPH, DELPHI, L3, OPAL, SLD, LEP Electroweak Working Group, SLD Electroweak Group, SLD Heavy Flavour Group), Phys. Rept. 427, 257 (2006), arXiv:hep-ex/0509008 .
  * Higgs [1964a] P. W. Higgs, Phys. Lett. 12, 132 (1964a).
  * Englert and Brout [1964] F. Englert and R. Brout, Phys. Rev. Lett. 13, 321 (1964).
  * Higgs [1964b] P. W. Higgs, Phys. Rev. Lett. 13, 508 (1964b).
  * Guralnik  _et al._ [1964] G. S. Guralnik, C. R. Hagen, and T. W. B. Kibble, Phys. Rev. Lett. 13, 585 (1964).
  * Aad  _et al._ [2012] G. Aad _et al._ (ATLAS), Phys. Lett. B 716, 1 (2012), arXiv:1207.7214 [hep-ex] .
  * Chatrchyan  _et al._ [2012] S. Chatrchyan _et al._ (CMS), Phys. Lett. B 716, 30 (2012), arXiv:1207.7235 [hep-ex] .
  * Chatrchyan  _et al._ [2013] S. Chatrchyan _et al._ (CMS), Phys. Rev. Lett. 110, 081803 (2013), arXiv:1212.6639 [hep-ex] .
  * Aad  _et al._ [2013] G. Aad _et al._ (ATLAS), Phys. Lett. B 726, 120 (2013), arXiv:1307.1432 [hep-ex] .
  * Langford [2021] J. M. Langford (ATLAS, CMS), PoS LHCP2020, 136 (2021).
  * ATL [2021] “ATLAS Collaboration, “Combined measurements of Higgs boson production and decay using up to $139$ fb-1 of proton-proton collision data at $\sqrt{s}=13$ TeV collected with the ATLAS experiment”, ATLAS-CONF-2021-053,” (2021).
  * Silveira and Zee [1985] V. Silveira and A. Zee, Phys. Lett. B 161, 136 (1985).
  * Pietroni [1993] M. Pietroni, Nucl. Phys. B 402, 27 (1993), arXiv:hep-ph/9207227 .
  * McDonald [1994] J. McDonald, Phys. Rev. D 50, 3637 (1994), arXiv:hep-ph/0702143 .
  * Lee [1973] T. D. Lee, Phys. Rev. D 8, 1226 (1973).
  * Haber and Kane [1985] H. E. Haber and G. L. Kane, Phys. Rept. 117, 75 (1985).
  * Kim [1987] J. E. Kim, Phys. Rept. 150, 1 (1987).
  * Peccei and Quinn [1977] R. D. Peccei and H. R. Quinn, Phys. Rev. Lett. 38, 1440 (1977).
  * Turok and Zadrozny [1991] N. Turok and J. Zadrozny, Nucl. Phys. B 358, 471 (1991).
  * Konetschny and Kummer [1977] W. Konetschny and W. Kummer, Phys. Lett. B 70, 433 (1977).
  * Cheng and Li [1980] T. P. Cheng and L.-F. Li, Phys. Rev. D 22, 2860 (1980).
  * Lazarides  _et al._ [1981] G. Lazarides, Q. Shafi, and C. Wetterich, Nucl. Phys. B 181, 287 (1981).
  * Schechter and Valle [1980] J. Schechter and J. W. F. Valle, Phys. Rev. D 22, 2227 (1980).
  * Magg and Wetterich [1980] M. Magg and C. Wetterich, Phys. Lett. B 94, 61 (1980).
  * Mohapatra and Senjanovic [1981] R. N. Mohapatra and G. Senjanovic, Phys. Rev. D 23, 165 (1981).
  * He  _et al._ [2009] X.-G. He, T. Li, X.-Q. Li, J. Tandean, and H.-C. Tsai, Phys. Rev. D 79, 023521 (2009), arXiv:0811.0658 [hep-ph] .
  * Grzadkowski and Osland [2010] B. Grzadkowski and P. Osland, Phys. Rev. D 82, 125026 (2010), arXiv:0910.4068 [hep-ph] .
  * Logan [2011] H. E. Logan, Phys. Rev. D 83, 035022 (2011), arXiv:1010.4214 [hep-ph] .
  * Boucenna and Profumo [2011] M. S. Boucenna and S. Profumo, Phys. Rev. D 84, 055011 (2011), arXiv:1106.3368 [hep-ph] .
  * He  _et al._ [2012] X.-G. He, B. Ren, and J. Tandean, Phys. Rev. D 85, 093019 (2012), arXiv:1112.6364 [hep-ph] .
  * Bai  _et al._ [2013] Y. Bai, V. Barger, L. L. Everett, and G. Shaughnessy, Phys. Rev. D 88, 015008 (2013), arXiv:1212.5604 [hep-ph] .
  * He and Tandean [2013] X.-G. He and J. Tandean, Phys. Rev. D 88, 013020 (2013), arXiv:1304.6058 [hep-ph] .
  * Cai and Li [2013] Y. Cai and T. Li, Phys. Rev. D 88, 115004 (2013), arXiv:1308.5346 [hep-ph] .
  * Chen  _et al._ [2014] C.-Y. Chen, M. Freid, and M. Sher, Phys. Rev. D 89, 075009 (2014), arXiv:1312.3949 [hep-ph] .
  * Guo and Kang [2015] J. Guo and Z. Kang, Nucl. Phys. B 898, 415 (2015), arXiv:1401.5609 [hep-ph] .
  * Wang and Han [2014] L. Wang and X.-F. Han, Phys. Lett. B 739, 416 (2014), arXiv:1406.3598 [hep-ph] .
  * Drozd  _et al._ [2014] A. Drozd, B. Grzadkowski, J. F. Gunion, and Y. Jiang, JHEP 11, 105 (2014), arXiv:1408.2106 [hep-ph] .
  * Ko  _et al._ [2014] P. Ko, Y. Omura, and C. Yu, JHEP 11, 054 (2014), arXiv:1405.2138 [hep-ph] .
  * Campbell  _et al._ [2015] R. Campbell, S. Godfrey, H. E. Logan, A. D. Peterson, and A. Poulin, Phys. Rev. D 92, 055031 (2015), [Erratum: Phys.Rev.D 101, 039905 (2020)], arXiv:1505.01793 [hep-ph] .
  * Drozd  _et al._ [2016] A. Drozd, B. Grzadkowski, J. F. Gunion, and Y. Jiang, JCAP 10, 040 (2016), arXiv:1510.07053 [hep-ph] .
  * von Buddenbrock  _et al._ [2016] S. von Buddenbrock, N. Chakrabarty, A. S. Cornell, D. Kar, M. Kumar, T. Mandal, B. Mellado, B. Mukhopadhyaya, R. G. Reed, and X. Ruan, Eur. Phys. J. C 76, 580 (2016), arXiv:1606.01674 [hep-ph] .
  * Arhrib  _et al._ [2020] A. Arhrib, R. Benbrik, M. El Kacimi, L. Rahili, and S. Semlali, Eur. Phys. J. C 80, 13 (2020), arXiv:1811.12431 [hep-ph] .
  * Engeln  _et al._ [2020] I. Engeln, P. Ferreira, M. M. Mühlleitner, R. Santos, and J. Wittbrodt, JHEP 08, 085 (2020), arXiv:2004.05382 [hep-ph] .
  * Azevedo  _et al._ [2021] D. Azevedo, P. Gabriel, M. Muhlleitner, K. Sakurai, and R. Santos, JHEP 10, 044 (2021), arXiv:2104.03184 [hep-ph] .
  * Biekötter  _et al._ [2023a] T. Biekötter, S. Heinemeyer, and G. Weiglein, Eur. Phys. J. C 83, 450 (2023a), arXiv:2204.05975 [hep-ph] .
  * Banik  _et al._ [2023] S. Banik, A. Crivellin, S. Iguro, and T. Kitahara, (2023), arXiv:2303.11351 [hep-ph] .
  * Biekötter  _et al._ [2023b] T. Biekötter, S. Heinemeyer, and G. Weiglein, (2023b), arXiv:2306.03889 [hep-ph] .
  * von Buddenbrock  _et al._ [2018] S. von Buddenbrock, A. S. Cornell, A. Fadol, M. Kumar, B. Mellado, and X. Ruan, J. Phys. G 45, 115003 (2018), arXiv:1711.07874 [hep-ph] .
  * Buddenbrock  _et al._ [2019] S. Buddenbrock, A. S. Cornell, Y. Fang, A. Fadol Mohammed, M. Kumar, B. Mellado, and K. G. Tomiwa, JHEP 10, 157 (2019), arXiv:1901.05300 [hep-ph] .
  * von Buddenbrock  _et al._ [2020] S. von Buddenbrock, R. Ruiz, and B. Mellado, Phys. Lett. B 811, 135964 (2020), arXiv:2009.00032 [hep-ph] .
  * Hernandez  _et al._ [2021] Y. Hernandez, M. Kumar, A. S. Cornell, S.-E. Dahbi, Y. Fang, B. Lieberman, B. Mellado, K. Monnakgotla, X. Ruan, and S. Xin, Eur. Phys. J. C 81, 365 (2021), arXiv:1912.00699 [hep-ph] .
  * Fischer  _et al._ [2022] O. Fischer _et al._ , Eur. Phys. J. C 82, 665 (2022), arXiv:2109.06065 [hep-ph] .
  * Crivellin  _et al._ [2021] A. Crivellin, Y. Fang, O. Fischer, A. Kumar, M. Kumar, E. Malwa, B. Mellado, N. Rapheeha, X. Ruan, and Q. Sha, (2021), arXiv:2109.02650 [hep-ph] .
  * ATL [2020] “ATLAS Collaboration, “Measurement of the properties of Higgs boson production at $\sqrt{s}$=13 TeV in the $H\to\gamma\gamma$ channel using 139 fb-1 of $pp$ collision data with the ATLAS experiment, ATLAS-CONF-2020-026,” (2020).
  * Aad  _et al._ [2020a] G. Aad _et al._ (ATLAS), Phys. Rev. Lett. 125, 061802 (2020a), arXiv:2004.04545 [hep-ex] .
  * Aad  _et al._ [2021a] G. Aad _et al._ (ATLAS), JHEP 10, 013 (2021a), arXiv:2104.13240 [hep-ex] .
  * Sirunyan  _et al._ [2021a] A. M. Sirunyan _et al._ (CMS), JHEP 07, 027 (2021a), arXiv:2103.06956 [hep-ex] .
  * Sirunyan  _et al._ [2020a] A. M. Sirunyan _et al._ (CMS), Phys. Rev. Lett. 125, 061801 (2020a), arXiv:2003.10866 [hep-ex] .
  * Sirunyan  _et al._ [2018a] A. M. Sirunyan _et al._ (CMS), JHEP 09, 046 (2018a), arXiv:1806.04771 [hep-ex] .
  * Sirunyan  _et al._ [2018b] A. M. Sirunyan _et al._ (CMS), JHEP 11, 152 (2018b), arXiv:1806.05996 [hep-ex] .
  * Sirunyan  _et al._ [2019] A. M. Sirunyan _et al._ (CMS), Phys. Lett. B 793, 320 (2019), arXiv:1811.08459 [hep-ex] .
  * CMS [2023a] “CMS Collaboration, “Search for a standard model-like Higgs boson in the mass range between 70 and 110$~{}\mathrm{GeV}$ in the diphoton final state in proton-proton collisions at $\sqrt{s}=13~{}\mathrm{TeV}$”, CMS-PAS-HIG-20-002,” (2023a).
  * CMS [2022a] “CMS Collaboration, “Searches for additional Higgs bosons and for vector leptoquarks in $\tau\tau$ final states in proton-proton collisions at $\sqrt{s}$ = 13 TeV”, CMS-HIG-21-001, CERN-EP-2022-137,” (2022a), arXiv:2208.02717 [hep-ex] .
  * ATL [2018] “ATLAS Collaboration, “Search for resonances in the 65 to 110 GeV diphoton invariant mass range using 80 fb-1 of $pp$ collisions collected at $\sqrt{s}=13$ TeV with the ATLAS detector”, ATLAS-CONF-2018-025,” (2018).
  * Aad  _et al._ [2022] G. Aad _et al._ (ATLAS), JHEP 08, 175 (2022), arXiv:2201.08269 [hep-ex] .
  * ATL [2023a] “ATLAS Collaboration, “Search for diphoton resonances in the 66 to 110 GeV mass range using 140 fb-1 of 13 TeV $pp$ collisions collected with the ATLAS detector”, ATLAS-CONF-2023-035,” (2023a).
  * Barate  _et al._ [2003] R. Barate _et al._ (LEP Working Group for Higgs boson searches, ALEPH, DELPHI, L3, OPAL), Phys. Lett. B 565, 61 (2003), arXiv:hep-ex/0306033 .
  * ATL [2023b] “ATLAS Collaboration, “Model-independent search for the presence of new physics in events including $H\rightarrow\gamma\gamma$ with $\sqrt{s}$ = 13 TeV pp data recorded by the ATLAS detector at the LHC”,” (2023b), arXiv:2301.10486 [hep-ex] .
  * Coloretti  _et al._ [2023] G. Coloretti, A. Crivellin, S. Bhattacharya, and B. Mellado, (2023), arXiv:2302.07276 [hep-ph] .
  * CMS [2022b] “CMS Collaboration, “Measurements of the Higgs boson production cross section and couplings in the W boson pair decay channel in proton-proton collisions at $\sqrt{s}$ = 13 TeV”,” (2022b), arXiv:2206.09466 [hep-ex] .
  * ATL [2022a] “ATLAS Collaboration, “Measurements of Higgs boson production by gluon$-$gluon fusion and vector-boson fusion using $H\rightarrow WW^{*}\rightarrow e\nu\mu\nu$ decays in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector”,” (2022a), arXiv:2207.00338 [hep-ex] .
  * Sirunyan  _et al._ [2018c] A. M. Sirunyan _et al._ (CMS), JHEP 11, 152 (2018c), arXiv:1806.05996 [hep-ex] .
  * Aad  _et al._ [2021b] G. Aad _et al._ (ATLAS), JHEP 10, 013 (2021b), arXiv:2104.13240 [hep-ex] .
  * Aad  _et al._ [2021c] G. Aad _et al._ (ATLAS), Eur. Phys. J. C 81, 178 (2021c), arXiv:2007.02873 [hep-ex] .
  * Aad  _et al._ [2020b] G. Aad _et al._ (ATLAS), Phys. Rev. Lett. 125, 061802 (2020b), arXiv:2004.04545 [hep-ex] .
  * Sirunyan  _et al._ [2020b] A. M. Sirunyan _et al._ (CMS), Phys. Rev. Lett. 125, 061801 (2020b), arXiv:2003.10866 [hep-ex] .
  * Sirunyan  _et al._ [2021b] A. M. Sirunyan _et al._ (CMS), JHEP 07, 027 (2021b), arXiv:2103.06956 [hep-ex] .
  * Aad  _et al._ [2020c] G. Aad _et al._ (ATLAS), Phys. Lett. B 801, 135148 (2020c), arXiv:1909.10235 [hep-ex] .
  * CMS [2023b] “CMS Collaboration, “Search for the lepton flavor violating decay of a Higgs boson in the $\mathrm{e}\mu$ final state in proton-proton collisions at $\sqrt{s}=13~{}\mathrm{TeV}$”, CMS-PAS-HIG-22-002,” (2023b).
  * Primulando  _et al._ [2023] R. Primulando, J. Julio, N. Srimanobhas, and P. Uttayarat, (2023), arXiv:2304.13757 [hep-ph] .
  * Afik  _et al._ [2023] Y. Afik, P. S. B. Dev, and A. Thapa, (2023), arXiv:2305.19314 [hep-ph] .
  * Fisher [1925] R. Fisher,  _Statistical Methods for Research Workers_ (Oliver and Boyd, Edinburgh, Scotland, 1925).
  * Gross and Vitells [2010] E. Gross and O. Vitells, Eur. Phys. J. C 70, 525 (2010), arXiv:1005.1891 [physics.data-an] .
  * Barlow [2020] R. J. Barlow, CERN Yellow Rep. School Proc. 5, 149 (2020), arXiv:1905.12362 [physics.data-an] .
  * von Buddenbrock  _et al._ [2019] S. von Buddenbrock, A. S. Cornell, E. D. R. Iarilala, M. Kumar, B. Mellado, X. Ruan, and E. M. Shrif, J. Phys. G 46, 115001 (2019), arXiv:1809.06344 [hep-ph] .
  * Khachatryan  _et al._ [2015a] V. Khachatryan _et al._ (CMS), Phys. Rev. D 92, 012004 (2015a), arXiv:1411.3441 [hep-ex] .
  * Aad  _et al._ [2015] G. Aad _et al._ (ATLAS), Eur. Phys. J. C 75, 476 (2015), [Erratum: Eur.Phys.J.C 76, 152 (2016)], arXiv:1506.05669 [hep-ex] .
  * Khachatryan  _et al._ [2015b] V. Khachatryan _et al._ (CMS), Eur. Phys. J. C 75, 212 (2015b), arXiv:1412.8662 [hep-ex] .
  * Aad  _et al._ [2016] G. Aad _et al._ (ATLAS), Eur. Phys. J. C 76, 6 (2016), arXiv:1507.04548 [hep-ex] .
  * Tumasyan  _et al._ [2022] A. Tumasyan _et al._ (CMS), Nature 607, 60 (2022), arXiv:2207.00043 [hep-ex] .
  * ATL [2022b] Nature 607, 52 (2022b), [Erratum: Nature 612, E24 (2022)], arXiv:2207.00092 [hep-ex] .
  * de Florian  _et al._ [2016] D. de Florian _et al._ (LHC Higgs Cross Section Working Group),  2/2017 (2016), 10.23731/CYRM-2017-002, arXiv:1610.07922 [hep-ph] .
  * Braaten and Leveille [1980] E. Braaten and J. P. Leveille, Phys. Rev. D 22, 715 (1980).
  * Sakai [1980] N. Sakai, Phys. Rev. D 22, 2220 (1980).
  * Inami and Kubota [1981] T. Inami and T. Kubota, Nucl. Phys. B 179, 171 (1981).
  * Gorishnii  _et al._ [1984] S. G. Gorishnii, A. L. Kataev, and S. A. Larin, Sov. J. Nucl. Phys. 40, 329 (1984).
  * Gorishnii  _et al._ [1991a] S. G. Gorishnii, A. L. Kataev, S. A. Larin, and L. R. Surguladze, Phys. Lett. B 256, 81 (1991a).
  * Gorishnii  _et al._ [1990] S. G. Gorishnii, A. L. Kataev, S. A. Larin, and L. R. Surguladze, Mod. Phys. Lett. A 5, 2703 (1990).
  * Gorishnii  _et al._ [1991b] S. G. Gorishnii, A. L. Kataev, S. A. Larin, and L. R. Surguladze, Phys. Rev. D 43, 1633 (1991b).
  * Djouadi  _et al._ [1998] A. Djouadi, P. Gambino, and B. A. Kniehl, Nucl. Phys. B 523, 17 (1998), arXiv:hep-ph/9712330 .
  * Degrassi and Maltoni [2005] G. Degrassi and F. Maltoni, Nucl. Phys. B 724, 183 (2005), arXiv:hep-ph/0504137 .
  * Passarino  _et al._ [2007] G. Passarino, C. Sturm, and S. Uccirati, Phys. Lett. B 655, 298 (2007), arXiv:0707.1401 [hep-ph] .
  * Actis  _et al._ [2009] S. Actis, G. Passarino, C. Sturm, and S. Uccirati, Nucl. Phys. B 811, 182 (2009), arXiv:0809.3667 [hep-ph] .
  * Spira  _et al._ [1992] M. Spira, A. Djouadi, and P. M. Zerwas, Phys. Lett. B 276, 350 (1992).
  * Chabab  _et al._ [2018] M. Chabab, M. C. Peyranère, and L. Rahili, Eur. Phys. J. C 78, 873 (2018), arXiv:1805.00286 [hep-ph] .
  * Fileviez Perez  _et al._ [2022] P. Fileviez Perez, H. H. Patel, and A. D. Plascencia, Phys. Lett. B 833, 137371 (2022), arXiv:2204.07144 [hep-ph] .
  * Cheng  _et al._ [2023] Y. Cheng, X.-G. He, F. Huang, J. Sun, and Z.-P. Xing, Nucl. Phys. B 989, 116118 (2023), arXiv:2208.06760 [hep-ph] .
  * Chen  _et al._ [2022] T.-K. Chen, C.-W. Chiang, and K. Yagyu, Phys. Rev. D 106, 055035 (2022), arXiv:2204.12898 [hep-ph] .
  * Rizzo [2022] T. G. Rizzo, Phys. Rev. D 106, 035024 (2022), arXiv:2206.09814 [hep-ph] .
  * Chao  _et al._ [2022] W. Chao, M. Jin, H.-J. Li, and Y.-Q. Peng, (2022), arXiv:2210.13233 [hep-ph] .
  * Wang  _et al._ [2022] J.-W. Wang, X.-J. Bi, P.-F. Yin, and Z.-H. Yu, Phys. Rev. D 106, 055001 (2022), arXiv:2205.00783 [hep-ph] .
  * Shimizu and Takeshita [2023] Y. Shimizu and S. Takeshita, (2023), arXiv:2303.11070 [hep-ph] .
  * Lazarides  _et al._ [2022] G. Lazarides, R. Maji, R. Roshan, and Q. Shafi, Phys. Rev. D 106, 055009 (2022), arXiv:2205.04824 [hep-ph] .
  * Senjanović and Zantedeschi [2023] G. Senjanović and M. Zantedeschi, Phys. Lett. B 837, 137653 (2023), arXiv:2205.05022 [hep-ph] .
  * Crivellin  _et al._ [2023] A. Crivellin, M. Kirk, and A. Thapa, (2023), arXiv:2305.03081 [hep-ph] .
  * Chen  _et al._ [2023] T.-K. Chen, C.-W. Chiang, and K. Yagyu, JHEP 06, 069 (2023), arXiv:2303.09294 [hep-ph] .
  * Ashanujjaman  _et al._ [2023] S. Ashanujjaman, S. Banik, G. Coloretti, A. Crivellin, B. Mellado, and A.-T. Mulaudzi, (2023), arXiv:2306.15722 [hep-ph] .
  * de Blas  _et al._ [2022] J. de Blas, M. Pierini, L. Reina, and L. Silvestrini, Phys. Rev. Lett. 129, 271801 (2022), arXiv:2204.04204 [hep-ph] .
  * Aaltonen  _et al._ [2022] T. Aaltonen _et al._ (CDF), Science 376, 170 (2022).
  * ATL [2023c] (2023c).
  * Aaij  _et al._ [2022] R. Aaij _et al._ (LHCb), JHEP 01, 036 (2022), arXiv:2109.01113 [hep-ex] .
  * Schael  _et al._ [2013] S. Schael _et al._ (ALEPH, DELPHI, L3, OPAL, LEP Electroweak), Phys. Rept. 532, 119 (2013), arXiv:1302.3415 [hep-ex] .


From Table <ref>, we could first observe a mismatch between the application domains identified in RQ$_1$ and those identified here. In the first place, this finding can already provide indications of the comprehensiveness of the automated approaches proposed so far: these only exploited datasets pertaining to a few domains such as `Computer Vision', `Decision Making', `Natural Language Processing', and `Automated Speech Recognition', hence possibly neglecting the additional domains identified in our review. At the same time, our finding may be a reflection of a lack of datasets that can be used to comprehensively investigate non-functional requirements of machine learning-enabled systems—our work focused on the datasets exploited by automated approaches rather than the full set of datasets available in the literature; hence we cannot provide conclusive remarks on the unavailability of datasets; as such, we limit ourselves to highlight a potential limitation that should be further investigated.

Looking at the second column of Table <ref>, we could observe that multiple datasets were considered for the analysis of the various classes of non-functional requirements, with efficiency and resiliency having the highest amount of related datasets. From this perspective, we could conclude that multiple datasets exist for experimenting with non-functional requirements of ML-enabled systems and might be further explored in the context of further research on the matter. Yet, we still point out that the automated approaches proposed in the literature missed the analysis of a large number of application domains, which might therefore require further attention in the future.

When considering Table <ref>, we could first confirm that some specific domains deserved more attention from the research community. Application domains such as `Computer Vision' and `Decision Making' were indeed confirmed to be the most investigated ones. Furthermore, researchers in these fields have experimented with multiple alternative solutions, assessing the suitability of several algorithms to deal with efficiency and resiliency. Interestingly, as shown in Table <ref>, the `Computer Vision' domain has been investigated through various solutions based on both convolutional and recurrent neural networks, while the other domains did not have broader investigations. The pieces of information provided by our study could inform researchers on the current state of the artificial intelligence algorithms applied to deal with non-functional requirements of machine learning-enabled systems, possibly informing them of additional perspectives to take into account.

By combining the results coming from our analyses, we could also provide some insights into the nature of the datasets available and the capabilities of the current automated approaches. As we learned from Table <ref>, most of the developed solutions were based on deep learning algorithms fed through the large datasets shown in Table <ref>. On the one hand, this information highlights that the datasets available were designed to be large enough to train both shallow and deep learning solutions able to deal with non-functional issues of machine learning-enabled systems: in this respect, our findings report that, despite the intrinsic limitations of the available datasets in terms of domains considered, these might be useful to experiment with multiple artificial intelligence solutions, hence representing potentially relevant benchmarks for the research community. On the other hand, deep learning solutions seem to be the most widely employed in most application domains, possibly indicating that the management of non-functional requirements of machine learning-enabled systems requires advanced learning techniques to properly support practitioners: along this side, further experimentation and comparisons between traditional and deep learning techniques might be worth to be conducted.

§.§.§ On the automated approaches and their relevance
Table <ref> lists the specific approaches proposed so far to deal with non-functional requirements of machine learning-enabled systems. The table classifies each approach based on multiple characteristics such as (1) the development phase it can be applied: we classified them based on whether they can be applied in the pre-processing stage of `Data Engineering' (Column `DE' in Table <ref>), in the in-processing stage of `Training' (`T' in Table <ref>), and  in the post-processing stage of `Post-training' (`PT' in Table <ref>); (2) the non-functional requirements targeted, i.e., a `' symbol indicates that an approach considered the non-functional requirement, while a `' symbol indicates that the approach had not taken a non-functional requirements into account. The table also reports a description of the currently available approaches, hence providing the reader with a quick summary of the capabilities of each of them. Looking at the table, we could first highlight that most of the efforts devoted by the SE4AI community focused on post-processing approaches, with a particular focus on software testing and automated program repair. Pre- and in-processing approaches are present in a lower extent, possibly suggesting that future research might target those aspects to further support practitioners. Secondly, we could also find out that accuracy is the most widely-considered non-functional requirement: as a matter of fact, the vast majority of the automated approaches focused on accuracy as a primary objective to safeguard—this indicates that accuracy is still the most relevant non-functional property to consider while developing automated approaches. Nonetheless, we could also observe a growing interest into the multi-objective optimization of non-functional requirements. In this respect, all the automated approaches available in literature considered non-functional requirements belonging to different classes as conflicting objectives and attempted to find a compromise among them.

One of the most frequent multi-objective optimization concerned accuracy and fairness: researchers have been discovering that an increase in terms of fairness typically leads to a decrease of accuracy 10.1145/3368089.3409697,10.1145/3468264.3468537. As a consequence, the approaches proposed attempted to devise solutions that may work on data to let the resulting model be accurate enough while preserving fairness. An example of these multi-objective optimization approaches is the Fair-SMOTE algorithm proposed by Chakraborty et al. 10.1145/3468264.3468537, which removes biased labels and rebalances internal distributions so that, based on sensitive attribute, examples are equal in positive and negative classes.

Similar contrasting objectives have been discussed with respect to the compromise between accuracy and resiliency 10.1145/3510003.3510231,10.1145/3551349.3556920,9676691,8746775,10.1145/2810103.2813687,usman2021nn,10.1145/3460120.3484818,8952197, accuracy and maintainability 10.1145/3510003.3510051,10.1145/3510003.3510163,9402039,10.1145/3293882.3330580. Also in these cases, researchers found that the accuracy of ML-enabled systems typically decreases when other non-functional attributes are considered. These observations justify and stimulate additional research on multi-objective optimization.

According to our analysis of the primary studies, we could also find out that non-functional requirements pertaining to the same class were often addressed collectively, showcasing how improvements in one non-functional requirement may have indirectly improved others. For instance, automated approaches aiming at mitigating bias to enhance fairness could indirectly improve accountability and ethics 10.1145/3368089.3409697,7961993. Similarly, efforts to boost robustness against adversarial samples could indirectly benefit security, flexibility, reliability, and availability usman2021nn,10.1145/3238147.3238187. In other terms, some of the currently available approaches may have direct and indirect implications on non-functional requirements. However, we could not find a general rule describing the indirect effect of the optimization of each non-functional requirement: we believe that further software analytics research might explore this matter, possibly providing a taxonomy able to explain the cause-effect relations among non-functional requirements.

Still in terms of indirect optimization, it is worth discussing the case of capacity, i.e., the amount of space required to store the model and any associated data. The optimization of this non-functional requirement leads to a model reduction where an algorithm reduces the overall size of the model, producing a smaller model that is later distributed. By analyzing the primary studies proposing model reduction techniques 10.1145/3551349.3556906,10.1145/3530019.3530025,10.1145/3506695, we could observe that capacity has not only indirect effects on non-functional requirements pertaining to its class (`Efficiency'), but also to other classes such as `Sustainability' and `Accuracy'. Indeed, the optimization of capacity typically leads to improvements in terms of performance, i.e., a smaller model has a more efficient throughput bound. At the same time, this has implications for the overall energy consumption of the model, i.e., a smaller model consumes less energy. Yet, optimizing capacity may imply a reduction of accuracy.

The case of capacity further justifies the need for additional research on the relations among non-functional requirements and, as such, ours represents a call for requirements engineering, software analytics, and empirical software engineering researchers.

As an additional perspective, we could understand that there is no approach able to comprehensively analyze the whole set of non-functional requirements classified in the context of RQ$_1$. As a matter of fact, only a few approaches optimized two or more non-functional requirements and, more interestingly, only a few approaches were deemed to be relevant for non-functional aspects such as efficiency, maintainability, and usability, hence suggesting that more research should be done on the optimization of those concerns. Furthermore, we could see that many of the challenges identified in RQ$_2$ are still not addressed by currently available approaches. In other terms, while the literature called for research in various non-functional requirements domains, there still seems to be a little automated tooling able to address it. For instance, we could observe little to no automated solutions available for solving privacy, post-deployment simulations, legal aspects, and adversarial learning. As such, the final outcome of our literature review points out, in the first place, the lack of automated approaches able to comprehensively consider the non-functional requirements of machine learning-enabled systems. Also, we let emerge the need for additional studies on the relations among non-functional requirements, which may effectively drive the definition of multi-objective optimization approaches. Finally, our work highlights that most of the challenges identified in the context of RQ$_2$ are still open and serve as a way to inform our research community on the next steps to perform to provide additional support to practitioners.

Automated approach to optimize non-functional requirements of ML-enabled systems.
`DE'=Data Engineering; `T'=Training; `PT'=Post-Training; `A'=Accuracy; `E'=Efficiency; `M'=Maintainability; `R'=Resiliency; `S'=Sustainability; `U'=Usability.

Approach     Description     A

– continued from previous page

PT     Approach     Description     A

11|r|Continued on next page

Fairway 10.1145/3368089.3409697     This paper discusses the issue of algorithmic discrimination and ethical bias in machine learning software. It proposes a method called Fairway to detect and mitigate bias in the training data and the learned model. The method combines pre-processing and in-processing approaches and removes ethical bias without significantly affecting predictive accuracy. The paper advocates for testing and mitigating bias to become a routine part of the machine learning software development life cycle

Deepstate 10.1145/3510003.3510231     The paper proposes DeepState, a test suite selection tool for RNN models to reduce data labeling and computation costs. DeepState selects data based on a stateful perspective of RNN and identifies possibly misclassified tests by capturing neuron state changes. The tool further provides a test selection method to obtain a test suite with strong fault detection and model improvement capabilities

Dare 10.1145/3551349.3556920     The paper proposes a novel model training framework called Dare that aims to improve the universal robustness of DL models. The proposed approach incorporates model transformation and data augmentation in a delta debugging fashion

MAAT 10.1145/3540250.3549093     This paper proposes a novel ensemble approach, MAAT, to improve the fairness-accuracy trade-off for ML software. MAAT combines models optimized for different objectives: fairness and ML accuracy.

Parfait-ML 10.1145/3510003.3510202     This paper investigates the impact of hyperparameters on fairness in ML algorithms. The authors propose three search-based software testing algorithms to identify hyperparameters that can improve fairness without sacrificing precision. They use statistical debugging to explain the role of these parameters in improving fairness and implement the proposed approach in the tool Parfait-ML.

Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules 10.1145/3510003.3510051
The paper proposes a method to decompose a CNN model for image classification into modules for each output class, enabling reusability and replaceability. Reusing and replacing these modules reduces CO2e emissions

k-PPD-ERT 9676691     This paper proposes a distributed extremely randomized trees (ERT) algorithm for machine learning on healthcare data without compromising patients' privacy. The proposed method, k-PPD-ERT, uses a cloud platform and is tested on medical datasets, including Heart Disease, Breast Cancer, and two mental health datasets.

DeepCorrect 8746775     This paper discusses the impact of image distortions, such as blur and noise, on pre-trained convolutional filters in deep neural networks used for computer vision tasks. They introduced a new approach called DeepCorrect. It applies small stacks of convolutional layers with residual connections to correct the worst affected filter activations without changing the rest of the pre-trained filter outputs. The accuracy results demonstrate that DeepCorrect significantly improves the robustness of DNNs against distorted images

Distributed Selective SGD 10.1145/2810103.2813687     The paper proposes a system that allows multiple parties to learn an accurate neural network model without sharing their datasets by selectively sharing small subsets of their models' key parameters during training. This system preserves privacy while still improving learning accuracy beyond what is achievable on their inputs

Towards Training Reproducible Deep Learning Models 10.1145/3510003.3510163     The paper proposes a systematic approach to training reproducible DL models, which is crucial for various tasks like training, testing, debugging, and auditing. DL models are challenging to reproduce due to issues like randomness in the software and non-determinism in the hardware. The proposed approach includes general criteria to evaluate reproducibility, a framework using record-and-replay and profile-and-patch techniques to mitigate software-related randomness and hardware-related non-determinism, and a reproducibility guideline

NNREPAIR usman2021nn     NNREPAIR tool use a technique for repairing neural network classifiers by fixing potentially faulty network parameters. It uses fault localization to identify the faulty parameters and then performs repairs using constraint solving to make small parameter modifications. The technique can improve the accuracy of models, fix security vulnerabilities caused by the poisoning of training data, and improve the robustness of the network against adversarial attacks.

AI-Lancet 10.1145/3460120.3484818     AI-Lancet proposes a systematic approach to trace and fixes errors in deep learning models that could lead to severe security issues. They locate error-inducing neurons and propose two methods, neuron-flip and neuron-fine-tuning, to fix the errors.

Apricot 8952197     The paper presents Apricot, a novel weight-adaptation approach to fixing imprecise deep learning models. Apricot generates a set of reduced DL models from the original DL model. In each iteration, it adjusts the weights of the input DL model towards the average weight of the reduced DL models that correctly classify a failing test case and/or away from that of the reduced DL models that misclassify the same test case.

FairNeuron 10.1145/3510003.3510087     The paper proposes a tool called FairNeuron, which automatically repairs DNN models to balance accuracy and fairness trade-offs by detecting neurons with contradictory optimization directions from accuracy and fairness training goals and achieving a trade-off by selective dropout. The approach is lightweight, scales to large models, and is more efficient than state-of-the-art methods.

RobOT 9402039     This paper proposes a testing framework called Robustness-Oriented Testing (RobOT) for DL systems that aims to improve model robustness. RobOT utilizes a novel quantitative measurement that evaluates the value of each test case in improving model robustness and the convergence quality of the model robustness improvement.

Search-based test and improvement of machine-learning-based anomaly detection systems 10.1145/3293882.3330580     This paper focuses on testing intrusion detection systems (IDS) against training attacks, which are designed to exploit the vulnerability of machine-learning-based anomaly detection systems. The authors propose a search-based approach to test IDS by making training attacks, and they also propose searching for countermeasures to increase the resilience of the tested IDS.

ASRTest 10.1145/3533767.3534391     The paper proposes a testing approach for deep neural network-driven automatic speech recognition (ASR) systems called ASRTest, which is built on the theory of metamorphic testing. The approach involves designing metamorphic relations for ASR systems and implementing transformation operators to simulate practical scenarios that generate speeches. Gini impurity is used to guide the generation process and improve testing efficiency.

Hybrid Repair 10.1145/3533767.3534408     The paper proposes HybridRepair, an annotation-efficient solution for repairing deep learning models with a mismatch between training data and field data distribution. The solution takes a holistic approach that combines a small amount of annotated data with a large amount of unlabeled data, leveraging semi-supervised learning techniques to boost training data density. The proposed method selectively annotates data in failure regions and propagates labels to neighboring data while using a selected part of the unlabeled data to avoid the impact of distribution shift on SSL solutions.

BET 10.1145/3533767.3534386     The paper proposes a Black-box Efficient Testing (BET) method for identifying defects in CNNs without requiring full knowledge of the target model. The BET method generates continuous perturbations in a black-box manner and uses a tunable objective function to guide the testing process. The method also employs an efficiency-centric policy to find error-inducing inputs within a fixed query budget

Security for distributed deep neural networks: Towards data confidentiality & intellectual property protection secrypt19     This paper proposes an approach for protecting distributed DNN based software assets, including the confidentiality of input and output data streams and safeguarding intellectual property, using fully homomorphic encryption (FHE). The authors evaluate the feasibility of this approach on a convolutional neural network (CNN) for image classification deployed.

Interpreting Deep Learning-based Vulnerability Detector Predictions Based on Heuristic Searching 10.1145/3429444     The article proposes a new explanation framework to identify the small number of tokens that contribute significantly to the detector's prediction, which can help domain experts understand and accept the detector's outputs. The experiments show that this framework has higher fidelity than existing methods, especially in real-world scenarios where features are not independent.

LEMNA 10.1145/3243734.3243792     The paper proposes a high-fidelity explanation method called LEMNA for security applications, which generates a small set of interpretable features to explain how the input sample is classified. LEMNA approximates a local area of the deep learning decision boundary using a simple interpretable model that handles feature dependency and nonlinear local boundaries to improve explanation fidelity.

HUDD 9439863     Heatmap-based unsupervised debugging of DNNs (HUDD) uses heatmaps and clustering to identify root causes of DNN errors automatically and supports retraining with related images. Evaluation in the automotive domain showed HUDD effectively identified root causes and improved DNN accuracy.

DeepRepair 9508369     To resolve errors in deep neural networks caused by a mismatch with real-world conditions after deployment, the proposed solution is style-driven data augmentation. DeepRepair, using style transfer, incorporates unknown error patterns into the training data. It successfully repairs convolutional and recurrent neural networks while maintaining better accuracy on clean datasets.

PLUM 9623049     To address inaccuracies in DL models, Plum is introduced as a novel hyperheuristic approach. Plum generates a set of DL model candidates using low-level repair strategies, evaluates them based on their fixing effects, and prioritizes the best strategy to output a fixed DL model.

TestRNN 9451178     TestRNN is a coverage-guided testing approach for recurrent neural networks (RNNs), specifically focusing on long short-term memory networks (LSTMs). The method employs three test metrics and a genetic algorithm to detect defects by exploring the internal behavior of LSTMs. testRNN demonstrates superior performance in capturing temporal semantics and handling input perturbation, providing meaningful and interpretable testing results..

NMTSloth 10.1145/3540250.3549102      NMTSloth is a technique that exploits a fundamental property in Neural Machine Translation (NMT) systems—computation efficiency is determined by output length. By making subtle changes to input sentences, NMTSloth significantly delays the generation of end-of-sentence tokens, leading to increased response latency and energy consumption in NMT systems. The technique shows substantial impacts on real-world mobile device battery power.

RULER 10.1145/3540250.3549169      RULER is a new model repair technique, by discriminating sensitive and non-sensitive attributes during test case generation for model repair. The generated cases are then used in training to improve DNN fairness. RULER balances the trade-off between accuracy and fairness by decomposing the training procedure into two phases and introducing a novel iterative adversarial training method for fairness

Fair-SMOTE 10.1145/3468264.3468537     The Fair-SMOTE algorithm is introduced to remove biased labels and balance internal data distributions based on sensitive attributes, effectively reducing bias without sacrificing accuracy.

DeepJanus 10.1145/3368089.3409730      DeepJanus assesses the quality of a deep learning system by examining its behavior frontier, i.e., distinct pairs of similar inputs that trigger different responses. Experimental results show that the frontier inputs provide developers with both quantitative and qualitative insights, helping to assess the quality of the DL system and identify problematic inputs.

CoEvA2 10.1145/3368089.3409739      CoEvA2 is a search-based method that generates valid opponent examples that satisfy domain constraints. CoEvA2 uses multi-objective search to simultaneously handle the constraints, execute the attack, and maximize the amount of overdraft required. CoEvA2 generates thousands of valid adversary examples by improving the system based on the examples produced, increasing robustness and making their attack fail.

Reducing DNN labelling cost using surprise adequacy: an industrial case study for autonomous driving  10.1145/3368089.3417065      The paper proposes a technique to reduce manual labeling cost during the development of a DNN-based object segmentation development by using Surprise Adequacy (SA) to predict model performance for unlabeled inputs. Engineers can trade off cost savings against acceptable inaccuracy levels in different development phases and scenarios.

DeepTest 10.1145/3180155.3180220      DeepTest is a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes. First, automatically generated test cases leveraging real-world changes in driving conditions like rain, fog, lighting conditions, etc. DeepTest systematically explores different parts of the DNN logic by generating test inputs that maximize the number of activated neurons. DNN accuracy can be improved by retraining with the synthetic data generated by DeepTest.

Aeqitas 10.1145/3238147.3238165      Aeqitas is an automated approach for validating fairness in machine-learning models used for decision-making in sensitive domains. Aeqitas identifies discriminatory inputs revealing fairness violations using probabilistic search. Leveraging the robustness of common models, it generates scalable tests that, when added to the training set, systematically improve fairness.

Adversarial Discrimination Finder 10.1145/3377811.3380331      Adversarial Discrimination Finder is a gradient-based algorithm for generating individual discriminatory instances in DNN. It has two parts: global generation and local generation. Gradients guide the crafting of discriminatory instances in both stages. The individual discriminatory instances generated by ADF are useful to improve the fairness of the DNN through retraining

DeepRoad 10.1145/3238147.3238187      DeepRoad is an unsupervised framework for testing DNN-based autonomous driving systems. Using generative adversarial networks (GANs), DeepRoad creates scenes with different weather conditions. The framework employs metamorphic testing to assess system consistency using synthetic images and validates input images by measuring their distance from training images using VGGNet features. Experimental results on three DNN-based autonomous driving systems demonstrate DeepRoad's ability to detect many inconsistent behaviors and improve system robustness through effective input validation.

ReMoS 10.1145/3510003.3510191      ReMoS is a relevant model slicing technique to reduce defect inheritance during transfer learning while retaining useful knowledge from the teacher model. Specifically, ReMoS computes a model slice (a subset of model weights) that is relevant to the student task based on the neuron coverage information obtained by profiling the teacher model on the student task. ReMoS can reduce inherited defects effectively and efficiently with minimal sacrifice of accuracy.

SafeCompress 10.1145/3551349.3556906      SafeCompress is a framework for test-driven sparse training. SafeCompress can automatically compress a large model into a small one, following the dynamic sparse training paradigm. Moreover, considering a representative attack, namely the membership inference attack (MIA), they developed a concrete mechanism for safe model compression called MIA-SafeCompress.

Fair Class Balancing 10.1145/3340531.3411980      The paper proposes a fair class balancing method that allows enhanced model fairness without using any information about sensitive attributes. They show that the method can achieve accurate prediction accuracy while concurrently improving fairness.

Clarify 10.1145/3447548.3467177      The paper introduces Amazon SageMaker Clarify, an explainability feature as part of Amazon SageMaker. It addresses the challenge of understanding the predictions and potential biases of machine learning models. Clarify offers insights into data and models by identifying biases and explaining predictions. The feature supports bias detection and computes feature importance throughout the ML lifecycle—during data preparation, model evaluation, and post-deployment monitoring.

3Our RQ$_3$ finally points out that the currently available support is limited and, indeed, most of the software engineering challenges identified are still open and worth further investigation. In addition, the currently available approaches do not often perform a comprehensive optimization of non-functional requirements; as such, practitioners would be required to deploy a complex selection of tools to optimize relevant requirements, not being able to rely on multi-objective approaches that may find trade-offs among desired design aspects. Last but not least, our research let emerge the presence of non-trivial cause-effect relations among non-functional requirements, which should be further explored to drive the definition of novel multi-objective optimization approaches.

§ DISCUSSION AND IMPLICATIONS

The results coming from our study provide a number of additional discussion points and implications that we further discuss in the following.

On the Relation between the acquired knowledge and the previous one. Our results allowed us to discover multiple aspects concerned with non-functional requirements of machine learning-enabled systems that extend the existing knowledge. The outcome of RQ$_1$ provided a comprehensive classification of non-functional requirements along with their classes. In this respect, our work extended the insights provided by Habibullah et al.  <cit.>, providing a larger overview of the issues that practitioners should consider while developing machine learning-enabled systems. Specifically, we noticed three major differences. First, Habibullah et al. <cit.> did not include `Sustainability' as a category of non-functional requirements but, rather, they limited the categorization to aspects connected to ethics and bias. Hence, our work extends the classification by contextualizing the aspects considered by Habibullah et al. <cit.> within the broader context of sustainability, explicitly referring to three dimensions such as environmental, economic, and social sustainability. Secondly, our systematic analysis led to the identification of four additional non-functional requirements connected to `Resiliency': robustness, reliability, behavioral, and flexibility. In this sense, we were able to enlarge the knowledge of the non-functional requirements of ML-enabled systems, building on top of the existing classification schemas to further enrich them with additional characteristics and attributes that should be optimized. Finally, we also noticed a substantial difference in the treatment of the category `Maintainability'. Habibullah et al. <cit.> combined the traditional concept of maintainability, i.e., the ability of a system to be modified, improved, and adapted, with the concept of usability, i.e., how effectively a user can learn and use the system. Based on our systematic work, we found out that the concepts would be better split into two separate categories. The rationale is twofold. On the one hand, maintainability and usability can be logically considered as two different non-functional requirements which pertain to distinct desirable attributes of ML-enabled systems. On the other hand, the optimization of these two attributes would require different methods and strategies and, for this reason, further research would benefit from a clear distinction between the two. The additional knowledge gathered by our work may serve as input to both researchers and practitioners. The former might take our review as a means to design empirical experimentation and approaches able to deal with the non-functional aspects that previous research did not consider. The former might instead take our classification as an input to increase their own awareness with respect to the problems arising when developing machine learning-enabled systems, other than reason on how to address those aspects in real-world use cases.

1Our classification of non-functional requirements may inspire further research on the matter, other than making practitioners aware of the multiple, multi-faceted concerns arising when developing machine learning-enabled software systems.

At the same time, it is also worth commenting on how the findings coming from RQ$_2$ and RQ$_3$ inform future research. Our research identified a set of over 20 software engineering challenges that should be further considered by our community. When comparing our findings with those reported by Horkoff <cit.>, we may first observe that the level of granularity of the challenges reported is, in general, different. Indeed, Horkoff <cit.> discussed a set of conceptual issues that may threaten the researchers' capabilities of effectively dealing with non-functional requirements of machine learning-enabled systems. For instance, the author reported that the understanding of the research community with respect to non-functional requirements of machine learning-enabled systems is fragmented and incomplete, which includes the lack of definitions of the key attributes affecting the way machine learning-enabled systems operate. On the one hand, our work was able to address some of the conceptual concerns raised by Horkoff <cit.>, offering a theoretical framework describing the definitions of non-functional requirements, how they have been explored so far, and what are the potential inter-relations among them. On the other hand, the set of challenges identified may be seen as a more concrete instantiation of the conceptual concerns reported by Horkoff <cit.>. Indeed, we were able to describe the specific challenges that the research community is called to address for each non-functional requirement, hence providing researchers with an actionable instrument to design further studies and investigations into the matter. In addition to that, it is also worth reporting that we could also corroborate one of the challenges identified by Horkoff <cit.>, namely the need for engineering machine learning-enabled systems for reducing internal errors and security. So, in conclusion, we could elaborate a larger, comprehensive catalog of challenges that have to do with multiple aspects of SE4AI and that naturally inform a number of software engineering research communities, from the one on requirements engineering till the one on software testing. As such, our work can be considered as a cross-cutting knowledge base for serving further research in software engineering. The outcomes of RQ$_3$ reinforce our conclusion: as many non-functional aspects were not still considered or partially taken into account by existing automated approaches, additional research would be worthy. Our overview of both challenges and automated approaches provides a ground for the development of additional software engineering instruments.

2Our work could identify a number of software engineering challenges that are not yet targeted by automated approaches, hence suggesting the future avenues that researchers can consider to further enlarge the knowledge and support provided to handle non-functional requirements of machine learning-enabled systems.

As a last point of discussion, we comment on how we envision our results have influence on the state of the practice. The outcome of RQ$_1$ may possibly be exploited to support the definition of standards and guidelines that attempt to describe how to deal with non-functional requirements of machine learning-enabled systems. For example, the definitions provided might be potentially useful to support and complement an existing, emerging standard such as the ISO/IEC 25059,[The ISO/IEC 25059 standard: <https://iso25000.com/index.php/en/iso-25000-standards/iso-25059>] which defines an initial taxonomy of non-functional properties relevant for machine learning-enabled systems. In this respect, we foresee improved implementations of the standard, which takes our findings into account, hence enlarging the definitions available for researchers and practitioners. Furthermore, when comparing our findings against the research that explored the current state of the industrial practice, we see some interesting complementarities. In particular, Amershi et al. <cit.> conducted a case study in Microsoft where they let emerge a catalog of best practices to make software engineering actionable for the development of machine learning-enabled systems. Among these practices, the authors pointed out the need for model debugging and interpretability and for model evolution, evaluation, and deployment. In the first place, the outcome of RQ$_1$ has the potential to make the best practices identified by Amershi et al. <cit.> more tangible: indeed, our findings provide a set of factors that strictly relate to the model debugging, interpretability, evolution, evaluation, and deployment which are potentially measurable and, therefore monitorable throughout the different development stages of machine learning-enabled systems. As an example, the retrainability attribute pertaining to the `Maintainability' cluster may impact the evolution and deployment of machine learning-enabled systems. Our systematic synthesis provides a theoretical foundation to build novel measurement and monitoring systems that may be employed to produce higher-quality machine learning-enabled systems. In addition, the findings of RQ$_2$ provide insights into the research gaps making the best practices identified by Amershi et al. <cit.> difficult to actually implement. Also in this case, our systematic synthesis may therefore provide researchers with an improved understanding of the next steps to pursue to better support the development and evolution of machine learning-enabled systems. Similarly, the automated approaches coming from RQ$_3$ may complement the insights by Amershi et al. <cit.>. On the one hand, our findings offer a comprehensive list of automated approaches that have been proposed in the literature to ease the application of the best practices, along with their potential limitations: the characteristics of these approaches might be assessed by practitioners to understand the limits and potentials of the currently available support, hence evaluating whether these might be actually transferred to industry. On the other hand, our results emphasize the gap between research and practice, indicating that only a few approaches may support practitioners: in this sense, our work may serve as a call to action for researchers.

3Our findings complement the current state of the practice in different manners. First, the outcome of RQ$_1$ may be useful to extend and/or complement emerging standards describing relevant non-functional requirements of machine learning-enabled systems. Second, the results of RQ$_1$, RQ$_2$, and RQ$_3$ can be combined to the pieces of information emerged from the state of the practice, highlighting the current research gaps that should be filled and the potential opportunities of technological transfer.

On the Inter-Relation among Non-Functional Requirements. According to our results, and specifically, those coming from RQ$_2$, there exist some inter-relations among non-functional requirements. These are not only concerned with the relation that accuracy has with other requirements but also with the innate interconnections between multiple non-functional aspects playing a role in the development of machine learning-enabled systems. In this respect, researchers in the area of requirements engineering and empirical software engineering might cooperate toward the development of novel taxonomies that may map the relations among the non-functional requirements of machine learning-enabled systems and how they impact each other. Our results also called for further research on the identification, management, and assessment of the trade-offs among multiple non-functional requirements. The results of RQ$_3$ confirmed the presence of cause-effect relations and indicated that the multi-objective optimization of non-functional requirements represents a growing trend that should be further explored. In this respect, the set of challenges identified in our work could provide insights into the next research avenues to pursue, pinpointing the multi-faceted engineering challenges to face and the non-functional requirements that should be further explored and analyzed by means of requirements engineering, software analytics, and empirical software engineering. At the same time, the results of RQ$_3$ further inform researchers and practitioners. On the one hand, we could point out that the available automated approaches could only satisfy subsets of non-functional requirements, hence not being comprehensive enough to properly support the holistic development of machine learning-enabled systems. On the other hand, practitioners could be informed of the support available, possibly highlighting opportunities for technological transfer. In any case, our work represents a call for a multi-objective, comprehensive investigation into the non-functional attributes of machine learning-enabled systems and their interconnections.

4Our findings suggest novel interconnections and research on the multi-objective optimization of non-functional requirements of machine learning-enabled systems. We call for a brand new research field focusing on empirically investigating the relations between non-functional requirements and how those relations may inform the development of automated approaches to optimize machine learning-enabled systems.

A Managerial Viewpoint. As a follow-up discussion, the research questions targeted by our work raise multiple trade-offs to consider when developing machine learning-enabled systems. As a matter of fact, the accuracy of artificial intelligence algorithms must necessarily be balanced with other aspects to create trustworthy software systems. On the one hand, this clearly represents a call for further research. On the other hand, our work may have implications on the managerial side: the constant and continuous search for trade-offs indeed represents a high-level challenges for project managers, who are required to monitor and handle multiple non-functional attributes throughout the evolution of machine learning-enabled systems. In the first place, the results produced by RQ$_1$ and RQ$_3$ may allow the reader to understand what is the set of non-functional attributes to actually take into account and the approaches currently available to optimize them, respectively. Secondly, the challenges identified in the context of RQ$_2$ should not only be considered from a technical perspective but also from a socio-technical and managerial one. Our work indeed calls for further research on the managerial strategies that would indicate the most appropriate management policies to apply when handling non-functional requirements. Also, we bring to the attention of software engineering and software project management researchers the lack of optimization approaches that may be contextual, dependent, and adaptable to the evolution of the system—we argue that those properties would be key to enabling the proper management of non-functional requirements over the software lifecycle.

5We call for research on managerial and socio-technical strategies to handle non-functional requirements throughout the evolution of machine learning-enabled systems. At the same time, our findings point out the need for optimization approaches specifically tailored to software evolution, hence being contextual, dependent, and adaptable.

Machine Learning Sustainability: Software Engineering Comes to Rescue. The results of both RQ$_2$ and RQ$_3$ identified sustainability as a non-functional requirement that has been somehow neglected by our research community. While previous work has started exploring the social side of sustainability, we could identify a lack of software engineering investigations into energy- and cost-related concerns. On the one hand, our work identified tiny machine learning <cit.> as a promising solution to some of the key challenges affecting the energy consumption of machine learning-enabled systems: as such, we call for additional investigations into these aspects. On the other hand, we also highlight a lack of experimentation in terms of costs: in this respect, further analysis of how to predict and manage costs, other than how to optimize non-functional approaches by taking those aspects into consideration, would be worthwhile.

5Sustainability concerns should be more carefully considered by the software engineering research community. There is a need for approaches to deal with energy consumption other than cost-aware non-functional requirements optimization strategies.

On the Currently Available Automated Support. Both RQ$_2$ and RQ$_3$ pointed out the limited amount of automated approaches, other than their low generalizability. We could indeed realize the limited support provided by researchers in terms of the analysis of specific contexts, architectures, and algorithms. We also elaborated on the need for novel approaches that could simultaneously address multiple non-functional requirements: in this sense, the observations provided in the context of RQ$_3$ might serve as a basis for further research on the matter. At the same time, we elicited additional needs in terms of automated support, especially related to software analytics instruments that could support practitioners while identifying, managing, and assessing the impact of non-functional requirements on both development processes and products. Hence, it is our hope that the findings reported may represent a call for researchers working in the field of automated software engineering to propose novel techniques and solutions able to support practitioners in a broader manner and in a way that faces the specific challenges pertaining to the management of non-functional requirements.

6Our findings clearly point out the limited automated support that practitioners have, other than the limited generalizability of the available solutions. Our work represents a call for software analytics and automated software engineering research efforts aiming at facing the specific challenges identified and providing novel, generalizable approaches to handle and optimize non-functional requirements of machine learning systems in practice.

§ THREATS TO VALIDITY

The validity of both the research method and conclusions drawn in our systematic literature review might have been threatened by some key considerations. This section overviews the potential limitations of our study and how these were mitigated when designing the study.

Literature selection. A critical challenge for a systematic literature review is identifying a consistent and comprehensive body of knowledge concerning the subject of interest. In this respect, we first approached the search process by adhering to well-established guidelines <cit.>. Nonetheless, we realized that these guidelines would not have been sufficient to extract the required body of knowledge and, as a consequence, might have led to missing key resources. To mitigate this threat, we, therefore, opted for a hybrid systematic analysis <cit.>: we complemented the set of primary studies identified through the guidelines by Kitchenham et al. <cit.> with additional resources coming from (1) the systematic screening of the research articles published in top-tier software engineering and artificial intelligence venues <cit.>; and (2) multiple snowballing iterations of the incoming and outcoming references of the primary studies identified <cit.>. In this respect, it is worth further discussing the seed search procedure conducted on artificial intelligence venues. As explained in Section <ref>, we had to limit ourselves to the analysis of a subset of all the top-tier conferences and journals in the field of artificial intelligence to make systematic scanning sustainable and feasible. In doing so, we solely considered the venues having a higher likelihood to include engineering or empirical pieces of work that might have been relevant for addressing our research questions: among 35,155 papers published within the selected venues between 2012 and 2022, we only found 12 potentially relevant articles, which reduced to seven after the quality assessment stage. We are aware that an extensive analysis of the artificial intelligence venues might have identified some additional relevant primary studies: this remains a limitation of our work that the reader must be aware of. However, our analyses revealed that the amount of papers published in artificial intelligence venues that describe or optimize non-functional requirements is somehow limited (only 0.02% of the papers scanned were finally included), possibly not justifying the effort that would have been required. In this sense, alternative research instruments, e.g., qualitative surveys or interviews with machine learning engineers, might be more useful to complement our findings and identify additional concerns or approaches used to optimize non-functional requirements.

In the second place, it is worth remarking that we performed the search on multiple databases such as ACM Digital Library, Scopus, and IEEEXplore. This was done to ensure wider coverage of the primary studies published in the literature.

The reliance on a hybrid systematic literature review, other than the multiple actions conducted to extend the search process, makes us confident of the completeness of the literature selection. However, for the sake of verifiability and replicability, our online appendix <cit.> contains all the data and material used to produce each intermediate search of our study. The interested reader might use the material to either assess the soundness/completeness of the process and further build on top of our results <cit.>.

Literature analysis and synthesis. Upon completing the search process, we applied several steps to ensure the inclusion of the relevant primary studies to address the research questions driving our systematic literature review. In the first place, we designed a set of exclusion criteria with the intent of filtering out the primary studies that did not fit our scope. In addition, we proceeded with the definition of inclusion criteria, other than a formal quality assessment of the suitability of each identified resource for the goals of our study.

These steps were conducted manually; therefore, the risk of subjectiveness and human error were the main limitations. While the first author mainly conducted all the activities, there are two considerations to make. In the first place, we designed the systematic literature review so that validation was conducted at the end of each step. The validation first involved the second author of the paper, who took the role of inspector: he was indeed called to verify the actions conducted by the first author and identify potential errors. At the end of this process, both authors opened a discussion and addressed the concerns raised in the validation.

Secondly, the two authors constantly worked together to define the research method and actions to make the study sound and reliable. In particular, they met weekly to discuss the advances of the systematic literature review and the potential limitations to address. These weekly meetings further limited potential concerns due to subjectiveness and human bias. Nonetheless, we acknowledge that the entire process has been conducted based on the knowledge and expertise of the authors, which might be limited. On the one hand, we released all the material produced to be as transparent as possible <cit.>. On the other hand, we can only report some background information - this may be useful to assess the study's validity and estimate the resources required to replicate our work. The two authors have a research experience of one and ten years, respectively. Both conduct or have already conducted quantitative and qualitative studies in the past, other than systematic literature/mapping studies on themes connected to software engineering for artificial intelligence, optimization of non-functional requirements, and software maintenance and evolution. Furthermore, they are both involved in the academic courses of Software Engineering, Fundamentals of Artificial Intelligence, and Software Engineering for Artificial Intelligence at the University of Salerno (Italy) - the second author is the lecturer of those courses, while the first is a teaching assistant.

The data analysis of the primary studies allowed us to address the research questions of the study, providing a picture of the current state of the art, synthesizing the research efforts conducted so far, identifying the areas that are still neglected or unexplored, and pointing out the next research directions that the software engineering research community should pursue to better assist practitioners in their activities. In doing so, we could analyze a bit more than two primary studies for each non-functional requirement, on average (69 articles concerned with 30 non-functional requirements). While this might indicate the limited generalizability of our findings, we do not see this point as a real limitation: our systematic review has indeed an intrinsic value which is independent of the number of primary resources finally considered. We are aware that future research may classify additional non-functional requirements or identify further challenges which are not documented in this paper: our ultimate goal is exactly that of stimulating more research on the matter by providing a foundational basis describing the current body of knowledge and the limitations that should be addressed in the future.

§ CONCLUSION

In this paper, we conducted a hybrid systematic literature review on non-functional requirements of machine learning-enabled systems—this was motivated by the increasing number of papers published on these aspects during the last ten years. We tackled three main research angles such as (1) the classification of the non-functional requirements investigated so far, (2) the challenges to face when dealing with them, and (3) the automated approaches proposed in literature to support practitioners when optimizing them in practice. The results of the systematic literature review do not only summarized the current knowledge on the matter, but also opened new horizons and challenges that our research community should consider. It is our hope that researchers and fresh Ph.D. Students might be inspired by our work and, further contributing to the field. All in all, our work brings the following major contributions:

* A systematic literature review on non-functional attributes of machine learning-intensive systems, which allowed to systematically classify them, other than identify the current open challenges and approaches to cope with them. These pieces of information may be exploited by researchers to define the next research steps to improve the currently available techniques and overall support provided to practitioners;

* A set of implications and take-away messages that researchers may use to address future research avenues on the management and optimization of non-functional requirements of machine learning-enabled systems;

* An online appendix containing all data and scripts used in the study, which might be used to extend the scope of the study or replicate our work.

The considerations and implications of this systematic literature review drive our future research agenda. We first aim at deepening our analysis of how the research community has investigated non-functional requirements so far, possibly looking for scientific papers that, despite not explicitly targeting non-functional requirements, tangentially improve non-functional properties of machine learning-enabled systems. We will also work toward the definition of novel approaches that could optimize the non-functional requirements of machine learning-enabled systems while keeping accuracy into account. In addition, we aim to design software analytics studies, surveys and semi-structured interviews with machine learning engineers to further understand the impact of non-functional requirements and the practitioners' expectations with respect to them.

§ CREDITS

Vincenzo De Martino: Formal analysis, Investigation, Data Curation, Validation, Writing - Original Draft, Visualization.
Fabio Palomba: Supervision, Validation, Writing - Review & Editing.

§ CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.

§ DATA AVAILABILITY

The data collected in the context of this systematic literature review, along with the scripts used to analyze and generate data, charts, and plots discussed when addressing our research goals, are publicly available at:  <cit.>

§ ACKNOWLEDGEMENT

The authors would like to thank the Associate Editor and Reviewers of TOSEM for their constructive and valuable inputs, which substantially improved the quality of this work. This work has been partially supported by the Qual-AI and FRINGE national research projects, which have been funded by the MUR under the PRIN 2022 and PRIN 2022 PNRR programs (Codes: D53D23008570006 and D53D23017340001, respectively).