# Observation of Nonclassical Photon Statistics in Single-Bubble
Sonoluminescence

Mohammadreza Rezaee  Department of Physics, University of Ottawa, 25
Templeton, Ottawa, Ontario, K1N 6N5 Canada  Yingwen Zhang  Department of
Physics, University of Ottawa, 25 Templeton, Ottawa, Ontario, K1N 6N5 Canada
National Research Council, 100 Sussex Dr, Ottawa, ON K1A 0R6, Canada  James L.
Harden  Department of Physics, University of Ottawa, 25 Templeton, Ottawa,
Ontario, K1N 6N5 Canada  Ebrahim Karimi  Department of Physics, University of
Ottawa, 25 Templeton, Ottawa, Ontario, K1N 6N5 Canada  National Research
Council, 100 Sussex Dr, Ottawa, ON K1A 0R6, Canada

A cavitation bubble inside a liquid, under a specific set of conditions, can
get trapped in an antinode of the ultrasonically driven standing wave and
periodically emits visible photons gaitan:90 ; gaitan1990frontiers . This
conversion of sound to light phenomenon, known as sonoluminescence, can be
seen with unaided eyes and occurs in multi- or single-bubble regimes. The
sonoluminescence radiation spectrum analysis attributes a temperature of about
6,000 Kelvin to the bubble flint:91 ; brenner:02 ; xu:10  – close to the sun’s
surface temperature. The bubble dynamics, thermodynamics, and other
physicochemical properties are well-explored and studied over the past decades
walton:84 ; prosperetti:88 ; putterman:00 . Notwithstanding that several
theories such as the thermal (black-body radiation) model jarman:60 ; finch:63
; roy:94 , shock wave theory wu:94 , phase transition radiation
tatartchenko:17a ; tatartchenko:17b , and vacuum quantum electrodynamics
schwinger:92 ; eberlein:96  – were proposed, the underlying physics of
sonoluminescence has hitherto remained a puzzle. Here, we experimentally
investigated the photon number statistics of the emitted photons from single-
bubble sonoluminescence (SBSL), employing two distinctly different techniques,
i.e., measuring multiphoton correlations and photon number (using a photon-
number resolving detector). Our findings show that the emitted photons from
SBSL possess sub-Poissonian statistics, indicating the nonclassical nature of
the SBSL. Our observation of sub-Poissonian statistics of emitted photons may
help to explain the physics of SBSL. In addition, considering the importance
of nonclassical light sources in quantum technologies, this discovery would be
an exciting milestone in paving the route towards a bright quantum photonics
source.

The dynamics and oscillation of an ultrasonically driven, micron-sized single
spherical bubble in a liquid medium kept in thermal equilibrium and ambient
pressure conditions can be explained by the Rayleigh-Plesset equation
rayleigh:17 ; plesset:48 . Other formalisms have been proposed to describe
bubble dynamics in different regimes prosperetti:88 ; flynn:75 ; keller:80 .
In particular, the physics governing the bubble dynamics changes dramatically
when the amplitude of the acoustic driving wave reaches a certain threshold.
Above this threshold, a bubble trapped in an antinode of a periodically driven
standing wave at ultrasound frequency undergoes an abrupt contraction,
resulting in extreme temperature and pressure rise and simultaneous light
emission. This nonlinear regime, defined by the sound wave’s pressure
amplitude and the gas concentration, is different for single-bubble
sonoluminescence gaitan:92  (SBSL) and multi-bubble sonoluminescence
marinesco:33  (MBSL) – in this work, we only investigated the SBSL. For SBSL,
the radius of a stably trapped air bubble in degassed water, when it emits
light, oscillates between 5 and 50 microns, a three order of magnitude change
in the bubble volume barber:92 . When the bubble periodically collapses to its
minimum radius, at the resonant frequency ($\approx 30$ kHz), a bright light
flash is emitted, containing about $10^{6}$ photons brenner:02  that propagate
in all directions ($4\pi$-steradian). Interestingly, although the time between
SBSL triggering states is several tens of microseconds, the emitted pulse
duration is in the range of $40-350$ picoseconds depending on the gas
concentration and the driving pressure amplitude gompf:97 ; hiller:98 ;
pecha:98 . The spectrum of SBSL light has been recorded and analysed for
different bubble regimes. Viewed as black-body radiation, the SBSL emission
suggests that the temperature inside the bubble is close to 6,000 K.
Notwithstanding existing theories, the physics of phonon-to-photon conversion
at the heart of the sonoluminescence effect has remained a puzzle for decades
maddox:93 . These theoretical models are either based on classical
electrodynamics of light generated during bubble collapse (e.g., due to
thermally-induced radiation vuong1999shock ; flannigan2005plasma ,
Bremsstrahlung radiation moss1997understanding ; frommhold1998electron  or
radiation from recombination of electrons and ions walton1984sonoluminescence
), or on quantum electrodynamics (e.g., radiation due to the quantum nature of
the vacuum state schwinger1994casimir ; eberlein1996theory ; milton1997casimir
; brevik:99 ; musha2011another ). In this study, we employ quantum optics and
novel instrument approaches in the hope to elucidate the underlying nature of
the light emitted from the collapsing bubble trapped in an acoustic standing
wave. We measured and analysed the photon statistics of SBSL using two
different experimental approaches: (1) an intensified time-stamping camera to
measure high-order temporal correlations among photons emitted from SBSL, and
(2) a photon number resolving camera to count the SBSL photons, and
consequently their statistics, directly. Both approaches show sub-Poissonian
photon statistics indicating the underlying mechanism of SBSL to be
nonclassical in nature. Enthralling, measurements based on the second-order
correlation function (Hanbury-Brown and Twiss) have been proposed to determine
the bubble’s physical features trentalange:96 , e.g., the light-emitting
region size, two-photon correlation measurements as a test for thermality
belgiorno:00  and the pulse width gompf:97 , but not to experimentally explore
the fundamental nature of SBSL.

Figure 1: Experimental setup to measure multiphoton correlation functions from
single-bubble sonoluminescence. (a) Schematic of the experimental setup to
generate and image single-bubble sonoluminescence onto a time-tagging camera
(TPX3CAM) that is made single- photon sensitive with an attached image
intensifier (Photonis Cricket with Hi-QE Red photocathode). Six mirrors are
used to reflect the emitted photons from the single-bubble, which propagate in
different directions onto different regions of the time-tagging camera. (b)
shows the single-bubble sonoluminescence light that is visible in the round
flask. Photo is taken by a monochrome industrial camera with CMOS Pregius
sensor using a zoom lens (8 mm EFL, f/1.4). (c) The raw image was taken of
photons from the SBSL detected by the camera from seven different directions,
($s_{1},s_{2},s_{3},s_{4},s_{5},s_{6},s_{7}$), including direct imaging of the
source spot at $s_{3}$, with an exposure time of 22 s. The propagation length
for all directions, excluding the central $s_{3}$, is about 300 mm. (d)
Schematic of the experimental setup for photon number statistics measurement.
(e) Zoomed-in image of the SBSL taken using a commercial camera. (f) The raw
image of sonoluminescence setup with SBSL visible in the middle of the flask
taken by the ORCA-Quest camera with an exposure time of 10 s (See Methods for
details of both experiments).

Figure 1 shows the schematic of our experimental setups. Inset (a) is the
schematic of the experiments performed using a photon arrival time-tagging
camera detector (TPX3CAM) nomerotski:19  and (b) is the photon number
resolving experiment performed using the ORCA-Quest camera from Hamamatsu. A
single air bubble is trapped in degassed water using a standing ultrasonic
wave created by a single piezoelectric transducer. The piezo element is
attached to the bottom of a $100$ ml spherical quartz flask. See section S1 of
Supplementary Information (S.I.) for more details on the experimental setup.
The air bubble is trapped when the resonance frequency of the water-filled
flask is in tune with the driving ultrasound frequency, which is $28.06$ kHz
in this case.

To explore the photon statistics through temporal correlations between SBSL
photons, emitted photons from different directions are collected and analysed
as depicted in Fig. 1-(a). As a first test, photons collected from two
different directions, reflected from one of the mirrors ($s_{4}$) and the
source ($s_{3}$), direct imaging of the SBSL light, were coupled into two
multimode optical fibres, prior to being detected by single-photon (avalanche
photodiode) detectors, the setup is not shown. Across many runs, a second-
order correlation $g^{(2)}$ between 0.8 to 0.9 was consistently measured,
while that measured for a multimode thermal light source and a LED-light were
always consistent with 1 (which is expected as the coherence time of classical
light sources are much less than the time resolution of the detector) – see S2
of the S.I.. Henceforth, we performed a more comprehensive set of tests, where
emitted photons from seven different directions were collected and analysed
simultaneously. Collected photons are imaged onto a time-tagging camera, which
independently offers an effective 7 nanosecond time-stamping resolution on
each of its $256\times 256$ pixels. Figure 1-(c) shows a raw image of the
photons from SBSL taken by the camera accumulated over 22 s. The photons from
seven different directions are detected as seven bright spots on the camera,
denoted by $s_{i}$, with $i=1-7$. $s_{3}$ is the direct image of the SBSL
captured by the camera, while the other six spots
$(s_{1},s_{2},s_{4},s_{5},s_{6},s_{7})$, are SBSL photons reflected by mirrors
towards the camera from six different directions. The average photon number
detected per pixel for the bright SBSL spots is below 0.03 photon per pulse
(or 840 photons per second), this eliminates the possibility of oversaturating
the camera chip with more than 1 photon per pulse per pixel resulting in
inaccurate photon statistics and, at the same time, guarantees adequate photon
statistics analysis can be performed. Temporal correlations of up to $g^{(5)}$
among all seven spots were measured, histograms showing the measured $g^{(n)}$
between different spots can be seen in Fig. 2. We can see the measured
$g^{(n)}$ is consistently below $1$, and decreases with increasing $n$ from
$g^{(2)}\simeq 0.82$ down to $g^{(5)}\simeq 0.6$. Note that for classical and
coherent light sources $g^{(n)}$ is expected to always be greater than or
equal to 1. $g^{(n)}<1$ is an indicator for nonclassical light and $g^{(n)}=0$
for a perfect single-photon source. However, the relation between sub-
Poissonian statistics and anti-bunching requires further care, e.g., see ref.
zou:90 . Details on the determination of $g^{(n)}$ and measured coincidences
can be found in the sections S2–S4 of S.I.

Figure 2: Summary of all calculated $g^{(n)}$ for all possible permutations of
the seven channels (e.g., ($s_{1}$,$s_{2}$); ($s_{1}$,$s_{3}$);
($s_{1}$,$s_{3}$,$s_{4}$); etc.) for multi-fold coincidences. For instance,
the first column shows the second-order correlation functions, $g^{(2)}$, for
all the possible permutations of channels organized in $n$-fold for $n=2-5$
combinations. The total data acquisition time is 22 seconds, with a $25$ ns
gate width. Two-fold column includes all the data points for all the possible
permutations, including two channels. The same logic is valid for all other
columns. Only events within a 10-pixel radius around the centre of each bright
spot (see Fig. 1-c) were taken into account in the data analysis and a
constant background of $\simeq 4.3$ photons per second per pixel was measured
and was subtracted in all calculations.

An additional note to add is that the $g^{(n)}$ measured here is a time
average of the more general $g^{(n)}(t)$ which is expected to be at a maximum
(for classical light) or minimum (for nonclassical light) at time $t=0$ and
decay/increase to 1 at $t\geqslant t_{c}$. Here, $t_{c}$ is the coherence time
of the light source. In this experiment, the measured $g^{(n)}$ will be the
average over a SBSL pulse duration which is expected to be in the order of
tens of picoseconds. Thus, we expect an even smaller $g^{(n)}$ to be measured
if a detector with a shorter timing resolution than that of the SBSL pulse is
used.

Studying photon statistics by employing Photon Number Resolving (PNR)
detectors to quantify the statistical properties is an alternative powerful
method in quantum optics for characterizing the nature of a light source. For
instance, a perfectly coherent light source with constant intensity exhibits
Poissonian photon statistics with $g^{(n)}$ = 1, while thermal and
nonclassical light sources, on the other hand, demonstrate super-Poissonian
$g^{(n)}>1$ and sub-Poissonian $g^{(n)}<1$ photon statistics, respectively
mandel_wolf_1995 ; fox2006quantum . Although experimental measurement schemes
can add electronic noise statistics to the inherent photon statistics, this
technique can still be employed to classify different light sources based on
the origin of their photon emission process hlouvsek:19 .

Photons emitted from a coherent light source with stable intensity and mean
photon number $\left\langle n\right\rangle$ exhibits Poisson distribution,
$P(n)={\left\langle n\right\rangle^{n}}/{n!}\,\,e^{-\left\langle
n\right\rangle}$, with a photon-number variance $(\Delta n)^{2}=\left\langle
n\right\rangle$. While single mode thermal light displays Bose-Einstein
distribution with a probability that relates to mean photon number according
to $P(n)={\left\langle n\right\rangle}^{n}/{(1+\left\langle
n\right\rangle)}^{n+1}$, and the variance $(\Delta n)^{2}=\left\langle
n\right\rangle+\left\langle n\right\rangle^{2}$. However, when multimode
thermal light with $N_{m}$ modes is considered, the variance becomes $(\Delta
n)^{2}=\left\langle n\right\rangle+\left\langle n\right\rangle^{2}/N_{m}$. For
many-mode thermal light, i.e. $N_{m}\gg 1$, the photon-number variance is
approximately equal to the mean photon number, $(\Delta
n)^{2}\approx\left\langle n\right\rangle$, similar to a Poissonian
distribution. On the other hand, sub-Poissonian photon statistics,
characterized by $(\Delta n)^{2}<\left\langle n\right\rangle$, cannot be
described by classical electrodynamics theory, and is a signature of quantum
light sources. Therefore, by comparing the variance in photon number
distribution and the mean photon number, one is able to determine whether a
light source is classical or quantum in nature. Figure 3-(a) shows the photon
number statistics observed for the SBSL, measured using a PNR camera. When
compared to a fitted Poisson curve with the same average photon number, one
can clearly observe the photon distribution is sub-Poissonian. Using the
relation $g^{(2)}=1+\left((\Delta n)^{2}-\left\langle
n\right\rangle\right)/{\left\langle n\right\rangle^{2}}$ , an average
$g^{(2)}=0.87\pm 0.02$ for SBSL was obtained after performing a series of
photon counting measurements. The measured $g^{(2)}$ via photon number
resolving detector is close to the previously measured ones, presented earlier
with the time-tagging camera. Since the minimum exposure time of the camera is
$172.8\leavevmode\nobreak\ \mu$s, which is approximately 5 SBSL pulses, the
measured $g^{(2)}$ value should be considered as an upper bound limit. For
comparison, in Fig. 3-(b) the result obtained from a broadband multimode
thermal source is presented. The measured photon statistics for the multimode
thermal source is clearly Poissonian. See S.I. for more detailed experimental
outcomes and statistical analysis results. The photon statistics were measured
for several different experiments taken over several weeks. Figure 3-(c) shows
$g^{(2)}$ and $(\Delta n)^{2}/\left\langle n\right\rangle$ for seventeen
different SBSL experiments. For all of these experiments, both $g^{(2)}$ and
$(\Delta n)^{2}/\left\langle n\right\rangle$ were observed to be below $1$,
and thus confirm that the emitted photons from SBSL, unlike a thermal source,
is sub-Poissonian.

Figure 3: Experimental measurement and statistical analysis results from
photon number resolving study for SBSL and multimode thermal source. (a) shows
photon-number statistics and the fitted Poisson curve for comparison. $P(n)$
is the probability of finding $n$ photons within the subsegment corresponding
to $172.8\leavevmode\nobreak\ \mu$s detection window vs measured photon-
number. The measured photon number distribution for SBSL is clearly narrower
than the fitted Poisson curve, a clear indication of sub-Poissonian photon
number statistics for emitted photons from SBSL. (b) multimode thermal state,
photon number distribution of a broadband thermal light source whose spatial
profile is first filtered using a single-mode optical fibre. As expected,
since the detection window is much longer than the coherence time of the light
source, the second-order correlation $g^{(2)}$ is averaged to 1 corresponding
to Poissonian statistics, and is clearly seen through the nearly perfect fit
of the Poisson curve. (c) shows the measured $g^{(2)}$ and $(\Delta
n)^{2}/\left\langle n\right\rangle$ for seventeen different SBSL experiments.
Consistently, we observed a $g^{(2)}<1$ and $(\Delta n)^{2}/\left\langle
n\right\rangle<1$, when we analysed emitted photons from SBSL experiment.

In summary, measurement and characterization of photon number statistics and
multichannel time-resolved photon counting were employed to probe the nature
of emitted light from SBSL. Results from studying higher-order temporal
correlations and direct measurement of the photon number statistics show a
clear indication of sub-Poissonian photon statistics, and manifested the
quantum nature of the emitted photons. The models based on classical
electrodynamics, such as thermal radiation, cannot lead to these observed sub-
Poissonian photon statistics – photons emitted from thermal sources possess
super-Poissonian statistics. Results from the measurements suggest photons
emitted in sonoluminescence should be originating from a quantum phenomenon.
So far, the dynamic Casimir effect (DCE) is the main candidate in quantum
theory proposed to describe SBSL schwinger1994casimir ; eberlein1996theory ;
milton1997casimir ; musha2011another . However, there have been studies
showing that DCE suffers from some major flaws such as requiring the bubble to
contract faster than the speed of light lambrecht1997comment ;
unnikrishnan1996comment ; garcia1997comment ; brevik:99 . We hope that our
observation of sub-Poissonian photon statistics will shed some light on the
physics of sonoluminescence, though providing a quantum theory to explain our
observations will require further investigation and is beyond the scope of
this work. Moreover, given the observed strong correlation among generated
photons from SBSL, a controlled SBSL would be a good candidate for
deterministic quantum photonic sources that is a genuine need in quantum
science and technology.

Method
Higher-order correlation measurements: Figure 1-(a) shows the setup schematic
view for multiphoton correlation measurements of the SBSL source. SBSL photons
from seven different directions are redirected towards the time-tagging camera
(TPX3CAM) and imaged onto seven different spots on the camera. The camera has
nanosecond time-resolution for each individual pixel and is made single-photon
sensitive with the aid of an attached image intensifier (Photonis’ Cricket
with Hi-QE Red photocathode). This camera, therefore, allows us to record each
detected photon’s time and pixel coordinates, and consequently, the
spatiotemporal correlation among detected photons. The camera and intensifier
system together have a timing resolution of approximately 7 ns. Thus, a
coincidence gating window of 25 ns was used, and only photons detected within
this gating window are considered as coincidence events and used in the
correlation calculations. The image intensifier has a spectral sensitivity
range between approximately 500 nm-900 nm, and we did not use any spectral
filters in our setup. Therefore, all photons in this spectral range that are
not absorbed by the water (and flask) were detected. Due to the image
intensifier converting individual photons into a flash of light, a cluster of
pixels on the camera is often illuminated by a single photon event. Thus, each
cluster must be regrouped into a single event employing a centroiding
algorithm ianzano:20 . Also, depending on the light intensity detected on each
pixel, each event’s measured Time-of-Arrival (ToA) may differ from pixel to
pixel even if they are from the same cluster. High photon intensity will cause
a steeper signal rise time and, therefore, cross the camera’s discrimination
threshold earlier than smaller signals, giving an earlier ToA. This time-walk
effect also must be corrected in the analysis to improve the camera’s timing
resolution ianzano:20 . After addressing this issue, a timing resolution of
approximately 7 ns is achieved.

Photon Number Statistics Measurement: To reveal the underlying nature of
sonoluminescence light, we directly investigated the statistics of emitted
photons and analysed the probability distribution using a photon number
resolving (PNR) camera. For this experiment, we utilized the ORCA-Quest
quantitative CMOS (qCMOS) camera from Hamamatsu, which can distinguish the
number of arrival photons up to 200 within each pixel at
$172.8\leavevmode\nobreak\ \mu$s time resolution. We ensured the photon
intensity was far below the photon number limit in all measurements. The
camera has 9.4 Megapixels ($4.6\leavevmode\nobreak\ \mu$m$\times
4.6\leavevmode\nobreak\ \mu$m pixel size), and when running in ultra-quiet
mode, the RMS readout noise is only 0.27 electrons. The PNR camera has high
quantum efficiency, 90% at $-20^{\circ}$C around the $475$ nm region, which is
also the most intense spectral region of SBSL. Minimizing different sources of
noise that may interfere with measuring the intrinsic photon statistic in the
experiment is key to unrevealing the fundamental nature of the light source
under study. Here, we took 10,000 images with $172.8\leavevmode\nobreak\ \mu$s
exposure time and constructed a photon number distribution histogram using the
photon number recorded on each pixel. To ensure the authentic creation of
actual photon distribution statistics and validate our approach, we compared
results obtained from SBSL to that of a stabilized broadband light source with
a black-body radiation spectrum (aka thermal source). The variance and mean
photon number were measured from the photon number distribution histogram and
was compared for 17 different data sets (see S.I.). Probability density of
photon detection is plotted against the mean photon number. Considering
Mandel’s Q-parameter Mandel:79  defined as $Q=((\Delta n)^{2}-\left\langle
n\right\rangle)/{\left\langle n\right\rangle}$ and $g^{(2)}=1+Q/\left\langle
n\right\rangle$ as the figures of merit makes it clear to distinguish between
classical ($Q>0$, $g^{(2)}>1$) and nonclassical ($Q<0$, $g^{(2)}<1$) light.

Data availability. The data that support the findings of this study are
available from the corresponding author, E.K.(ekarimi@uottawa.ca), upon
reasonable request.

## References

  * [1] Dario Felipe Gaitan.  An experimental investigation of acoustic cavitation in gaseous liquids.  PhD thesis, The University of Mississippi, 1990.
  * [2] DF Gaitan, LA Crum, MF Hamilton, and DT Blackstock.  Frontiers of nonlinear acoustics.  In 12th ISNA, pages 459–463, 1990.
  * [3] Edward B Flint and Kenneth S Suslick.  The temperature of cavitation.  Science, 253(5026):1397–1399, 1991.
  * [4] Michael P Brenner, Sascha Hilgenfeldt, and Detlef Lohse.  Single-bubble sonoluminescence.  Reviews of Modern Physics, 74(2):425, 2002.
  * [5] Hangxun Xu and Kenneth S Suslick.  Molecular emission and temperature measurements from single-bubble sonoluminescence.  Physical Review Letters, 104(24):244301, 2010.
  * [6] Alan J Walton and Geo T Reynolds.  Sonoluminescence.  Advances in Physics, 33(6):595–660, 1984.
  * [7] Andrea Prosperetti, Lawrence A Crum, and Kerry W Commander.  Nonlinear bubble dynamics.  The Journal of the Acoustical Society of America, 83(2):502–514, 1988.
  * [8] Seth J Putterman and Keith R Weninger.  Sonoluminescence: How bubbles turn sound into light.  Annual Review of Fluid Mechanics, 32(1):445–476, 2000.
  * [9] Peter Jarman.  Sonoluminescence: a discussion.  The Journal of the Acoustical Society of America, 32(11):1459–1462, 1960.
  * [10] RD Finch.  Sonoluminescence.  Ultrasonics, 1(2):87–98, 1963.
  * [11] RA Roy.  Physical aspects of sonoluminescence from acoustic cavitation.  Ultrasonics Sonochemistry, 1(1):S5–S8, 1994.
  * [12] CC Wu and Paul H Roberts.  A model of sonoluminescence.  Proceedings of the Royal Society of London. Series A: Mathematical and Physical Sciences, 445(1924):323–349, 1994.
  * [13] Vitali A Tatartchenko et al.  Sonoluminescence as the peta radiation.  Optics and Photonics Journal, 7(02):27, 2017.
  * [14] Vitali A Tatartchenko.  Sonoluminescence as the peta radiation, part two.  Optics and Photonics Journal, 7(11):197–220, 2017.
  * [15] Julian Schwinger.  Casimir energy for dielectrics.  Proceedings of the National Academy of Sciences of the United States of America, 89(9):4091, 1992.
  * [16] Claudia Eberlein.  Sonoluminescence as quantum vacuum radiation.  Physical Review Letters, 76(20):3842, 1996.
  * [17] Lord Rayleigh.  Viii. on the pressure developed in a liquid during the collapse of a spherical cavity.  The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 34(200):94–98, 1917.
  * [18] MS Plesset.  Dynamics of cavitation bubbles.  J Appl Mech, 16:228–231, 1948.
  * [19] Hugh G Flynn.  Cavitation dynamics. i. a mathematical formulation.  The Journal of the Acoustical Society of America, 57(6):1379–1396, 1975.
  * [20] Joseph B Keller and Michael Miksis.  Bubble oscillations of large amplitude.  The Journal of the Acoustical Society of America, 68(2):628–633, 1980.
  * [21] D Felipe Gaitan, Lawrence A Crum, Charles C Church, and Ronald A Roy.  Sonoluminescence and bubble dynamics for a single, stable, cavitation bubble.  The Journal of the Acoustical Society of America, 91(6):3166–3183, 1992.
  * [22] N Marinesco and JJ Trillat.  Action of supersonic waves upon the photographic plate.  Proc R Acad Sci, 196:858–860, 1933.
  * [23] Bradley P Barber and Seth J Putterman.  Light scattering measurements of the repetitive supersonic implosion of a sonoluminescing bubble.  Physical Review Letters, 69(26):3839, 1992.
  * [24] B Gompf, R Günther, G Nick, R Pecha, and W Eisenmenger.  Resolving sonoluminescence pulse width with time-correlated single photon counting.  Physical Review Letters, 79(7):1405, 1997.
  * [25] Robert A Hiller, Seth J Putterman, and Keith R Weninger.  Time-resolved spectra of sonoluminescence.  Physical Review Letters, 80(5):1090, 1998.
  * [26] R Pecha, B Gompf, G Nick, ZQ Wang, and W Eisenmenger.  Resolving the sonoluminescence pulse shape with a streak camera.  Physical Review Letters, 81(3):717, 1998.
  * [27] John Maddox.  Sonoluminescence in from the dark.  Nature, 361(6411):397, 1993.
  * [28] Vi Q Vuong, Andrew J Szeri, and David A Young.  Shock formation within sonoluminescence bubbles.  Physics of Fluids, 11(1):10–17, 1999.
  * [29] David J Flannigan and Kenneth S Suslick.  Plasma formation and temperature measurement during single-bubble cavitation.  Nature, 434(7029):52–55, 2005.
  * [30] William C Moss.  Understanding the periodic driving pressure in the rayleigh–plesset equation.  The Journal of the Acoustical Society of America, 101(2):1187–1190, 1997.
  * [31] Lothar Frommhold.  Electron-atom bremsstrahlung and the sonoluminescence of rare gas bubbles.  Physical Review E, 58(2):1899, 1998.
  * [32] Alan J Walton and Geo T Reynolds.  Sonoluminescence.  Advances in Physics, 33(6):595–660, 1984.
  * [33] Julian Schwinger.  Casimir light: field pressure.  Proceedings of the National Academy of Sciences, 91(14):6473–6475, 1994.
  * [34] Claudia Eberlein.  Theory of quantum radiation observed as sonoluminescence.  Physical Review A, 53(4):2772, 1996.
  * [35] Kimball A Milton and Y Jack Ng.  Casimir energy for a spherical cavity in a dielectric: Applications to sonoluminescence.  Physical Review E, 55(4):4207, 1997.
  * [36] Iver Brevik, Valery N Marachevsky, and Kimball A Milton.  Identity of the van der waals force and the casimir effect and the irrelevance of these phenomena to sonoluminescence.  Physical Review Letters, 82(20):3948, 1999.
  * [37] Takaaki Musha et al.  Another possibility of sonoluminescence due to the cherenkov radiation from the zpf field in a water bubble.  Natural Science, 3(3):249–254, 2011.
  * [38] S Trentalange and SU Pandey.  Bose–einstein correlations and sonoluminescence.  The Journal of the Acoustical Society of America, 99(4):2439–2441, 1996.
  * [39] Francesco Belgiorno, Stefano Liberati, Matt Visser, and Dennis W Sciama.  Sonoluminescence: two-photon correlations as a test of thermality.  Physics Letters A, 271(5-6):308–313, 2000.
  * [40] Andrei Nomerotski.  Imaging and time stamping of photons with nanosecond resolution in timepix based optical cameras.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 937:26–30, 2019\.
  * [41] XT Zou and L Mandel.  Photon-antibunching and sub-poissonian photon statistics.  Physical Review A, 41(1):475, 1990.
  * [42] Leonard Mandel and Emil Wolf.  Optical Coherence and Quantum Optics.  Cambridge University Press, 1995.
  * [43] Anthony Mark Fox, Mark Fox, et al.  Quantum optics: an introduction, volume 15.  Oxford university press, 2006.
  * [44] Josef Hloušek, Michal Dudka, Ivo Straka, and Miroslav Ježek.  Accurate detection of arbitrary photon statistics.  Physical Review Letters, 123(15):153604, 2019.
  * [45] Astrid Lambrecht, Marc-Thierry Jaekel, and Serge Reynaud.  Comment on “sonoluminescence as quantum vacuum radiation”.  Physical Review Letters, 78:2267, 1997.
  * [46] CS Unnikrishnan and Shomeek Mukhopadhyay.  Comment on “sonoluminescence as quantum vacuum radiation".  Physical Review Letters, 77(22):4690, 1996.
  * [47] N Garcia and AP Levanyuk.  Comment on “sonoluminescence as quantum vacuum radiation".  Physical Review Letters, 78(11):2268, 1997.
  * [48] Christopher Ianzano, Peter Svihra, Mael Flament, Andrew Hardy, Guodong Cui, Andrei Nomerotski, and Eden Figueroa.  Fast camera spatial characterization of photonic polarization entanglement.  Scientific Reports, 10(1):6181, 2020.
  * [49] L. Mandel.  Sub-poissonian photon statistics in resonance fluorescence.  Opt. Lett., 4(7):205–207, Jul 1979.

Supplementary Information accompanies this manuscript.

Acknowledgments We like to acknowledge the help we received from the Hamamatsu
Photonics and the chance to test the ORCA-Quest qCMOS camera. E.K. would like
to thank K. Heshami for fruitful discussion and feedback to characterise the
quantum features of the SBSL, and R. Fickler and A. Abas for the first attempt
to design the sonoluminescence setup. This work was supported by Canada
Research Chairs (CRC), Canada First Research Excellence Fund (CFREF) Program,
and NRC-uOttawa Joint Centre for Extreme Quantum Photonics (JCEP) via High
Throughput and Secure Networks Challenge Program at the National Research
Council of Canada. M.R. would like to acknowledge the support of Mitacs
Accelerate Industrial Postdoctoral Fellowship.

Author Contributions E.K: Conceived the idea; M.R., Y.Z., J.H. and E.K.:
Designed the experiment; M.R. and Y.Z.: Performed the experiment and collect
the data; M.R., Y.Z. and E.K.: Analysed the data; All authors discussed the
results and contributed to the text of the manuscript.

Author Information The authors declare no competing financial interests.
Correspondence and requests for materials should be addressed to E.K.
(ekarimi@uottawa.ca).

Supplementary Information for:
Observation of Nonclassical Photon Statistics in Single-Bubble
Sonoluminescence

##  S1  Experimental setup for generating single-bubble sonoluminescence

Different methods are suggested to transfer the ultrasound energy to the
liquid to observe sonoluminescence. Various configurations which successfully
implemented so far include using an ultrasound horn immersed in the liquid,
having two piezoelectric elements glued to the opposite sides of the flask
facing each other, and using a single piezo transducer attached to the bottom
of the flask, to name a few. In this experiment, a single piezoelectric
transducer attached to the bottom of a $\leavevmode\nobreak\ 100$ ml round
bottom quartz flask ($63$ mm external diameter) using epoxy was used. Another
transducer was epoxied to the flask’s side to work as the microphone to find
the system’s resonance frequency. Experiments should be performed in a dark
lab and inside a black box to minimize any stray light. Stray lights must be
minimized since the sonoluminescence light is faint and therefore not easy for
the naked eye or the commercial camera (Canon EOS 550D) to detect. The drive
signal is generated using a waveform generator (Agilent 33220A) and amplified
in the next stage using an audio amplifier. A successful experiment’s main
steps include water preparation, signal conditioning, and finding the vessel’s
containing the liquid medium resonance condition. Although not necessary, only
water was used (according to other studies, sulfuric acid makes brighter
sonoluminescence light than degassed water) for the sake of simplicity and
safety. Water should be free of large particle contamination. In general, the
results from qualitative investigations suggest around $\leavevmode\nobreak\
5-10^{\circ}$C; sonoluminescence light is more intense. Observed
sonoluminescence light intensity is a function of temperature since the speed
of acoustic waves propagating in the water as well as the amount of dissolved
gas are temperature dependent, among other possible reasons. Figure S1 shows
the general outline of the experiment and its main components.

Figure S1: Sonoluminescence characterisation experimental setup. The 100 ml
round flask is filled with degassed water at room temperature. The sound wave
at 28.06 kHz is generated using the function generator, and the audio
amplifier intensifies the signal amplitude. The amplified signal is sent to
the solenoid and, from there, to the piezoelectric element. We can tune the
circuit resonance frequency to match that of the water-filled flask using the
movable ferrite rod core in the inductor. Monitoring the microphone signal’s
amplitude using the oscilloscope can ensure the proper resonance frequency is
applied. The time-stamping camera images sonoluminescence light reflected by
the six mirrors and the source light.

The function (frequency) generator output signal should be stable (frequency
and amplitude) around $25-30$ kHz. For the setup used in this experiment,
$28.06$ kHz turned out to be the correct resonance frequency. The signal is
amplified using an audio amplifier, and sent to an inductor formed from a wire
spool ($35$ mH) with a ferrite rod inside. This variable solenoid helps to
tune the circuit. From the inductor, the signal is connected to the
piezoelectric transducer. It is essential to find the right frequency for
trapping bubbles since there is more than one resonance frequency. At the
proper resonance frequency, the flask is very sensitive, and exerting force to
its surface causes a drastic change in the signal’s amplitude from the
microphone. Also, a hydrophone can be utilized to read the acoustic field
amplitude in the flask (probably more accurate and expensive). One can fine-
tune the frequency and adjust the inductance by moving the ferrite rod in and
out of the solenoid. When the right frequency is found, the driving signal’s
amplitude should increase from the audio amplifier. Observing the bubble’s
behaviour using a commercial camera and reading the microphone amplitude, the
amplitude should be increased until the bubble stops jittering and becomes
steady. The acoustic field’s threshold amplitude is the right amount of sound
energy to trap the bubble and see the light without losing it. The flask is
mounted on a translational stage connected to a 3-dimensional stage for fine-
tuning the light source for further quantum optics experiments.

Figure S2: Sonoluminescence light in degassed water. The small dot
approximately at the centre of the flask is the sonoluminescence light,
visible to the unaided eye in the dark. Photo is taken using the ORCA-Quest
camera 5 s exposure time. Note that during data taking, the enclosure were
sealed properly to avoid any stray light entering to the box.

Figure S2 is a picture of the sonoluminescence round flask with the glowing
bubble in the middle taken ORCA-Quest camera with 5 s exposure time. The light
appears as a small faint dot, almost located in the middle of the flask. In a
completely dark room, the light is visible with the naked eye.

##  S2 Second-order correlation function measured with single-photon avalanche
photodiodes

The initial second-order correlation function $g^{(2)}$ measurements were
performed using two single-photon avalanche photodiodes ($\tau$-SPAD-50)
between two channels ($s_{3}$-the source and $s_{4}$). The emitted photons
from the SBSL in two channels $s_{3}$ and $s_{4}$ are coupled into multimode
optical fibres utilizing two $10\times$ microscope objectives, and then
detected by two single-photon avalanche photodiodes (APD). The APDs quantum
efficiency at 550 nm is about 70% with a dark count and deadtime of $50$
counts per second and $50$ ns, respectively. A time tagging unit (Swabian
instruments “Time Tagger 20”) is used to measure the coincidences between the
two APDs. Results of this coincidence measurement showed strong time-
correlation between detected photons, a clear indication of photon pair
generation. The measured FWHM is 5 ns. The measured coincidences and $g^{(2)}$
(see S5 for the calculation of $g^{(2)}$) is shown in Table S1. This
measurement was a motivation to further explore the higher-order correlations
function by switching to a time-tagging camera.

Table S1:  Coincidences and $g^{(2)}$ measured for SBSL with multimode fibers and APDs. Background in the Singles 1 and 2 are 56 counts/s and 48 counts/s, respectively. | Set 1 | Set 2 | Set 3 | Set 4 | Set 5 | Set 6 | Set 7
---|---|---|---|---|---|---|---
Integration time (s) | 250 | 150 | 120 | 100 | 100 | 200 | 200
Singles 1 (background subtracted) | 81,041 | 54,059 | 52,713 | 46,242 | 52,491 | 107,514 | 61,378
Singles 2 (background subtracted) | 74,592 | 48,677 | 44,796 | 36,487 | 42,849 | 88,333 | 84,273
Measured Coincidences | 748 | 509 | 634 | 517 | 708 | 1538 | 768
$g^{(2)}$ | 0.87 | 0.82 | 0.91 | 0.87 | 0.89 | 0.91 | 0.84

##  S3 Higher-order correlation function measured with time-tagging camera

A time-tagging camera (TPX3CAM) with an attached image intensifier (Photonis’
Cricket with Hi-QE Red photocathode) was used for the time-correlation setup
for measuring higher orders of correlation between the SBSL photons. The
quantum efficiency of the image intensifier was measured to be approximately
5% on average in the 500-900 nm wavelength region and the camera offers the
time-stamping capability for individual pixels ($256\times 256$ pixels with
$55\times 55\leavevmode\nobreak\ \mu\,m^{2}$ pixel size) of the camera’s
active detection area. These features made this experiment feasible and
offered a new approach to characterise sonoluminescence light. Though the
camera inherently has a 1.5 ns time resolution, however, after clustering and
time-over-threshold (TOT) corrections, as discussed in the Methods section of
the main test, a timing resolution of approximately 7 ns (FWHM) is achieved.
Intensified camera integrated with the intensifier can detect and time-tag
approximately 10 M photons per second across all its pixels. Time correlations
were recorded by using six mirrors around the flask to reflect the
sonoluminescence light towards the camera, overall, seven photon spots on the
camera. Along with the source spot, nearly at the centre of the flask, this
configuration can simultaneously record the position and arrival time of
photons at seven different points (seven light detection channels). The
current setup can be employed for multichannel photon correlation
measurements. Although this camera has been designed and built with high
energy and particle physics applications such as imaging ions in mind, it
proved extremely useful in quantum optics experiments.

##  S4 Multiphoton time-correlated photon counting

In order to characterise the state of light, we measured the second-order
intensity correlation function, $g^{(2)}$, as well as higher orders up to five
to investigate the nature of the emitted light. Statistical measurements and
calculating temporal intensity correlation functions of two-or more photon
channels can reveal a lot of information regarding the nature of the field.
The second-order correlation function, also known as the degree of second-
order coherence, $g^{(2)}(\tau)$ can be given in terms of the intensity $I(t)$
as

$\displaystyle g^{2}(\tau)=\frac{\left\langle
I(t)I(t+\tau)\right\rangle}{\left\langle I(t)\right\rangle^{2}}\,$ (S1)

where $I(t)$ is also proportional to the photon number. It is extremely
important when it comes to characterising the statistical properties of the
emission source since it is a reliable indicator of the quantum or classical
nature of light source.

The n-th order correlation (normalized Glauber’s correlation functions)
function is defined as,

$\displaystyle g^{n}(t_{1},t_{2},\ldots,t_{n})=\frac{\left\langle
I_{1}(t_{1})I_{2}(t_{2})\ldots I_{n}(t_{n})\right\rangle}{\left\langle
I_{1}(t_{1})\right\rangle\left\langle
I_{2}(t_{2})\right\rangle\ldots\left\langle I_{n}(t_{n})\right\rangle}\,$ (S2)

since there is no position dependency in this experiment, one can ignore the
position dependency and the correlation function will be merely a function of
time. For this experiment, since the detector timing resolution is longer than
the SBSL pulse duration, our measured $g^{(n)}$ is time averaged over the
entire pulse. In terms of the physically measured parameters, the single
channel and coincidence counts, the 2nd order correlations are calculated
according to

$\displaystyle g^{2}_{(i,j)}=\frac{C_{(i,j)}\,R\,T}{S_{i}\,S_{j}}\,$ (S3)

where $C_{(i,j)}$ is the total coincidence counts within the data recording
time $T$ between channels $(i,j)$, $S_{i}$ and $S_{j}$ are the single channels
counts (background subtracted) within time $T$ , $R=28,060$ Hz is the SBSL
pulse repetition rate and $T=22$ s.

For n-th order correlations, this is given by:

$\displaystyle
g^{(n)}_{(i,j)}=\frac{C_{(n)}\left(R\,T\right)^{n-1}}{\Pi_{i=1}^{n}S_{i}}\,$
(S4)

where $C_{(n)}$ is the $n$-fold coincidence between channels
$(s_{i}...s_{n})$. The number of background subtracted photon detection events
over $22$ seconds for the channels/spots from $s_{1}$ to $s_{7}$ are
$124,627$; $119,499$; $243,295$; $109,486$; $111,898$; $77,269$ and $75,708$,
respectively. Here, only events within a 10-pixel radius around the centre of
each bright spot were taken into account; an average background of $4.3$
photons per second per pixel was measured, an equivalent of $30,678$ photons
in $22$ seconds within the $10$-pixel radius circle. The result of coincidence
counts for all the possible permutations of channels are summarised in Fig.
S4. Using the in-house developed code for analysing the data, coincidence is
calculated for all the possible permutations between different channels.
Panels include different possible orders of permutation of channels.

Although using avalanche photodiodes offers detection with higher time-
resolution, employing the time-tagging camera helps to reduce the experimental
complexity of multi-channel coincidence measurements. Although it is simple to
increase the number of spots/channels, a much longer data acquisition time
will be required for ($n>5$)-channel coincidence detections, subjecting the
measurement to instabilities in the SBSL source.

Figure S3: Summary of all measured coincidences for all possible permutations
of the seven channels (e.g., (1,2); (1,3); (1,3,4); etc.) for multi-fold
coincidences. Combination of all coincidence counts for different permutation
on all seven channels, possible for n-fold coincidence measurements for n=2-5.
The total integration time is 22 seconds with a 25 ns gate width. Only events
within a 10-pixel radius around the centre of each bright spot were taken into
account in the analysis. The exponential decay of the coincidence counts by
increasing the number of folds is a characteristic that can be attributed to
the camera’s quantum efficiency.

##  S5 Photon number resolving measurement

We recorded an area containing $100\times 100$ pixels on the PNR camera with
the SBSL spot at around 10 pixels in diameter. Every pixel shows a detection
of a few photons per frame, well within the range of the camera
specifications.

Variance and mean photon numbers were acquired from statistical analysis and
compared for 17 data sets to determine the inherent nature of photon
statistics. Considering Mandel $Q$ parameter defined as $Q=\left((\Delta
n)^{2}-\left\langle n\right\rangle\right)/\left\langle
n\right\rangle=\left\langle n\right\rangle(g^{2}-1)$, makes it clear to
distinguish between classical and nonclassical light. Negative $Q$ is an
indication of nonclassical light and only possible if the statistics is sub-
Poissonian. For simplicity, we also introduced another figure of merit
$E={(\Delta n)^{2}}/{\left\langle n\right\rangle}$ and obviously an $E<1$ is a
clear sign of sub-Poissonian photon statistics. To calculate the second-order
correlation function:

$\displaystyle
g^{(n)}=\frac{\left\langle\hat{a}^{\dagger}(t_{1})\ldots\hat{a}^{\dagger}(t_{n})\leavevmode\nobreak\
\hat{a}(t_{n})\ldots\hat{a}(t_{1})\right\rangle}{\left\langle\hat{a}^{\dagger}(t_{1})\leavevmode\nobreak\
\hat{a}(t_{1})\right\rangle\ldots\left\langle\hat{a}^{\dagger}(t_{n})\leavevmode\nobreak\
\hat{a}(t_{n})\right\rangle}\,$ (S5)

we employed $g^{(2)}=1+\left((\Delta n)^{2}-\left\langle
n\right\rangle\right)/\left\langle n\right\rangle^{2}$ which by comparing the
results obtained from a stabilized thermal source, it does produce reliable
results for this experiment. Table S2 shows the summary of results from 17
measurements and the statistical analysis using the maximum-likelihood
approach performed subsequently.

Table S2:  Summary of statistical analysis of recorded photons using a photon number resolving camera for 17 different data runs. STDEV is the standard deviation and for each histogram, we recorded $10,000$ sequential images with $172.8\leavevmode\nobreak\ \mu$s exposure time. Only one pixel with highest intensity was considered in this analysis. Sample # | $\left\langle n\right\rangle$ | $(\Delta n)^{2}$ | $g^{(2)}$ | $Q$ | $E$ | STDEV |  $g^{(2)}$ uncertainty
---|---|---|---|---|---|---|---
1 | 5.36 | 3.49 | 0.93 | -0.35 | 0.65 | 1.87 | 0.01
2 | 3.51 | 1.95 | 0.87 | -0.45 | 0.55 | 1.39 | 0.03
3 | 3.50 | 1.96 | 0.87 | -0.45 | 0.56 | 1.40 | 0.03
4 | 3.52 | 1.88 | 0.87 | -0.47 | 0.53 | 1.37 | 0.03
5 | 2.95 | 1.60 | 0.84 | -0.46 | 0.54 | 1.26 | 0.05
6 | 2.90 | 1.68 | 0.86 | -0.42 | 0.58 | 1.30 | 0.05
7 | 3.76 | 1.93 | 0.87 | -0.49 | 0.51 | 1.39 | 0.03
8 | 3.54 | 2.12 | 0.89 | -0.40 | 0.60 | 1.46 | 0.03
9 | 2.64 | 0.95 | 0.76 | -0.64 | 0.36 | 0.98 | 0.06
10 | 2.61 | 0.99 | 0.76 | -0.62 | 0.38 | 0.99 | 0.06
11 | 6.00 | 3.62 | 0.93 | -0.40 | 0.60 | 1.90 | 0.01
12 | 5.17 | 2.78 | 0.91 | -0.46 | 0.54 | 1.67 | 0.02
13 | 3.99 | 2.51 | 0.91 | -0.37 | 0.63 | 1.58 | 0.03
14 | 3.80 | 2.47 | 0.91 | -0.35 | 0.65 | 1.57 | 0.03
15 | 3.42 | 1.56 | 0.84 | -0.54 | 0.46 | 1.25 | 0.04
16 | 3.42 | 2.11 | 0.89 | -0.38 | 0.62 | 1.45 | 0.04
17 | 4.15 | 2.67 | 0.91 | -0.36 | 0.64 | 1.64 | 0.02


# AROhI: An Interactive Tool for Estimating ROI of Data Analytics

Noopur Zambare1, Jacob Idoko2, Jagrit Acharya2,and Gouri Ginde2  1Dept. of
Mechanical Engineering, Indian Institute of Technology Jodhpur, India
Email<EMAIL_ADDRESS>2Dept. of Electrical and Software Engineering,
University of Calgary, Canada
Email: {jacob.idoko, jagrit.acharya1<EMAIL_ADDRESS>
###### Abstract

The cost of adopting new technology is rarely analyzed and discussed, while it
is vital for many software companies worldwide. Thus, it is crucial to
consider Return On Investment (ROI) when performing data analytics. Decisions
on ”How much analytics is needed”? are hard to answer. ROI could guide
decision support on the What?, How?, and How Much? Analytics for a given
problem. This work details a comprehensive tool that provides conventional and
advanced ML approaches for demonstration using requirements dependency
extraction and their ROI analysis as use case. Utilizing advanced ML
techniques such as Active Learning, Transfer Learning and primitive Large
language model: BERT (Bidirectional Encoder Representations from Transformers)
as its various components for automating dependency extraction, the tool
outcomes demonstrate a mechanism to compute the ROI of ML algorithms to
present a clear picture of trade-offs between the cost and benefits of a
technology investment.

###### Index Terms:

Data Analysis, Return on Investment, Machine Learning

##  I Introduction

Most Machine Learning (ML) solutions are evaluated based on the algorithm’s
performance measures, such as accuracy and AUC-ROC (Area Under the Receiver
Operating Characteristic) Curve, precision and recall. However, ML algorithms
for a particular problem typically consume more computational resources than
classical algorithms for the same problem. Furthermore, these methods require
large training (annotated) datasets, which tends to become an effort and cost-
intensive aspect. Hence, decisions on ”How much is needed?” are hard to
answer; however, ROI could be used as guidance for decision support in
deciding What? How? and How Much? of analytics for a given problem[1].

Our contributions to this work are as follows:

  * •

Provide a method to compare and evaluate the ML approaches not only using ML
performance measures but also based on ROI criteria (based on our prior
research work [1], [2])

  * •

Propose a comprehensive interactive tool design to visualize the ROI and ML
performance measurement trade-offs.

  * •

Demonstrate how users can change cost factors and view the projections of
changing ROI for decision-making through a dashboard-like interactive
interface.

  * •

Demonstrate the utility of the solution application through a requirements
dependency extraction use case.

  * •

Developed and hosted AROhI tool111Excluding URL to tool for double-blind
review on the AWS instance for everyone to use.

##  II Related Work

In recent years, the assessment of the ROI for ML solutions across various
domains has increased. This section explores existing research, tools, and
frameworks, focusing on ROI estimation in data analytics, particularly for ML
and AI applications.

###  II-A Existing frameworks for measuring the business value of Data
Analytics

Several frameworks have been developed to enhance ROI estimation within data
analytics and ML projects. These tools provide methodologies to estimate AI-
driven solutions’ financial impact and value. Some notable examples among them
include:

  * •

Net Present Value (NPV): NPV calculates the present value of a data analytics
project’s expected future cash flows. It considers the initial investment,
expected returns, and the time value of money [3][4].

  * •

Cost-Benefit Analysis: CBA compares the costs associated with implementing and
maintaining a data analytics project with the benefits it generates. It
considers tangible and intangible costs and benefits to determine the overall
ROI [5].

  * •

Return on Investment Ratio: A simple ratio that compares the net benefits of a
data analytics project to its costs. It is expressed as a percentage and
provides a quick snapshot of the project’s financial performance [6].

  * •

Break-Even Analysis: This model determines the point at which the benefits of
a data analytics project equal its costs. It helps identify the time it takes
for the project to reach a break-even point and start generating positive
returns [7].

  * •

Value-Based ROI: This framework focuses on measuring the value created by a
data analytics project in terms of its impact on key performance indicators
(KPIs) or strategic objectives. It goes beyond financial metrics to assess the
project’s overall value proposition [8].

  * •

Total Cost of Ownership (TCO): TCO considers the total lifecycle costs of a
data analytics project, including upfront investment, operational costs,
maintenance, and potential upgrades. It provides a comprehensive view of the
project’s financial impact [9].

###  II-B Industry applications of ROI

ROI estimation tools and frameworks are commonly used in various sectors to
improve decision-making processes and boost business outcomes with the help of
data analysis and ML. Some specific uses in the industry include:

  * •

Finance: ROI estimating tools and frameworks play an important role in finance
by optimizing investment strategies and predicting trends in the market. Lee
et al. (2022) show that AI models such as LSTM improve decision-making in real
estate investments and reveal high ROI [10]. Net Present Value (NPV) and ROI
ratio are used to measure AI/ML-based investment strategies’ financial
feasibility and performance [11].

  * •

Healthcare: Healthcare is a field that AI and data analytics have greatly
influenced in enhancing patients’ quality of life. Pandey et al. (2021)
highlight AI’s impact on enhancing ROI by improving the efficiency of
operations while at the same time reducing the costs involved [12]. Cost
Benefit Analysis (CBA) analyzes the economic benefits of healthcare
performance indicators such as patient recovery rates and the use of resources
[13].

  * •

Manufacturing and Aerospace: AI and robotics integration in manufacturing and
aerospace increases productivity and operational efficiency, thereby
delivering rapid ROI. Marjanović et al. (2018) show that integrating value and
sustainability assessments in design space exploration through ML optimizes
design processes and reduces costs [14]. On the other hand, Figy (2021)
discusses how Labplas achieved an ROI of less than one year by redesigning
custom production machines with EtherCAT, robotics, and AI for quality
control. This resulted in a 15 to 35% increase in machine productivity [15].
Break-even Analysis is used to identify the point at which the benefits of the
new technology equal its costs. At the same time, Total Cost Ownership (TCO)
evaluates the full lifecycle costs against the gains in productivity.

However, to our knowledge, no tool has been developed yet that can be used to
determine the value of data analytics for any given problem through a no-code
interaction interface. This study is a step towards achieving such a feat.

Figure 1: Conceptual design of the AROhI tool and various technology
components of it

##  III Need for ROI estimation

###  III-A The accuracy fallacy

In most ML projects, accuracy is often the most reported metric for evaluating
the performance of a model. However, this can be misleading and result in a
fallacy that undermines the actual value of a model. The ”accuracy fallacy”
here is that a model’s high accuracy equates to reliability and performance,
which is rarely true. Accuracy measures how often a model’s predictions are
correct. However, it does not account for the cost of errors, such as false
positives and false negatives. For instance, a model that predicts rare events
can achieve high accuracy by simply predicting the majority class, which does
not provide meaningful insight into the minority class. Accuracy also fails to
account for the balance between recall (the ability to find all relevant
instances) and precision (the ability to find relevant instances) [16]. In
most cases, recall should carry more weight than precision, as noted by Berry
et al. in their empirical evaluation of tools for requirements engineering
tasks [17]. These limitations call for more robust evaluation metrics that
align with business objectives.

To truly evaluate a model’s effectiveness, it is necessary to estimate its
ROI. This involves considering factors such as lift, which measures how much
better the model is compared to random guessing, and the specific costs
associated with different types of errors [18]. In summary, it captures the
multiplicative improvement to operations and the increased return for your
efforts, i.e., ROI. Wafiq Syed, a Data Product Manager at Walmart, notes, ”The
most important metric for your model’s performance is the business metric that
it is supposed to influence” [18]. Metrics such as accuracy reports, such as
”How often is the model correct?” for both positive and negative cases, make a
bad model look good. For example, if only 1 percent of the customers buy, a
model that predicts ”no” for every customer achieves 99% accuracy but fails to
predict a potential customer correctly.

###  III-B Evaluating Machine Learning solutions beyond performance

While there is a broad spectrum of ML techniques for task automation, not all
techniques are economically viable in all the scenarios, considering the size
of training data and the investment in effort. When in doubt, simple ML
techniques are recommended over complex ones [19], [20]. However, what if a
method existed to estimate such investments and compute the benefits when
other complex algorithms are chosen? Interestingly, scholarly research
exploring the ROI of ML is scarce [21] [22]. However, a few discussions and
approaches have been proposed in the gray literature, especially on the
business front, to weigh the benefits of AI/ML-based software solutions in
terms of economic value (dollars). For instance, [23] emphasizes the need for
ROI as a measure for evaluating the benefits of ML. On the other hand, in
another blog [24], a method is proposed to arrive at ROI economic value, which
emphasizes the cost incurred due to prediction outcomes of the ML technique.
Although these approaches are intriguing and align well with our agenda, they
need more empirical rigour and evaluation. They do not consider the cost of
data accumulation and pre-processing, a large part of applied ML effort [25].

Hence, ML techniques must be evaluated beyond performance measures for
effective decision-making. In other words, it is necessary to understand how
far one should chase the accuracy parameter when the cost of achieving the
differential is substantial.

##  IV The AROhI Tool

The AROhI (Data Analytics and its Return On hInvestment)222AROHI also means
something which/who is always striving for something more in Sanskrit language
tool provides users with a cloud-deployed dashboard, enabling estimation of
the ROI of Data Analytics. This tool is based on our prior research and
experiments [1] [2] [26]. This no-code tool enables utilizing ML in technology
investment-based decision-making through advanced data analysis and ROI
estimations, thus empowering non-programmer decision-makers to create and
customize ML-driven analytics. The detailed working of the tool has been
captured in a demonstration video through the anonymous link to
folder333https://anonymous.4open.science/r/ROI_tool-1DC8/README.md. This tool
design is also inspired by other tools in the AI realm such as by Labba et al.
[27]

Architecture: Figure 1 shows the conceptual design of this tool with various
technology components used in its development, and figure 2 shows its user
interface and capabilities. AROhI has a front-end user interface that was
developed using the ReactJs framework. The interactive interface allows end
users to upload the dataset and provide additional settings on algorithm
selection and cost factors for ROI computation. REST APIs communicate with the
backend server developed in the Flask framework. Communication between front
end (UI) and back end are managed through REST APIs. The backend server was
developed using Python programming language, utilizing the Amazon S3 database
for data storage. ML algorithms are developed using SciKit Learn, an open-
source software library.

Machine Learning Algorithms:  AROhI comprises supervised learning algorithms,
including Logistic Regression, naive Bayes, Decision Trees, Support Vector
Classifier (SVC), Random Forest Classifier, and fine-tuned BERT. The tool also
provides an option to use semi-supervised learning, such as Active learning.
The user can modify several settings of the active learning algorithm,
including the threshold, maximum iteration, test size, sampling type,
classifier type, resampling, and manual annotations count. Active learning
[28] attempts to maximize a model’s performance gain while annotating the
fewest samples possible [29]. With increasing data, the demand and
computational cost of data analysis using ML algorithms have gradually risen.
Hence, considering both accuracy and investments, it is important to consider
a holistic approach to evaluate these algorithms. This approach considers
broader business impacts, thus offering a more thorough evaluation of costs
and benefits for data analysis.

Figure 2: AROhI Tool’s user Interface and various screens of the tool enable
users to interact and draw conclusions

Features and Capabilities: Descriptive analysis of the input dataset is shown
in the front end (UI) utilizing various charts and graphs (visualization:
discussed in use case Section V). The data set is further preprocessed through
a data pipeline. Upon uploading the data, the data analysis component (➋ of
Figure 2) enables the end user to observe the features present within the data
and further drill down through interactive charts to visualize statistical
information through bar graphs. Users can further choose the desired dependent
and independent variables (features).

Figure 3: Overall workflow diagram of AROhI tool: Enables end user to choose
the desired supervised or semi-supervised method to evaluate its usefulness
for their problem and ROI analysis

As shown in the flow diagram of in Figure 3, AROhI provides an option for the
end user to procure additional annotations through employing semi-supervised
ML such as Active Learning [28] if the dataset is smaller or utilizing
advanced ML methods such as Fine-tuned BERT (currently implementation is work
in progress) over conventional ML algorithms (discussed in this work).

Furthermore, the dataset is processed through a natural language processing
(NLP) pipeline. This pipeline includes lemmatization (a process that uses
vocabulary and morphological analysis of words to remove inflectional endings
and return words to their dictionary form) [30], and vectorization (a
technique in NLP to map vocabulary words to a vector of real numbers.) as part
of the text processing.

The processed data goes to the ML Algorithms during the next stage. Training
and testing subsets are created based on the training size provided by the
user. We have designed two learning techniques, including supervised and semi-
supervised algorithms. In supervised learning, annotated data is used to
predict the values of the output [31][32]. Most of the time, the use cases
need more sufficient labelled data. Often, labelling data is expensive where
unlabeled data is available in abundance [32]. Semi-supervised learning
addresses these issues and works better than supervised learning[32].

ML models are evaluated based on their accuracy, 10-fold cross-validation
score, confusion matrix and F1 score. Additionally, precision and recall are
also shown using incremental train set size. Beginning with the 20% of the
training set, models are developed to capture their performance by
incrementally growing (increasing the train by 10%) training set size to
capture its impact on F1 accuracy and ROI. Users can monitor variations in
accuracy when training sizes are changed, as the classification matrices are
provided to help users better understand their model. The user can also
compare all the models based on F1 score, recall and precision in the user
interface.

For ROI computations, end users are provided with slider bars to choose
various cost factors (discussed in Section IV) so that the incremental (with
increasing train set) ROI estimates corresponding to the performance of the
chosen models are determined. Using sliders, interaction with cost factors
configuration generates dynamic graphs for comparative ROI and F1 score
analysis for varying training sizes based on user input. The generated graphs
are visualized in the user interface as shown in Figure 1.

The relationship between the framework of AROhI in Figure 1 and the AROhI
software (tool) in Figure 2 is as follows:

  * •

In Figure 1, the first component ➊ corresponds the figure 2, depicting data
import. Next, the data analysis is performed in component ➋ of Figure 2.

  * •

Through REST APIs, the front end communicates with the backend shown in
component ➋ of the figure 1. Data passes through the NLP pipeline, leading to
the process in component ➌ of Figure 2, where ML model selection is made. A
comparison of various models built using scikit-learn is shown in component ➍.

  * •

In the last component of Figure 1 where ROI computations and configurations of
costs happen corresponds to the component ➎ of Figure 2. Finally, the results
are sent to the front end (UI) using REST APIs.

TABLE I: Parameters used for ROI computation |  Parameters |  Symbol | Unit
---|---|---|---
Cost |  Data gathering time |  $C_{\mathrm{dg}}$ | Minutes
|  Pre-processing time |  $C_{\mathrm{pp}}$ | Minutes
|  Evaluation time |  $C_{\mathrm{e}}$ | Minutes
|  Labeling time |  $C_{\mathrm{l}}$ | Minutes
|  Resource Cost |  $C_{\mathrm{resource}}$ | $ per hour
Benefit |  Value per True Positive instance |  $B_{\mathrm{reward}}$ | $
|  Penalty per False Negative instance |  $B_{\mathrm{penalty}}$ | $
Other |  H |  Human resources | Number
|  $N_{\mathrm{train}}$ |  Size of the training set | Number
|  $N_{\mathrm{test}}$ |  Size of the testing set | Number
|  N |  $N_{\mathrm{train}}$ \+ $N_{\mathrm{test}}$ | Number

ROI Modeling:  Following a typical approach to the ML life cycle, Planning,
Data Preparation, Execution and Validation are determined as various phases to
further associate cost factors with each one as shown in Table I. Table II
depicts the parameter settings used. These parameters and settings are derived
from the work by [1]. Users can configure fixed costs, labelling costs,
resource costs, reward per true positive instance, penalty per false negative
instance, and the number of human resources used. The fixed cost and labelling
cost together form the data preprocessing cost [1].

TABLE II: Parameter settings for Binary Requirements Dependency Classification Parameters |  Value
---|---
$C_{\mathrm{fixed}}=C_{\mathrm{dg}}+C_{\mathrm{pp}}+C_{\mathrm{e}}$ |  1 min/sample
$C_{\mathrm{l}}$ |  0.5 min/sample
$C_{\mathrm{resource}}$ |  $400/hr
$B_{\mathrm{reward}}$ |  $500/TP
$B_{\mathrm{penalty}}$ |  $500/FN
H |  1
N |  4105

To demonstrate cost-effectiveness, we focused on financial metrics such as
ROI, which is similar to the approach used to evaluate the profitability of
software based on real-world cases [33][1].

The ROI formula is given by:

$\text{ROI}=\frac{\text{Benefit}-\text{Cost}}{\text{Cost}}$ (1)

The benefit calculation formula is given by:

$\text{Benefit}=\text{TP}\times B_{\mathrm{reward}}-\text{FN}\times
B_{\mathrm{penalty}}$ (2)

The cost calculations are made by using:

$\text{Cost}=N\%\times\frac{(C_{\text{fixed}}+C_{\text{l}})}{60}\times H\times
C_{\text{resource}}$ (3)

Unit for benefit and cost is the human effort in person-hour[2].

###  IV-A Evaluation Metrics

We are letting users evaluate their results using the following metrics:

  * •

Cross validation:  It is statistical method of evaluating Furthermore,
comparing learning algorithms by dividing data into two segments: one used to
learn or train a model and the other used to validate the model [34]. We have
used k-fold cross-validation with k = 10.

  * •

Confusion Metrics: We obtained a $2\times 2$ matrix for each model indicating
Actual Negative, Actual Positive, Predicted Negative and Predicted Positive
[2].

  * •

F1 Score: It is calculated using actual and predicted classes.

$\text{F1 score}=\frac{(2\times TP)}{2\times TP+FP+FN}$ (4)

  * •

ROI: As shown in Equation (1), it is a financial metric used to evaluate the
profitability of an investment.

##  V Use case

To demonstrate the utility of this tool, we utilize the Requirements
Dependency Extraction problem for the dataset from the Mozilla family of
products [35] as a use case (such as Firefox browser, Thunderbird email
client, etc.). A requirement is a condition or capability a user needs to
solve a problem or achieve an objective. It is also a condition or capability
that must be met or possessed by a system or system component to satisfy a
contract, standard, specification, or other formally imposed documents [36].
To demonstrate the utility of our solution, we performed binary classification
of requirements dependency types using a subset of the entire dataset for
training and testing purposes. This process was iterated by gradually
increasing the proportion of the training set and storing the results for
further calculations. This allows users to understand the trade-off between
the F1 score and ROI and the eventual break-even point of positive returns.

TABLE III: Samples of dependency pairs from our use case dataset. Dependency type |  Requirement 1 |  Requirement 2
---|---|---
DEPENDENT |  implement quantum bar design update behind pref |  shift bookmark toolbar downwards when urlbar is focused in new tab
update navigation toolbar overflow panel for photon |  make bookmark toolbar item blend in better in overflow panel
INDEPENDENT |  offer choice to open natively with content disposition attachment view/show in browser |  remove old onboarding add on prefs
context menu option to open link/image using external application |  add automated test for item from bookmark toolbar can be sorted by name

###  V-A Dataset

Our work adopts the Binary Requirements Dependency Classification concept
presented by [26] and [2]. Given a set of requirements R and a pair of
requirements $(r,s)\in R\times R$, we have considered two requirements as
INDEPENDENT if addressing one of them does not have any logical or practical
implications for addressing the other. Otherwise, they are labelled as
DEPENDENT[26]. Table III represents pairs of requirements dependencies from
our dataset.

###  V-B Execution

  * •

To select the initial subset of the data, we used its 80%. We kept 20% for
testing and validation from the subset of training data depicted in ➊ of
Figure 2.

  * •

component ➋ of Figure 2 shows that requirements and their descriptions were
selected during feature selection. Figure 1 shows the tool’s workflow when the
user inputs the data through front end (UI). However, users can customize the
selection using the dropdown in the tool. The chart is displayed on the front
end to determine the characteristics and quantify the number of various
requirements in the data.

  * •

In the next component, the user can compare the model results for semi-
supervised shown in 3b and supervised learning in 3a2 using different
classifiers.

  * •

In Active Learning, the user can choose from available sampling techniques,
threshold, resampling technique, manual annotation count and classifier type.
Component ➌ of Figure 2 illustrates, how the ML component of the tool works.
This component stores the true positive and false negative count. Next is to
compare different models regarding F1 score, recall and precision shown in
component ➍, Figure 2. The final component in ➎ is to calculate benefit and
cost using the data the user selects through slider and ML models stored data.

TABLE IV: F1 and ROI for the settings in Table II | Logistic regression | Naïve Bayes | Random Forest | SVM | Decision Tree
---|---|---|---|---|---
Training size | F1 | ROI | F1 | ROI | F1 | ROI | F1 | ROI | F1 | ROI
20% | 0.50 | 311.50 | 0.52 | 1249.00 | 0.52 | 936.50 | 0.49 | -1.00 | 0.60 | -938.50
30% | 0.54 | 832.33 | 0.53 | 832.33 | 0.51 | 415.67 | 0.49 | -1.00 | 0.72 | 2082.33
40% | 0.53 | 311.50 | 0.54 | 624.00 | 0.54 | 467.75 | 0.49 | -1.00 | 0.70 | 1717.75
50% | 0.57 | 749.00 | 0.56 | 499.00 | 0.53 | 374.00 | 0.50 | 124.00 | 0.69 | 749.00
60% | 0.58 | 519.83 | 0.56 | 311.50 | 0.54 | 311.50 | 0.51 | 103.17 | 0.64 | -1.00
70% | 0.59 | 356.14 | 0.60 | 445.43 | 0.55 | 266.86 | 0.51 | 88.29 | 0.68 | -90.29
80% | 0.62 | 389.63 | 0.57 | 233.38 | 0.60 | 311.50 | 0.55 | 155.25 | 0.68 | 311.50
90% | 0.68 | 276.78 | 0.64 | 207.33 | 0.68 | 276.78 | 0.59 | 137.89 | 0.72 | 207.33

###  V-C Results

Table V shows the accuracies of various supervised learning models where
execution time denotes the time taken to execute the process from triggering
the model for training for the initial set training size defined by the user
(0.8 in this case). We also performed the 10-fold cross-validation during
training of this initially set data. Table IV shows the variation of F1 score
and ROI with training data size.

For Logistic Regression: The best accuracy can be reached with a dataset of up
to 60%. Beyond that, the ROI drops. Although the highest accuracy is with 90%
of the dataset, the ROI drops sharply with the given settings:

Naive Bayes: The highest ROI is with just 20% of the dataset, so investing
more is not useful. although NB’s performance becomes comparable with others
at a certain point, it does not yield value beyond 40% at .54 accuracy. Thus,
NB might not be a good algo for the given problem.

Random Forest: Accuracy does not grow until 80% of the train set however,
similar to NB, ROI is maximum at just 20% of the dataset, so even RF might not
be a good algorithm to choose from.

SVC: Although ROI is positive only after 50% of the dataset is procured for
training. Also, the accuracy is poor, and the algorithm struggles to
understand the dataset’s underlying pattern, making this one an inferior
choice. Also, beyond 80%, the ROI drastically decreases while accuracy
increases marginally. Also, it shows that the break-even is achieved only
after investing in 40% training data acquisition.

Decision Tree: Achieves best accuracy of .72 just with 20% of the dataset with
the highest ROI, making this one the most valuable and effective algorithms
for the given problem and values.

###  V-D ROI Sensitivity Analysis

As an outlook to varying input costs, we envision enhancing the tool further
to show how ROI varies by varying the various cost factors from Table II. For
example, we fixed the training size to 80% for Logistic Regression under
supervised learning and then increased Resource Cost from $400 to $440 (10%
increase), and ROI decreased from 389.62 to 354.11 ( 9.11% drop). A detailed
analysis has been explained and shown in our past work [2] through heat map
visualizations. Enabling such visualization remains part of the AROhI tool’s
feature list. As show in Figures 5, 5, 7, and 7 to visualize such analysis
dynamically.

TABLE V: Accuracy of Supervised Learning Models fixing train size = 0.8 Model | Accuracy % | CV score % |  Execution Time (sec)
---|---|---|---
Logistic Regression | 96.50 | 96.16 | 1.6053
Decision Tree Classifier | 94.82 | 95.95 | 2.3645
Naive Bayes | 92.39 | 96.07 | 0.781
SVC | 95.89 | 96.07 | 9.3548
Random Forest Classifier | 95.28 | 96.16 | 21.7949

Figure 4: The login based UI shows descriptive analysis of the uploaded data

Figure 5: Various graphs showing relative train set and various ML measures
such as F1-Score, Precision and Recall

Figure 6: For the chosen cost factors, ROI vs F1 score charts are generated
dynamically showing at what point break even is achieved and how the ROI
varies with increasing train set for ML algorithms

Figure 7: For various settings using the sliders on the left side, Active
Learning based algorithms can be used to accumulate additional annotations
incrementally. This is work in progress

##  VI Limitations

This research has several limitations: (i) The use case evaluation is
restricted to a small dataset that uses less intense computational resources.
Rigorous testing of the tool remains part of our future work. However, based
on our prior work, we are confident the results are generalizable. (ii) The
tool currently supports classification problems; Extending to other problem
types remains out of scope. (iii) The tool is trained on independent and
dependent types of requirements, which we plan to extend to other multi-class
classification problems in the future. (iv) Other aspects of the ML, such as
bias, fairness, and carbon footprint, have implications for ROI. Thus, instead
of evaluating the models based on performance measures and ROI alone, we
consider these aspects while computing ROI. However, this remains out of scope
at this point.

##  VII Conclusion and Future Work

Based on our prior research work, in this study, we propose the AROhI tool to
compute the ROI of data analytics and compare it with the F1-score, thus
enabling end users (stakeholders and decision-makers). This tool provides an
extensive and interactive interface to analyze and compare various data
analysis techniques using ML based on accuracy, costs and benefits, thus
providing a holistic ROI-driven approach to any classification problem. In our
future work, we aim to overcome the current limitations of our tool and
improve its performance and applicability across a broader range of datasets
and ML as well as LLM models. Further, we plan to address the following areas:

  1. 1.

We intend to generalize our approach to all dependency classification
datasets, regardless of the specific column names used within such datasets.
This will ensure that our model is versatile and adaptable to different data
types.

  2. 2.

In addition, we plan to apply unsupervised learning algorithms, improving the
model’s ability to detect underlying patterns without relying on labelled
data.

  3. 3.

Furthermore, we intend to extend our methodology to incorporate advanced
language models such as BERT, GPT, and other Large Language Models (LLMs) for
ROI calculations.

  4. 4.

Along with ROI and accuracy, we intend to introduce a metric to evaluate bias
and fairness in the dataset.

## References

  * [1] R. for double blind review, “Beyond accuracy: Roi-driven data analytics of empirical data,” in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–6, 2020.
  * [2] R. for double blind review, “Is bert the new silver bullet?-an empirical investigation of requirements dependency classification,” in 2021 IEEE 29th international requirements engineering conference workshops (REW), pp. 136–145, IEEE, 2021.
  * [3] E. F. Brigham, Financial management: Theory and practice.  Cengage Learning Canada Inc, 2016.
  * [4] T. Koller, M. Goedhart, and D. Wessels, Valuation: measuring and managing the value of companies.  John Wiley & Sons, 2010.
  * [5] D. Baracskay, “Cost-benefit analysis: Concepts and practice,” 1998.
  * [6] J. J. Phillips, Return on investment in training and performance improvement programs.  Routledge, 2012.
  * [7] J. G. Siegel, J. K. Shim, and S. Hartman, “Schaum’s quick guide to business formulas: 201 decision-making tools for business, finance, and accounting students,” (No Title), 1998.
  * [8] G. B. Stewart, “The quest for value: a guide for senior managers,” (No Title), 1991.
  * [9] L. M. Ellram, “Total cost of ownership: an analysis approach for purchasing,” International Journal of Physical Distribution & Logistics Management, vol. 25, no. 8, pp. 4–23, 1995.
  * [10] J.-h. Lee, H.-b. Kim, and G.-e. Shim, “Prediction model of real estate roi with the lstm model based on ai and bigdata,” International journal of advanced smart convergence, vol. 11, no. 1, pp. 19–27, 2022.
  * [11] Ginimachine, “Understanding the roi of implementing ai in financial services.” https://ginimachine.com/blog/the-roi-of-implementing-ai-in-financial-services/, 2024.
  * [12] S. Pandey, S. Gupta, and S. Chhajed, “Roi of ai: Effectiveness and measurement,” INTERNATIONAL JOURNAL OF ENGINEERING RESEARCH & TECHNOLOGY (IJERT) Volume, vol. 10, 2021.
  * [13] R. J. Brent, “Cost-benefit analysis versus cost-effectiveness analysis from a societal perspective in healthcare,” International Journal of Environmental Research and Public Health, vol. 20, no. 5, p. 4637, 2023.
  * [14] A. Bertoni, S. K. Dasari, S. I. Hallstedt, P. Andersson, et al., “Model-based decision support for value and sustainability assessment: Applying machine learning in aerospace product development,” in DS 92: Proceedings of the DESIGN 2018 15th International Design Conference, pp. 2585–2596, 2018.
  * [15] J. Figy, “Pc-based control boosts sterile sampling bag production,” Control Engineering, vol. 68, no. 2, pp. 22–25, 2021.
  * [16] J. DiMatteo, D. M. Berry, and K. Czarnecki, “Requirements for monitoring inattention of the responsible human in an autonomous vehicle: The recall and precision tradeoff.,” in REFSQ Workshops, 2020.
  * [17] D. M. Berry, J. Cleland-Huang, A. Ferrari, W. Maalej, J. Mylopoulos, and D. Zowghi, “Panel: Context-dependent evaluation of tools for nl re tasks: Recall vs. precision, and beyond,” in 2017 IEEE 25th International Requirements Engineering Conference (RE), pp. 570–573, 2017.
  * [18] E. Siegel, The AI Playbook: Mastering the Rare Art of Machine Learning Deployment.  MIT Press, 2024.
  * [19] P. Domingos, “A few useful things to know about machine learning,” Communications of the ACM, vol. 55, no. 10, pp. 78–87, 2012.
  * [20] C. Bird, T. Menzies, and T. Zimmermann, The art and science of analyzing software data.  Elsevier, 2015.
  * [21] S. Nagrecha and N. V. Chawla, “Quantifying decision making for data science: from data acquisition to modeling,” EPJ Data Science, vol. 5, no. 1, p. 27, 2016.
  * [22] M. Ferrari et al., “Roi in text mining projects,” WIT Transactions on State-of-the-art in Science and Engineering, vol. 17, 2005.
  * [23] www.altexsoft.com, “Measuring roi for machine learning and data science.” https://www.altexsoft.com/blog/business/how-to-estimate-roi-and-costs-for-machine-learning-and-data-science-projects/.  Accessed: July 2024.
  * [24] “Roi of machine learning.” https://towardsdatascience.com/return-on-investment-for-machine-learning-1a0c431509e#:~:text=Just%20like%20with%20any%20investment,cost%20of%20mistakes%20and%20accuracy. Accessed: July 2024.
  * [25] I. Figalist, C. Elsner, J. Bosch, et al., “An end-to-end framework for productive use of machine learning in software analytics and business intelligence solutions,” in Inter. Conf. on Product-Focused Software Process Improvement, pp. 217–233, Springer, 2020.
  * [26] R. for blind review, “How much data analytics is enough? the roi of machine learning classification and its application to requirements dependency classification,” arXiv preprint arXiv:2109.14097, 2021.
  * [27] C. Labba, A. Alcouffe, E. Crubézy, and A. Boyer, “Iarch: An ai tool for digging deeper into archaeological data,” in 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 22–29, IEEE, 2023.
  * [28] B. Settles, “Active learning literature survey,” 2009.
  * [29] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang, “A survey of deep active learning,” ACM computing surveys (CSUR), vol. 54, no. 9, pp. 1–40, 2021.
  * [30] L. Zhao, W. Alhoshan, A. Ferrari, and K. J. Letsholo, “Classification of natural language processing techniques for requirements engineering,” arXiv preprint arXiv:2204.04282, 2022.
  * [31] T. Hastie, R. Tibshirani, J. Friedman, T. Hastie, R. Tibshirani, and J. Friedman, “Overview of supervised learning,” The elements of statistical learning: Data mining, inference, and prediction, pp. 9–41, 2009.
  * [32] Y. Reddy, P. Viswanath, and B. E. Reddy, “Semi-supervised learning: A brief review,” Int. J. Eng. Technol, vol. 7, no. 1.8, p. 81, 2018.
  * [33] S. Stradowski and L. Madeyski, “Costs and benefits of machine learning software defect prediction: Industrial case study,” 2024.
  * [34] P. Refaeilzadeh, L. Tang, and H. Liu, “Cross-validation,” Encyclopedia of database systems, pp. 532–538, 2009.
  * [35] Mozilla.org, “Bugzilla: bug-tracking system.” https://www.bugzilla.org/, Apr. 2020.
  * [36] “Ieee standard glossary of software engineering terminology,” ANSI/ IEEE Std 729-1983, pp. 1–40, 1983.