# Stimulated Raman lineshapes in the large light-matter interaction limit

G. Batignani1,2, G. Fumero1, E. Mai1, M. Martinati1, T. Scopigno1,2,3
<EMAIL_ADDRESS>1Dipartimento di Fisica, Universitá di Roma
“La Sapienza”, Roma, I-00185, Italy  2Istituto Italiano di Tecnologia, Center
for Life Nano Science @Sapienza, Viale Regina Elena, 291, I-00161, Italy  3
Istituto Italiano di Tecnologia, Graphene Labs, Via Morego, 30, Genova,
I-16163, Italy

##  I abstract

Stimulated Raman scattering (SRS) represents a powerful tool for accessing the
vibrational properties of molecular compounds or solid state systems. From a
spectroscopic perspective, SRS is able to capture Raman spectra free from
incoherent background processes and typically ensures a signal enhancement of
several orders of magnitude with respect to its spontaneous counterpart. Since
its discovery in 1962, SRS has been applied to develop technological
applications, such as Raman-based lasers, frequency shifters for pulsed
sources and Raman amplifiers. For the full exploitation of their potential,
however, it is crucial to have an accurate description of the SRS processes
under the large gain regime. Here, by taking as an example the stimulated
Raman spectrum of a model solvent, namely liquid cyclohexane, we discuss how
the spectral profiles and the lineshapes of Raman excitations critically
depend on the pump excitation regime. In particular, we show that in the large
light-matter interaction limit the Raman gain undergoes an exponential
increase (decrease) in the red (blue) side of the spectrum, with the Raman
linewidths that appear sharpened (broadened).

##  II Introduction

Stimulated Raman scattering was first reported in 1962, when Ng and Woodbury,
gating a ruby laser for giant pulses production, detected an extra line
shifted by an amount corresponding to the stretching mode of nitrobenzene, the
material used for the Kerr shutter [1]. The observation was rationalized as
the inelastic diffusion of (the Ruby emitted) light from a medium
(nitrobenzene) coherently stimulated by the joint interaction of two
propagating photons (the original Ruby 694.3nm photons and some Raman -Stokes-
shifted ones), whose frequency difference matches one of the roto-vibrational
frequencies of the material [2]. Building on its coherent nature and chemical
sensitivity, SRS has been widely exploited to investigate the structural
properties and the photo-induced dynamics of molecules and solid state
materials. In particular, the advent of ultrafast lasers has boosted novel
spectro-microscopy approaches during the last thirty years [3, 4, 5, 6, 7, 8,
9, 10, 11].

In a typical spectroscopic configuration, SRS experiments require the presence
of two external laser pulses temporally and spatially overlapped on the
sample, namely a picosecond, narrowband ($\approx 10-20$ cm-1) Raman pump
(RP), which ensures high spectral resolution, and a femtosecond broadband
probe pulse (PP) [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Upon the
system interaction with the RP, the PP impulsively stimulates vibrational
coherences of the molecular modes, whose excitation energies correspond to the
difference between the Raman and probe frequencies. The stimulated Raman
signal is in fact induced by the third-order Raman susceptibility and is
engraved on top of the PP, with the Raman modes detected as modifications of
the PP spectral profile. Heterodyne detection hence efficiently suppresses any
incoherent isotropic fluorescence background, and the net effect can be
measured as the ratio of the probe spectral profile in presence and in absence
of the Raman pump. Under the electronically off-resonant condition, i.e. when
the incoming laser wavelengths are tuned far from any of the electronic
absorption edges of the sample, higher energy RP photons are converted to
lower energy photons in the red side spectral components of the PP. The
opposite occurs in the blue side of the PP, where higher energy PP photons are
converted to lower energy RP photons. For these reasons Raman gains and Raman
losses are measured in the red and in the blue side of the SRS spectrum [24],
respectively. In parallel with microscopy and spectroscopy, technological
applications based on SRS have been explored with the aim to realize Raman-
based lasers [25, 26, 27, 28, 29], frequency shifters [30, 31] and Raman
amplifiers [32, 33]. Inelastic scattering of electromagnetic radiation
provides indeed a convenient method to synthesize and amplify ultrashort laser
pulses, which is in general a challenging task, with a unique versatility. The
idea underneath Raman-based light emitting devices is to use the phonons
response as the mediator for optical manipulation instead of electronic
transitions, which are exploited in conventional laser technologies. For
example, in Raman amplification, coherent phonons mediate the energy transfer
from higher-frequency pump photons to lower-frequency signal photons [34],
similarly to the process discussed for the broadband probe in the SRS
spectroscopic approach. Taking advantage of Raman emission bands, which can be
thousands of wavenumbers shifted with respect to the pump frequency, it is
possible to switch toward different spectral regions. In this respect, since
the first gaseous SRS laser demonstrated in 1963 [35], multidisciplinary
scientific efforts have been committed to the quest for ideal materials with
optimal trade-off between large Raman gain and broad spectral range [36].

For both spectroscopic investigations and the development of optical devices,
it is important to understand the mechanisms underlying an efficient
generation of the SRS signal, which leads to large Raman gains. In particular,
modifications in the lineshapes of the SRS spectra may reflect important
details of the nonlinear process regardless of the specific application under
consideration: in spectroscopy, lineshapes affect and possibly rule the
spectral resolution, while in laser applications, spectral properties impacts
directly on the generated bandwidth.

Since the SRS signal intensity is proportional to the incoming RP energy, the
Raman gain ($RG$), defined as the ratio of the probe spectral profile in
presence and in absence of the Raman pump, can be increased varying the RP
fluence. In the low RP energy fluence, $RG$ gain depends linearly on the pump
intensity and can be calculated [37, 38] as
$\left[RG(\omega)-1\right]\propto-\Im\left[P^{(3)}(\omega)/E_{P}(\omega)\right]$,
where $\Im(z)$ denotes the imaginary part of $z$, $E_{P}(\omega)$ is the probe
field and $P^{(3)}(\omega)$ is the third-order polarizability responsible for
the SRS process. If the pump energy is increased beyond the linear threshold,
this dependence is complicated by the nonlinear nature of the SRS process.
Here we study the SRS response under nonlinear, high fluence pumping
conditions. We measure the spectral modifications induced by a RP fluence
beyond the linear regime of the $RG$ in cyclohexane, a liquid solvent used as
a prototypical Raman scatterer. We interpret the experimental measurements by
means of a microscopic treatment of the generated nonlinear polarization,
showing that an overall sharpening of the Raman spectrum and a relative
enhancement of the strongest Raman bands with respect to the weakest ones
occur in the large light-matter interaction limit.

##  III Experimental Setup

The Raman and probe pulses used for the SRS experiment are synthesized from
the same source, a Ti:sapphire laser, which generates transform limited 40 fs
centered at 800 nm, with an energy of 3.2 mJ and 1 kHz of repetition rate. In
order to generate the PP, a small portion of the laser is focused on a
sapphire 3 mm crystal producing a broadband (450-1000 nm) white light
continuum (WLC) via supercontinuum generation [39]. For the synthesis of the
RP, a commercial two-stage optical parametric amplifier [40] (Light Conversion
TOPAS-C) is used to produce tunable IR-visible pulses, which are then
frequency doubled by spectral compression (SC) via second harmonic and sum
frequency generation [41, 42, 43] in a 25 mm-thick BBO crystal. The picosecond
Raman pulses generated by SC are centered at $\lambda_{R}\approx$ 580 nm and
are characterized by a temporal profile unfavorable for SRS [44, 45], which is
then rectified by a double-pass (2f) spectral filter, further narrowing the
pump spectrum. The RP fluence can be adjusted by a variable neutral density
filter on the optical path. The pulses, which are linearly polarized with
parallel fields, are then focused on the sample in a non-collinear geometry
($\approx 5^{\circ}$) and the PP spectrum is monitored on a charge-coupled
device (CCD) upon frequency dispersion by a spectrometer (Acton Spectra Pro
2500i). In order to record the spectra of successive probe pulses in presence
and in absence of the RP, a synchronized chopper blocks the RP at 500 Hz.
Further details on the experimental setup are reported in [43, 44, 46].

##  IV Results and Discussion

SRS measurements have been performed on a liquid solvent, namely cyclohexane
($\mathrm{C_{6}H_{12}}$), and are reported in Figure 1 as a function of the
temporal delay $\Delta T$ between the Raman and the probe pulses for different
excitation RP fluences. The spontaneous Raman spectrum of cyclohexane [47] is
also reported for comparison.

Figure 1:  Red side SRS spectra of liquid cyclohexane acquired at two
different fluences of the Raman pump (350 nJ in panel a and 1400 nJ in panel
b) measured varying the temporal delay between the RP and the PP. For positive
time delays, the PP precedes the RP maximum providing an increased spectral
resolution. SRS spectra measured in the blue side are reported in panels c and
d (with 500 nJ and a and 1400 nJ of RP excitation energy). Slices at selected
time delays (indicated in ps in the legends) are reported in the bottom
panels. The spontaneous Raman (SR) spectrum of cyclohexane, recorded with a
continuous wave laser at 364 nm, is reported (in arbitrary units) in panels
(a)-(b) for comparison [47].

In order to investigate the SRS lineshape in the large light-matter
interaction limit, we focused on the high frequency spectral region, where the
large cross-section of the $\mathrm{C-H}$ bond vibrations generates strong
Raman signals. In addition, the overlapping Raman bands around 2900 cm-1 offer
the chance to investigate the dependence of the spectral resolution as a
function of the Raman pump fluence. In order to avoid cross phase modulation
artifacts [39, 48, 49], we limited the RP energy below 2 $\mu J$, using a
long, 10 mm-thick, optical glass cuvette to increase the SRS response. The SRS
spectra collected for PP spectral components red shifted with respect to the
RP wavelength ($\tilde{\nu}<\tilde{\nu}_{R}$) are reported in panels (a-b),
while the blue side spectra ($\tilde{\nu}>\tilde{\nu}_{R}$) are shown in
panels (c-d). As discussed in Ref. [24], at odds with the spontaneous Raman
case, the blue side spectra are originated from initial molecular populations
in the ground vibrational state and hence the amplitude of the high frequency
mode does not vanish at room temperature. A comparison of the red and the blue
side spectra is reported in Figure 2-a. As expected, the Raman gain profiles
reach the maximum intensity for slightly positive time delays ($\Delta T>0$),
i.e. for a probe pulse preceding the RP maximum, in agreement with previous
observations [50, 51].

Notably, for the RP-PP temporal delays that provide the maximum Raman gain,
increasing the RP energy by a factor of 4 (from 350 nJ to 1400 nJ) results in
red-side Raman bands with an amplitude $\approx$ 7 times larger, while an
increase of the RP energy by a factor of $\approx$ 3 (from 500 nJ to 1400 nJ)
generates blue-side Raman loss with relative amplitudes that increase by a
$\approx$ 1.4 factor. In addition, a careful inspection of Figure 1 reveals
different spectral lineshapes and relative peak intensities for different pump
fluences.

Figure 2:  SRS spectra acquired at low and high fluences ($\Phi$) in the red
and in the blue side of the spectrum are compared in panel (a) as a function
of the absolute Raman shift for overlapped Raman and probe pulses. The Raman
gain normalized to the corresponding maximum is reported in panel (b) for
selected time delays reported in the legend in ps at high and low fluences
(continuous and dashed lines). Notably, the spectral lineshapes and the
relative peak intensities show pronounced differences varying the fluence
regime, with the strongest Raman peak that undergoes an enhancement and a
sharpening with respect to the weakest ones. This effect is emphasized in
panel (c), where we report the difference between the normalized Raman spectra
of panel (b) acquired at high vs low fluence.

This is emphasized in Figure 2-b, where the red side SRS spectra normalized to
their corresponding maximum are shown for selected time delays at low and high
RP fluences (dashed and continuous lines, respectively). Specifically, the
strongest Raman peak at 2938 cm-1 undergoes an enhancement and a sharpening
with respect to the weaker 2924 cm-1 neighboring mode, which can be barely
resolved in the high fluence regime. The difference between normalized SRS
spectra, shown in Figure 2-c, further stresses the significant difference (up
to 20%) between the low and the high fluence spectral profiles. In Figure 3,
the integrated area of the SRS profiles as a function of the RP-PP delay are
shown for the red and the blue side.

Figure 3:  Integrated area of the measured SRS signal as a function of the RP-
PP delay in the red and in the blue side of the spectrum for the low and high
fluence regimes.

In order to rationalize the peculiar behavior of such experimental
observations, we modeled the SRS response by a perturbative expansion of the
light-matter interaction in terms of the electromagnetic fields. Starting from
the expression of the Maxwell’s equations inside a dielectric medium, it is
readily obtained in Gaussian units [37]

$\nabla\times\nabla\times
E(\mathrm{r},t)+\frac{n^{2}}{c^{2}}\frac{\partial^{2}E(\mathrm{r},t)}{\partial
t^{2}}=-\frac{4\pi}{c^{2}}\frac{\partial^{2}P^{(3)}(\mathrm{r},t)}{\partial
t^{2}}$ (1)

where $P^{(3)}$ is the third-order nonlinear polarization responsible for the
SRS process, while the corresponding term for the linear polarization has been
included in the refractive index $n$. The RP and PP fields can be conveniently
expressed in terms of their temporal envelopes $\mathcal{A}_{R}(z,t)$ and
$\mathcal{A}_{P}(z,t)$ as
$E_{R/P}(z,t)=\mathcal{A}_{R/P}(z,t)\,e^{i(k_{R/P}\,z-\omega_{R/P}\,t)}+c.c.$,
where $\hat{z}$ is the propagation direction and $\omega_{R/P}$ is the carrier
frequency of the Raman/Probe pulse, respectively. Analogously, the nonlinear
polarization in the $R/P$ field frame can be written as
$P^{(3)}_{R/P}(z,t)=\mathcal{P}_{R/P}^{(3)}(z,t)\,e^{i(k_{R/P}\,z-\omega_{R/P}\,t)}+c.c.$
Under the slowly varying envelope approximation (SVEA), namely if
$\frac{\partial^{2}}{\partial
t^{2}}\mathcal{A}_{R/P}(z,t)\ll\omega_{R/P}\,\frac{\partial}{\partial
t}\mathcal{A}_{R/P}(z,t)\ll\omega^{2}_{R/P}\,\mathcal{A}_{R/P}(z,t)$ and
$\frac{\partial^{2}}{\partial
t^{2}}\mathcal{P}^{(3)}_{R/P}(z,t)\ll\omega_{R/P}\,\frac{\partial}{\partial
t}\mathcal{P}^{(3)}_{R/P}(z,t)\ll\omega^{2}_{R/P}\,\mathcal{P}^{(3)}_{R/P}(z,t)$,
the second derivative of $\mathcal{A}_{R/P}(z,t)$ with respect to $z$ as well
as the first and second derivatives with respect to time of the two envelopes
in Eq. 1 can be neglected. As a result, it holds

$ik_{R/P}\frac{\partial\mathcal{A}_{R/P}(t)}{\partial
z}=-\frac{2\pi}{c^{2}}\omega_{R/P}^{2}\,\mathcal{P}^{(3)}_{R/P}(t)\,e^{i\mathrm{\Delta
k}\cdot\mathrm{z}}$ (2)

where $\Delta k$ is the wavevector mismatch [52]. The last equation can be
expressed in the frequency domain as

$ik_{R/P}\frac{\partial\mathcal{A}_{R/P}(\omega)}{\partial
z}=-\frac{2\pi}{c^{2}}\omega_{R/P}^{2}\,\mathcal{P}^{(3)}_{R/P}(\omega)\,e^{i\mathrm{\Delta
k}\cdot\mathrm{z}}$ (3)

and it holds $\mathcal{A}_{R/P}(\omega)=E_{R/P}(\omega+\omega_{R/P})$ as well
as $\mathcal{P}^{(3)}_{R/P}(\omega)=P^{(3)}_{R/P}(\omega+\omega_{R/P})$.

It is worth to stress that, in principle, the third-order nonlinear
polarization $P^{(3)}$ that generates the Raman signal results from different
processes, corresponding to the different permutations of the fields-matter
interactions [38, 53, 54, 55]. Under the non-resonant regime considered in the
present work, the concurring processes, responsible for the SRS gains and
losses in the Raman and probe fields, are depicted in Figure 4. Briefly,
diagrams a-b describe the stimulation of a vibrational coherence by two
consecutive interactions between the sample and the RP and PP fields, followed
by a second interaction with the RP and a free induction decay. Similarly,
diagrams c-d describe a SRS Raman loss/gain in the RP field, generated by a
double interaction with the red/blue shifted spectral components of the PP.

Figure 4:  Energy level diagrams accounting for SRS gains in the red and
losses in the blue side of the probe spectrum (a and b, respectively). Two
interactions with the Raman and probe fields stimulate a vibrational
coherence, which is then probed by a subsequent interaction with the Raman
field and a free-induction decay. Diagrams accounting for loss and gains on
the Raman pump are depicted in the bottom panels (c and d, respectively).
Labels $g$, $g^{\prime}_{k}$ denote the ground state and a vibrationally
excited level. Dashed and continuous arrows are used to indicate interactions
with the bra and ket sides of the density matrix, respectively.

Focusing on the SRS signal generated on the probe field, the third-order
polarization originated by the a-b diagrams can be expressed in the frequency
domain as [37]

$P^{(3)}_{a}(\omega)=\left(\frac{i}{\hbar}\right)^{3}\sum_{k}\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\int_{0}^{+\infty}d\tau_{3}E_{R}(t-\tau_{3})\\\
\int_{0}^{+\infty}d\tau_{2}E_{P}(t-\tau_{2}-\tau_{3})\int_{0}^{+\infty}d\tau_{1}E^{*}_{R}(t-\tau_{1}-\tau_{2}-\tau_{3})\\\
|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}e^{-i\tilde{\omega}_{ge}\tau_{1}}e^{-i\tilde{\omega}_{gg^{\prime}_{k}}\tau_{2}}2\pi\,g(\tau_{2})e^{-i\tilde{\omega}_{eg^{\prime}_{k}}\tau_{3}}$
(4)

and

$P^{(3)}_{b}(\omega)=\left(\frac{i}{\hbar}\right)^{3}\sum_{k}\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\int_{0}^{+\infty}d\tau_{3}E_{R}(t-\tau_{3})\\\
\int_{0}^{+\infty}d\tau_{2}E_{R}^{*}(t-\tau_{2}-\tau_{3})\int_{0}^{+\infty}d\tau_{1}E_{P}(t-\tau_{1}-\tau_{2}-\tau_{3})\\\
|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}e^{-i\tilde{\omega}_{eg}\tau_{1}}e^{-i\tilde{\omega}_{g^{\prime}_{k}g}\tau_{2}}2\pi\,g(\tau_{2})e^{-i\tilde{\omega}_{eg}\tau_{3}}$
(5)

where the summation over $k$ runs over the different investigated Raman active
modes; $\tau_{i}$ represents a time interval between two consecutive
radiation-matter interactions,
$\tilde{\omega}_{ij}=\omega_{i}-\omega_{j}-i\gamma_{ij}$, with
$\gamma_{ij}=\tau_{ij}^{-1}$ that indicates the dephasing time of the
$\ket{i}\bra{j}$ coherence, $\mu_{ij}$ is the dipole matrix element between
the $i$-$j$ states, and $2\pi\,g(\tau)$ is an inhomogeneous dephasing function
additional to the natural exponential damping (which can be neglected in the
homogeneous regime). The $g$, $g^{\prime}_{k}$, $e$ labels indicate the
molecular ground state, the vibrationally excited and the off-resonant
electronically excited levels, respectively. Importantly, the SRS signal is
generated collinearly with the probe pulse direction
($k_{P}=k_{P}+k_{R}-k_{R}^{*}$) and the process is self phase-matched ($\Delta
k=0$).

By recasting the RP/PP envelopes and the inhomogeneous dephasing function in
terms of their Fourier transform

$E_{R/P}(t)=\int_{-\infty}^{\infty}\frac{d\omega}{2\pi}\,E_{R/P}(\omega)\,e^{-i\omega
t}$
$g(\tau)=\int_{-\infty}^{\infty}\frac{d\omega_{D}}{2\pi}\,g(\omega_{D})\,e^{-i\omega_{D}t}$

it is possible to simplify Eqs. 4, 5 to

$\begin{gathered}P^{(3)}_{a}(\omega)=\left(\frac{i}{\hbar}\right)^{3}\sum_{k}|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,\frac{1}{(2\pi)^{3}}\int_{-\infty}^{\infty}d\omega_{D}\,g(\omega_{D})\\\
\int_{-\infty}^{\infty}d\omega_{1}E^{*}_{R}(\omega_{1})\int_{-\infty}^{\infty}d\omega_{2}E_{P}(\omega_{2})\int_{-\infty}^{\infty}d\omega_{3}E_{R}(\omega_{3})\\\
\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\int_{0}^{+\infty}d\tau_{3}\int_{0}^{+\infty}d\tau_{2}\int_{0}^{+\infty}d\tau_{1}\\\
e^{i\omega_{1}(t-\tau_{1}-\tau_{2}-\tau_{3})}e^{-i\omega_{2}(t-\tau_{2}-\tau_{3})}e^{-i\omega_{3}(t-\tau_{3})}\\\
e^{-i\tilde{\omega}_{ge}\tau_{1}}e^{-i(\tilde{\omega}_{gg^{\prime}_{k}}+\omega_{D})\tau_{2}}e^{-i\tilde{\omega}_{eg^{\prime}_{k}}\tau_{3}}\end{gathered}$
(6)

and

$\begin{gathered}P^{(3)}_{b}(\omega)=\left(\frac{i}{\hbar}\right)^{3}\sum_{k}|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,\frac{1}{(2\pi)^{3}}\int_{-\infty}^{\infty}d\omega_{D}\,g(\omega_{D})\\\
\int_{-\infty}^{\infty}d\omega_{1}E_{P}(\omega_{1})\int_{-\infty}^{\infty}d\omega_{2}E_{R}^{*}(\omega_{2})\int_{-\infty}^{\infty}d\omega_{3}E_{R}(\omega_{3})\\\
\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\int_{0}^{+\infty}d\tau_{3}\int_{0}^{+\infty}d\tau_{2}\int_{0}^{+\infty}d\tau_{1}\\\
e^{-i\omega_{3}(t-\tau_{3})}e^{i\omega_{2}(t-\tau_{2}-\tau_{3})}e^{-i\omega_{1}(t-\tau_{1}-\tau_{2}-\tau_{3})}\\\
e^{-i\tilde{\omega}_{eg}\tau_{1}}e^{-i(\tilde{\omega}_{g^{\prime}_{k}g}+\omega_{D})\tau_{2}}e^{-i\tilde{\omega}_{eg}\tau_{3}}\end{gathered}$
(7)

where all the temporal integrals can be analytically simplified as follows:

$\begin{gathered}P^{(3)}_{a}(\omega)=\left(\frac{1}{\hbar}\right)^{3}\sum_{k}|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,\frac{1}{(2\pi)^{2}}\int_{-\infty}^{\infty}d\omega_{D}\,g(\omega_{D})\\\
\int_{-\infty}^{\infty}d\omega_{1}E^{*}_{R}(\omega_{1})\int_{-\infty}^{\infty}d\omega_{2}E_{P}(\omega_{2})\int_{-\infty}^{\infty}d\omega_{3}E_{R}(\omega_{3})\\\
\frac{\delta(\omega+\omega_{1}-\omega_{2}-\omega_{3})}{(\tilde{\omega}_{eg^{\prime}_{k}}+\omega_{1}-\omega_{2}-\omega_{3})(\tilde{\omega}_{gg^{\prime}_{k}}+\omega_{1}-\omega_{2})(\tilde{\omega}_{ge}+\omega_{1})}\end{gathered}$
(8)

and

$\begin{gathered}P^{(3)}_{b}(\omega)=\left(\frac{1}{\hbar}\right)^{3}\sum_{k}|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,\frac{1}{(2\pi)^{2}}\int_{-\infty}^{\infty}d\omega_{D}\,g(\omega_{D})\\\
\int_{-\infty}^{\infty}d\omega_{1}E_{P}(\omega_{1})\int_{-\infty}^{\infty}d\omega_{2}E_{R}^{*}(\omega_{2})\int_{-\infty}^{\infty}d\omega_{3}E_{R}(\omega_{3})\\\
\frac{\delta(\omega-\omega_{3}+\omega_{2}-\omega_{1})}{(\tilde{\omega}_{eg}-\omega_{3}+\omega_{2}-\omega_{1})(\tilde{\omega}_{g^{\prime}_{k}g}+\omega_{2}-\omega_{1})(\tilde{\omega}_{eg}-\omega_{1})}\end{gathered}$
(9)

These last equations can be recast by taking advantage of the Dirac delta,
simplifying the integral over the probe field, obtaining

$\begin{gathered}P^{(3)}_{a}(\omega)=\sum_{k}\frac{|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}}{(2\pi)^{2}\hbar^{3}}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}d\omega_{D}\,d\omega_{1}d\omega_{3}\\\
\frac{g(\omega_{D})E_{R}(\omega_{3})E^{*}_{R}(\omega_{1})E_{P}(\omega+\omega_{1}-\omega_{3})}{(\tilde{\omega}_{eg^{\prime}_{k}}-\omega)(\tilde{\omega}_{gg^{\prime}_{k}}-\omega+\omega_{3})(\tilde{\omega}_{ge}+\omega_{1})}\end{gathered}$
(10)

and

$\begin{gathered}P^{(3)}_{b}(\omega)=\sum_{k}\frac{|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}}{(2\pi)^{2}\hbar^{3}}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}d\omega_{D}\,d\omega_{2}d\omega_{3}\\\
\frac{g(\omega_{D})E_{P}(\omega-\omega_{3}+\omega_{2})E_{R}^{*}(\omega_{2})E_{R}(\omega_{3})}{(\tilde{\omega}_{eg}-\omega)(\tilde{\omega}_{g^{\prime}_{k}g}-\omega+\omega_{3})(\tilde{\omega}_{eg}-\omega-\omega_{2}+\omega_{3})}\end{gathered}$
(11)

that are the convolution of the third-order polarizations
$P^{(3)}_{a/b}(\omega)$ valid for the homogeneous lineshape regime with the
additional dephasing function $g(\omega_{D})$.

Considering a monochromatic Raman pulse,
$E_{R}(t)=\mathcal{A}_{R}^{0}e^{-i\omega_{R}t}$,
$E_{R}(\omega)=\mathcal{A}_{R}^{0}\,2\pi\delta(\omega-\omega_{R})$ and
neglecting hereafter the inhomogeneous dephasing
($g(\omega_{D})=\delta(\omega_{D})$), the integral in Eqs. 10 and 11 can be
simplified, leading to the following analytical relations

$\begin{gathered}P^{(3)}_{a}(\omega)=\sum_{k}\frac{\hbar^{-3}\,|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,|\mathcal{A}_{R}^{0}|^{2}\,E_{P}(\omega)}{(\tilde{\omega}_{eg^{\prime}_{k}}-\omega)(\tilde{\omega}_{gg^{\prime}_{k}}-\omega+\omega_{R})(\tilde{\omega}_{ge}+\omega_{R})}=\\\
\chi^{(3)}_{a}(\omega_{R},\omega)|\mathcal{A}_{R}^{0}|^{2}E_{P}(\omega)\end{gathered}$
(12)

and

$\begin{gathered}P^{(3)}_{b}(\omega)=\sum_{k}\frac{\hbar^{-3}\,|\mu_{ge}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}\,|\mathcal{A}_{R}^{0}|^{2}\,E_{P}(\omega)}{(\tilde{\omega}_{eg}-\omega)(\tilde{\omega}_{g^{\prime}_{k}g}-\omega+\omega_{R})(\tilde{\omega}_{eg}-\omega)}=\\\
\chi^{(3)}_{b}(\omega_{R},\omega)|\mathcal{A}_{R}^{0}|^{2}E_{P}(\omega)\end{gathered}$
(13)

Combining Eqs. 3 and 12, it is possible to calculate the spectrally dispersed
SRS response in the red/blue side of the spectrum, accordingly to

$ik_{P}\frac{\partial\mathcal{A}_{P}(z,\omega)}{\partial
z}=-\frac{2\pi}{c^{2}}\omega_{P}^{2}\chi^{(3)}_{a/b}(\omega_{R},\omega)|\mathcal{A}_{R}^{0}|^{2}\mathcal{A}_{P}(z,\omega)$
(14)

that can be integrated analytically:

$\begin{cases}\mathcal{A}_{P}(z,\omega)=\mathcal{A}_{P}(0,\omega)\,e^{+i\frac{2\pi\omega_{P}\chi^{(3)}_{a/b}(\omega_{R},\omega)I_{R}z}{c\,n_{P}}}\\\
I_{P}(z,\omega+\omega_{P})=I_{P}(0,\omega+\omega_{P})\,e^{-\frac{2\pi\omega_{P}\Im\left[\chi^{(3)}_{a/b}(\omega_{R},\omega)\right]I_{R}z}{cn_{P}}}\end{cases}$
(15)

Hence, the measured Raman gain
$RG(\omega)=\frac{I_{P}^{R_{on}}(\omega)}{I_{P}^{R_{of\\!f}}(\omega)}=\frac{I_{P}(z,\,\omega)}{I_{P}(0,\,\omega)}$
is equal to

$RG_{\hbox{red/blue}}(\omega)=e^{-\frac{2\pi\omega_{P}\Im\left[\chi^{(3)}_{a/b}(\omega_{R},\,\omega-\omega_{P})\right]I_{R}z}{cn_{P}}}$
(16)

which, for small gains, can be expanded to the first order in $I_{R}z$ as
$RG_{\hbox{red/blue}}(\omega)\approx
1-\frac{2\pi\omega_{P}\Im\left[\chi^{(3)}_{a/b}(\omega_{R},\omega-\omega_{P})\right]I_{R}z}{cn_{P}}$
, so that it depends linearly on the imaginary part of $\chi^{(3)}_{a/b}$. We
note that in the low excitation regime, the $\mathcal{A}_{P}(z,\omega)$ in the
right hand side of Eq. 14 can be considered constant, retrieving the Raman
Gain definition reported in the introduction for low RP excitation regimes.

It is worth to stress that, under the low RP excitation regime, the linear
dependence of the RG on the probed frequency $\omega_{P}$ in Eq. 16 generates
blue side spectra with an amplitude stronger with respect to the red side.
Notably, while in the red side $-\Im\left(\chi^{(3)}_{a}\right)$ corresponds
to the sum of positive Raman bands and hence generates signals that
exponentially grow as a function of the sample length and RP intensity, in the
blue side $-\Im\left(\chi^{(3)}_{b}\right)$ is a negative quantity, leading to
exponentially decaying Raman gains.

Figure 5: Normalized red-side SRS lineshapes obtained from Eq. 16 varying the
fluence of a monochromatic RP (the corresponding relative amplitudes are 1.0,
1.7, 3.0, 4.0), calculated considering three Raman modes (at 2800, 2825 and
2860 cm-1) with different amplitudes and 30 cm-1 full width half maxima. The
black continuous line indicates the spontaneous Raman spectrum, while the
dashed lines represent the different Raman components. The inset shows the
corresponding Raman gain in the the red and in the blue side of the spectrum.

As illustrated in Figure 5, such exponential gain alters the relative
intensity and the lineshapes of the different modes. Specifically, the
strongest Raman modes are enhanced with respect to the weakest ones, and the
overall Raman profile is sharpened.

Figure 6:  Experimental cyclohexane SRS spectra (circles) are compared with
the modeled ones (dashed lines) in the red side of the spectrum under the low
and the high excitation regimes (panels a and b, corresponding to a 350 nJ and
1400 nJ Raman pump energy, respectively) and in the blue side (panels c and d,
measured with a 500 nJ and 1400 nJ RP energy). Traces acquired at different
temporal delays between the Raman and the probe fields (indicated in the
legends) have been vertically offset by a constant factor.

For non-monochromatic Raman pump pulses, Eq. 16 is no longer valid to
calculate the SRS spectrum, which hence has to be computed by numerically
propagating the probe field by Eq. 3 coupled to Eqs. 10-11. Since the PP is
itself acting as a source of the $P^{(3)}_{a/b}(\omega)$, this can be
conveniently done by a symmetrized split-step method in which numerical
integrations are performed at small steps $d\\!z$. Firstly, the PP field
propagation at $d\\!z$ is calculated

$E(z,\omega)\xrightarrow{P^{(3)}\left[E(z,\omega)\right]}\hat{E}(z+d\\!z,\omega)$

and then the propagated field $\hat{E}(z+d\\!z,\omega)$ is used to numerically
re-evaluate the $P^{(3)}_{a/b}(\omega)$ to correct the probe field
propagation:

$E(z,\omega)\xrightarrow{P^{(3)}\left[\frac{E(z,\omega)+\hat{E}(z+d\\!z,\omega)}{2}\right]}E(z+d\\!z,\omega)$

In Figure 6 the experimental signal (circles) is compared with the globally
fitted one (dashed lines) in the red (panels a-b) and in the blue (panels c-d)
side of the spectrum by such procedure, showing a good agreement.
Specifically, homogeneous lineshapes ($g(\tau)=(2\pi)^{-1}$) have been used
for all the Raman modes under investigation, while the Raman pump has been
modeled considering a transform limited Gaussian temporal/spectral profile
(with 1.2 ps of time duration), which reflects the dependence of the SRS area
(reported in Figure 3) as a function the RP-PP temporal delay. In fact, as
discussed in Appendix A, such dependence can be exploited to directly
determine the RP intensity temporal profile. The cyclohexane Raman mode
positions [56] have been used to calibrate the RP wavelength, while the
integration step used is 50 $\mu$m. The dephasing times
$\tau_{g^{\prime}_{k}g}$ and the relative amplitudes
($|\mu_{eg}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}$) extracted from the global fit
procedure are reported in Table 1.

Peak Position (cm-1) | Dephasing Time | Amplitude (AU)
---|---|---
2853 | 1.3 ps | 0.82
2890 | 0.3 ps | 0.64
2922 | 1.3 ps | 0.41
2938 | 0.7 ps | 1.0
Table 1: Peak positions ($\tilde{\nu}_{g^{\prime}_{k}g}$), dephasing times
($\tau_{g^{\prime}_{k}g}$) and relative amplitudes
($|\mu_{eg}|^{2}|\mu_{eg^{\prime}_{k}}|^{2}$) of the cyclohexane high-
frequency modes, obtained from the global fit of the SRS spectra.

The unique experimental control knob for tuning the effective spectral
resolution is the time delay between the Raman and the probe field [50]: upon
the stimulation of the vibrational coherences at the arrival time of the probe
pulse (Figure 4), the time window for probing the Raman excitations is
determined by the residual temporal envelope of the Raman pulse and by the
vibrational dephasing time. For this reason large (positive) time delays
between the RP-PP pair can help to increase the spectral resolution, at the
expense of a decreased peak amplitude.

The modeled signal is able to capture the dependence of the SRS spectra both
on the RP-PP temporal delay and, most importantly, on the RP fluence. In
particular, under the large light-matter interaction limit recorded at the
highest RP energy, the SRS gain becomes exponential and the Raman profiles
undergoes a spectral narrowing in the red side of the spectrum (as opposed to
an exponential decrease and a broadening in the blue side). It is worth to
stress that, while the natural width of the measured modes appears narrower
under the high fluence regimes, the effective spectral resolution (intended as
the capability of distinguishing different modes) of the linear gain regime is
maintained, since the overall spectral profile is undergoing the same
exponential gain, and not the different spectral components [57]. In fact, as
exemplified in the monochromatic RP limit, the total RG is
$RG(\omega)=e^{-\frac{2\pi\omega_{P}\Im\left[\sum_{k}\chi^{(3)}_{k}\right]I_{R}z}{cn_{P}}}$,
which does not correspond to the sum of the different Raman profiles
$\sum_{k}e^{-\frac{2\pi\omega_{P}\Im\left[\chi^{(3)}_{k}\right]I_{R}z}{cn_{P}}}$.
Notably, under the low fluence regime, the expansion of the $RG$ to the first
order in $I_{R}z$ leads to a signal that corresponds to the usual sum of the
individual Raman profiles, equal to the spontaneous Raman response.

##  V Conclusions

In summary, we have studied the stimulated Raman scattering response in the
large light-matter interaction limit. Measuring the SRS spectrum of liquid
cyclohexane in the $\mathrm{C-H}$ bond vibrations spectral region (2800-3000
cm-1), we have shown that for large RP fluences and long interaction regions
of the pulses inside the sample, an exponential gain is reached in the red
side of the spectrum, as opposed to an exponential decrease in the blue side.
Under such regime, the red-side (blue-side) spectral lineshapes appear
narrower (broader) with respect to the linear gain regime, while the effective
resolution, relevant for spectroscopic applications, is maintained. Building
on a perturbative treatment of the signal generation, we have shown how to
model the data, taking into account for the pulse fluences, temporal profiles
and relative time delays. We anticipate that the presented results can be
exploited for the rational design of novel optical devices based on SRS, such
as Raman-based lasers, frequency shifter and Raman amplifier.

##  Appendix A Dependence of the SRS Area on the Raman pump temporal profile

An electronically non-resonant SRS experiment performed under the low fluence
regime can be exploited for directly accessing the RP temporal profile, by
measuring the area of the Raman profiles as a function of the RP-PP temporal
delay. For convenience, the expression of the red-side third-order
polarization in Eq.4 can be recast by switching the integration variables from
the time intervals $\tau_{n}$ to the absolute times at which the interactions
with the fields occur, i.e. $\tau_{n}=t_{n+1}-t_{n}$. By considering an
impulsive probe pulse arriving at $t=T$, i.e.
$E_{P}(t)=\mathcal{A}^{0}_{P}\,2\pi\delta(t-T)$, and a Raman pump field
$E_{R}(t)$ whose envelope is centered around $t=0$, it holds

$P^{(3)}_{a}(\omega)\propto(i)^{3}\sum_{k}\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\int_{-\infty}^{t}dt_{3}\,E_{R}(t_{3})\\\
\int_{-\infty}^{t_{3}}dt_{2}\,\mathcal{A}^{0}_{P}\,\delta(t_{2}-T)\int_{-\infty}^{t_{2}}dt_{1}\,E^{*}_{R}(t_{1})\\\
e^{-i\tilde{\omega}_{ge}(t_{2}-t_{1})}e^{-i\tilde{\omega}_{gg^{\prime}_{k}}(t_{3}-t_{2})}e^{-i\tilde{\omega}_{eg^{\prime}_{k}}(t-t_{3})}$
(17)

and hence, by adding the proper Heaviside step function in order to simplify
the Dirac delta, the last equation reads

$P^{(3)}_{a}(\omega)\propto(i)^{3}\mathcal{A}^{0}_{P}\sum_{k}\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\\\
\int_{-\infty}^{t}dt_{3}\,\theta(t_{3}-T)\,E_{R}(t_{3})\,e^{-i\tilde{\omega}_{eg^{\prime}_{k}}(t-t_{3})}\,e^{-i\tilde{\omega}_{gg^{\prime}_{k}}(t_{3}-T)}\\\
\int_{-\infty}^{T}dt_{1}\,E^{*}_{R}(t_{1})\,e^{-i\tilde{\omega}_{ge}(T-t_{1})}$
(18)

Under non-resonant conditions, the dominant contributions from the integrals
over $t_{3}$ and $t_{1}$ come for $t_{3}=t$ and $t_{1}=T$, hence

$P^{(3)}_{a}(\omega)\propto-i\int_{-\infty}^{\infty}dt\,e^{i\omega
t}\theta(t-T)\,E_{R}(t)\,E^{*}_{R}(T)\,e^{-i\tilde{\omega}_{gg^{\prime}_{k}}(t-T)}$

where the summation over the different normal modes has been omitted for
simplicity. Under the low fluence regime, the area $A$ of the SRS profiles can
be directly obtained by integrating the heterodine-detected signal
$RG(\omega)-1=S(\omega)\propto-\Im\left(E^{*}_{P}(\omega)P^{(3)}_{a}(\omega)/|E_{P}(\omega)|^{2}\right)$
over the detection frequency $\omega$. Since
$E^{*}_{P}(\omega)=\mathcal{A}^{0}_{P}e^{i\omega T}$, it holds

$\begin{gathered}A=-\int_{-\infty}^{\infty}d\omega\,\,\Im\left(E^{*}_{P}(\omega)P^{(3)}_{a}(\omega)\right)\propto\\\
\Re\Big{(}\int_{-\infty}^{\infty}d\omega\,e^{i\omega
T}\int_{-\infty}^{\infty}dt\,e^{i\omega t}\\\
E_{R}(t)\,\theta(t-T)\,E^{*}_{R}(T)\,e^{-i\tilde{\omega}_{gg^{\prime}_{k}}(t-T)}\Big{)}\propto\\\
\Re\Big{(}\int_{-\infty}^{\infty}dt\,2\pi\delta(t-T)\,E_{R}(t)\,\theta(t-T)\,E^{*}_{R}(T)\,e^{-i\tilde{\omega}_{gg^{\prime}_{k}}(t-T)}\Big{)}\propto\\\
\Re\Big{(}E_{R}(T)E^{*}_{R}(T)\Big{)}=I_{R}(T)\end{gathered}$ (19)

which is the intensity of the Raman pump at the arrival time of the probe
pulse. Similar considerations can be derived also for the blue side of the SRS
spectrum. Hence, the measurement of the area under the SRS bands as a function
of the RP-PP delay directly provides the temporal profile of the Raman pump
intensity.

##  Appendix B Acknowledgement

This project has received funding from the PRIN 2017 Project, Grant No.
201795SBA3-HARVEST, and from the European Union’s Horizon 2020 research and
innovation program Graphene Flagship under Grant Agreement No. 881603. G.B.
and T.S. acknowledge the ‘Progetti di Ricerca Medi 2019’ and the ‘Progetti di
Ricerca Medi 2020’ grants by Sapienza Universitá di Roma.

## References

  * [1] E. J. Woodbury and W. K. Ng.  Ruby laser operation in near ir.  Proc. Inst. Radio Eng., 50(11):2367, 1962.
  * [2] N. Bloembergen and Y. R. Shen.  Coupling between vibrations and light waves in raman laser media.  Phys. Rev. Lett., 12(18):504–507, 1964.
  * [3] M. Yoshizawa, Y. Hattori, and T. Kobayashi.  Femtosecond time-resolved resonance raman gain spectroscopy in polydiacetylene.  Phys. Rev. B, 49:13259–13262, 1994.
  * [4] S. Ruhman, A.G. Joly, and K.A. Nelson.  Coherent molecular vibrational motion observed in the time domain through impulsive stimulated raman scattering.  IEEE J. Quant. Electron., 24(2):460–469, 1988.
  * [5] P. Kukura, D. W. McCamant, S. Yoon, D. B. Wandschneider, and R. A. Mathies.  Structural observation of the primary isomerization in vision with femtosecond-stimulated raman.  Science, 310(5750):1006–1009, 2005.
  * [6] P. Kukura, D. W McCamant, and R. A Mathies.  Femtosecond stimulated raman spectroscopy.  Annu. Rev. Phys. Chem., 58:461–488, 2007.
  * [7] M. Kowalewski, K. Bennett, K. E. Dorfman, and S. Mukamel.  Catching conical intersections in the act: Monitoring transient electronic coherences by attosecond stimulated x-ray raman signals.  Phys. Rev. Lett., 115:193003, 2015.
  * [8] R. C. Prince, R. R. Frontiera, and E. O. Potma.  Stimulated raman scattering: From bulk to nano.  Chem. Rev., 117(7):5070–5094, 2016.
  * [9] D. Polli, V. Kumar, C. M. Valensise, M. Marangoni, and G. Cerullo.  Broadband coherent raman scattering microscopy.  Laser Photonics Rev., 12(9):1800020, 2018.
  * [10] G. Batignani, C. Ferrante, G. Fumero, and T. Scopigno.  Broadband impulsive stimulated raman scattering based on a chirped detection.  J. Phys. Chem. Lett., 10(24):7789–7796, 2019.
  * [11] G. Fumero, C. Schnedermann, G. Batignani, T. Wende, M. Liebel, G. Bassolino, C. Ferrante, S. Mukamel, P. Kukura, and T. Scopigno.  Two-dimensional impulsively stimulated resonant raman spectroscopy of molecular excited states.  Phys. Rev. X, 10:011051, 2020.
  * [12] C. Fang, R. R. Frontiera, R. Tran, and R. A. Mathies.  Mapping GFP structure evolution during proton transfer with femtosecond raman spectroscopy.  Nature, 462(7270):200, 2009.
  * [13] A. Weigel and N.P. Ernsting.  Excited stilbene: Intramolecular vibrational redistribution and solvation studied by femtosecond stimulated raman spectroscopy.  J. Phys. Chem. B, 114(23):7879–7893, 2010.
  * [14] H. Kuramochi, S. Takeuchi, and T. Tahara.  Ultrafast structural evolution of photoactive yellow protein chromophore revealed by ultraviolet resonance femtosecond stimulated raman spectroscopy.  J. Phys. Chem. Lett., 3(15):2025–2029, 2012.
  * [15] M. Quick, A. L. Dobryakov, S. A. Kovalenko, and N. P. Ernsting.  Resonance femtosecond-stimulated raman spectroscopy without actinic excitation showing low-frequency vibrational activity in the s 2 state of all-trans $\beta$-carotene.  J. Phys. Chem. Lett., 6(7):1216, 2015.
  * [16] G. Batignani, D. Bossini, N. Di Palo, C. Ferrante, E. Pontecorvo, G. Cerullo, A. Kimel, and T. Scopigno.  Probing ultrafast photo-induced dynamics of the exchange energy in a heisenberg antiferromagnet.  Nat. Photonics, 9(8):506–510, 2015.
  * [17] G. Batignani, E. Pontecorvo, C. Ferrante, M. Aschi, C. G. Elles, and T. Scopigno.  Visualizing excited-state dynamics of a diaryl thiophene: Femtosecond stimulated raman scattering as a probe of conjugated molecules.  J. Phys. Chem. Lett., 7(15):2981–2988, 2016.
  * [18] D. R. Dietze and R. A. Mathies.  Femtosecond stimulated raman spectroscopy.  ChemPhysChem, 17(9):1224–1251, 2016.
  * [19] C. R. Hall, J. Conyard, I. A. Heisler, G. Jones, J. Frost, W. R. Browne, B. L. Feringa, and S. R. Meech.  Ultrafast dynamics in light-driven molecular rotary motors probed by femtosecond stimulated raman spectroscopy.  J. Am. Chem. Soc., 139(21):7408–7414, 2017.
  * [20] Y. Hontani, M. Kloz, T. Polívka, M. K. Shukla, R. Sobotka, and J. T. M. Kennis.  Molecular origin of photoprotection in cyanobacteria probed by watermarked femtosecond stimulated raman spectroscopy.  J. Phys. Chem. Lett., 9(7):1788–1792, 2018.
  * [21] C. J. Otolski, A. M. Raj, V. Ramamurthy, and C. G. Elles.  Ultrafast dynamics of encapsulated molecules reveals new insight on the photoisomerization mechanism for azobenzenes.  J. Phys. Chem. Lett., 10(1):121–127, 2019.
  * [22] G. Batignani, E. Pontecorvo, D. Bossini, C. Ferrante, G. Fumero, G. Cerullo, S. Mukamel, and T. Scopigno.  Modeling the ultrafast response of two-magnon raman excitations in antiferromagnets on the femtosecond timescale.  Ann. Phys., 531(12):1900439, 2019.
  * [23] C. Ferrante, G. Batignani, E. Pontecorvo, L. C. Montemiglio, M. H. Vos, and T. Scopigno.  Ultrafast dynamics and vibrational relaxation in six-coordinate heme proteins revealed by femtosecond stimulated raman spectroscopy.  J. Am. Chem. Soc., 142(5):2285–2292, 2020.
  * [24] U. Harbola, S. Umapathy, and S. Mukamel.  Loss and gain signals in broadband stimulated-raman spectra: Theoretical analysis.  Phys. Rev. A, 88:011801(R), 2013.
  * [25] F. Benabid, J. C Knight, G. Antonopoulos, and P. St. J. Russell.  Stimulated raman scattering in hydrogen-filled hollow-core photonic crystal fiber.  Science, 298(5592):399–402, 2002.
  * [26] M. A. Ferrara and L. Sirleto.  Integrated raman laser: A review of the last two decades.  Micromachines, 11(3):330, 2020.
  * [27] X. Shen, H. Choi, D. Chen, W. Zhao, and A. M. Armani.  Raman laser from an optical resonator with a grafted single-molecule monolayer.  Nat. Photonics, 14(2):95–101, 2020.
  * [28] A. I. Adamu, Y. Wang, M. S. Habib, M. K. Dasa, J. E. Antonio-Lopez, R. Amezcua-Correa, O. Bang, and C. Markos.  Multi-wavelength high-energy gas-filled fiber raman laser spanning from 1.53 $\mu$m to 2.4 $\mu$m.  Opt. Lett., 46(3):452–455, 2021.
  * [29] Z. Wang, B. Gu, Y. Chen, Z. Li, and X. Xi.  Demonstration of a 150-kw-peak-power, 2-ghz-linewidth, 1.9-$\mu$m fiber gas raman source.  Appl. Opt., 56(27):7657–7661, 2017.
  * [30] C. Vicario, M. Shalaby, A. Konyashchenko, L. Losev, and C. P. Hauri.  High-power femtosecond raman frequency shifter.  Opt. Lett., 41(20):4719–4722, 2016.
  * [31] K. Légaré, R. Safaei, G. Barrette, L. Arias, P. Lassonde, H. Ibrahim, B. Vodungbo, E. Jal, J. Lüning, N. Jaouen, Z. Tao, A. Baltuška, F. Légaré, and G. Fan.  Raman red-shift compressor: A simple approach for scaling the high harmonic generation cut-off.  Advanced Photonics Res., 2(11):2100113, 2021.
  * [32] L. Sirleto and M. A. Ferrara.  Fiber amplifiers and fiber lasers based on stimulated raman scattering: A review.  Micromachines, 11(3):247, 2020.
  * [33] Z. Wang, F. Yu, W. J. Wadsworth, and J. C. Knight.  Efficient 1.9 $\mu$m emission in h2-filled hollow core fiber by pure stimulated vibrational raman scattering.  Laser Phys. Lett., 11(10):105807, 2014.
  * [34] M.G. Raymer and J. Mostowski.  Stimulated raman scattering: unified treatment of spontaneous initiation and spatial propagation.  Phys. Rev. A, 24(4):1980, 1981.
  * [35] R.W. Minck, R.W. Terhune, and W.G. Rado.  Laser-stimulated raman effect and resonant four-photon interactions in gases h2, d2, and ch4.  Appl. Phys. Lett., 3(10):181–184, 1963.
  * [36] L. Sirleto, M. A. Ferrara, T. Nikitin, S.i Novikov, and L. Khriachtchev.  Giant raman gain in silicon nanocrystals.  Nat. Commun., 3(1), 2012.
  * [37] S. Mukamel.  Principles of Nonlinear Spectroscopy.  Oxford University Press, New York, 1995.
  * [38] K. E. Dorfman, B. P. Fingerhut, and S. Mukamel.  Time-resolved broadband raman spectroscopies: A unified six-wave-mixing representation.  J. Chem. Phys, 139(12):124113, 2013.
  * [39] G. Agrawal.  Nonlinear Fiber Optics.  Academic Press, 2013.
  * [40] C. Manzoni and G. Cerullo.  Design criteria for ultrafast optical parametric amplifiers.  J. Opt., 18(10):103501, 2016.
  * [41] M. A. Marangoni, D. Brida, M. Quintavalle, G. Cirmi, F. M. Pigozzo, C. Manzoni, F. Baronio, A. D. Capobianco, and G. Cerullo.  Narrow-bandwidth picosecond pulses by spectral compression of femtosecond pulses in second-order nonlinear crystals.  Opt. Express, 15(14):8884, 2007.
  * [42] M. Marangoni, D. Brida, M. Conforti, A. D. Capobianco, C. Manzoni, F. Baronio, G. F. Nalesso, C. De Angelis, R. Ramponi, and G. Cerullo.  Synthesis of picosecond pulses by spectral compression and shaping of femtosecond pulses in engineered quadratic nonlinear media.  Opt. Lett., 34(3):241, 2009.
  * [43] E. Pontecorvo, S.M. Kapetanaki, M. Badioli, D. Brida, M. Marangoni, G. Cerullo, and T. Scopigno.  Femtosecond stimulated raman spectrometer in the 320-520nm range.  Opt. Express, 19:1107–1112, 2011.
  * [44] E. Pontecorvo, C. Ferrante, C. G. Elles, and T. Scopigno.  Spectrally tailored narrowband pulses for femtosecond stimulated raman spectroscopy in the range 330-750 nm.  Opt. Express, 21:6866–6872, 2013.
  * [45] D. P. Hoffman, D. Valley, S. R. Ellis, M. Creelman, R. A. Mathies.  Optimally shaped narrowband picosecond pulses for femtosecond stimulated raman spectroscopy.  Opt. Express, 21(18):21685, 2013.
  * [46] G. Batignani, C. Ferrante, and T. Scopigno.  Accessing excited state molecular vibrations by femtosecond stimulated raman spectroscopy.  J. Phys. Chem. Lett., 11(18):7805–7813, 2020.
  * [47] S. E. J. Bell, E. S. O. Bourguignon, and A. Dennis.  Analysis of luminescent samples using subtracted shifted raman spectroscopy.  Analyst, 123(8):1729–1734, 1998.
  * [48] S. Lim, B. Chon, H. Rhee, and M. Cho.  Spectral modulation of stimulated raman scattering signal: Beyond weak raman pump limit.  J. Raman Spectrosc., 49(4):607–620, 2018.
  * [49] G. Batignani, G. Fumero, E. Pontecorvo, C. Ferrante, S. Mukamel, and T. Scopigno.  Genuine dynamics vs cross phase modulation artefacts in femtosecond stimulated raman spectroscopy.  ACS Photonics, 6(2):492–500, 2019.
  * [50] S. Yoon, D. W. McCamant, P. Kukura, R. A. Mathies, D. Zhang, and S.Y. Lee.  Dependence of line shapes in femtosecond broadband stimulated raman spectroscopy on pump-probe time delay.  J. Chem. Phys, 122(2):024505, 2005.
  * [51] C. Ferrante, G. Batignani, G. Fumero, E. Pontecorvo, A. Virga, L. C. Montemiglio, G. Cerullo, M. H. Vos, and T. Scopigno.  Resonant broadband stimulated raman scattering in myoglobin.  J. Raman Spectrosc., 49(6):913–920, 2018.
  * [52] R. W. Boyd.  Nonlinear Optics, Third Edition.  Academic Press, 2008.
  * [53] S.Y. Lee, D. Zhang, D. W. McCamant, P. Kukura, and R. A. Mathies.  Theory of femtosecond stimulated raman spectroscopy.  J. Chem. Phys., 121(8):3632, 2004.
  * [54] G. Batignani, G. Fumero, S. Mukamel, and T. Scopigno.  Energy flow between spectral components in 2d broadband stimulated raman spectroscopy.  Phys. Chem. Chem. Phys., 17(16):10454–10461, 2015.
  * [55] G. Batignani, C. Sansone, C. Ferrante, G. Fumero, S. Mukamel, and T. Scopigno.  Excited-state energy surfaces in molecules revealed by impulsive stimulated raman excitation profiles.  J. Phys. Chem. Lett., 12(38):9239–9247, 2021.
  * [56] T. Shimanouchi.  Tables of molecular vibrational frequencies. Consolidated volume I.  National Bureau of Standards, 1972.
  * [57] Z. Wu, Z. Wang, X. Sun, L. Zhang, M. Xu, and X. Xu.  Insight into the liquid structure of water and sodium chloride solutions using stimulated raman scattering.  Phys. Rev. Applied, 13:024030, 2020.


# M18K: A Comprehensive RGB-D Dataset and Benchmark for Mushroom Detection and
Instance Segmentation

Abdollah Zakeri  Mulham Fawakherji  Jiming Kang  Bikram Koirala  Venkatesh
Balan  Weihang Zhu  Driss Benhaddou  Fatima A. Merchant

###### Abstract

Automating agricultural processes holds significant promise for enhancing
efficiency and sustainability in various farming practices. This paper
contributes to the automation of agricultural processes by providing a
dedicated mushroom detection dataset related to automated harvesting, growth
monitoring, and quality control of the button mushroom produced using Agaricus
Bisporus fungus. With over 18,000 mushroom instances in 423 RGB-D image pairs
taken with an Intel RealSense D405 camera, it fills the gap in mushroom-
specific datasets and serves as a benchmark for detection and instance
segmentation algorithms in smart mushroom agriculture. The dataset, featuring
realistic growth environment scenarios with comprehensive annotations, is
assessed using advanced detection and instance segmentation algorithms. The
paper details the dataset’s characteristics, evaluates algorithmic
performance, and for broader applicability, we have made all resources
publicly available including images, codes, and trained models via our GitHub
repository https://github.com/abdollahzakeri/m18k

###### keywords:

Public Mushroom Dataset, Automated Mushroom Harvesting, Mushroom Detection,
Computer Vision, Smart Agriculture

††journal: Computers and Electronics in Agriculture

[c]organization=Department of Computer Science, addressline=University of
Houston, city=Houston, state=TX, country=US [t]organization=Department of
Engineering Technology, addressline=University of Houston, city=Houston,
state=TX, country=US

##  1 Introduction

The integration of Artificial Intelligence (AI)-driven automation into
agricultural processes has gained significant momentum, offering opportunities
for increased efficiency and sustainability in various farming practices
Károly and Galambos (2022); Baisa and Al-Diri (2022); Arjun et al. (2022);
Nguyen et al. (2023). Among these, the automatic harvesting of edible
mushrooms in large farms has emerged as an area with substantial potential for
streamlining operations and reducing labor-intensive tasks.

The development of accurate and robust mushroom detection methods is essential
for the success of mushroom harvesting systems, aiming to identify and locate
individual mushrooms within complex and densely populated environments. While
computer vision and object detection technologies have seen significant
advancements, a major challenge remains due to the lack of publicly available
datasets specifically tailored for mushroom detection. This impediment hinders
systematic evaluation and comparison of different mushroom detection
algorithms and methodologies. To address this, synthetic mushroom datasets
have been introduced Anagnostopoulou et al. (2023); Károly and Galambos
(2022), serving as a valuable tool for pre-training models. However, these
datasets fall short in capturing the full breadth of challenges encountered in
real-world scenarios. They often fail to accurately represent various lighting
conditions, the diversity of mushrooms at different stages of growth, and the
wide range of backgrounds and orientations that mushrooms can be found in.
Moreover, issues such as densely populated areas, various occlusions of
mushrooms, and different soil patterns are not comprehensively covered, which
means that, although useful for the initial stages of model training, these
datasets do not provide sufficient robustness for detection models in real-
world applications. Consequently, for a detection model to be truly effective
in practical scenarios, further fine-tuning on a dataset of real images is
essential, ensuring the model can navigate the complexities and variability of
natural environments. To address this research gap, we present a comprehensive
dataset of mushroom images specifically designed to serve as a benchmark for a
valid comparison of mushroom detection methods and to improve the quality of
mushroom localization in images. Our dataset consists of 423 RGB-D images,
captured using an Intel RealSense D405 RGB-D camera. Both the RGB and depth
images have a resolution of 1,280x720 pixels, ensuring consistent image
quality and compatibility across the dataset.

(a) BB Sample Image

(b) BB Sample Ground Truth

(c) BB Sample Depth

(d) WB Sample Image

(e) WB Sample Ground Truth

(f) WB Sample Depth

Figure 1: Sample of dataset images, ground truth instance segmentation masks,
and depth images for Baby Bella (BB) and White Button (WB) mushroom images

The dataset comprises two varieties of mushrooms commonly encountered in
large-scale cultivation: white button mushrooms and baby bella (cremini)
mushrooms. It consists of 173 RGB and depth image pairs of white button
mushrooms, containing a total of 4,217 annotated instances, and 250 RGB and
depth image pairs of baby bella mushrooms, containing a total of 13,955
annotated instances. The significant number of instances and images for each
mushroom type provides a diverse and comprehensive dataset for benchmarking
and evaluation.

Each image in the dataset encompasses various lighting conditions, capturing
mushrooms at different growth stages, and encompasses a range of backgrounds
and orientations. Accompanying the RGB images, the depth maps provide valuable
depth information, allowing researchers to explore depth-based mushroom
detection techniques for improved accuracy and robustness. The dataset further
includes precise annotations for each image, providing detailed bounding boxes
and segmentation mask annotations for individual mushrooms.

The creation of this mushroom dataset serves multiple objectives. First, it
facilitates evaluating and comparing existing mushroom detection algorithms,
enabling researchers to identify their strengths, weaknesses, and limitations.
Second, it establishes a standardized benchmark for assessing the performance
of novel mushroom detection methods, fostering the development of advanced
mushroom detection methods for various tasks such as automatic mushroom
harvesting, mushroom growth monitoring, and quality and quantity measurements.

Furthermore, the availability of this dataset addresses the lack of publicly
accessible mushroom datasets, encouraging collaboration and promoting
innovation in the field. Providing a shared platform for researchers to
evaluate their techniques, encourages reproducibility and accelerates progress
in mushroom detection technologies through the collective exchange of
knowledge and resources.

Figure 2: RGB-D Point Cloud Sample

In the subsequent sections of this paper, we provide a detailed description of
the dataset, covering the annotation methodology, and comprehensive
descriptive statistics. Additionally, we provide an in-depth analysis of the
dataset, highlighting its unique characteristics and potential challenges.
Furthermore, we demonstrate the utility of the dataset by evaluating state-of-
the-art object detection algorithms and discussing their performance within
the context of our benchmark. The introduction of the dedicated mushroom
dataset in this study will play a pivotal role in the advancement of automatic
mushroom harvesting systems. Our dataset, consisting of 423 RGB and depth
image pairs, each with a resolution of 1280x720 pixels, along with precise
annotations, provides a diverse and comprehensive resource for researchers to
evaluate, compare, and develop mushroom detection methods. A sample of the
dataset images, ground truth instance segmentation masks, and depth images are
plotted in Fig. 1. We anticipate that this dataset will significantly
contribute to the progress and innovation in the field of edible mushroom
smart and sustainable farming.

##  2 Literature Review

Several articles and papers have been published regarding the application of
image processing and machine learning algorithms for mushroom detection,
quality assessment, disease detection, growth monitoring, and automatic
harvesting. These works can be grouped into three main categories as follows.

Table 1:  Summary of task, method, and results of similar studies leveraging mushroom detection. None of the datasets used in similar studies are publicly available.  Author |  Task |  Method |  Dataset Size |  Results
---|---|---|---|---
Baisa and Al-Diri (2022) |  Detection, localization, and 3D pose estimation |  Segmentation using AC, Detection using CHT, 3D localization using depth information |  - |  98.99% Precision 99.29% Recall
Jareanpon et al. (2024) |  Fungal disease detection |  DenseNet201, ResNet50, Inception V3, VGGNet19 |  2000 images |  94.35% Precision 89.47% F1-score
Lee et al. (2019) |  Detection, and maturity classification |  Faster R-CNN for detection, SVM for maturity classification |  920 time-lapse image sets |  42.00% Precision 82.00% Recall 56.00% F1-score 70.93% Maturity classification accuracy
Moysiadis et al. (2023) |  Mushroom growth monitoring |  YOLOv5 and Detectron2 |  1128 images, 4271 Mushrooms |  76.50% F-1 Score 70.00% Accuracy
Nadim et al. (2019) |  Mushroom quality control |  Neural network and fuzzy logic |  250 images |  95.60% Accuracy
Olpin et al. (2018) |  Detection |  RCNN and RFCN |  310 images |  92.16% Accuracy
Retsinas et al. (2023) |  Detection and 3D pose estimation |  segmentation using a k-Medoids approach based on FPFH and FCGF |  Synthetic 3D dataset |  99.80% MAP at 25.00% IOU
Vizhanyo and Felfoldi Vízhányó and Felföldi (2000) |  Disease Detection |  LDA |  - |  85.00% True Classification Rate
Wang et al. (2018) |  Automatic sorting |  Watershed, Canny, Morphology |  - |  97.42% Accuracy
Wang et al. (2022) |  Detection and growth monitoring |  YOLOv5 + CBAM + BiFPN |  - |  99.24% MAP
Wei et al. (2022) |  Detection and growth monitoring |  YOLOv5 + ASPP + CIOU |  - |  98.00% Accuracy
Yang et al. (2022) |  Detection |  MaskRCNN |  - |  95.06% AP at 50.00% IOU
Zahan et al. (2022) |  Disease Classification |  AlexNet, GoogleNet, ResNet15 |  2536 images |  90.00% Accuracy 89.00% Precision 90.00% Recall 90.00% F1-score

###  2.1 Mushroom Detection and Localization

There is a growing body of literature dedicated to mushroom detection and
localization, each focusing on different algorithms for mushroom detection and
localization, using techniques from classical image processing to advanced
deep learning and computer vision.

The YOLO (You Only Look Once) algorithm, a significant advancement introduced
in 2016, is a cornerstone in this field. It simplified object detection into a
single regression problem, directly predicting bounding boxes and class
probabilities, thus enhancing efficiency in detection tasks. Several studies
adapted the YOLO algorithm for mushroom detection, yielding notable results.

Wei et al. (2022) developed Recursive-YOLOv5, significantly surpassing the
original YOLOv5 with 98% accuracy, though at the cost of almost doubling the
network’s parameters. Wang et al. (2022) introduced Mushroom-YOLO, an improved
version of YOLOv5, achieving a mean average precision of 99.24% by integrating
Convolutional Block Attention Module (CBAM) and Bidirectional Feature Pyramid
Network (BiFPN) into the architecture. Olpin et al. (2018) used Region-Based
Convolutional Networks for mushroom detection, finding that the Region-based
Convolutional Neural Network(RCNN) model was more accurate (92.162%) than the
Region-based Fully Convolutional Network (RFCN) (87.631%), albeit slower. Yang
et al. (2022) enhanced Agaricus Bisporus recognition using Mask RCNN,
achieving an Average Precision (AP50) of 95.061%, though the operation was
slow. Retsinas et al. (2023) created a vision module for 3D pose estimation of
mushrooms, resulting in high retrieval scores (up to 99.80% Mean Average
Precision at 25% Intersection over Union) and accurate pose estimation (mean
angle error as low as 8.70°). Lee et al. (2019) automated the selective
harvesting process, using the Faster R-CNN model for identification and a 3D
point cloud for segmentation, achieving 70.93% accuracy in maturity
identification. Baisa and Al-Diri Baisa and Al-Diri (2022) focused on robotic
picking, detecting, and estimating the 3D pose of mushrooms with high
precision (98.99%) and recall (99.29%), using RGB-D data. These studies
demonstrate advancements in mushroom detection and localization, showcasing
various algorithmic improvements. While they present significant progress,
challenges such as computational efficiency, dataset quality, and the need for
further accuracy and speed enhancements remain.

###  2.2 Mushroom Growth Monitoring And Quality Assessment

Lu et al. (2019) developed a system using YOLOv3 for monitoring mushroom
growth in greenhouses. The system could estimate mushroom cap size, count,
growth rate, and predict harvest times. It demonstrated effectiveness in
identifying growth patterns and provided real-time updates via a mobile
interface. Lu and Liaw Lu and Liaw (2020) introduced an image measurement
algorithm utilizing YOLOv3, coupled with a Score-Punishment algorithm for
accurate mushroom cap diameter measurement. The method excelled in accuracy
over traditional methods but faced challenges with unclear mushroom contours
or soil particles on caps. Nadim et al. (2019) proposed an image processing
system which assessed mushroom quality based on color, area, weight, and
volume, using data mining, neural networks, and fuzzy logic. It achieved a
95.6% correct detection rate but required image pre-processing to counteract
quality issues from normal imaging conditions.

Moysiadis et al. (2023) automated the monitoring and classification of oyster
mushrooms in greenhouses. They used YOLOv5 for detection and classification,
and Detectron2 for tracking growth, showing potential improvements in
harvesting times but faced difficulties in detecting small mushrooms. Wang et
al. (2018) developed an automatic sorting system for white button mushrooms,
using image processing to measure the pileus diameter. The system achieved
high grading speed and accuracy, significantly improving over manual grading
methods.

Benhaddou et al. (2023) implemented a computer vision system to estimate
mushroom yield and quality in a commercial farm setting, utilizing the
Circular Hough Transform algorithm. They achieved detailed tracking of
mushroom growth across 1960 frames over 20 days, demonstrating the system’s
ability to monitor size distribution and development trends. This
technological application offers a promising tool for optimizing cultivation
and harvesting strategies by providing precise data on mushroom growth
patterns and potential yield estimations.

###  2.3 Mushroom Disease Detection

Zahan et al. (2022) employed deep learning models to classify mushroom
diseases, with ResNet15 showing the best performance in accuracy and
precision. The study emphasized the potential of deep learning in agricultural
management. Vizhanyo and Felfoldi Vízhányó and Felföldi (2000) used advanced
image analysis to distinguish diseased spots on mushroom caps. Their method
showed an 85% correct classification ratio, effectively differentiating
between brown blotch and ginger blotch diseases.

Jareanpon et al. (2024) developed an intelligent farm system for real-time
detection of fungal diseases in Lentinus. The system integrated environment
control, an imaging robot, and a deep learning-based prognosis system,
achieving high precision in maintaining optimal conditions and disease
detection.

In conclusion, the literature review section of this paper highlights a range
of innovative approaches to mushroom detection, quality assessment, growth
monitoring, and disease detection using image processing and machine learning
algorithms. Table 1 lists these studies and their results. The studies vary in
focus, ranging from detection and localization to quality and disease
detection, each contributing valuable insights and advancements in the field.
Notably, while these works demonstrate significant progress, there remains a
gap in the standardization of methodologies and results comparison, compounded
with challenge of data drift Mirza et al. (2024). This gap is primarily due to
the lack of a publicly available benchmark dataset. Based on the current
literature Yin et al. (2022), and to the best of the authors’ knowledge, there
are no publicly available annotated edible mushroom datasets. A standardized
dataset would provide a common ground for evaluating different algorithms and
techniques, ensuring comparability and consistency across studies. The
availability of such a dataset would benefit the field, allowing for more
robust and reliable comparisons of methods and results, driving forward the
development of more efficient and effective solutions for mushroom cultivation
and processing in the agricultural sector.

##  3 Labeling Process

The images were initially labeled using the Segment Anything Model (SAM)
automatic mask generation feature Kirillov et al. (2023). This feature
provides several configurations and post-processing steps to ensure the best
quality of automatically generated masks for specific purposes. For instance,
a grid of n*n (n=32 by default) positive points is placed on the original
image and randomly cropped versions of it, and the generated masks are
filtered by their confidence scores and stability within different confidence
thresholds. The automatic mask generation algorithm considers a mask to be
stable if thresholding the probability map at $0.5-\delta$ and $0.5+\delta$
results in similar masks. Additionally, masks can be filtered by their minimum
and maximum area and a certain maximum intersection over union (IOU) threshold
can be set to avoid generation of overlapping masks. Nevertheless, these
configurations do not guarantee the generation of perfect ground truth masks
and can only be used as a starting point. These generated masks were
meticulously edited manually during several steps to guarantee the perfection
of the dataset ground truth masks. Initially, all the false positive masks
were removed and the missing masks were added. Later, the intersection of all
pairs of masks in the same image was calculated for all the images in the
dataset, and the overlapping masks were removed manually.

##  4 Data Description

Our dataset includes 423 RGBD images, 250 of which belong to Baby Bella
mushrooms and the remaining 173 belong to the White Button mushrooms. A sample
of the dataset images, depth maps, and label masks is plotted in Fig.1. Both
the RGB images and the depth images have a resolution of 1280*720 pixels, and
hence, no alignment will be required for using the RGB and depth images as a
single RGB-D input. A sample 3D point cloud image is plotted in Fig. 2. The
camera was placed at vertical distances of 27 cm (about 10.63 in) and 15 cm
(about 5.91 in) above the cultivation beds during image acquisition of baby
bella and white button mushrooms, respectively.

Figure 3: Scatter plots of instances mask area vs bounding box diagonal length
for (A) baby bella mushrooms and (B) white button mushrooms.

Our dataset consists of 4217 white button mushroom instances and 13,955 baby
bella mushroom instances in total. These instances include a wide range of
mushroom sizes for both mushroom categories which makes our dataset suitable
for training detection models aiming for various tasks such as mushroom growth
monitoring and mushroom automatic harvesting. Histograms of mask areas and
bounding box diagonal length of mushroom instances are plotted in Fig. 4 and
scatter plots of instances mask area vs bounding box diagonal length are
plotted in Fig. 3. As can be seen in the scatter plots, a fraction of the
mushroom instances are above the scatter plot’s trend line, meaning they have
a small area compared to their diagonal length. These instances usually belong
to the mushrooms occluded by neighboring mushrooms in the cluster, causing
only a portion of them to be visible, having crescent-shaped masks as opposed
to fully visible mushrooms that have convex-shaped masks. Both figures 3 and 4
show much larger masks for white button mushrooms which is caused by the
smaller vertical distance between the camera and cultivation bed at the time
of image acquisition.

Figure 4: Histograms of Mask Areas and Bounding Box Diagonal Lengths;
Histograms (A) and (B) show pixel mask areas of Baby Bella (BB) and White
Button (WB) mushrooms accordingly. Histograms (C) and (D) show bounding box
diagonal lengths for BB and WB mushroom instances accordingly.

To study the separability of the two mushroom classes under various color
spaces, we have plotted scatter plots visualizing the distribution of
instances of each class in three color spaces RGB, HSV, and LAB in Fig. 5.
Each data point on the scatter plots represents a single instance and the mean
value of all the pixels of the instance’s mask for each color channel were
translated as x, y, and z coordinates of the 3D scatter plots. The two
clusters seem more intertwined in RGB color space compared to HSV and LAB.

(a) RGB

(b) HSV

(c) LAB

Figure 5: Color distribution of mushroom instances in various color spaces;
Judging only by the color distributions, LAB and HSV color spaces are more
suitable than RGB for classifying WB and BB instances. WB mushrooms are shown
in orange and BB mushrooms are shown in blue.

##  5 Benchmarking and Results

In this section, we provide a detailed analysis of several detection and
instance segmentation algorithms tested on our mushroom dataset. The models
were rigorously trained on a robust hardware setup consisting of a DGX-2
cluster of 16 Nvidia V-100 GPUs, each equipped with 32GB of memory, ensuring
substantial computational power for handling extensive data and complex model
architectures. All models underwent thorough evaluations, with their
performance presented in Table 2. Our assessment utilized metrics such as F1
score, average precision (AP), and average recall (AR) for all models
regardless of the task being detection or instance segmentation.

(a) BB Sample Image

(b) WB Sample Image

Figure 6: Prediction results of MaskRCNN-ResNet-50 architecture on (a) baby bella images, and (b) white button mushroom images Table 2: Performance of different object detection and instance segmentation models on M18K dataset - Params is number of model parameters in Million, and C is number of input channels Model Name | Backbone | Params |  C | Task |  Size |  F1 |  AP |  AR
---|---|---|---|---|---|---|---|---
FasterRCNN Ren et al. (2015) | efficientnet_b1 | 86.6 |  3 | Det |  1280 |  0.824 |  0.807 |  0.842
|  MobileNet-V3 Howard et al. (2019) | 18.9 |  3 | Det |  1280 |  0.825 |  0.808 |  0.843
|  ResNet-50 He et al. (2016) | 43.3 |  3 | Det |  1280 |  0.894 |  0.880 |  0.909
MaskRCNN He et al. (2017) |  EfficientNet-B1 Tan and Le (2020) | 91.6 |  3 | Det |  1280 |  0.819 |  0.802 |  0.836
|  |  |  | Seg |  1280 |  0.812 |  0.797 |  0.828
| MobileNet-V3 | 64.8 |  3 | Det |  1280 |  0.816 |  0.800 |  0.834
|  |  |  | Seg |  1280 |  0.813 |  0.798 |  0.828
| ResNet-50 | 45.9 |  3 | Det |  1280 |  0.891 |  0.876 |  0.905
|  |  |  | Seg |  1280 |  0.881 |  0.866 |  0.896
| ResNet-50 | 171 |  4 | Det |  1280 |  0.863 |  0.846 |  0.881
|  |  |  | Seg |  1280 |  0.861 |  0.846 |  0.876
RT-DETR-L Lv et al. (2023) | ResNet-50 | 31.9 |  3 | Det |  640 |  0.245 |  0.210 |  0.295
|  |  |  |  |  1280 |  0.907 |  0.891 |  0.923
RT-DETR-X | ResNet-50 | 65.4 |  3 | Det |  640 |  0.120 |  0.099 |  0.153
|  |  |  |  |  1280 |  0.906 |  0.891 |  0.922
YOLOV8-L |  CSPDarknet Wang et al. (2020) | 43.7 |  3 | Det |  640 |  0.735 |  0.693 |  0.782
|  |  |  |  |  1280 |  0.880 |  0.869 |  0.892
|  |  |  | Seg |  640 |  0.700 |  0.659 |  0.745
|  |  |  |  |  1280 |  0.857 |  0.846 |  0.868
YOLOV8-M | CSPDarknet | 25.9 |  3 | Det |  640 |  0.732 |  0.690 |  0.779
|  |  |  |  |  1280 |  0.868 |  0.857 |  0.880
|  |  |  | Seg |  640 |  0.711 |  0.669 |  0.760
|  |  |  |  |  1280 |  0.844 |  0.833 |  0.856
YOLOV8-N | CSPDarknet | 3.2 |  3 | Det |  640 |  0.775 |  0.739 |  0.815
|  |  |  |  |  1280 |  0.887 |  0.873 |  0.901
|  |  |  | Seg |  640 |  0.748 |  0.712 |  0.788
|  |  |  |  |  1280 |  0.859 |  0.846 |  0.874
YOLOV8-S | CSPDarknet | 11.2 |  3 | Det |  640 |  0.775 |  0.743 |  0.809
|  |  |  |  |  1280 |  0.880 |  0.868 |  0.892
|  |  |  | Seg |  640 |  0.753 |  0.720 |  0.789
|  |  |  |  |  1280 |  0.857 |  0.845 |  0.869
YOLOV8-X | CSPDarknet | 68.3 |  3 | Det |  640 |  0.724 |  0.667 |  0.790
|  |  |  |  |  1280 |  0.887 |  0.875 |  0.898
|  |  |  | Seg |  640 |  0.695 |  0.638 |  0.763
|  |  |  |  |  1280 |  0.858 |  0.847 |  0.869
YOLOV9-C Wang et al. (2024) | GELAN-C | 25.5 |  3 | Det |  640 |  0.760 |  0.726 |  0.797
|  |  |  |  |  1280 |  0.884 |  0.872 |  0.897
YOLOV9-E | GELAN-E | 58.1 |  3 | Det |  640 |  0.736 |  0.702 |  0.774
|  |  |  |  |  1280 |  0.877 |  0.867 |  0.887

The data in Table 2 provides details of the performance of a variety of models
with distinct backbone architectures, benchmarked on object detection and
instance segmentation tasks. These models were trained on image resolutions of
640 * 360 and 1280 * 720. The MaskRCNN with a ResNet-50 backbone surpassed the
other networks in instance segmentation, achieving a mean Average Precision
(mAP) of 0.866 and an Average Recall (AR) of 0.896 when processing 1280-sized
RGB images. When adapted for RGB-D input, which increased the network’s
parameters over three times the original, there was no enhancement in
performance. This lack of improvement may stem from the mandatory re-
initialization of weights following the modifications to accommodate the added
depth channel, which precluded the use of existing pre-trained weights that
typically contribute to superior performance via fine-tuning on a specific
dataset. Furthermore, the expanded parameter count makes the network less
suitable for real-time use where hardware resources are constrained. Due to
the lack of improvement in performance while adding the depth channel, other
networks were not tested with RGB-D images. Furthermore, The RT-DETR Large
model exhibited superior object detection performance on 1280-sized images,
with a mAP of 0.907 and an AR of 0.923. In contrast, both RT-DETR variants
performed poorly on 640-sized images, potentially due to the network not being
as effective at capturing the necessary details at a reduced resolution given
the small number of dataset images. An intriguing finding was that the
smallest and largest YOLO-V8 versions, YOLOV8-Nano with 3.2 million
parameters, and YOLOV8-X with 68.3 million parameters, achieved nearly
identical mAPs of 0.873 and 0.875, respectively. This suggests that beyond a
certain threshold, additional parameters may yield diminishing returns in
performance improvement, and that smaller models may be sufficiently
expressive for our given task or more effective in generalizing from the small
number of training data. Fig. 6 provides a visual comparison of the prediction
results for Mask R-CNN with a ResNet-50 backbone trained on RGB images of size
1280. Fig. 6(a) and Fig. 6(b) show the prediction results on RGB images for BB
(Baby Bella) and WB (White Buttons) mushroom instances, respectively. Overall,
the results indicate that while all models performed well across various
metrics, the choice of the backbone and the balance between the number of
parameters and input channels play a significant role in the trade-off between
accuracy and computational efficiency. These insights can inform the
development and optimization of mushroom detection algorithms tailored for
agricultural automation.

##  6 Conclusion

In this work, we have presented a dedicated mushroom detection dataset
tailored to train and evaluate object detection and instance segmentation
methods for various purposes that require localization of mushrooms in images
such as automated mushroom harvesting, mushroom quality assessment, and
disease detection. Through comprehensive data collection and meticulous
labeling processes, the dataset encompasses a wide spectrum of scenarios and
challenges inherent in real-world mushroom growth environments including
clusters of mushrooms with a variety of densities, images with different
lighting conditions, occasional occlusion of mushroom instances by soil
particles, and mushrooms growing at the edge of cultivation beds. The diverse
nature of the dataset, including instances of both white button and baby bella
mushrooms, allows for the robust training and benchmarking of detection and
instance segmentation algorithms.

Our evaluation of 11 different models highlights their respective strengths
and weaknesses, shedding light on their suitability for mushroom detection and
instance segmentation tasks. Furthermore, our exploration of color spaces
provides insights into potential improvements in classification accuracy. By
sharing this dataset and evaluation results, we hope to catalyze research and
innovation in the domain of smart edible mushroom farming, fostering
collaborations and accelerating progress in detection technologies. The
availability of this benchmark dataset, our code, and trained models
contributes to the development of advanced mushroom detection techniques,
paving the way for increased efficiency and sustainability in automated
mushroom harvesting systems.

## Acknowledgements and funding

This work is partially supported by the United States Department of
Agriculture grants #13111855, 13424031, and 2023-51300-40853, and the
University of Houston Infrastructure Grant. The authors would also like to
acknowledge Monterey Mushroom Farms, Madisonville, Texas for their
collaboration on this study.

## References

  * Anagnostopoulou et al. (2023) Anagnostopoulou, D., Retsinas, G., Efthymiou, N., Filntisis, P., Maragos, P., 2023.  A Realistic Synthetic Mushroom Scenes Dataset, pp. 6282–6289.
  * Arjun et al. (2022) Arjun, A.D., Chakraborty, S.K., Mahanti, N.K., Kotwaliwale, N., 2022.  Non-destructive assessment of quality parameters of white button mushrooms (Agaricus bisporus) using image processing techniques.  Journal of Food Science and Technology 59, 2047–2059.  doi:10.1007/s13197-021-05219-w.
  * Baisa and Al-Diri (2022) Baisa, N.L., Al-Diri, B., 2022.  Mushrooms detection, localization and 3d pose estimation using rgb-d sensor for robotic-picking applications.  ArXiv .
  * Benhaddou et al. (2023) Benhaddou, D., Balan, V., Garza, A.D.L., Merchant, F., 2023.  Estimating mushroom yield and quality using computer vision, in: 2023 International Wireless Communications and Mobile Computing (IWCMC), pp. 562–567.  doi:10.1109/IWCMC58020.2023.10182999.
  * He et al. (2017) He, K., Gkioxari, G., Dollar, P., Girshick, R., 2017.  Mask r-cnn, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV).
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Howard et al. (2019) Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H., 2019.  Searching for mobilenetv3, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  * Jareanpon et al. (2024) Jareanpon, C., Khummanee, S., Sriputta, P., Scully, P., 2024.  Developing an Intelligent Farm System to Automate Real-time Detection of Fungal Diseases in Mushrooms.  CURRENT APPLIED SCIENCE AND TECHNOLOGY , e0255708–e0255708doi:10.55003/cast.2023.255708.
  * Károly and Galambos (2022) Károly, A.I., Galambos, P., 2022.  Automated dataset generation with blender for deep learning-based object segmentation, in: 2022 IEEE 20th Jubilee World Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 000329–000334.  doi:10.1109/SAMI54271.2022.9780790.
  * Kirillov et al. (2023) Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R., 2023.  Segment anything.  arXiv:2304.02643.
  * Lee et al. (2019) Lee, C.H., Choi, D., Pecchia, J., He, L., Heinemann, P., 2019.  Development of A Mushroom Harvesting Assistance System using Computer Vision.
  * Lu and Liaw (2020) Lu, C.P., Liaw, J.J., 2020.  A novel image measurement algorithm for common mushroom caps based on convolutional neural network.  Computers and Electronics in Agriculture 171, 105336.  doi:10.1016/j.compag.2020.105336.
  * Lu et al. (2019) Lu, C.P., Liaw, J.J., Wu, T.C., Hung, T.F., 2019.  Development of a Mushroom Growth Measurement System Applying Deep Learning for Image Recognition.  Agronomy 9, 32.  doi:10.3390/agronomy9010032. number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
  * Lv et al. (2023) Lv, W., Xu, S., Zhao, Y., Wang, G., Wei, J., Cui, C., Du, Y., Dang, Q., Liu, Y., 2023.  Detrs beat yolos on real-time object detection.  arXiv:2304.08069.
  * Mirza et al. (2024) Mirza, S., Nguyen, V.D., Mantini, P., Shah, S.K., 2024.  Data quality aware approaches for addressing model drift of semantic segmentation models, in: VISIGRAPP (3: VISAPP), pp. 333–341.
  * Moysiadis et al. (2023) Moysiadis, V., Kokkonis, G., Bibi, S., Moscholios, I., Maropoulos, N., Sarigiannidis, P., 2023.  Monitoring Mushroom Growth with Machine Learning.  Agriculture 13, 223.  doi:10.3390/agriculture13010223. number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
  * Nadim et al. (2019) Nadim, M., Ahmadifar, H., Mashkinmojeh, M., yamaghani, M.R., 2019.  Application of Image Processing Techniques for Quality Control of Mushroom.  Caspian Journal of Health Research 4, 72–75.  doi:10.29252/cjhr.4.3.72.
  * Nguyen et al. (2023) Nguyen, V., Ho, T.A., Vu, D.A., Anh, N.T.N., Thang, T.N., 2023.  Building footprint extraction in dense areas using super resolution and frame field learning, in: 2023 12th International Conference on Awareness Science and Technology (iCAST), pp. 112–117.  doi:10.1109/iCAST57874.2023.10359290.
  * Olpin et al. (2018) Olpin, A.J., Dara, R., Stacey, D., Kashkoush, M., 2018.  Region-Based Convolutional Networks for End-to-End Detection of Agricultural Mushrooms, in: Image and Signal Processing, pp. 319–328.  doi:10.1007/978-3-319-94211-7\\_35.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, in: Advances in Neural Information Processing Systems.
  * Retsinas et al. (2023) Retsinas, G., Efthymiou, N., Anagnostopoulou, D., Maragos, P., 2023.  Mushroom Detection and Three Dimensional Pose Estimation from Multi-View Point Clouds.  Sensors (Basel, Switzerland) 23, 3576.  doi:10.3390/s23073576.
  * Tan and Le (2020) Tan, M., Le, Q.V., 2020.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv:1905.11946.
  * Vízhányó and Felföldi (2000) Vízhányó, T., Felföldi, J., 2000.  Enhancing colour differences in images of diseased mushrooms.  Computers and Electronics in Agriculture 26, 187–198.  doi:10.1016/S0168-1699(00)00071-5.
  * Wang et al. (2020) Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., Yeh, I.H., 2020.  Cspnet: A new backbone that can enhance learning capability of cnn, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  * Wang et al. (2024) Wang, C.Y., Yeh, I.H., Liao, H.Y.M., 2024.  Yolov9: Learning what you want to learn using programmable gradient information.  arXiv:2402.13616.
  * Wang et al. (2018) Wang, F., Zheng, J., Tian, X., Wang, J., Niu, L., Feng, W., 2018.  An automatic sorting system for fresh white button mushrooms based on image processing.  Computers and Electronics in Agriculture 151, 416–425.  doi:10.1016/j.compag.2018.06.022.
  * Wang et al. (2022) Wang, Y., Yang, L., Chen, H., Hussain, A., Ma, C., Al-gabri, M., 2022.  Mushroom-YOLO: A deep learning algorithm for mushroom growth recognition based on improved YOLOv5 in agriculture 4.0, in: 2022 IEEE 20th International Conference on Industrial Informatics (INDIN), pp. 239–244.  doi:10.1109/INDIN51773.2022.9976155.
  * Wei et al. (2022) Wei, B., Zhang, Y., Pu, Y., Sun, Y., Zhang, S., Lin, H., Zeng, C., Zhao, Y., Wang, K., Chen, Z., 2022.  Recursive-YOLOv5 Network for Edible Mushroom Detection in Scenes With Vertical Stick Placement.  IEEE Access 10, 40093–40108.  doi:10.1109/ACCESS.2022.3165160. conference Name: IEEE Access.
  * Yang et al. (2022) Yang, S., Huang, J., Yu, X., Yu, T., 2022.  Research on a Segmentation and Location Algorithm Based on Mask RCNN for Agaricus Bisporus, in: 2022 2nd International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI), pp. 717–721.  doi:10.1109/CEI57409.2022.9950157.
  * Yin et al. (2022) Yin, H., Yi, W., Hu, D., 2022.  Computer vision and machine learning applied in the mushroom industry: A critical review.  Computers and Electronics in Agriculture 198, 107015.  doi:10.1016/j.compag.2022.107015.
  * Zahan et al. (2022) Zahan, N., Hasan, M.Z., Uddin, M.S., Hossain, S., Islam, S.F., 2022.  Chapter 10 - A deep learning-based approach for mushroom diseases classification, in: Khan, M.A., Khan, R., Ansari, M.A. (Eds.), Application of Machine Learning in Agriculture. Academic Press, pp. 191–212.  doi:10.1016/B978-0-323-90550-3.00005-9.