# Parallelization of frequency domain quantum gates: manipulation and
distribution of frequency-entangled photon pairs generated by a 21 GHz silicon
micro-resonator

Antoine Henry1,2  Dario Fioretto2  Lorenzo M. Procopio3  Stéphane Monfray4
Frédéric Boeuf4  Laurent Vivien2  Eric Cassan2  Carlos Ramos2  Kamel
Bencheikh2  Isabelle Zaquine1  Nadia Belabas2  1LTCI, Télécom Paris, Institut
Polytechnique de Paris, 19 place Marguerite Perey, 91120, Palaiseau, France
2Université Paris-Saclay, CNRS,Centre for Nanosciences and Nanotechnology, UMR
9001, 10 Boulevard Thomas Gobert, 91120, Palaiseau, France
3 Weizmann Institute of Science, Rehovot 7610001, Israel
4STMicroelectronics SAS, 850 rue Jean Monnet, 38920 Crolles, France
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Harnessing the frequency dimension in integrated photonics offers key
advantages in terms of scalability, noise resilience, parallelization and
compatibility with telecom multiplexing techniques. Integrated ring resonators
have been used to generate frequency-entangled states through spontaneous
four-wave-mixing. However, state-of-the-art integrated resonators are limited
by trade-offs in size, number of frequency modes and spectral separation. We
have developed silicon ring resonators with a foot-print below 0.05 mm2
providing more than 70 frequency channels separated by 21 GHz. We exploit the
narrow frequency separation to parallelize and independently control 34 single
qubit-gates with off-the-shelf electro-optic devices. This allows to fully
characterize 17 frequency-bin maximally-entangled qubit pairs by performing
quantum state tomography. We demonstrate for the first time a fully connected
5-user quantum network in the frequency domain. These results are a step
towards a new generation of quantum circuits implemented with scalable silicon
photonics technology, for applications in quantum computing and secure
communications.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Frequency encoding provides a resource-efficient way to access a high-
dimensional Hilbert space within a single spatial mode, opening the way for
scalable quantum information processing. In this scheme, photons can be
generated in a superposition of different frequency modes. The superpositions
are very robust against phase noise in long-distance propagation. Photon pairs
can be generated through non-linear interaction such as spontaneous parametric
down conversion (SPDC) or spontaneous four wave mixing (SFWM). The frequency
bins are created by external filtering of a wideband nonlinear source [1, 2,
3, 4], or by exploiting the inherently discrete frequencies of a resonator [5,
6, 7, 8].

The silicon-on-insulator (SOI) technology provides key advantages for the
generation of correlated and entangled photon pairs, including the scalability
and the availability of a wide library of high-performance optic and
optoelectronic devices. Indeed, silicon photonics has been identified as an
enabling technology for quantum information[9]. In particular, it is possible
to generate bi-photon frequency combs through spontaneous four wave mixing
(SFWM) in high-quality-factor microresonators [10, 11, 12, 13]

Moreover in the telecom wavelength range, where the SOI is transparent and
efficient for SFWM, off-the-shelf filters, demultiplexers and modulators are
available and long-distance interaction between future quantum processors or
communication nodes can be achieved using existing classical telecom
infrastructures.

The frequency bins can be manipulated using non-linear processes like optical
frequency conversion[14, 15]. However, this approach may be hampered by
limited configurability and by the optical noise from the pump required to
generate the nonlinear phenomena. When generated in the telecom wavelengths,
near 1550 nm, frequency bins can be manipulated using off-the-shelf telecom
devices such as electro-optic phase modulators (EOM)[16, 17, 18] and
programmable filters (PF) [19, 5, 3, 7]. It has been shown that the
combination of two EOMs and a PF allows arbitrary qubit transformation if the
qubit mode spacing $\Delta f$ is equal to the Radio Frequency $\Omega$ (RF)
driving the EOMs [20, 21]. The reconfigurability of such quantum frequency
processors allows for various applications[22, 23, 24].

The limited bandwidth of standard telecom EOMs limits the maximum qubit
frequency spacing, $\Delta f$, to a few tens of gigahertz. In addition, low
$\Delta f$ boosts the spectral efficiency, i. e. maximizes the achievable
Hilbert dimension for a given available source bandwidth. Yet, achieving such
a narrow frequency spacing with integrated resonators is a challenging task.

Spectral separation of 40-50 GHz has been achieved using silicon nitride ring
resonators with a radius near 500 um, yielding a foot-print exceeding 1 mm2
[5, 25, 6]. Frequency separation of only $\Delta f$ $\approx$ 20 GHz has been
recently achieved by combining silicon rings with resonances separated by 200
GHz [7, 8]. This clever design nevertheless gives rise to variable qubit mode
spacing and limits the number of achievable frequency modes.

The generated frequency-entangled quantum states can be characterized with
electro-optic modulators and programmable filters. A single modulator does not
allow for a unitary control of a photonic qubit, but permits the quantum state
tomography of high dimensional states [5, 25, 6, 7, 8, 3]. Quantum state
tomography can be performed with parallelizable unitary operation on entangled
qubits if the spacing is lower than the RF bandwidth. In [1], frequency-
entangled qubits from a periodically poled lithium niobate SPDC source
filtered by a fibered etalon frequency comb with a spacing of 25 GHz and
quantum state tomography is performed with the [EOM-PF-EOM] configuration. The
parallelization of two gates allowed the control of two independent frequency
qubits.

In this paper, we report the parallelization of 34 tunable electro-optic
frequency domain quantum gates, all implemented with a single [EOM-PF-EOM]
configuration. We develop to this end a SOI spiral ring-resonator with a foot-
print below 0.05 mm2 and a frequency channel separation $\Delta f$ = 21 GHz.
The rings are fabricated using STMicroelectronics’ silicon photonics R&D and
manufacturing platform based on 300-mm SOI wafers and 193-nm-deep-ultraviolet
(DUV) lithography, ensuring compatibility with large-scale production[26]. The
narrow spectral separation allows photon pair generation through SFWM on more
than 70 frequency modes over a 1.4 THz. In addition, the 21 GHz spectral
separation allows implementing parallel and arbitrary qubit transformations,
based on [EOM-PF-EOM] scheme implemented with commercially available electro-
optic devices. We perform quantum state tomography on frequency domain
maximally entangled photons. Based on this approach we perform a proof-of-
concept-demonstration of a fully-connected quantum network, where 5 users
share a secure key with every other user, using frequency-bin entangled qubits
generated by our broadband silicon photon pair source and controlled by 34
parallel tunable electro-optic quantum gates.

##  2 21 GHz Silicon On Insulator spiral micro-resonator

Our photon pairs are generated through SFWM in a SOI micro-resonator (MR).

The integrated photonic devices were fabricated using STMicroelectronics’
silicon photonics R&D and manufacturing platform based on 300 mm SOI wafers.
The structures were defined with DUV lithography and transferred to the
silicon layer with reactive ion etching[26]. A 2 mm-thick PMMA layer was
deposited over the chip surface for protection. The thickness of the guiding
silicon layer is 300 nm. A waveguide width of 700 nm is chosen to yield small
anomalous dispersion near 1550 nm wavelength. The ring is shaped as spiral to
reduce the footprint of the device. The spiral waveguide length is set to 3.54
mm to yield a free-spectral-range (FSR) near 21 GHz, which determines the
frequency channel separation ($\Delta f$). The waveguide bendings follow a
Bezier trajectory to minimize losses due to mode mismatch between straight and
curved waveguides[27]. The spiral resonator has a size of 165 $\mu$m by 255
$\mu$m (<0.05 mm2). The sample temperature is tuned and stabilized by a
Peltier module at 25°$\pm 0.01$C. Cleaved SMF28 fibers, set at a 15° incidence
angle, are used to couple light in and out of the chip, through single-etch
grating couplers. The measured "fiber-to-fiber" insertion loss is 7.6 dB, i.
e. 3.8 dB per coupler including propagation loss. An experimental value of
$\text{FSR}=21.18\pm 0.85$ GHz is obtained by measuring the frequency spacing
of adjacent resonances from 1526.7 nm to 1565 nm. The resonator modes exhibit
a full width-at-half-maximum (FWHM) of 600 MHz, leading to a quality factor
$Q\simeq 3.10^{5}$. We tune the optical pump frequency $\omega_{p}$ on a
resonator mode so that signal and idler photons are emitted on symmetric
resonances at frequency $\frac{\omega_{p}}{2\pi}\pm n\cdot\text{FSR}$
($n\in\mathbb{N}$). As a result, the measured FSR varies less than 20 MHz over
the whole measured SFWM spectrum where the correlated pairs can be observed
from 1526.7 to 1553 nm. The quantum state $\ket{\phi}$ of the generated photon
pairs is therefore

$\ket{\phi}=\frac{1}{\sqrt{N}}\sum_{n=1}^{N}e^{i\alpha_{n}}\ket{I_{n}}\ket{S_{n}},$
(1)

where $I_{n}$ and $S_{n}$ stand for idler and signal frequencies with
$I_{n}=\frac{\omega_{p}}{2\pi}-n\cdot\text{FSR}$ and
$S_{n}=\frac{\omega_{p}}{2\pi}+n\cdot\text{FSR}$, respectively. $\alpha_{n}$
corresponds to the bi-photon residual spectral phase[6], within the 600 MHz
linewidth of the resonances. The ring resonator intrinsically produces
frequency-bin entangled qudits of dimension $N$ where $N$ is limited here by
the bandwidth of the programmable filters (5 THz) and phase matching
conditions.

Figure 1: a. Set up for measurement of the Joint Spectral Intensity. BP: Band-
pass filter, NF: Notch filter, PF: programmable filter, PC polarization
controller, SNSPDs: superconducting single-photon detectors, b. Joint spectral
intensity measurement for every accessible signal-idler pair from $n=3$ to
$n=83$.

To characterize the spectral correlation, we measure the Joint Spectral
Intensity of the biphoton state. We use the setup shown in Fig. 1.a. A
bandpass filter (BP) is used to filter out the amplified spontaneous emission
of the CW pump laser at 1540 nm up to 40 dB. Two fibered Bragg notch filters
with a bandwidth of 80 GHz are used at the output of the SOI MR to filter out
70 dB of the laser light and an additional filtering of the laser light is
done with the programmable filter, ensuring more than 100 dB pump rejection.
The spectrally correlated photon pairs are spatially separated by the same
programmable filter, sent to Superconducting Nanowire Single Photon Detectors
(SNSPDs) with a 70 % quantum efficiency, and analyzed using the Swabian
TimeTagger Ultra (TT). In this paper, the coincidence are taken in a 1ns
windows. The dead time of the detectors is $\approx$20 ns, allowing maximum
efficiency for the range of count rates observed in this work (in the order of
40 kHz). The combined time jitter of the SNSPDs and the TT is around 120 ps.

In Fig.1.b., we plot the number of coincidences between the signal and idler
paths as a function of the selected resonance number $n$ associated with the
frequency pairs $I_{n},S_{n}$. This spectral distribution of the photon pairs
corresponds to the diagonal elements of the Joint Spectral Intensity. We
deduce an internal brightness of 15.87$\pm 0.03\times 10^{6}$ pairs/s, and
measure an heralded $g^{(2)}(0)=0.057\pm 0.007$, for a on-chip power of 0.75
mW. The low FSR of our resonator allows access to a larger number of
resonances compared to previous works [6, 25, 5], providing possibilities of
parallelization of operations over a larger number of qubits and perspectives
for the processing of higher dimensional quantum states. The long cavity
nonetheless leads to more propagation losses. The overall decrease of the
coincidences with increasing $n$ is linked to the spectral transmission of the
grating couplers.

##  3 Frequency-domain Quantum State Tomography

In this section, we demonstrate quantum state tomography of a frequency
2-dimensional maximally entangled state produced by the 21 GHz SOI micro-
resonator. A first programmable filter PF1 (see Fig. 2) selects two adjacent
signal and idler mode pairs from the high-dimensional two-photon state of Eq.
1 coming out of the resonator to produce the biphoton maximally entangled
state $\frac{1}{\sqrt{2}}(\ket{I_{n}S_{n}}+\ket{I_{n+1}S_{n+1}})$.

To perform the tomography of this state, single qubit rotations in the
frequency domain are necessary. We implement those using EOMs and PFs as
demonstrated in [21] (see Appendix. A). The setup is shown in Fig.2 and uses
these devices sequentially in a [EOM-PF-EOM] configuration. RF driving of the
EOM at $\Omega$=FSR is possible here because the low FSR of our micro-
resonator is compatible with the 40 GHz bandwidth of the EOM. With this
configuration, the [EOM-PF-EOM] device shown on Fig. 2 can achieve parallel
independent manipulation of signal and idler photons.

Figure 2: Set up for the quantum state tomography. PF: Programmable filters,
EOM: Electro-optic phase modulators. Insets are the action of the programmable
filters on the frequency modes. PF1 is used both as an amplitude filter to
select the four modes of the two qubits, and as a phase gate implementing a
phase $\phi_{i}$ and $\phi_{s}$ on the frequency modes $I_{n}$ and $S_{n}$.
The boxed device implement Identity or Hadamard gate on the qubits. All the
projections required for the tomography are accessible with these two gates.
PF3 selects two modes $I_{p}$ and $S_{q}$, where p,q $\in${n,n+1}.

Fig.2 shows a simplified setup for the quantum state tomography. In the full
setup, polarizers are present before each EOM. Before each detector, a
polarizer is also added to control the input polarization and a fibered notch
filter around 900 nm is placed to suppress parasitic calibration light coming
from the PF.

Previously reported tomography in the spectral domain used a single EOM [25,
6, 5, 7]. Such a simple setup can be used when the frequency spacing between
two modes is too large to match the RF driving frequency $\Omega$ and then
corresponds to a multiple of $\Omega$. This setup has the advantage of low
loss but does not allow independent unitary operations on parallel qubits.  In
[1], quantum state tomography was performed using two parallel tunable quantum
gate with the same [EOM-PF-EOM] configuration. A Bayesian method based on
measurements only in the $\mathbb{Z}$ and $\mathbb{X}$ bases was used to
reconstruct the density matrix and a fidelity of 0.92$\pm$0.01 to a
$\ket{\Psi^{+}}$ entangled state was obtained.

Our logical qubits are defined as follows: $\ket{0^{X}_{n}}=\ket{X_{n}}$ and
$\ket{1^{X}_{n}}=\ket{X_{n+1}}$, where $X=S,\,I$ refers to the logical signal
or idler qubit (see Fig. 2). To perform the quantum state tomography, we need
to project the two qubits onto four state vectors belonging to three different
bases $\mathbb{Z}=\left\\{\ket{0^{X}_{n}},\ket{1^{X}_{n}}\right\\}$,
$\mathbb{X}=\left\\{\ket{+^{X}_{n}},\ket{-^{X}_{n}}\right\\}$ and
$\mathbb{Y}=\left\\{\ket{+i^{X}_{n}},\ket{-i^{X}_{n}}\right\\}$, where
$\ket{\pm^{X}_{n}}=\frac{1}{\sqrt{2}}\left(\ket{0^{X}_{n}}\pm\ket{1^{X}_{n}}\right)$
and $\ket{\pm i^{X}_{n}}=\frac{1}{\sqrt{2}}\left(\ket{0^{X}_{n}}\pm
i\ket{1^{X}_{n}}\right)$. The quantum gate (see Fig. 2) allows us to choose
between the $\mathbb{Z}$ and $\mathbb{X}$ bases. To access the $\mathbb{Y}$
basis, we use PF1 to apply a relative phase shift $\phi_{i}$ ($\phi_{s}$)
between the modes of the idler (signal) qubit. $C_{a,b}$ denotes the
coincidence numbers corresponding to the projections on vectors $\ket{a}$ and
$\ket{b}$ from the three different bases $\mathbb{X}$, $\mathbb{Y}$ and
$\mathbb{Z}$. Table 1 shows the recorded coincidences for the 16 projections
performed on the two-qubit state for n = 34.

Projections | Coinc. | Projections | Coinc.
---|---|---|---
$C_{0,0}$ | 1548 | $C_{+,0}$ | 716
$C_{0,1}$ | 36 | $C_{+,1}$ | 767
$C_{0,+}$ | 622 | $C_{+,+}$ | 1275
$C_{0,+i}$ | 663 | $C_{+,+i}$ | 608
$C_{1,0}$ | 22 | $C_{+i,0}$ | 837
$C_{1,1}$ | 1553 | $C_{+i,1}$ | 695
$C_{1,+}$ | 692 | $C_{+i,+}$ | 723
$C_{1,+i}$ | 664 | $C_{+i,+i}$ | 42
Table 1: Coincidences for the two-photon projections $C_{a,b}$ integrated for
125 seconds, in a coincidence window of 1 ns.

Fig. 3 shows the reconstructed density matrix. Comparing it to the density
matrix of a maximally entangled state
$\ket{\phi^{+}}=\frac{1}{\sqrt{2}}\left(\ket{I_{34},S_{34}}+\ket{I_{35},S_{35}}\right)$
we obtain a fidelity $\mathcal{F}=0.961\pm 0.007$. The errors are calculated
using Monte-Carlo methods.

Figure 3: Numerical reconstruction of the experimental density matrix of a
two-qubit frequency-bin entangled state generated by the SOI resonator + PF1.
a.: real part, and b.: imaginary part.

Having checked the fidelity of the produced frequency-entangled state, we take
advantage of the broad bandwidth of our source and the versatility of the
[EOM-PF-EOM] setup to parallelize the measurement of such high quality
entangled states by performing 34 quantum gates on 17 frequency-entangled
qubit pairs, in a first implementation of frequency-encoded quantum
communication protocol.

##  4 Frequency-bin entangled photons for fully connected networks

In this section, we show proof-of-principle of a fully connected network of up
to 5 users in which every user can share a secure key with every other user,
using frequency-bin entangled qubits generated by our broadband photon pair
source. In previous implementations or proof-of-principle experiments based on
time-bin or polarization-based entangled sources [28, 29, 30, 31], the main
challenge was phase or polarization stabilization. In addition to previously
mentionned assets, the frequency degree of freedom that we harness here has
the advantage of requiring no phase stabilization for superposition analysis.

In this setup, polarization control is nevertheless necessary when entering
the modulators. Recent advances have proposed schemes using polarization
diversity electro-optic modulators (EOMs) to address this challenge [32],
which is a relevant and additional asset for real world applications. As we
encode frequency qubits on distinct pairs of adjacent frequencies, one can use
a PF as a demultiplexer to spectrally separate and distribute the pairs to
respective users to create a network. For each mode pair, compensation for the
bi-photon residual phase is required in order to produce the desired
$\ket{\phi^{+}}$ states [6].

To validate our scheme, we first show that the parallelization of the quantum
gate enables measurement of entanglement for qubits separated by two guards
modes, allowing for a measured crosstalk $\leq 10^{-3}$ between adjacent
quantum gates (see appendix A). Fig. 4 shows the fidelity for the accessible
frequency-bin entangled pairs. We start from the $n=10$ resonance from the
pump frequency, to avoid any residual pump laser leakage. The fidelity is
higher than $0.8$ for 14 pairs. The few lower values are related to the dips
observed in the coincidence spectrum shown in Fig. 1 around the 50th and 70th
resonances from the pump.

Figure 4: Fidelity to a maximally entangled state for several frequency-bin
entangled photon pairs. The x axis correspond to the distance of the frequency
qubit to the pump frequency

Important metrics for quantum networks are the key rate and qubit error rate.
We deduce these parameters from our coincidence measurements, using the method
proposed in [33]. We use the coincidences in the $\mathbb{Z}$ basis
($C_{0,0}$, $C_{0,1}$, $C_{1,0}$, $C_{1,1}$) and the $\mathbb{X}$ basis
($C_{+,+}$, $C_{+,-}$, $C_{-,+}$ and $C_{-,-}$) to compute the raw coincidence
rate, QBER and sifted key rate (see appendix B).

Figure 5:  a.: Raw coincidences (bars) and qubit error rate (QBER) (dots)
between two users, and b.: sifted key rate, calculated using the method in
[29] as a function of $n$, spectral detuning from the pump

Fig. 5.a shows, for each accessible pair, the number of raw coincidences
$R_{raw}$ and the qubit error rate QBER. By comparing it with Fig. 4, we see
that as expected, the qubits with a lower raw key rate (and higher QBER) are
also the ones with the lower fidelity to an entangled state. A given pair
achieves secure quantum communication only if the corresponding QBER is lower
than the positive key rate threshold of 11% [34]. Among the 17 accessible
pairs, only 12 satisfy this condition. A sifted key rate in bit/s can also be
derived from the raw rate $R_{raw}$ using the method in [29] as depicted in
Fig. 5.b. The inset in Fig. 5.b. shows how the distribution of 10 photon pairs
is used to create a node-free quantum network of 5 users.

In this proof of principle experiment, the achieved key rates (0.5 to 2.5
bits/second) are limited by the performances of the source, as well as the
insertion loss of the devices (14 dB for the 3 PFs and the 2 EOMs).
Integrating these components on the same chip would help to limit losses and
improve the overall insertion loss of the [EOM-PF-EOM] configuration to
73%[35]. Several methods are proposed to achieve the integration of elements
allowing the manipulation of frequency-bin qubits [36, 37]. It is also worth
noting that the number of accessible pairs could be increased by setting the
pump frequency at the center of the PF bandwidth.

##  5 Conclusion

In this paper, we introduce a SOI micro-resonator for producing photon pairs
at telecom wavelengths through SFWM and encoding frequency-bin qubits on
adjacent pairs of resonances. Due to the broadband emission of the source and
the 21 GHz FSR, we can generate up to 17 pairs of frequency entangled qubits.
We manipulate these qubits using quantum gates based on a single set of
devices composed of one programmable filter between two electro-optic phase
modulators, enabling independent control of each qubit. This leverages the
ability to use these reprogrammable quantum gates for quantum information
applications.

We demonstrate quantum state tomography with these quantum gates on parallel
qubits, and assess a fidelity to a $\ket{\phi^{+}}$ entangled state of over
0.8 for 14 of them.

Finally, we demonstrate a local proof-of-concept of a fully connected network
and compute the key rate and QBER of 17 photon pairs, taking into account the
threshold for attacks on the quantum channels and error correction. We can
distribute 10 photon pairs and create a fully-connected network of up to 5
users.

To scale up this approach, it is possible to encode and manipulate higher-
dimensional quantum states with minimal additional resources using this setup,
for example, by adding a single additional harmonic to the fundamental RF tone
that drives the EOMs. Moreover, on-chip integration of electro-optic devices
and programmable filters are a promising lead to reduce the losses (currently
around 8.5 dB for our quantum gates implementation) [35]. On-chip integration
has been proposed, for instance, using a lithium niobate resonator driven by
RF signals [36, 37]. Our work thus offers perspectives for scalable frequency-
domain architectures for high-dimensional and resource-efficient quantum
communications

## Ackowledgments

This work has been supported by Region Ile-de-France in the framework of DIM
SIRTEQ. Lorenzo M. Procopio acknowledges the European Union’s Horizon 2020
research under the Marie Skłodowska-Curie grant agreement No 800306.

## References

  * [1] H.-H. Lu, J. M. Lukens, N. A. Peters, B. P. Williams, A. M. Weiner, and P. Lougovski, “Quantum interference and correlation control of frequency-bin qubits,” Optica 5, 1455–1460 (2018). [lu18].
  * [2] P. Imany, O. D. Odele, M. S. Alshaykh, H.-H. Lu, D. E. Leaird, and A. M. Weiner, “Frequency-domain Hong–Ou–Mandel interference with linear optics,” Optics Letters 43, 2760–2763 (2018). [Imany18].
  * [3] M. C. Ponce, A. L. M. Muniz, M. Huber, and F. Steinlechner, “Unlocking the frequency domain for high-dimensional quantum information processing,” (2022). ArXiv:2206.00969 [quant-ph].
  * [4] A. Khodadad Kashi and M. Kues, “Spectral Hong–Ou–Mandel Interference between Independently Generated Single Photons for Scalable Frequency-Domain Quantum Processing,” Laser & Photonics Reviews 15, 2000464 (2021). [Khodadad21].
  * [5] M. Kues, C. Reimer, P. Roztocki, L. R. Cortés, S. Sciara, B. Wetzel, Y. Zhang, A. Cino, S. T. Chu, B. E. Little, D. J. Moss, L. Caspani, J. Azaña, and R. Morandotti, “On-chip generation of high-dimensional entangled quantum states and their coherent control,” Nature 546, 622–626 (2017).
  * [6] H.-H. Lu, K. V. Myilswamy, R. S. Bennink, S. Seshadri, M. S. Alshaykh, J. Liu, T. J. Kippenberg, D. E. Leaird, A. M. Weiner, and J. M. Lukens, “Full quantum state tomography of high-dimensional on-chip biphoton frequency combs with randomized measurements,” arXiv:2108.04124 [physics, physics:quant-ph] (2021). ArXiv: 2108.04124.
  * [7] M. Clementi, F. A. Sabattoli, M. Borghi, L. Gianini, N. Tagliavacche, H. El Dirani, L. Youssef, N. Bergamasco, C. Petit-Etienne, E. Pargon, J. E. Sipe, M. Liscidini, C. Sciancalepore, M. Galli, and D. Bajoni, “Programmable frequency-bin quantum states in a nano-engineered silicon device,” Nature Communications 14, 176 (2023).
  * [8] M. Borghi, N. Tagliavacche, F. A. Sabattoli, H. E. Dirani, L. Youssef, C. Petit-Etienne, E. Pargon, J. E. Sipe, M. Liscidini, C. Sciancalepore, M. Galli, and D. Bajoni, “A reconfigurable silicon photonics chip for the generation of frequency bin entangled qudits,” (2023). ArXiv:2301.08475 [quant-ph].
  * [9] J. Wang, F. Sciarrino, A. Laing, and M. G. Thompson, “Integrated photonic quantum technologies,” Nature Photonics 14, 273–284 (2020).
  * [10] X. Zhang, Y. Zhang, C. Xiong, and B. J. Eggleton, “Correlated photon pair generation in low-loss double-stripe silicon nitride waveguides,” Journal of Optics 18, 074016 (2016).
  * [11] F. Samara, A. Martin, C. Autebert, M. Karpov, T. J. Kippenberg, H. Zbinden, and R. Thew, “High-rate photon pairs and sequential Time-Bin entanglement with Si ${}_{\textrm{3}}$ N ${}_{\textrm{4}}$ microring resonators,” Optics Express 27, 19309 (2019).
  * [12] Z. Yin, K. Sugiura, H. Takashima, R. Okamoto, F. Qiu, S. Yokoyama, and S. Takeuchi, “Frequency correlated photon generation at telecom band using silicon nitride ring cavities,” Optics Express 29, 4821 (2021).
  * [13] S. Clemmen, K. P. Huy, W. Bogaerts, R. G. Baets, P. Emplit, and S. Massar, “Continuous wave photon pair generation in silicon-on-insulator waveguides and ring resonators,” Optics Express 17, 16558 (2009).
  * [14] T. Kobayashi, R. Ikuta, S. Yasui, S. Miki, T. Yamashita, H. Terai, T. Yamamoto, M. Koashi, and N. Imoto, “Frequency-domain Hong–Ou–Mandel interference,” Nature Photonics 10, 441–444 (2016).
  * [15] M. G. Raymer, S. J. van Enk, C. J. McKinstrie, and H. J. McGuinness, “Interference of two photons of different color,” Optics Communications 283, 747–752 (2010).
  * [16] L. Olislager, E. Woodhead, K. Phan Huy, J.-M. Merolla, P. Emplit, and S. Massar, “Creating and manipulating entangled optical qubits in the frequency domain,” Physical Review A 89, 052323 (2014).
  * [17] L. Olislager, J. Cussey, A. T. Nguyen, P. Emplit, S. Massar, J.-M. Merolla, and K. P. Huy, “Frequency-bin entangled photons,” Physical Review A 82, 013804 (2010).
  * [18] M. Bloch, S. W. McLaughlin, J.-M. Merolla, and F. Patois, “Frequency-coded quantum key distribution,” Optics Letters 32, 301 (2007).
  * [19] J. M. Lukens and P. Lougovski, “Frequency-encoded photonic qubits for scalable quantum information processing,” Optica 4, 8–16 (2017).
  * [20] H.-H. Lu, J. M. Lukens, N. A. Peters, O. D. Odele, D. E. Leaird, A. M. Weiner, and P. Lougovski, “Electro-Optic Frequency Beamsplitters and Tritters for High-Fidelity Photonic Quantum Information Processing,” Physical Review Letters 120, 030502 (2018). ArXiv: 1712.03992.
  * [21] H.-H. Lu, E. M. Simmerman, P. Lougovski, A. M. Weiner, and J. M. Lukens, “Fully Arbitrary Control of Frequency-Bin Qubits,” Physical Review Letters 125, 120503 (2020).
  * [22] H.-H. Lu, J. M. Lukens, B. P. Williams, P. Imany, N. A. Peters, A. M. Weiner, and P. Lougovski, “A controlled-NOT gate for frequency-bin qubits,” npj Quantum Information 5, 1–8 (2019).
  * [23] H.-H. Lu, N. Klco, J. M. Lukens, T. D. Morris, T. D. Morris, A. Bansal, A. Ekström, G. Hagen, G. Hagen, T. Papenbrock, T. Papenbrock, A. M. Weiner, M. J. Savage, and P. Lougovski, “Subatomic Many-Body Physics Simulations on a Quantum Frequency Processor,” in _Conference on Lasers and Electro-Optics (2019), paper FTh3A.6,_ (Optical Society of America, 2019), p. FTh3A.6.
  * [24] H.-H. Lu, N. B. Lingaraju, D. E. Leaird, A. M. Weiner, and J. M. Lukens, “High-dimensional discrete Fourier transform gates with a quantum frequency processor,” Optics Express 30, 10126 (2022).
  * [25] P. Imany, J. A. Jaramillo-Villegas, O. D. Odele, K. Han, D. E. Leaird, J. M. Lukens, P. Lougovski, M. Qi, and A. M. Weiner, “50-GHz-spaced comb of high-dimensional frequency-bin entangled photons from an on-chip silicon nitride microresonator,” Optics Express 26, 1825–1840 (2018).
  * [26] F. Boeuf, S. Crémer, E. Temporiti, M. Ferè, M. Shaw, C. Baudot, N. Vulliet, T. Pinguet, A. Mekis, G. Masini, H. Petiton, P. L. Maitre, M. Traldi, and L. Maggi, “Silicon photonics r&d and manufacturing on 300-mm wafer platform,” J. Lightwave Technol. 34, 286–295 (2016).
  * [27] P. T. Do, C. Alonso-Ramos, X. Le Roux, I. Ledoux, B. Journet, and E. Cassan, “Wideband tunable microwave signal generation in a silicon-micro-ring-based optoelectronic oscillator,” Scientific Reports 10, 6982 (2020).
  * [28] S. K. Joshi, D. Aktas, S. Wengerowsky, M. Lončarić, S. P. Neumann, B. Liu, T. Scheidl, G. C. Lorenzo, Z. Samec, L. Kling, A. Qiu, M. Razavi, M. Stipčević, J. G. Rarity, and R. Ursin, “A trusted-node-free eight-user metropolitan quantum communication network,” Science Advances 6, eaba0959 (2020). ArXiv: 1907.08229.
  * [29] F. Appas, F. Baboux, M. I. Amanti, A. Lemaítre, F. Boitier, E. Diamanti, and S. Ducci, “Flexible entanglement-distribution network with an AlGaAs chip for secure communications,” npj Quantum Information 7, 1–10 (2021).
  * [30] E. Fitzke, L. Bialowons, T. Dolejsky, M. Tippmann, O. Nikiforov, T. Walther, F. Wissel, and M. Gunkel, “Scalable Network for Simultaneous Pairwise Quantum Key Distribution via Entanglement-Based Time-Bin Coding,” PRX Quantum 3, 020341 (2022).
  * [31] S. Wengerowsky, S. K. Joshi, F. Steinlechner, H. Hübel, and R. Ursin, “An entanglement-based wavelength-multiplexed quantum communication network,” Nature 564, 225–228 (2018).
  * [32] O. E. Sandoval, N. B. Lingaraju, P. Imany, D. E. Leaird, M. Brodsky, and A. M. Weiner, “Polarization diversity phase modulator for measuring frequency-bin entanglement of a biphoton frequency comb in a depolarized channel,” Optics Letters 44, 1674 (2019).
  * [33] C. Autebert, J. Trapateau, A. Orieux, A. Lemaître, C. Gomez-Carbonell, E. Diamanti, I. Zaquine, and S. Ducci, “Multi-user quantum key distribution with entangled photons from an AlGaAs chip,” Quantum Science and Technology 1, 01LT02 (2016).
  * [34] N. Lütkenhaus, “Security against individual attacks for realistic quantum key distribution,” Physical Review A 61, 052304 (2000).
  * [35] B. E. Nussbaum, A. J. Pizzimenti, N. B. Lingaraju, H.-H. Lu, and J. M. Lukens, “Design Methodologies for Integrated Quantum Frequency Processors,” (2022). ArXiv:2204.12320 [physics, physics:quant-ph].
  * [36] Y. Hu, M. Yu, D. Zhu, N. Sinclair, A. Shams-Ansari, L. Shao, J. Holzgrafe, E. Puma, M. Zhang, and M. Lončar, “On-chip electro-optic frequency shifters and beam splitters,” Nature 599, 587–593 (2021).
  * [37] S. Buddhiraju, A. Dutt, M. Minkov, I. A. D. Williamson, and S. Fan, “Arbitrary linear transformations for photons in the frequency synthetic dimension,” Nature Communications 12, 2401 (2021).

##  Appendix A Frequency-domain quantum gates

###  A.1 Gate characterization

In this section, we use a combination of EOMs, and PFs to create frequency-bin
quantum gates[20, 21], that can be parallelized to a large number of qubits.
The setup in Fig. 6.a. presents the quantum gate, composed of two electro-
optic phase modulators iXblue MPZ-ln-40 (EOM1 and EOM2), driven by a multi-
channel RF sine wave generator Anapico APMS33G, and one phase-only
programmable filter Finisar WaveShaper 1000A (PF). The overall insertion loss
of the quantum gate is around 9 dB from the input of the first modulator to
the output of the second. The time dependent phase shift $\phi(t)$ applied by
the EOM to the optical wave is proportional to the sine wave produced by the
RF generator :

$\phi(t)=\mu\cos\left(\Omega t+\theta\right),$ (2)

where the RF frequency $\Omega$ determines the mode separation between the
generated frequency modes $\ket{\omega_{n}}=\ket{\omega_{0}+n\Omega}$, and
$\mu$ is the modulation index, proportional to the RF voltage V :
$\mu=\pi\frac{V}{V_{\pi}}$. $V_{\pi}$ is the voltage giving rise to a $\pi$
phase shift. The specific settings of the modulators and programmable filter
allowing to implement the quantum gate (tomography) are the following: both
modulators must be driven with the modulation index $\mu=0.81$ and the
relative phase between them must be set to $\pi$. This operation is achieved
by using the multi-channel available on the generator, on which we can set a
relative phase, ranging from 0 to $\pi$. It is noteworthy that compensation of
the dispersion is required in order to keep this relative RF phase shift
constant at all optical frequencies within the PF bandwidth. Another PF is
used as a wavelength selecting switch (WSS) after the quantum gate to select
the frequency mode to be detected by the IR photo-diode (PD). Fig. 6.b. Shows
the principle of the operation performed. We define a frequency-bin qubit as a
qubit for which the information is encoded on two frequency modes, here
generically called $\ket{\omega_{0}}$ and $\ket{\omega_{1}}$. The goal of the
quantum gate, is apply a controlled rotation on the qubit, here represented by
the coefficients $\beta$ and $\delta$. To characterize the quantum operation,
we use the fidelity $\mathcal{F}$ and the success probability $\mathcal{P}$
[20]:

$\mathcal{F}=\frac{Tr(W^{\dagger}T)Tr(T^{\dagger}W)}{Tr(W^{\dagger}W)Tr(T^{\dagger}T)},\;\;\mathcal{P}=\frac{Tr(W^{\dagger}W)}{Tr(T^{\dagger}T)}$
(3)

Where the fidelity corresponds to how close the realiezd operation ($W$) is to
the ideal case ($T$), and the success probability measures the unitarity of
the operation, accounting for the energy loss towards unwanted frequency
modes.

Fig. 6.c. represents the phase pattern applied by the PF in order to tune the
operation of the gate. A step phase of height $\alpha$ is applied between the
two qubit modes. In order to characterize the spectral processing implemented
by this single qubit gate, we use coherent light as it is sufficient to
characterize a single qubit gate.

Figure 6: Classical characterization of the quantum gate. a. : principle of a
frequency-domain operation. b. : principle of a frequency domain quantum gate.
c: phase pattern applied by the PF d. : measured tunability of the quantum
gate operation

6d. shows the effect of the phase difference $\alpha$ applied by the PF
between the two qubit modes on the quantum gate, evolving from Identity
($\alpha=0$) to Hadamard operation ($\alpha=\pi$). The success probability
experimentally reaches $\mathcal{P}=0,95$ implies that the intrinsic
transmission of the quantum gate is slightly below 0.5 but the splitting
between the two modes is balanced, leading to a Fidelity $\mathcal{P}=0,99$.
The success probability does not take into account the insersion loss of each
devices, around 8.5 dB (3.5 for the PF, and 2.5 for each modulators).

###  A.2 Gate parallelization

In these experiments, the mode spacing is taken to be 21 GHz. This is
compliant with the resolution of the programmable filters (10 GHz), and the
accessible modulation bandwidth of commercial EOMs (40 GHz). The phase pattern
depicted in Fig. 6.c. shows a step phase applied between the two modes of the
qubit. By applying a phase change over several pairs of frequency modes, it is
possible to parallelize independent quantum operations over several frequency-
bin qubits.

Figure 7:  a : light coupling from mode $\ket{\omega_{0}}$ (black) or
$\ket{\omega_{1}}$ (red) into 9 neighboring modes. The grey shaded areas
represent the guard modes that are not used for parallelization. b.
Transmission measurement for 9 frequency modes when a Hadamard gate is applied
to two qubits separated by two guard modes.

It is important, for parallelization, that two consecutive gates are isolated
from each other. To determine what is the minimum desired spacing between each
qubits, we send light through the quantum gate, tuned to realize the Hadamard
transformation. We define the crosstalk between the qubit as the portion of
light being converted from one qubit to another. The experimental results are
shown in Fig. 7.a, where the black(red) lines correspond to the measured
light, when we input light in the mode $\omega_{0}$($\omega_{1}$). Fig. From
this figure, we see that two separation modes between consecutive operations
are enough to ensure less than $10^{-3}$ crosstalk between two qubits.
Fig.7.b. shows the intensity transmission matrix of a 10 dimensional frequency
space, where we apply two Hadamard gates on the frequency modes
$\omega_{0},\,\omega_{1}$, and $\omega_{4},\,\omega_{5}$. This leverages the
possibility to operate in parallel on many qubits. To allow the
parallelization of the quantum gates for the whole C-band, it is necessary to
compensate for the dispersion in the fiber between the two modulators. To this
end, the PF in the middle of the [EOM-PF-EOM] setup applies a supplementary
phase shift over the whole PF spectrum corresponding to a negative dispersion
of -0.4 ps/nm.

##  Appendix B Evaluating the performances for quantum key distribution

We use the coincidences in the $\mathbb{Z}$ basis ($C_{0,0}$, $C_{0,1}$,
$C_{1,0}$, $C_{1,1}$) and the $\mathbb{X}$ basis ($C_{+,+}$, $C_{+,-}$,
$C_{-,+}$ and $C_{-,-}$) to compute the raw coincidence rate, QBER and sifted
key rate. The raw numbers of coincidences in these bases are equal to [29]

$\displaystyle
C_{\mathbb{Z}}=\frac{1}{2}\left(C_{0,0}+C_{0,1}+C_{1,0}+C_{1,1}\right)$ (4)
$\displaystyle
C_{\mathbb{X}}=\frac{1}{2}\left(C_{+,+}+C_{+,-}+C_{-,+}+C_{-,-}\right).$ (5)

If the integration time is $\tau$, the raw coincidence rate is then

$R_{raw}=\frac{1}{2}\frac{\left(C_{\mathbb{Z}}+C_{\mathbb{X}}\right)}{\tau}.$
(6)

The qubit error rate (QBER) is the ratio of accidental to total coincidences
over the two bases

$e=\frac{C_{0,1}+C_{1,0}+C_{+,-}+C_{-,+}}{C_{\mathbb{Z}}+C_{\mathbb{X}}}$ (7)

We compute the sifted key rate using formulas in [33] accounting for error
correction.


# Academic Search Engines: Constraints, Bugs, and Recommendations

Zheng Li  Queen’s University Belfast18 Malone RdBelfastNorthern IrelandUKBT9
5AF<EMAIL_ADDRESS>0000-0002-9704-7651 and  Austen Rainer  Queen’s
University Belfast18 Malone RdBelfastNorthern IrelandUKBT9 5AF
<EMAIL_ADDRESS>0000-0001-8868-263X

###### Abstract.

Background: Academic search engines (i.e., digital libraries and indexers)
play an increasingly important role in systematic reviews however these
engines do not seem to effectively support such reviews, e.g., researchers
confront usability issues with the engines when conducting their searches.
Aims: To investigate whether the usability issues are bugs (i.e., faults in
the search engines) or constraints, and to provide recommendations to search-
engine providers and researchers on how to tackle these issues. Method: Using
snowball-sampling from tertiary studies, we identify a set of 621 secondary
studies in software engineering. By physically re-attempting the searches for
all of these 621 studies, we effectively conduct regression testing for 42
search engines. Results: We identify 13 bugs for eight engines, and also
identify other constraints. We provide recommendations for tackling these
issues. Conclusions: There is still a considerable gap between the search-
needs of researchers and the usability of academic search engines. It is not
clear whether search-engine developers are aware of this gap. Also, the
evaluation, by academics, of academic search engines has not kept pace with
the development, by search-engine providers, of those search engines. Thus,
the gap between evaluation and development makes it harder to properly
understand the gap between the search-needs of researchers and search-features
of the search engines.

academic search engine, regression testing, systematic review, usability
evaluation, user interaction

††copyright: none††ccs: Information systems Search interfaces††ccs: General
and reference Evaluation††ccs: General and reference Empirical studies

##  1\. Introduction

Academic search engines are increasingly important for research in all
disciplines. Correspondingly, search engine evaluation is significant both to
help users (researchers) to select appropriate search engines and to help
producers to improve those engines (Vaughan, 2004). (We use the term “search
engine” to refer to both content providers, such as the ACM Digital Library,
and context indexers, such as Google Scholar.) In the empirical software
engineering domain, evidence-based software engineering (EBSE) heavily relies
on academic search engines for evidence collection. Evaluations of search
engines are sometimes provided through researchers’ own experience. For
example, it has been reported that different search engines significantly vary
in scope, underlying model, user interface, syntax and filtering mechanisms
(Dybå et al., 2007; Shakeel et al., 2018). Such diversity of engine, with the
consequential constraints from such diversity, frequently requires that the
“standard” search string constructed by the researcher has to be modified and
adapted for each engine.

Researchers’ evaluations are often, but not always, shared as “side
information” in the secondary studies. Such evaluations are sparse on details,
often being more like short experience reports. Although researchers
individually may report the problems they encounter to the search-engine
developers, as suggested by Kruger et al. (Krüger et al., 2020), there will
still be a lack of holistic view for the search engine developers to
understand the professional users’ needs and to improve their products and
services. We therefore decided to perform an extensive evaluation of academic
search engines, in order to provide systematic feedback and evidence-based
recommendations to both researchers and the relevant engine developers. For
the current paper, we focus on the usability of search engines, as usability
and its evaluation are widely recognised as important components of search
engine development (Vrana, 2007; Khoo et al., 2012).

As with our previous research (Li, 2021), we adopted a snowball sampling
technique to identify secondary studies, and obtained 621 valid and unique
samples. These 621 studies employed a total of 42 unique digital libraries and
indexers. For each of these 621 studies, we re-ran the longest search string
in terms of the number of Boolean operators. Each re-run constitutes one “user
interaction”. Considering the diversity in search strings, settings and
operations of the 621 studies, each of our user interactions can be viewed as
independent test cases for search engine evaluation. Thus, our collection of
test cases may be treated as an extensive regression testing, to gain first-
hand experience and close observations on search engine usability.111The
detailed testing logs are shared at http://doi.org/10.5281/zenodo.4447488

This paper reports the results from our regression testing. In addition to
listing search engine constraints, we highlight 13 bugs in eight engines.
These bugs do not appear to have been reported previously. By comparing the
originally published “user interactions” with our own experience, we reveal
some of the changes in search engines and suggest future directions for the
whole search-engine community. Accordingly, this work makes a threefold
contribution.

  * •

By fixing the bugs and taking the evidence-based recommendations, the academic
search engine developers can improve their products and services to better
satisfy the diverse needs of different users. As argued in (Li, 2021), all the
stakeholders in the scientific community should take responsibility for
improving the research quality rather than purely emphasise the obligation of
researchers.

  * •

Before the search engine developers fix those bugs and relax the constraints,
researchers can refer to our testing results to mitigate the user interaction
limits. The available workarounds emerged from this work include avoiding
EBSE-unfriendly search engines, bypassing the known bugs, and designing search
strings that tolerate the existing engine constraints.

  * •

By raising the problems of the current search engines, this work can act as a
bridge between the testing community and the search engine community.
Considering the inefficiency and fatigue effect of using a large number of
random test cases, we argue the need of a systematic test case suite for
evaluating the engine usability, and our identified bugs can inspire and
motive the design of efficient test cases.

##  2\. Related Work

With the growth of search engines, researchers have developed various models
and frameworks to facilitate evaluating their usability. For example, by
summarising the existing discussions about the term “usability” in the search
engine context, Jeng (Jeng, 2005) proposed a usability evaluation model that
comprises four dimensions, such as effectiveness, efficiency, satisfaction,
and learnability. By focusing on the user interfaces, Lai et al. (Lai et al.,
2014) developed a reference model and identified top five criteria (namely
ease of use, searching, language, presentation, and design) for evaluating
search engines. By defining the quality components of usability, Nielsen
(Nielsen, 2012) pointed out another five criteria for usability testing,
including learnability, efficiency, memorability, errors, and satisfaction.
Our work only focuses on one evaluation dimension, i.e. effectiveness, as
explained and justified later.

It has been identified that those reference models and criterion frameworks
mainly contribute to the theoretical landscape from the researcher’s
perspective (Li and Liu, 2019), while they offer little handy guidance to
practitioners about evaluating the usability of search engines in action. In
fact, it is argued that the usability assessment should be an evidence-based
practice, via “carefully observing users interacting with the system in a
realistic way” (Kelly, 2014). Note that the widely employed usability testing
methods like questionnaire and survey (e.g., (Xie, 2008)) do not satisfy this
argument in terms of close observations. In contrast, our research is an
empirical study that focuses on the real-time interaction between the user and
search engines (i.e. evidence-based practice), rather than on the user’s post-
usage feedback.

Some other researchers also emphasised user interactions for academic search
engine evaluation, and they further clarified information resource, interface,
and tasks as three interaction components (Li and Liu, 2019). However, the aim
of their research was to explore the relationships between the user
interaction and the search engine evaluation. Differently, our research mainly
aims to explore evidence-based recommendations for engine developers to
improve their products and services. In particular, although our empirical
study replicates search activities of 621 secondary studies, the replication
only enables real-time observation on limited users (i.e. the authors).
Therefore, this research reflects only one attribute of usability –
effectiveness (i.e. how usable) that is concerned with functional completion
of user tasks (Buchanan and Salako, 2009). There is no doubt that the other
non-functional attributes (i.e. how easy to use) would require observations on
the real-time interaction of a crowd of users.

In the context of EBSE, there are early research efforts dedicated to
investigating user interactions against academic search engines (Bailey et
al., 2007; Chen et al., 2010). However, these investigations only chose
limited engines, and mainly aimed to raise the researchers’ understanding and
awareness of the inconsistent or unpredictable engine behaviours. A typical
example is the development of a caution guide for systematic reviewers, based
on the features of five search engines only (Singh and Singh, 2017). To the
best of our knowledge, this research is the first investigation that conducts
extensive user interaction testing, both in terms of number of tests and the
number of search engines tested.

##  3\. Regression Testing of the Search Engine Constraints

Table 1. Current Constraints of Academic Search Engines String Length Constraints |  Boolean Operation Constraints |  Search Field Constraints
---|---|---
• DBLP has a maximum limit of 127 characters. • Google Scholar has a maximum limit of 256 characters. • IGI Global has a maximum limit of 100 characters. • JSTOR has a maximum limit of 200 characters. • ScienceDirect has a maximum limit of 500 characters. • SpringerLink does not work well with lengthy search strings (about 1800 characters). |  • ACM DL does not support the proximity operators. • Google Scholar does not recognise wildcards and the Boolean operator AND in its Title search. • IEEE Xplore has a maximum limit of seven wildcards in one string. • ScienceDirect does not support wildcard search. • ScienceDirect has a maximum limit of eight Boolean operators per search field. • Web of Science (WoS) supports left-hand wildcard only for the Topic, Title, and Identifying Code searches. |  • Google Scholar and SpringerLink do not support zonal settings, except for the Title search. • ScienceDirect does not support exclusive abstract search or keywords search. • ScienceDirect, SpringerLink and Wiley Online Library (Wiley OL) do not support command line search. • SpringerLink’s title search is not recommended, as it does not work as expected. • Wiley OL does not support disjunction between multiple search fields.

We sampled 621 EBSE secondary studies and tried replicating the reported user
interactions, i.e. searches. Our first-hand replication trials can be viewed
as regression testing of the relevant search engines. The sampling methodology
and rationale remain the same as our previous work (Li, 2021). To avoid
duplication, and due to publication constraints, we do not repeat them here.
Although we do not report the methodology we do explain, in Section 4, how to
reproduce the bugs we have identified.

After excluding the inapplicable cases (e.g., the retired or inaccessible
search engines), we conducted tests on 42 digital libraries and indexing
platforms. We notice that not all the published user interactions are
replicable, even by trying to adapt the searches and configurations to the
individual interfaces of different search engines. By collecting and analysing
the reasons that hindered us from replicating the previous practices, we
identify concerning differences, even conflicts, between the researchers’
needs and the search engine constraints.

For example, to maximise the coverage of the relevant primary studies, the de
facto guidelines (Kitchenham and Charters, 2007; Petersen et al., 2015) for
secondary studies generally require exhaustively selecting and combining
search terms. This will inevitably result in long search strings.
Unfortunately, the well-designed search strings could not be acceptable or
executable in practice, due to search engines’ string length limits and/or
Boolean operation limits. Similar to the Boolean operation limits, the pre-
designed search fields on the engine interfaces may also constrain the
combinations of search terms. Therefore, we summarise these engine constraints
both researchers and practitioners should be aware of, as listed in Table 1.

In particular, we also identified three digital libraries that are generally
unhelpful, even disruptive to, systematic reviews:

  * •

AIS eLibrary is not recommended for complex search strings, due to its bug of
parentheses interpretation.

  * •

CiteSeerX is not recommended in general, due to its unexplainable search
behaviours.

  * •

DBLP is not recommended in general, due to its unique query syntax and the
lack of advanced search.

Note that although CiteSeerX is statistically one of the popular digital
libraries employed by the existing EBSE researchers (in 10.6% of, or 66 out of
621, sample studies), we do not recommend it to be used, because of its
strange behaviours during the user interaction (see Bug 4.3).

##  4\. Bugs of the Tested Search Engines

Our tests also found 13 bugs in eight engines. Since “accurately recording the
steps leading to a bug is an absolute must” (Hamilton, 2017), we use concise
demonstrations (presented in boxes) to describe each bug and to show how it
can be reproduced. Whilst researchers wait on search engine developers to fix
these bugs, researchers need to be aware of, and avoid, these bugs when
employing the relevant search engines.

###  4.1. The Bugs of ACM Digital Library

Although ACM DL offers flexible field-specific search including an Edit Query
window, it is not convenient to manipulate complex multi-field search, not to
mention that the Edit Query window is invisible by default. Unlike IEEE
Xplore, ACM DL’s advanced search interface does not enable Boolean operator
selection between different fields, i.e. the logic combination of search
fields is fixed by AND. To relax this constraint and adjust search strings in
the Edit Query window, users will have to conduct a useless search first to
make the Edit Query window visible. Thus, this inconvenience can be viewed as
a bug especially from the perspective of new users. By taking the search for
Title:(software) OR Abstract:(engineering) as an example, we demonstrate how
to reproduce this bug in Bug 4.1.

Bug #1 Steps to reproduce the bug of multi-field search in ACM DL: (Last test
on July 20, 2022) (1) Go to the interface of ACM DL’s Advanced Search
(https://dl.acm.org/search/advanced). (2) In the Search Within panel, select
“Title” as the search field and input the search term software. (3) Click the
plus icon (i.e. add search field) next to the search term. (4) Select
“Abstract” as the added search field and input the search term engineering.
---
Note that there is no chance to use the Boolean operator OR, and the Edit
Query window does not exist.
(5) Click the Search button, and it returns results for [Publication Title:
software] AND [Abstract: engineering]. (6) Click the Edit Search button and
return to the interface of Advanced Search.
Note that a View Query Syntax panel appears at the bottom of the interface.
(7) Expand the View Query Syntax panel, and the Edit Query window appears and
contains Title:(software) AND Abstract:(engineering). (8) Replace AND with OR,
and click the Search button again. (9) The search results are finally correct
for [Publication Title: software] OR [Abstract: engineering].

When it comes to the single-field search, ACM DL seems to have a hidden bug
when dealing with the unnecessary outermost parentheses of search strings.
Given a search string enclosed within a pair of outermost parentheses,
although the initial search results have nothing wrong, re-searching after
editing the string in the Term field (instead of in the Query window) will
combine both string versions. The simplified steps to reproduce this bug are
specified in Bug 4.1. It should be noted that the outermost parentheses are
indeed redundant for search strings, whereas they are not a syntax error.

Bug #2. Steps to reproduce the bug of outermost parentheses interpretation in
ACM DL: (Last test on July 20, 2022) (1) Open ACM DL’s homepage
(https://dl.acm.org/). (2) Input (software) in the default Quick Search field.
(3) Click the Search button, and it returns results for: All: software. (4)
Click the Edit Search button and go to the advanced search interface.
---
Note that the Search Term field shows software without parentheses.
(5) Replace software with engineering in the Search Term field, remain the
Search Term field active and press the Enter key. (6) The results are returned
for: [All: software] AND [All: engineering].
Note that [All: engineering] is expected.

In addition, there are inconsistent behaviours of ACM DL’s default Boolean
operation. By default, ACM DL assumes an OR relationship between search terms.
However, there will be clear difference between the search results from an
operator-specific string and from the string’s alternative version that relies
on the default operation. The difference can be demonstrated through the steps
in Bug 4.1.

Bug #3. Steps to reproduce the bug of default Boolean operation in ACM DL:
(Last test on July 21, 2022) (1) Open ACM DL’s homepage (https://dl.acm.org/).
(2) Input software engineering in the default Quick Search field. (3) Click
the Search button, and it returns “446,113 Results”. (4) Reopen ACM DL’s
homepage. (5) Input software OR engineering in the default Quick Search field.
(6) Click the Search button, and it returns “448,696 Results”.
---

###  4.2. The Bug of AIS eLibrary

Although AIS eLibrary offers the +/- buttons to include additional criteria,
parentheses can be unavoidable for grouping search terms to satisfy
sophisticated logics. Unfortunately, it seems that AIS eLibrary cannot
interpret parentheses properly, while there is little information about
whether or not parentheses are accepted in its search mechanism. The steps to
reproduce this bug are described in Bug 4.2.

Bug #4. Steps to reproduce the bug of parentheses interpretation in AIS
eLibrary: (Last test on July 21, 2022) (1) Go to the interface of AIS
eLibrary’s Advanced Search (https://aisel.aisnet.org/do/search/advanced/). (2)
Input (software) AND (engineering) in the field next to All Fields.
---
Note that the search string is immediately interpreted as ( software AND
(engineering ) below the text “Advanced Search”.
(3) Click the Search button, and it returns “0 results”.

###  4.3. The Bug of CiteSeerX

According to our trials, CiteSeerX shows various unexplainable search
behaviours, which makes it unreliable to support usually complex automated
search in EBSE. Furthermore, it is even difficult for us to isolate specific
bugs from those strange behaviours. Therefore, we try to merge our
observations in a concise way into Bug 4.3.

Bug #5. Steps to reproduce the unexplainable search behaviours of CiteSeerX:
(Last test on July 23, 2022) (1) Go to the interface of CiteSeerX’s Advanced
Search (http://citeseer.ist.psu.edu/advanced_search). (2) If Title search for
software engineering, it will return “Your search … did not match any
documents”. (3) If Title search for "software engineering", it will return
“Results 1 - 10 of 25”. (4) If Title search for software OR engin*, it will
return “Results 1 - 10 of 47”. (5) If Title search for software OR
engineering, it will return “Results 1 - 10 of 1,073”. (6) If Title search for
(software OR engineering), it will return “Results 1 - 10 of 72,011”. (7) If
search for (software OR engineering) AND empirical, it will always return
“Results 1 - 10 of 109” no matter using which field (even the Author or Venue
field).
---

In fact, its early version (namely CiteSeer) was also identified to have some
inconsistent and unexplainable search behaviours (Brereton et al., 2007). We
are afraid that the development of CiteSeerX has made the situation even
worse. Thus, we do not recommend CiteSeerX as a search engine for EBSE studies
until the aforementioned bug is fixed.

###  4.4. The Bugs of IEEE Xplore

We experienced two bugs in IEEE Xplore during this research. The first one is
when conducting field-specific search, if the search string includes
parentheses, the opening parenthesis will be treated as part of its adjacent
search term. This can be a hidden problem in the case of long and complex
search strings, because users will merely receive “No results found” and may
not notice the parenthesis issue. A simple demo of this bug is specified in
Bug 4.4.

Bug #6. Steps to reproduce the bug of opening parenthesis interpretation in
IEEE Xplore: (Last test on July 27, 2022) (1) Go to the interface of IEEE
Xplore’s Advanced Search (https://ieeexplore.ieee.org/search/advanced). (2)
Input (software AND engineering) in the Search Term field. (3) Select
“Abstract” as the search field. In fact, any search field can be used for this
demonstration including the default “All Metadata”. (4) Click the Search
button, and it returns “No results found for ("Abstract":(software AND
"Abstract":engineering))”.
---
Note that (software has been treated as a whole term.

The second bug is the HTTP failure response in the scenario of using a
wildcard (*) with less than three characters. In fact, if this scenario
involves a single-word term only (e.g., en*), IEEE Xplore can give a user-
friendly hint about requiring “at least 3 valid characters for each keyword”.
However, when this scenario exists in a multi-word search term or in a multi-
term search string, users will receive a failure response of “400 Bad Request”
instead of any meaningful message. Even though this case may also be
considered as a string bug (i.e. the user’s mistake), the meaningless response
can be, and should be, avoided in IEEE Xplore. The simplest workflow of
reproducing this bug is shown below.

Bug #7. Steps to reproduce HTTP error 400 in IEEE Xplore: (Last test on July
27, 2022) (1) Open IEEE Xplore’s homepage (https://ieeexplore.ieee.org/). (2)
Input software AND en* in the default Search field. (3) Click the Search
button, and it returns “Http failure response for
https://ieeexplore.ieee.org/rest/search: 400 Bad Request”. (4) Input "software
en*" in the default search field. (5) Click the Search button again, and it
still returns “Http failure response for
https://ieeexplore.ieee.org/rest/search: 400 Bad Request”.
---

###  4.5. The Bugs of ScienceDirect

ScienceDirect is able, to some extent, to automatically balance the unpaired
quotes and parentheses in search strings. In particular, the automatic
balancing of unpaired parentheses seems to supplement the missing one at an
outermost position. During the verification of this mechanism, we identified a
bug related to outermost parentheses in ScienceDirect, i.e. enclosing a
relatively complex search string within a pair of parenthesis will deliver
significantly less results than using the original string. It should be noted
that this bug is different from the outermost parentheses bug in ACM DL (cf.
Bug 4.1), and the redundant parentheses have nothing to do with any syntax
error. We demonstrate this bug via the steps in Bug 4.5.

Bug #8. Steps to reproduce the bug of outermost parentheses interpretation in
ScienceDirect: (Last test on July 28, 2022) (1) Open ScienceDirect’s homepage
(https://www.sciencedirect.com/). (2) Input software OR engineering AND
empirical in the Keywords field. (3) Click the Search button, and it returns
“3,686,632 results”. (4) Go back to ScienceDirect’s homepage, and input
(software OR engineering AND empirical) in the Keywords field. (5) Click the
Search button, and it returns “525,335 results”.
---
Note that searching for software OR (engineering AND empirical) and (software
OR engineering) AND empirical will return “3,686,634 results” and “754,504
results” respectively.

There is no doubt that all the search engines have defined stop words to
improve the search efficiency. However, ScienceDirect seems to have
inconsistent sets of stop words for different search fields. For example,
usable and usability are not searchable in most field-specific cases, whereas
they are not in the stop word list (ScienceDirect, 2021) and they are indeed
searchable when searching all parts of the articles. The concise reproduction
steps are listed in Bug 4.5. Despite that multiple studies (e.g., (de Lima
Salgado and Freire, 2014)) have employed usability as a search term, a bigger
concern is about if ScienceDirect maintains an out-of-date list of more
unknown stop words.

Bug #9. Steps to reproduce the bug of stop words in ScienceDirect: (Last test
on July 28, 2022) (1) Open the interface of ScienceDirect’s Advanced Search
(https://www.sciencedirect.com/search). (2) Input usability in the top field.
(3) Click the Search button, and it returns “111 results”. (4) Go back to
ScienceDirect’s Advanced Search, and input usability in the Title field. (5)
Click another field to make the Title field inactive, then the bottom of Title
field shows a tip message “Check for typos or errors”, and the top-left panel
shows a warning message “This search will not return any results.” (6) Click
the Search button, nothing happens, or it returns “No results found.”
---
Note that usability is treated as a valid search term in the top field, while
as a stop word in the other fields.

###  4.6. The Bug of Scopus

It is known that curly quotes (“ ”) may not be interpreted properly as
quotation marks in search strings, unless search engines specifically accept
them. In Scopus, surprisingly, curly quotes will be treated differently via
different search entries: Its advanced search accepts curly quotes while the
basic search does not. This inconsistency in a single search engine should be
filed as a bug that can be reproduced through the steps in Bug 4.6.

Bug #10. Steps to reproduce the bug of curly quote interpretation in Scopus:
(Last test on July 24, 2022) (1) Open Scopus’s basic (default) Documents
search interface (https://www.scopus.com/search/form.uri?display=basic#basic).
(2) Input ‘‘software engineering’’ in the Search Documents field.
---
Note that the search term software engineering is enclosed within a pair of
curly quotes.
(3) Click the Search button, and it returns some amount of hits with TITLE-
ABS-KEY(“software AND engineering”). (4) Click the Go Back button of the web
browser, and go back to the basic search interface. (5) Click the Advanced
Document Search link, and go to the advanced search interface.
Note that the Enter Query String field shows TITLE-ABS-KEY(“software
engineering”) by default.
(6) Click the Search button, and it returns much less hits with TITLE-ABS-
KEY("software engineering").
Note that the quotation mark format has been changed automatically from curly
quotes into straight quotes.

###  4.7. The Bugs of SpringerLink

As mentioned previously, title search is the only field-specific search option
in SpringerLink. Although we do not treat any functional limit as a bug, it is
worth reminding practitioners of the misleading title search functionality of
SpringerLink: Firstly, its title search does not work with Boolean operators,
unless one of the search terms is enclosed within a pair of quotation marks ("
"). Secondly, most of the title search results do not match the original
search string. The steps to reproduce this bug are specified in Bug 4.7.

Bug #11. Steps to reproduce the bug of title search in SpringerLink: (Last
test on July 22, 2022) (1) Open the interface of SpringerLink’s Advanced
Search (https://link.springer.com/advanced-search). (2) Go to the field “where
the title contains” and input the search string software AND engineering. (3)
Click the Search button, and it returns “0 Result(s)”. (4) Go back and remove
AND from the search string. (5) Click the Search button, and it returns “2,239
Result(s)”.
---
Note that software engineering has been treated as a single search term.
(6) Go back and change the search string into software AND "engineering". (7)
Click the Search button, and it returns “25,801 Result(s)”.
Note that most results do not contain the both search terms in the publication
titles.

Another bug is related to SpringerLink’s vague capacity for handling extremely
long search strings (the threshold seems to be about 1800 characters according
to our tests). Although SpringerLink does not seem to have any constraint on
string length, refining the default search results from long strings will be
rejected with an HTTP error 431 “Request Header Fields Too Large”. Since this
error may have various reasons at the server side (Okta, 2021), we are not
able to reveal the root cause of this bug, but only giving its reproduction
steps as shown in Bug 4.7.

Bug #12. Steps to reproduce HTTP error 431 in SpringerLink: (Last test on July
21, 2022) (1) Open SpringerLink’s homepage (https://link.springer.com/). (2)
Input a lengthy search string (over about 1800 characters) in, or directly
copy the search string from (Ferreira et al., 2017) to, the Search field. (3)
Click the Search button, and the search results should be able to show up. (4)
Refine the search results by clicking an item (e.g., “Article”) within the
left side Refine Your Search panel. (5) The page returns “Bad Message 431” and
“reason: Request Header Fields Too Large”.
---

###  4.8. The Bug of Wiley Online Library

Although the repeatability of filter settings is not measured in this
research, we have carefully tried filtering the publishing venues for those
studies who employed venue-specific automated search. During those trials, we
observed a bug in Wiley OL: The pre-set filter of publishing venue will not be
applied to the search results. It should be noted that the dysfunctional venue
filtering can be misleading or at least a waste of time for users, even though
it can be fixed by selecting filters again on the results page. The steps to
reproduce this bug are specified in Bug 4.8.

Bug #13. Steps to reproduce the bug of venue filtering in Wiley OL: (Last test
on July 28, 2022) (1) Open the interface of Wiley OL’s Advanced Search
(https://onlinelibrary.wiley.com/search/advanced). (2) Input software in the
top search field. (3) Input Software: Practice and Experience in the venue
field under “Published in”. (4) Click the Search button, and then scroll down
to the bottom of the Filters panel in the search results page.
---
Note that all the publishing venues are listed, which mean that the pre-set
venue filter is not applied to the results.

##  5\. Recommendations to the Developers of Academic Search Engines

Since search engines would never stop evolving, a possible concern is about
the consistency issues in the user interaction due to the changing conditions
(i.e. the search engines’ interfaces and/or functionalities). We believe that
this concern implies a pessimistic assumption about the influence of search
engine evolution. It is true that some engines’ functional or interface
changes have made previous user interactions non-replicable. However, the
evolution of some others has brought more user-friendly features. For
instance, the generic string in (Bianchi et al., 2015) was previously required
to be customised for ACM DL, while it can now be used directly without any
modification. Therefore, we decided to briefly summarise the past changes of
academic search engines, and then use them to strengthen our evidence-based
recommendations about the future evolution directions.

###  5.1. Publications Have a Memory: The Past Changes

Although we do not physically track the historical changes in the existing
digital libraries and indexing platforms, we have observed various differences
between the previously published details and our own user experience. These
differences can then be used to reveal the evolution of relevant search
engines, to some extent. From the perspective of enhancing replicability (that
may indicate the improvement of backward compatibility of a single engine
and/or the improvement of compatibility among multiple engines), we
distinguish the past changes between evolution and devolution of search
engines.

####  5.1.1. Evolution

Firstly, we see a trend in unifying the interface of advanced search. Although
there is still inconsistent functionality (e.g., ACM DL supports command
search, while Wiley OL does not), multiple digital libraries seem to have
employed the same and new interface design, such as ACM DL
(https://dl.acm.org/search/advanced), INFORMS PubsOnLine
(https://pubsonline.informs.org/search/advanced), Wiley OL
(https://onlinelibrary.wiley.com/search/advanced), and World Scientific
(https://www.worldscientific.com/search/advanced).

Secondly, there is a clear trend in unifying the Boolean operators for
combining search terms. The lowercase format used in ACM DL (Haselberger,
2016) and the angle-bracketed lowercase version used in IEEE Xplore (Budgen et
al., 2011) have all been changed into the capital-case Boolean operators that
are the standard format in almost all the search engines.

Thirdly, there also seems a trend in unifying zonal settings. For instance,
the keyword search was not an option in ACM DL and in Wiley OL at least before
2015 (Yang et al., 2016). Furthermore, ACM DL’s full text search was
unavailable at least until the year of 2011 (Riņķevičs and Torkar, 2013).
Although the search field of Review in ACM DL (Lucas et al., 2009) is removed,
not many secondary studies have employed this field in their search
activities.

At last, some search engine constraints documented in the previous
publications are relaxed. For example, nowadays users are able to input a long
and complex string in any search field of ACM DL, which was not supported
before according to (Chen et al., 2009). Similarly, IEEE Xplore did not
support lengthy search strings even in the year of 2015 (Soomro et al., 2016),
while it works well now. Particularly, in this research, we observed that IEEE
Xplore’s maximum limit of wildcards (*) was increased from six to seven in the
past year.

####  5.1.2. Devolution

It is understandable that none of the publishers and indexers will claim their
changes to be devolution. Unfortunately, the changes of some search engines
have introduced more troubles (or at least constraints) to the current
automated searches. One of the typical issues is the removal of popular search
fields. For example, the keyword search and abstract search used to be
functional in SpringerLink (e.g., (Kusumo et al., 2012)), while they are both
unavailable at the time of this research. The available title search is
misleading: Firstly, SpringerLink’s title search does not work with Boolean
operators, unless one of the search terms is enclosed within a pair of
quotation marks (" "). Secondly, most of the title search results may not
match the search string. Thus, we have to claim that SpringerLink does not
(well) support metadata search.

Another typical issue is the increased constraints of the searching features.
For instance, ScienceDirect used to not only support long strings (e.g., (Khan
et al., 2011)), but also to allow wildcards (*) (e.g., (Garcia et al., 2014))
and more than eight Boolean operators (e.g., (Santos et al., 2018)) to be used
in search strings. Nevertheless, these features have all been limited,
according to our tests.

###  5.2. Recommendations for the Future Evolution

After summarising the aforementioned observations and our user experience in
the automated search tests, we come up with two groups of recommendations for
evolving search engines in the future. The first group is towards researcher-
friendly evolution, including:

  * •

Relaxing the current constraints. As mentioned previously, the search engine
diversity and constraints frequently require modifying (e.g., shortening or
adapting) well-designed search strings, while it is not a trivial task to mix
major terms and synonyms properly (Imtiaz et al., 2013). Considering that
systematic reviews pervasively need long and sophisticated search strings, it
had better relax the search engine constraints of string length and Boolean
operations. In addition, offering the feature of command line search would be
particularly useful for manipulating and running long and sophisticated search
strings.

  * •

Syntax validation of search strings. Human beings make mistakes. As
demonstrated by Bug 4.6, researchers can incidentally introduce various logic
and stylistic errors especially to long and complex search strings. Therefore,
offering syntax validation will significantly facilitate researchers to use
correct strings to conduct automated search. In fact, our tests have found
that Scopus and WoS are equipped with relatively well developed mechanisms of
syntax validation.

  * •

Unified and standard syntax and syntax rules. There is no doubt that diverse
syntax and rules of different search engines (e.g., DBLP and Engineering
Village) will inevitably increase the workloads of building search strings,
even if researchers are able to equivalently translate one into another.
Therefore, we urge unified and standard syntax and syntax rules to improve the
user experience across multiple search engines. In fact, as mentioned in
Section 5.1.1, most search engines have been evolving along this direction.

  * •

Unified and standard front-end interfaces. Compared with the syntax
inconsistency, the diverse search engine interfaces and engine features seem
to bother researchers more frequently and cause more issues (e.g., search
string adaptation) in automated search. Thus, unifying interfaces will at
least ease the user operations when a user needs to switch from one search
engine to another.

  * •

Fully featured APIs. Unlike ad hoc reviews, systematic reviews generally
employ automated search for and access massive number of papers. Ideally,
programmably searching and obtaining the search results will make automated
search really automatic. Unfortunately, compared with the promising progress
for automating paper selection (Shakeel et al., 2019), the current automated
searches still largely rely on manual operations via search engines’ user
interfaces. Note that crawling web pages is widely prohibited by digital
libraries, while not all the search engines offer APIs. Among the limited
available APIs, most of them are not fully featured (e.g., only metadata
search is supported).

By facilitating operations and by reducing human mistakes, the researcher-
friendly evolution will indirectly help enhance the replicability of user
interactions. In contrast, the second group of recommendations is specifically
towards replicability-oriented evolution, including:

  * •

Backward compatible updates. As mentioned previously, replicating user
interactions relies not only on the same input, but also on the same/similar
environments or conditions. If a search engine is updated without remaining
compatible with its earlier functional features, it will be difficult to
expect previous practices to be replicable on the same search engine. For
example, according to the explanation (Elsevier, 2020), the increased search
constraints of ScienceDirect may be a compromise to improve its engine
performance, which however has significantly sacrificed the replicability of
user interaction in many existing studies (Li, 2021).

  * •

Unified and standard back-end functional features. As mentioned above, the
inconsistent search engine features and interfaces not only complicate the
search process but also incur replicability problems. For example, search
string adaptation has been identified as a major factor that threatens search
replications (Krüger et al., 2020). Moreover, the engine features will have
cascaded influences on the design and the usage of APIs. Considering that
academic publications are well structured data with standard metadata, it is
feasible to unify the functional features of academic search engines, to
better support the unification of interfaces.

Overall, our recommendations about unification and standardisation further
require extensive collaboration among those publishers and indexers. Although
there has been spontaneous consensus among them, the community may need an
independent committee to help them cooperate (and also possibly negotiate)
with each other more efficiently.

##  6\. Discussion and Conclusions

###  6.1. Discussion

It is well recognised that the evaluation of search engines (especially with
respect to digital libraries) is both useful and challenging, because it has
various criteria and standards for verifying different attributes, aspects,
and components from different perspectives (Li and Liu, 2019). We acknowledge
that our work only conducts user interaction testing and reflects the
effectiveness part of search engine usability, without measuring the other
evaluation dimensions.

However, given our research resource (manpower) limit, it is impractical to
cover a wide range of evaluation dimensions for such extensive testing work.
On the other hand, it is common for researchers to intentionally exclude
measures from search engine evaluation in practice. For example, recall and
precision have frequently been omitted in large scale testing (Vaughan, 2004),
because exhaustive judgements for document relevance against every single
request are infeasible with large-size literature collections. Therefore, the
theoretical evaluation models and frameworks (see Section 2) may need a
divide-and-conquer fashion to guide practices.

Following the divide-and-conquer fashion, we have chosen to investigate the
most critical attribute (i.e. effectiveness) of search engines’ usability. As
one of the three main themes of search engine evaluation (the other two are
the value of collection and the system performance) (Xie, 2008), usability
measures how usable and how easy-to-use a search engine is. Before proving a
search engine to be effectively usable, it will be meaningless to discuss its
easiness of usage.

###  6.2. Conclusions and Future Work

Our study leads to two conclusions. First, there remains a considerable gap
between the search-needs of researchers and the usability of academic search
engines. It is not clear whether search-engine developers are aware of this
continuing gap. An early tertiary study has already highlighted academic
search engines’ poor support to systematic reviews (Brereton et al., 2007).
Our study provides a recent complement to the tertiary review, and shows that
problems with search engines remain.

Second, the evaluation, by academics, of academic search engines has not kept
pace with the development, by search-engine providers, of those search
engines. This gap between evaluation and development makes it harder to
properly understand the gap between the search-needs of researchers and
search-features of the search engines.

Compared with general search engines, the academic engines (should) have
unique features that are particularly designed for the specific requirements
of its (academic) users, e.g., that the results of a search are exhaustive, or
that the results are ranked in a way more relevant and reliable for research,
rather than in a way that is more convenient or quicker to process. In other
words, academics have a higher performance requirement of their search
engines, compared to public usage of search engines, and search-engine
providers should have an appreciation for these specific performance
requirements.

Driven by the conclusions, we argue the needs of dedicated efforts on
evaluating academic search engines. Therefore, our next-step work will be
distilling a systematic yet concise suite of test cases from this random-
sample-based regression testing. Then, we plan not only to share the test case
suite with colleague researchers, but also to contact the relevant search
engine providers and help them address the diverse needs from the scientific
community of which they are also part.

###### Acknowledgements.

This research is supported in part by startup funding from Queen’s University
Belfast. We also thank reviewers, in advance, for their reviews.

## References

  * (1)
  * Bailey et al. (2007) John Bailey, Cheng Zhang, David Budgen, Mark Turner, and Stuart Charters. 2007.  Search Engine Overlaps: Do they agree or disagree?. In _Proceedings of the 2nd International Workshop on Realising Evidence-Based Software Engineering (REBSE 2007)_. IEEE Computer Society, Minneapolis, MN, USA, 1–6.
  * Bianchi et al. (2015) Thiago Bianchi, Daniel Soares Santos, and Katia Romero Felizardo. 2015.  Quality Attributes of Systems-of-Systems: A Systematic Literature Review. In _Proceedings of the 3rd IEEE/ACM International Workshop on Software Engineering for Systems-of-Systems (SESoS 2015)_. IEEE Press, Florence, Italy, 23–30.
  * Brereton et al. (2007) Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007.  Lessons from applying the systematic literature review process within the software engineering domain.  _Journal of Systems and Software_ 80, 4 (April 2007), 571–583.
  * Buchanan and Salako (2009) Steven Buchanan and Adeola Salako. 2009.  Evaluating the usability and usefulness of a digital library.  _Library Review_ 58, 9 (October 2009), 638–651.
  * Budgen et al. (2011) D. Budgen, A. J. Burn, O. P. Brereton, B. A. Kitchenham, and R. Pretorius. 2011.  Empirical evidence about the UML: A systematic literature review.  _Software: Practice and Experience_ 41, 4 (April 2011), 363–392.
  * Chen et al. (2009) Lianping Chen, Muhammad Ali Babar, and Ciaran Cawley. 2009\.  A Status Report on the Evaluation of Variability Management Approaches. In _Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering (EASE 2009)_. BCS Learning & Development Ltd., Durham University, UK, 1–10.
  * Chen et al. (2010) Lianipng Chen, Muhammad Ali Babar, and He Zhang. 2010\.  Towards an evidence-based understanding of electronic data sources. In _Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering (EASE 2010)_. BCS, Keele University, UK, 135–138.
  * de Lima Salgado and Freire (2014) André de Lima Salgado and André Pimenta Freire. 2014.  Heuristic Evaluation of Mobile Usability: A Mapping Study.  In _Human-Computer Interaction. Applications and Services. HCI 2014_ , Masaaki Kurosu (Ed.). Lecture Notes in Computer Science, Vol. 8512. Springer, Cham, Heraklion, Crete, Greece, 178–188.
  * Dybå et al. (2007) Tore Dybå, Torgeir Dingsøyr, and Geir K. Hanssen. 2007\.  Applying Systematic Reviews to Diverse Study Types: An Experience Report. In _Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM 2007)_. IEEE Computer Society, Madrid, Spain, 225–234.
  * Elsevier (2020) Elsevier. 2020.  New ScienceDirect advanced search delivers fast and powerful results.  https://www.elsevier.com/librarians/article-news/new-sciencedirect-advanced-search-delivers-fast-and-powerful-results.
  * Ferreira et al. (2017) Thiago Nascimento Ferreira, Silvia Regina Vergilio, and Jerffeson Teixeira de Souza. 2017\.  Incorporating user preferences in search-based software engineering: A systematic mapping study.  _Information and Software Technology_ 90 (October 2017), 55–69.
  * Garcia et al. (2014) Cecilia Garcia, Abraham Dávila, and Marcelo Pessoa. 2014\.  Test Process Models: Systematic Literature Review. In _Proceedings of the 14th International Conference on Software Process Improvement and Capability Determination (SPICE 2014)_. Springer, Cham, Vilnius, Lithuania, 84–93.
  * Hamilton (2017) Brian Hamilton. 2017\.  The Importance of Steps to Reproduce a Bug.  https://blog.testlodge.com/the-importance-of-steps-to-reproduce-a-bug/.
  * Haselberger (2016) David Haselberger. 2016\.  A literature-based framework of performance-related leadership interactions in ICT project teams.  _Information and Software Technology_ 70 (February 2016), 1–17.
  * Imtiaz et al. (2013) Salma Imtiaz, Muneera Bano, Naveed Ikram, and Mahmood Niazi. 2013.  A Tertiary Study: Experiences of Conducting Systematic Literature Reviews in Software Engineering. In _Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE 2013)_. ACM Press, Porto de Galinhas, Brazil, 177–182.
  * Jeng (2005) Judy Jeng. 2005\.  What Is Usability in the Context of the Digital Library and How Can It Be Measured.  _Information Technology and Libraries_ 24, 2 (June 2005), 47–56.
  * Kelly (2014) Elizabeth Joan Kelly. 2014\.  Assessment of Digitized Library and Archives Materials: A Literature Review.  _Journal of Web Librarianship_ 8, 4 (September 2014), 384–403.
  * Khan et al. (2011) Siffat Ullah Khan, Mahmood Niazi, and Rashid Ahmad. 2011\.  Barriers in the selection of offshore software development outsourcing vendors: An exploratory study using a systematic literature review.  _Information and Software Technology_ 53, 7 (July 2011), 693–706.
  * Khoo et al. (2012) Michael Khoo, Diana Kusunoki, and Craig MacDonald. 2012\.  Finding Problems: When Digital Library Users Act as Usability Evaluators. In _Proceedings of the 45th Hawaii International Conference on System Sciences (HICSS 2012)_. IEEE Computer Society, Maui, HI, USA, 1615–1624.
  * Kitchenham and Charters (2007) Barbara A. Kitchenham and Stuart Charters. 2007.  _Guidelines for Performing Systematic Literature Reviews in Software Engineering_.  Technical Report EBSE 2007-01. School of Computer Science and Mathematics, Keele University and Durham University.
  * Krüger et al. (2020) Jacob Krüger, Christian Lausberger, Ivonne von Nostitz-Wallwitz, Gunter Saake, and Thomas Leich. 2020.  Search. Review. Repeat? An empirical study of threats to replicating SLR searches.  _Empirical Software Engineering_ 25 (January 2020), 627–677.
  * Kusumo et al. (2012) Dana S. Kusumo, Mark Staples, Liming Zhu, He Zhang, and Ross Jeffery. 2012.  Risks of off-the-shelf-based software acquisition and development: A systematic mapping study and a survey. In _Proceedings of the 16th International Conference on Evaluation and Assessment in Software Engineering (EASE 2012)_. IET press, Ciudad Real, Spain, 233–242.
  * Lai et al. (2014) Chin-Feng Lai, Po-Sheng Chiu, Yueh-Min Huang, Tzung-Shi Chen, and Tien-Chi Huang. 2014.  An evaluation model for digital libraries’ user interfaces using fuzzy AHP.  _The Electronic Library_ 32, 1 (January 2014), 83–95.
  * Li and Liu (2019) Yuelin Li and Chang Liu. 2019.  Information Resource, Interface, and Tasks as User Interaction Components for Digital Library Evaluation.  _Information Processing and Management_ 56, 3 (May 2019), 704–720.
  * Li (2021) Zheng Li. 2021.  Stop Building Castles on a Swamp! The Crisis of Reproducing Automatic Search in Evidence-based Software Engineering. In _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER 2021)_. IEEE Press, Madrid, Spain, 16–20.
  * Lucas et al. (2009) Francisco J. Lucas, Fernando Molina, and Ambrosio Toval. 2009\.  A systematic review of UML model consistency management.  _Information and Software Technology_ 51, 12 (December 2009), 1631–1645.
  * Nielsen (2012) Jakob Nielsen. 2012\.  Usability 101: Introduction to Usability.  https://www.nngroup.com/articles/usability-101-introduction-to-usability/.
  * Okta (2021) Okta. 2021.  HTTP Error 431: Definition, Status, Causes & Solutions.  https://www.okta.com/identity-101/http-error-431/.
  * Petersen et al. (2015) Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015\.  Guidelines for conducting systematic mapping studies in software engineering: An update.  _Information and Software Technology_ 64 (August 2015), 1–18.
  * Riņķevičs and Torkar (2013) K. Riņķevičs and R. Torkar. 2013.  Equality in cumulative voting: A systematic review with an improvement proposal.  _Information and Software Technology_ 55, 2 (February 2013), 267–287.
  * Santos et al. (2018) José Amancio M. Santos, João B. Rocha-Junior, Luciana Carla Lins Prates, Rogeres Santos do Nascimento, Mydiã Falcão Freitas, and Manoel Gomes de Mendonça. 2018\.  A systematic review on the code smell effect.  _Journal of Systems and Software_ 144 (October 2018), 450–477.
  * ScienceDirect (2021) ScienceDirect. 2021\.  How do I use the advanced search?  https://service.elsevier.com/app/answers/detail/a_id/25974/supporthub/sciencedirect/.
  * Shakeel et al. (2018) Yusra Shakeel, Jacob Krüger, Ivonne von Nostitz-Wallwitz, Christian Lausberger, Gabriel Campero Durand, Gunter Saake, and Thomas Leich. 2018.  (Automated) Literature Analysis - Threats and Experiences. In _Proceedings of the 13th IEEE/ACM International Workshop on Software Engineering for Science (SE4Science 2018)_. ACM Press, Gothenburg, Sweden, 20–27.
  * Shakeel et al. (2019) Yusra Shakeel, Jacob Krüger, Ivonne von Nostitz-Wallwitz, Gunter Saake, and Thomas Leich. 2019\.  Automated Selection and Quality Assessment of Primary Studies: A Systematic Literature Review.  _ACM Journal of Data and Information Quality_ 12, 1 (November 2019).  Article 4.
  * Singh and Singh (2017) Paramvir Singh and Karanpreet Singh. 2017.  Exploring Automatic Search in Digital Libraries - A Caution Guide for Systematic Reviewers. In _Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE 2017)_. ACM Press, Karlskrona, Sweden, 236–241.
  * Soomro et al. (2016) Arjumand Bano Soomro, Norsaremah Salleh, Emilia Mendes, John Grundy, Giles Burch, and Azlin Nordin. 2016.  The effect of software engineers’ personality traits on team climate and performance: A Systematic Literature Review.  _Information and Software Technology_ 73 (May 2016), 52–65.
  * Vaughan (2004) Liwen Vaughan. 2004\.  New measurements for search engine evaluation proposed and tested.  _Information Processing and Management_ 40, 4 (July 2004), 677–691.
  * Vrana (2007) Radovan Vrana. 2007\.  The Importance of Usability in Development of Digital Libraries. In _Proceedings of the 1st International Conference The Future of Information Sciences (INFuture2007)_. Zagreb, Croatia, 379–387.
  * Xie (2008) Hong Iris Xie. 2008\.  Users’ evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment.  _Information Processing and Management_ 44, 3 (May 2008), 1346–1373.
  * Yang et al. (2016) Chen Yang, Peng Liang, and Paris Avgeriou. 2016.  A systematic mapping study on the combination of software architecture and agile development.  _Journal of Systems and Software_ 111 (January 2016), 157–184.