# Two-tone spectroscopy of high-frequency quantum circuits with a Josephson
emitter

A. Peugeot1, H. Riechert1, S. Annabi1, L. Balembois2, M. Villiers3, E.
Flurin2, J. Griesmar1, E. Arrighi1, J.-D. Pillet1∗, L. Bretheau1 These authors
supervised equally this work.
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>1 Laboratoire de Physique de la Matière
condensée, CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, 91120
Palaiseau, France  2 Quantronics group, Université Paris-Saclay, CEA, CNRS,
SPEC, 91191 Gif-sur-Yvette Cedex, France  3 Laboratoire de Physique de l’Ecole
normale supérieure, Centre Automatique et Systèmes, Mines Paris, Inria, ENS-
PSL, Université PSL, CNRS, Sorbonne Université, Paris, France

###### Abstract

We perform two-tone spectroscopy on quantum circuits, where high-frequency
radiation is generated by a voltage-biased superconductor-normal-
superconductor Josephson junction and detection is carried out by an ancillary
microwave resonator. We implement this protocol on two different systems, a
transmon qubit and a $\lambda/4$ resonator. We demonstrate that this two-tone
Josephson spectroscopy operates well into the millimeter-wave band, reaching
frequencies larger than 80 GHz, and is well-suited for probing highly coherent
quantum systems.

##  I Introduction

Circuit Quantum Electrodynamics (cQED) is based on microwave measurements and
control of superconducting circuits [1, 2]. In practice however, the
accessible frequencies are typically smaller than 30 GHz, due to the limited
range of commercially available instruments and components and the difficulty
to route high frequency signals in a cryostat. Increasing this upper limit
could help to probe novel quantum systems in hybrid architectures [3] as well
as elucidate decoherence mechanisms caused by the environment at high
frequency [4]. It would also open new routes towards the development of high-
frequency qubits that can operate at higher temperatures [5]. In order to take
a step forward in this direction, one can exploit the AC Josephson effect [6,
7] to generate high-frequency radiation directly at the cold-stage level. When
biased at voltage $V_{J}$, a Josephson junction indeed radiates photons at
frequency $f_{J}=2eV_{J}/h$, thereby working as an on-chip tunable microwave
source [8, 9, 10, 11, 12, 13, 14, 15, 16]. This phenomenon enables broad-band
spectroscopy that extends from a few tens of MHz up to the THz regime. Its
principle relies on power conservation and is quantumly described by the
theory of dynamical Coulomb blockade [17, 18]. Each time a photon of energy
$hf_{J}$ is absorbed and dissipated in the junction’s electromagnetic
environment, a Cooper pair of energy $2eV_{J}$ inelastically flows across the
junction. Consequently, when $f_{J}$ matches a resonance of the target system,
one can detect a current $I_{J}=2e\Gamma_{J}$ that is directly proportionnal
to the photon emission rate $\Gamma_{J}$.

Absorption Josephson spectroscopy has been employed for characterizing
macroscopic systems, such as ensembles of cobalt atoms [19] or microwave
resonators [20, 21, 22, 23, 24], as well as elementary quantum objects,
including Cooper pair boxes [25, 26, 27, 28] and Andreev bound states [29, 30,
31, 32]. This spectroscopy technique is however challenging as it is sensitive
to the whole electromagnetic environment of the junction and relies on low-
frequency measurements of small DC currents. Even more crucially, this
approach is not suited to probe highly coherent quantum systems like
superconducting qubits, as its detection sensitivity decreases dramatically
for long coherence time. Indeed, the dissipation rate of $\sim 1$ photon per
coherence time interval induces an upper bound for $\Gamma_{J}$. A typical
$T_{1}\sim 100$ $\mu$s would thus amount in detecting a current of a few fA,
four orders of magnitude below the resolution of the best power-absorption
Josephson spectrometer [24].

In this work, we develop a new paradigm for Josephson spectroscopy, inspired
by cQED, where the detection is realized using an ancillary system (see Fig.
1a). First, we utilize a superconductor-normal-superconductor (SNS) Josephson
junction as an on-chip microwave emitter (red). We couple such an SNS emitter
to a quantum object of interest with frequency $f_{q}$ (green). This system is
in turn dispersively coupled to a microwave cavity that resonates at frequency
$f_{r}$ and acts as a detector (blue). When driven at resonance $f_{J}=f_{q}$,
the quantum object gets excited with $n_{q}$ photons, which induces a
frequency shift $\chi n_{q}$ of the cavity that can be detected using
sensitive and fast radiofrequency techniques. Following this approach, our
goal is to perform two-tone Josephson spectroscopy at high-frequency. As a
proof of concept, we implement this strategy to probe two coherent though very
different quantum objects, a superconducting transmon qubit that resonates
around 6 GHz, and electromagnetic modes of a $\lambda/4$ resonator up to $83$
GHz.

##  II SNS Josephson junction as a microwave emitter

Figure 1:  (a) Principle of two-tone Josephson spectroscopy. A Josephson
emitter (red) emits photons at frequency $f_{J}=2eV_{J}/h$ onto the system of
interest (green) that resonates at frequency $f_{q}$. A microwave resonator
(blue) with frequency $f_{r}$ is dispersively coupled to the system of
interest and serves as a detector. When the system of interest is excited, the
resonator’s frequency shifts, which is detected by a probe tone. (b) Current-
voltage characteristic $I_{J}(V_{J})$ of an SNS Josephson junction measured at
7 mK. The grey region $|V_{J}|\leq 23\leavevmode\nobreak\ \mu$V is not
accessible as the current falls below the retrapping current and the junction
jumps to the zero-voltage state. Inset: Scanning electron microscope image of
the device. The normal part is a Ti($5$ nm)-Au($15$ nm) bilayer and the
superconducting part is a $55$ nm-thick layer of Nb. (c) Josephson power $P$
as a function of both frequency $f$ and voltage $V_{J}$ across the emitter.
The Josephson radiation is here emitted by an Al($100$ nm)-Hf($7$ nm)-Al($100$
nm) junction with $I_{c}\approx 400$ nA and measured using a spectrum analyzer
by integrating the power in a $2$ MHz bandwidth around $f$. Cuts corresponding
to the blue ($V_{J}=8.9$ $\mu$ V), orange ($V_{J}=9.45$ $\mu$ V) and green
($V_{J}=10$ $\mu$ V) dashed lines are shown in the top left inset. Bottom
right inset: simplified version of the setup, where a $4-8$ GHz band-pass
filter protects the junction from the $1.5$ K noise emitted by the HEMT.

Our Josephson emitter consists of two superconducting electrodes connected by
a normal metallic wire, forming an SNS Josephson junction (see micrograph in
Fig. 1b and details of fabrication in appendix A). We conduct measurements
either on Nb-Au-Nb or Al-Hf-Al structures. In the latter case, although the Hf
wire might actually become superconducting at the base temperature of our
experiments [33], it essentially behaves like a normal metal above its
critical current at finite voltage.

We initially characterize our devices through DC electrical measurements and
extract their current-voltage characteristic $I_{J}(V_{J})$ (see Fig. 1b and
appendix C). Depending on the thickness, width, length and composition of the
normal wire connecting the two superconductors, the critical current $I_{c}$
can vary from a few hundreds nA to a few tens of $\mu$A. When biased above
$I_{c}$, the junction switches and acquires a finite normal resistance
$R_{n}$, ranging from about 10 to a few hundreds of Ohms depending on the
devices. Due to Joule heating, retrapping occurs at a current significantly
smaller than $I_{c}$. The thermalization is mostly ensured by the normal metal
wings, which act as thermal baths and maintain the emitters temperature below
a few hundreds of mK. In most of our designs, the $I_{c}R_{n}$ products is
around 100 $\^{A}\mu V$, which is consistent with results of other experiments
with similar geometries [34, 35]. This suggests that our SNS junctions possess
a minigap $2E_{g}\sim 25$ GHz and are likely to exhibit additional dissipation
beyond this frequency [36].

The operation of SNS microwave sources relies on the AC Josephson effect, as
they carry, when voltage-biased, an oscillating current at the Josephson
frequency $f_{J}=2eV_{J}/h$. The voltage-to-frequency conversion ratio is thus
given by the inverse of the magnetic flux quantum $\phi_{0}^{-1}=2e/h\approx
0.5$ GHz$/\mu$V. Consequently, Josephson junctions can emit microwave photons
ranging from a few GHz up to a few hundreds of GHz. This upper bound is set by
the value of the superconducting gap, which can reach $\sim 180$ GHz, $1.4$
THz and $1.8$ THz for Al, Nb and NbN respectively, the three most commonly
used materials in superconducting circuits. Beyond this limit, the amplitude
of the microwave signal is expected to decrease slowly [37, 38]. In the
$I_{J}(V_{J})$ characteristic of Fig. 1b, the AC Josephson current is averaged
out and thus invisible. To detect it, we measure the power spectral density
emitted by the junction in the 4-8 GHz band by routing the signal with a bias
tee to a cryogenic HEMT amplifier anchored at 4K. In the power spectra
presented in Fig. 1c, we observe an emission peak whose frequency coincides
with $\phi_{0}^{-1}V_{J}$, up to the 1% precision of our bias circuit. This
voltage dependence enables us to unambiguously attribute the signal to the AC
Josephson effect.

The peak has a gaussian shape with a standard deviation $\sigma\approx 5$ MHz,
which we attribute to a $10$ nV RMS noise on the bias voltage. Such thin
linewidth, obtained thanks to significant filtering (details in appendix B),
is comparable to the best performances achieved so far [39, 40, 41, 42].
Additionally, we observe a frequency dependence of the signal amplitude within
the $4-8$ GHz bandwidth of our detection chain, with a large decrease of the
signal around $7.3$ GHz, probably due to a parasitic resonance on the junction
chip or in our setup. This illustrates how challenging it is to efficiently
route the signal emitted by the junction in the desired direction and to
determine it quantitatively. By integrating the signal from Fig. 1 over the
full linewidth and comparing it to the noise floor added by the HEMT
amplifier, we find an emission power of $\sim-130$ dBm. This microwave power
corresponds to the emission of about $10^{7}$ photons per second, _i.e._ of
the order of magnitude of the read-out pulses used in circuit-QED experiments.
Note that the power emitted by the SNS junction strongly depends on the
environment it is embedded into and might change for each experiment. At
larger voltage, we observe an additional broadband emission due to shot noise
(see appendix C), whose impact is negligible in the following experiments.

These DC and AC measurements therefore show that SNS junctions function
properly as microwave sources, at least in the $4-8$ GHz range. Interestingly,
SNS microwave sources offer several advantages compared to more standard
tunnel Josephson junctions. First, their $I_{J}(V_{J})$ characteristic is
mostly linear, with $R_{n}$ remaining relatively small. Their impedance thus
weakly changes with $V_{J}$ and, as such, does not significantly affect other
components to which it is coupled. This aspect is crucial for applications
like two-tone spectroscopy, where a sensitive microwave cavity interacts with
the junction (see appendix G). Second, one can mention the relative ease of
their fabrication process compared to tunnel junctions when using niobium as a
superconductor [43, 44, 45]. Finally, the low backaction from the environment
on their $I_{J}(V_{J})$ characteristics make them simpler to use as their
resistance does not significantly vary depending on the objects to which they
are coupled. In the following, we use these newly developped on-chip SNS
microwave emitters to perform two-tone spectroscopy of quantum circuits.

##  III Two-tone Josephson spectroscopy of a transmon qubit

Figure 2:  Two-tone Josephson spectroscopy of a transmon qubit. (a) Schematic
of the experimental setup. An Al-Hf-Al SNS Josephson emitter (red), mounted in
its own sample holder, emits photons at frequency $f_{J}$. These photons are
routed via an SMA cable to another sample holder towards the drive line of a
transmon qubit (green), which is dispersively coupled to a microwave resonator
(blue). By probing the resonator close to its resonant frequency $f_{r}$, one
can detect the excitation of the qubit. Though not represented, a $4-8$ GHz
bandpass filter is also present in the setup. (b) Reflection coefficient $R$,
normalized by its value in absence of the Josephson drive, as a function of
the emitted Josephson frequency $f_{J}$. Experimental data are shown as blue
dots and fits as solid lines. The latter corresponds to the sum of two
gaussians of standard deviation $\sigma$ centered at $f_{q}$ and $f_{q}-\chi$,
with a relative weight $p_{1}$ given by the finite population of the Fock
state $|n=1\rangle$ in the resonator. Fit parameters: $f_{q}=5.9055$ GHz $\pm
0.1$ MHz, $\chi=11.8\pm 0.8$ MHz, $p_{e}=0.13\pm 0.02$, $\sigma=5.4\pm 0.1$
MHz.

As a first proof of concept, we employ two-tone Josephson spectroscopy on a
well-known quantum system: a transmon qubit. While its frequency $f_{q}$ may
be relatively low, it allows us to validate our technique by comparing it to
measurements obtained using conventional microwave instrumentation. The
transmon is capacitively coupled to a microwave resonator of frequency $f_{r}$
for dispersive readout (see Fig. 2a). The qubit and cavity being off-
resonantly coupled with a large enough detuning, they can be described by the
Hamiltonian $hf_{r}a^{\dagger}a+hf_{q}|e\rangle\langle e|-h\chi
a^{\dagger}a|e\rangle\langle e|$, where $a^{\dagger}$ and $a$ are the ladder
operators of the cavity, $|g\rangle$ and $|e\rangle$ the ground and excited
state of the qubit, and $\chi$ is the dispersive frequency shift. The cavity
frequency is thus conditioned to the state of the qubit, which provides the
basis for performing its two-tone spectroscopy. Details on the design and
fabrication of this circuit, which was initially created for microwave photon
counting are given in references [46, 47].

We first characterize the transmon circuit using standard microwave
instruments and thus access to its characteristic frequencies and coherence
times (see appendix E). We then turn to two-tone Josephson spectroscopy using
our on-chip SNS microwave emitter ($I_{c}\approx 400$ nA, see Fig. 5 in
appendix C). For this experiment, the emitter and the transmon are separated
on two independent chips, each mounted in its own sample-holder, connected to
each other by an SMA cable (see Fig. 2a). We emit Josephson radiation onto the
transmon while probing the resonator via reflectometry close to its resonant
frequency $f_{r}=8.177$ GHz. As we vary $f_{J}$, we observe a clear
suppression of the reflection coefficient $R$ at the qubit frequency
$f_{q}=5.9055$ GHz, demonstrating that it gets excited by the SNS microwave
source (see Fig. 2b).

The transition linewidth with $\sigma=5.4$ MHz is much larger than the
intrinsic qubit decay rate. This width is set by the voltage noise across the
SNS emitter and is similar to the one measured in Fig. 1c. On top of that, the
resonant dip displays an asymmetry, as a second transition can be excited at
frequency $f_{J}=f_{q}-\chi$, with $\chi=11.8$ MHz. The qubit frequency indeed
depends on the number of photons $n$ in the cavity due to the dispersive
coupling. As a consequence, we observe a transition between states
$|g,1\rangle$ and $|e,1\rangle$, owing to the finite thermal population
$p_{1}\approx 0.1$ of the Fock state $|n=1\rangle$. We therefore demonstrate
that our on-chip SNS microwave emitter can be used to detect an elementary
quantum system, a transmon qubit that resonates around $6$ GHz, using two-tone
Josephson spectroscopy.

##  IV Two-tone Josephson spectroscopy of high-frequency modes

Figure 3:  Two-tone Josephson spectroscopy of high-frequency modes. (a)
Schematic of the circuit, and (b) illustration of the device layout (with
niobium in white and silicon in black). The $\lambda/4$ resonator (blue) is
inductively coupled at its end, both to a superconducting loop closed by an
SNS junction (RF SQUID in pink) and to an SNS Josephson emitter (red). Both
Josephson junctions are Nb-Au-Nb structures. A flux line, visible at the
bottom, is used to adjust the magnetic flux $\Phi$ in the RF SQUID loop. (c)
Diagram illustrating the principle of two-tone spectroscopy based on the
cross-Kerr effect between electromagnetic modes within the same resonator. The
fundamental mode $f_{0}$ is continuously monitored with a probe tone $f_{r}$,
while the $m$-index mode is irradiated at frequency $f_{J}$. At resonance
$f_{J}=f_{m}$, mode $m$ gets excited with $n_{m}$ photons. Consequently, the
fundamental mode frequency shifts to $f_{0}-\chi_{0,m}n_{m}$ and a phase shift
$\delta\varphi_{r}$ of the reflected signal is observed. (d) Phase shift
$\delta\varphi_{r}$ of the probe tone as a function of $f_{J}$. Orange
vertical lines serve as markers for the expected frequencies $f_{m}$ of the
resonator’s harmonics. Close to each line, a resonance corresponding to an
electromagnetic mode of the resonator is observed. Inset: detailed view around
a resonance at 83 GHz, interpreted as the $m=8$ mode of the resonator. The
signal-to-noise ratio was here optimized by carefully selecting the probe
tone’s frequency. Note that for these measurements, a systematic error in
measuring $V_{J}$ is corrected by referencing the position of mode $m=6$
(indicated with a yellow star), enabling a more precise calibration of
$f_{J}$. (e) Detailed view of the resonance for mode $m=6$, measured at
$\Phi=0$ (solid line) and $\phi_{0}/2$ (dashed line). The observed sign change
of the dispersive coupling $\chi_{0,6}$ confirms that this resonance
corresponds to a mode of the resonator.

To further test this novel detection scheme, we now probe the electromagnetic
modes of a $\lambda/4$ resonator. The advantage of such a system is that it
has a large number of regularly spaced modes, which is well-suited to explore
two-tone Josephson spectroscopy at high frequency and over a large range. The
resonator, shown in Fig. 3a-b, is made out of Niobium and resonates at its
harmonic frequencies $f_{m}$, where $m\in\mathbb{N}$ is the mode index. It is
inductively coupled to an RF SQUID, which consists of an SNS Josephson
junction enclosed in a superconducting loop threaded by a magnetic flux
$\Phi$. This introduces some non-linearity into the circuit, which results in
cross-Kerr interaction between the resonant modes. The system can thus be
described by the Hamiltonian
$\sum_{m}hf_{m}a^{\dagger}_{m}a_{m}+\sum_{m,m^{\prime}}h\chi_{m,m^{\prime}}a^{\dagger}_{m}a_{m}a^{\dagger}_{m^{\prime}}a_{m^{\prime}}$,
where $\chi_{m,m^{\prime}}$ are the cross-Kerr frequencies. Therefore, when
mode $m$ gets excited with 1 photon, the fundamental mode’s frequency $f_{0}$
is shifted by $\chi_{0,m}$. This provides the basis to perform a two-tone
Josephson spectroscopy of the high-frequency modes, while the fundamental mode
of the resonator is used as a detector (see Fig. 3c).

As before, we use a voltage-biased SNS Josephson junction as a microwave
emitter. It is positioned on the same chip as the $\lambda/4$ resonator, with
which it is inductively coupled. We emit Josephson radiation while probing the
fundamental mode of the resonator at frequency $f_{r}\approx f_{0}$. Fig. 3d
shows the phase shift $\delta\varphi_{r}$ of the reflected signal as a
function of the Josephson frequency $f_{J}$. The spectrum exhibits regularly
spaced resonances at frequencies $f_{J}\approx f_{m}=(2m+1)f_{0}$, with
$f_{0}=4.95$ GHz, which correspond to the high-frequency modes of the
$\lambda/4$ resonator (see also appendix F). Other resonances appear in our
signal, but are not clearly identified at this stage. We believe they might be
geometric resonances of the circuit that would couple to the resonator.
Additionally, a slowly varying background is noticeable. It is attributed to a
flux change within the loop terminating the resonator, induced by the DC
current carried by the Josephson emitter.

The measured linewidth of the different transitions, which is roughly constant
and about $\sim 500$ MHz, is much larger than the intrinsic decay rate of the
modes that we estimate to be $\sim 350$ kHz based on the measured quality
factor $Q=14000$ of the fundamental mode. It is again limited by the Josephson
emission linewidth, which is here almost two orders of magnitude larger than
previously observed. This larger linewidth is due to a larger voltage noise
across the SNS emitter, owing to the different filtering setup needed for this
experiment (see appendix B). Within this architecture, we are able to detect
the modes up to $m=8$, corresponding to $83$ GHz. We don’t detect modes $m\geq
9$, which we attribute to dissipation in the emitter junction shunting the
high-frequency Josephson radiation. In essence, the normal resistance $R_{N}$
of the junction and its coupling inductance $L\sim 100$ pH induce a cutoff
frequency at $R_{N}/L\sim 15$ GHz above which the signal decreases as
$1/(f_{J})^{2}$, until it is too low to be detectable within a reasonable
averaging time. Although it limits our sensitivity at high frequencies, this
cutoff is less drastic than the $LC$ cutoff of tunnel junction-based
spectrometers, where the sensitivity is suppressed as $1/(f_{J})^{4}$ [29].
This cutoff frequency can be increased either by opting for an emitter with
higher normal resistance or by reducing the junction’s length to increase the
size of its minigap. This problem is however unlikely to arise when probing a
more nonlinear system, such as Andreev bound states, where the dispersive
coupling $\chi$ with the resonator should be much larger [48].

To further confirm that the identified resonances correspond to the modes of
the resonator, we adjust the flux $\Phi$ threading the RF SQUID loop. The
primary consequence is the modulation of the fundamental mode’s frequency
$f_{0}$, with a measured magnitude of $1.5$ MHz. Consequently, the higher
harmonic modes $f_{m}$ are also expected to modulate with $\Phi$. However in
practice, this modulation is not observable in the two-tone Josephson
spectrum, as it is much smaller than our resolution, limited by the Josephson
emission width. Nonetheless, changing the flux from $\Phi=0$ to
$\Phi=\phi_{0}/2$ dramatically alters the spectrum, as illustrated in Fig. 3e
for resonance $m=6$ around $64$ GHz. Instead of a peak, a resonant dip is
observed at $\Phi=\phi_{0}/2$, demonstrating an inverted frequency shift. This
change can be understood as a reversal of the Josephson inductance of the RF
SQUID when varying the phase drop across the junction [49]. This demonstrates
that the resonances detected using two-tone Josephson spectroscopy indeed
correspond to the modes of the $\lambda/4$ resonator.

##  V Conclusion and perspectives

These measurements demonstrate that SNS Josephson junctions behave as on-chip
broadband microwave emitters, capable of performing two-tone spectroscopy on
superconducting quantum circuits at unprecedented frequencies. With this
innovative technique, we were able to probe both a transmon qubit resonating
around $6$ GHz and the electromagnetic modes of a $\lambda/4$ resonator up to
$83$ GHz. Two-tone Josephson spectroscopy is here well adapted to detect these
highly coherent systems, with lifetimes $\sim 10\leavevmode\nobreak\ \mu$s and
$\sim 0.5\leavevmode\nobreak\ \mu$s respectively. All indications suggest that
this technique could operate at higher frequencies, albeit necessitating some
adjustments in both the emitter properties and its coupling with the system
under study.

More fundamentally, two-tone Josephson spectroscopy could be instrumental in
detecting fermionic or bosonic excitations in mesoscopic systems, such as
Andreev bound states in superconducting quantum dots [50, 51] or topological
Weyl band structures in multiterminal Josephson junctions [52, 53, 54]. It
could also prove valuable for studying quasiparticle generation [55] and
recombination in exotic 2D superconductors [56, 57, 58, 59]. Another promising
axis is to explore the quantum nature of Josephson radiation [39, 60, 40, 41,
61, 62, 63, 64, 65]. This could lead to the development of novel Josephson
photonics devices, such as quantum-limited amplifiers [60] or single-photon
detectors [66, 62], operating at high frequencies. More generally, the
architectures that we developed could be used to probe the high-frequency
environment of standard superconducting qubits and how it induces decoherence
on them. Finally, an exciting research direction involves sharpening the
emitted Josephson frequency, achievable either by reducing voltage noise or by
employing clever injection locking techniques [15, 67, 68, 69, 16]. This
advancement would enable time-domain measurement and control at unprecedented
frequencies, thereby opening up a whole new realm for high-frequency cQED.

###### Acknowledgements.

We first want to emphasize the invaluable help of the late F. Portier and P.
Jacques from the Nanoelectronics Group on low-noise electronics. We
acknowledge valuable discussions with the Quantronics Group, in particular
with C. Urbina for sharing the initial idea of two-tone Josephson
spectroscopy. Special thanks to Ç. Girit, J.-L. Smirr, and Z. Leghtas on
microwave measurements. Gratitude is extended to the SPEC of CEA-Saclay for
their help on nanofabrication, and to D. Roux and R. Mohammedi from LPMC for
their technical support. JDP acknowledges support of Agence Nationale de la
Recherche through grant ANR-20-CE47-0003. LB acknowledges support of the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (grant agreement No. 947707). This work has
been supported by the French ANR-22-PETQ-0003 grant under the France 2030
plan.

##  Appendix A Sample fabrication

Figure 4:  Schematics of the experimental setups for (a) Josephson emission
measurements, (b) two-tone Josephson spectroscopy of a transmon qubit and (c)
of a $\lambda/4$ resonator.

Resonator fabrication begins with sputtering $150$ nm of Nb onto an oxidized
high-resistivity silicon wafer, which is subsequently diced into $10\times 11$
mm2 chips. For processing a single chip, a layer of S1813 resist is spun on
top of the Nb. Following this, the negative of the resonator pattern is
defined using a laser writer and developed in MF319. Subsequently, reactive
ion etching is utilized to etch away the Nb through the resist mask. The
resist is then removed by immersing the chip in acetone maintained at
$60^{\circ}$C for 1 hour, followed by sonication in an ultrasonic bath.

Our SNS junctions are fabricated via metallic deposition on the same chip
using a suspended resist mask. For this, a bilayer of MAA EL6 resist is spun
at $4000$ rpm, followed by a layer of PMMA A6 at a slower speed of $3000$ rpm.
This results in a bilayer with a thin bottom layer ($210$ nm) and a very thick
top layer ($520$ nm). The SNS junction mask is defined using e-beam
lithography. The bilayer is then developed in a 1:3 MIBK/IPA solution for 1
minute, which provides a suspended mask. The chip is then loaded into a metal
evaporator. Before metal deposition, we perform an ion-milling step using an
Argon plasma to remove the layer of Nb oxide at the chip surface, which could
hinder proper electrical contact between the SNS junction and the voltage
leads. Initially, $5$ nm of Ti is evaporated to serve as a sticking layer for
subsequent metals. This is followed by a $15$ nm Au deposition. The sample is
then tilted by $22^{\circ}$, and $60$ nm of Nb is evaporated. Nb is deposited
almost everywhere over the gold layer but not at the constriction level, where
it is blocked by the mask of PMMA due to the angle, thus preventing from
shorting the SNS junction. The pressure inside the loadlock is maintained
below a few $10^{-7}$ mbar throughout the process to ensure a clean SN
interface. Finally, the resist is lifted off after evaporation in a hot
acetone bath.

##  Appendix B Measurement setups

In Fig. 4, we represent the three implementations of the experiments presented
in this manuscript.

The setup in Fig. 4a corresponds to one used for the measurement of the
Josephson emission, _i.e._ , the experimental data presented in Fig. 1c. The
power emitted by the junction is directed, via a bias tee, towards a power
spectrum analyzer (PSA) through a ZX60-83LN12+ room-temperature amplifier as
well as a 4-8 GHz HEMT amplifier from LNF anchored at 4K. The junction is
protected from the noise emitted by the HEMT by means of three 4-8 GHz
cryogenic isolators, a 12 GHz low-pass filter from $K\&L$, and a 4-8 GHz
bandpass filter from Micro-Tronics. This setup also allows for the measurement
of the current-voltage characteristic of the Josephson emitter. For this, a
voltage is applied with a Yokogawa GS200 source through a $\div 10^{4}$
voltage divider, and both voltage and current are measured using SP1004
voltage amplifiers from Basel Precision Instrument and a Yokogawa DL350
oscilloscope. The current is determined by measuring the voltage across a
resistor in series with the Josephson junction, whose resistance value (97
$\Omega$) was measured during the run. For all DC lines, we use Thermocoax
coaxial cables, RC lowpass filters and $\pi$-filters, both at 10 mK, to
protect the junctions from the noise. Detailed references of most components
can be found in reference [70].

Figure 5:  DC and AC characterization of the Al($100$ nm)-Hf($7$ nm)-Al($100$
nm) Josephson junction. (a) Current-voltage characteristic $I_{J}(V_{J})$. The
switching current is 385 nA and its normal resistance is 75 $\Omega$. (b)
Excess noise $S_{P}$ emitted by the junction at 5 GHz, in units of the HEMT
noise background $k_{B}T_{N}$. The red arrow indicates the AC Josephson effect
peak.

In Fig. 4b, we show the setup used for the two-tone Josephson spectroscopy of
a transmon qubit (Fig. 2). The DC part of the setup is similar than before,
but the RF part is changed significantly. This time the RF port of the bias
tee directs the Josephson emission towards the readout port of a transmon
qubit through a 4-8 GHz bandpass filter, a 12 GHz lowpass filter and a
circulator. The same port is also used for cavity spectroscopy by sending a
microwave signal from a ZNB20 vector network analyzer (VNA) through a -60 dB
attenuator, a 12 GHz low-pass filter, and then through the -20 dB port of a
directional coupler. The latter allows the VNA signal to be combined with the
one emitted by the junction, enabling two-tone spectroscopy of the transmon
qubit. Once the microwave signal from the VNA is reflected off the cavity, it
is redirected back to the VNA, passing through three 4-8 GHz isolators, a 12
GHz low-pass filter, and a similar amplification chain as before.

The final setup, depicted in Fig. 4c, corresponds to the two-tone Josephson
spectroscopy experiment of the $\lambda/4$ resonator shown in Fig. 3. Although
not fully represented, the DC part is similar to the previous two setups, with
a few minor modifications. Additionally, this setup includes a flux line that
allows a current to flow close to the superconducting loop terminating the
resonator, enabling us to adjust the magnetic flux in the loop. The
spectroscopy of the fundamental mode of the resonator is performed in a
standard manner, with a VNA connected to an attenuated input line and an
output line with a similar amplification chain as before.

##  Appendix C Characterization of SNS junctions

The current-voltage characteristic of the Al-Hf-Al junction, used for the
experiments described in Fig. 1 and 2, is shown in Fig. 5a. Unlike our Nb-Au-
Nb junctions, it was not designed for an efficient evacuation of Joule power.
Consequently, it exhibits a significantly lower retrapping current compared to
its switching current. In order to characterize the Al-Hf-Al junction as a
microwave source, we measured the emitted power in a narrow band centered at 5
GHz as a function of the bias voltage (see Fig. 5b). The narrow peak close to
10 $\mu V$, indicated by a red arrow, corresponds to the AC Josephson effect.
The continuously increasing background corresponds to incoherent noise, due to
shot noise and thermal noise, and remains lower than the AC Josephson emission
in the voltage bias range of interest.

##  Appendix D Calibration of the Josephson frequency

In order to properly calibrate the frequency $f_{J}$ of the signal emitted by
the SNS junction, it is crucial to accurately know the voltage $V_{J}$ across
its terminals. As we apply a voltage $V_{b}$ to the bias line of the junction,
we simultaneously measure $V_{J}$ using a low-noise amplifier. As depicted in
Fig. 6, these two voltages are related by the equation $V_{J}^{2}=\epsilon
V_{b}^{2}-V_{0}^{2}$, which is consistent with the RCSJ model [71]. Here
$\epsilon$ is a constant that depends on the attenuation of our line and
$V_{0}$ is a constant extracted from the fit. This relation allows us to
determine the actual value of $V_{J}$, and thus the Josephson frequency
$f_{J}=2eV_{J}/h$, without constantly monitoring it when performing two-tone
Josephson spectroscopy. Since our amplifier gain has a systematic error of
about $1\%$, we use in Section IV the $64.35$ GHz mode from Fig. 3d as a
reference to calibrate $f_{J}$ even more precisely.

Figure 6:  Voltage $V_{J}$ across the Nb-Au-Nb SNS junction measured as a
function of the bias voltage $V_{b}$. This allows us to calibrate de Josephson
frequency $f_{J}$, which is displayed on the right y-axis.

##  Appendix E Standard RF measurements of the transmon qubit

The complete design of the circuit investigated in Section III, which mostly
consists in a transmon qubit capacitively coupled to a microwave resonator, is
described in refs. [46, 47]. In a preliminary run, we characterized this
circuit-QED architecture using standard microwave instruments. Fig. 7 shows
the resonator and transmon qubit spectra. We could thus extract the
characteristic frequencies $f_{r0}\approx 8.1775$ GHz, $f_{q0}\approx 5.9136$
GHz and $\chi_{0}\approx 10.7$ MHz. The qubit frequency and dispersive shift
are slightly different than the one we identified later using two-tone
Josephson spectroscopy, which we attribute to the transmon’s ageing between
the two runs.

Figure 7:  Resonator and transmon qubit spectra. (a) One-tone spectroscopy of
the resonator: reflection amplitude as a function of frequency for the qubit
in the ground state (blue curve) and in an excited mixed state following a
saturation pulse at the qubit resonant frequency (orange curve). (b) Two-tone
spectroscopy of the transmon qubit. Here the excitation tone is provided by a
commercial microwave source.

Going further, we have probed the qubit quantum coherence by performing
operations in the time domain. The strong coupling between qubit and cavity
allows one to perform single-shot measurements of the qubit state. We could
thus measure the probability $P_{|e\rangle}$ of the qubit to be in the excited
state as a function of time. Using Rabi oscillations, one can calibrate $\pi$
and $\pi/2$-pulses. Thus, by applying a $\pi$-pulse that prepares the qubit in
state $|e\rangle$, we could measure after a variable delay an exponential
decay trace with an energy relaxation time $T_{1}\approx
39.9\leavevmode\nobreak\ \mu$s (Fig. 8a). Finally, we have measured the
dephasing for an equal qubit superposition $(|g\rangle+|e\rangle)/\sqrt{2}$,
by applying two $\pi/2$-pulses detuned at $f_{d}-f_{q}\approx 1$ MHz and
separated by a varying delay time. Fig. 8b shows the corresponding Ramsey
oscillations, which indicate coherent precession of the qubit state in the
Bloch sphere around the z-axis. From the decay, we could extract a dephasing
time $T_{2}^{*}\approx 16.2\leavevmode\nobreak\ \mu$s. This value being much
smaller than $2T_{1}$, our qubit coherence is dominated by pure dephasing with
a characteristic time $T_{\phi}\approx 20\leavevmode\nobreak\ \mu$s.

Figure 8:  Time-domain measurements of the transmon qubit. (a) $T_{1}$
relaxation curve: qubit population as a function of delay time
$\tau_{\mathrm{relax}}$ after a $\pi$-pulse, which was calibrated from the
Rabi oscillations. The solid line is a fit to an exponential decay that yields
$T_{1}\approx 39.9\leavevmode\nobreak\ \mu$s. (b) Ramsey oscillations: qubit
population as a function of delay time between two $\pi/2$-pulses
$\tau_{\mathrm{Ramsey}}$ detuned at $f_{d}-f_{q}\approx 1$ MHz. The fitting
(solid line) yields $T_{2}^{*}\approx 16.2\leavevmode\nobreak\ \mu$s.

##  Appendix F Complementary two-tone Josephson spectroscopy measurements

In Fig. 9a, we present two-tone Josephson spectroscopy measurements performed
in an additional sample similar to the one of Fig. 3. The mutual inductance
between the resonator and the superconducting loop was however designed to be
smaller, which reduces Kerr effect. A dip in the amplitude of the reflected
signal indicates the position of the $m=0$ mode of the microwave resonator. As
we sweep the bias voltage, the Josephson frequency happens to match the
frequency of a higher mode. As the latter is populated by microwave photons,
the $m=0$ resonance shifts by a few kHz, which allows the detection of high
frequency modes.

In Fig. 9b, we present measurements performed on another device, where the
coupling between the Josephson emitter and the resonator was made much
stronger. We detect many more resonances than in the measurements presented in
the main text; however, they do not correspond to the bare resonances
$f_{m}=(2m+1)f_{0}$ of the $\lambda/4$ resonator. We believe that, with the
coupling between the emitter and the resonator being stronger, the modes of
the resonator are perturbed by the emitter, and in this case, the spectroscopy
is too invasive not to perturb the system that we probe.

Figure 9:  (a) Two-tone Josephson spectroscopy for a sample similar to the one
of Fig. 3, with a smaller coupling between the resonator and the
superconducting loop. Reflection coefficient as a function of the bias voltage
$V_{b}$. The frequency $\delta f$ is referred to the resonance frequency of
the cavity in the absence of bias voltage. The green arrows indicate the modes
$m=2$, $3$, $4$ and $5$ that we detect in this experiment. (b) Two-tone
Josephson spectroscopy for a sample with a much larger emitter-detector
coupling. In this sample we detect many more modes than the harmonic
resonances of the cavity, which we attribute to hybridization of the resonator
and the emitter junction circuit lines.

##  Appendix G Two-tone Josephson spectroscopy with a tunnel Josephson
junction

One of the reasons we opt for an SNS junction over an SIS tunnel junction as
the Josephson emitter is due to the SNS junction’s relatively constant
impedance when a voltage is applied across its terminals, compared to the
tunnel junction. In Fig. 10, we show measurements of a two-tone Josephson
spectroscopy, similar to Fig. 3d, but conducted with an SIS junction as the
emitter. We observed significant signal variations with numerous structures
resembling resonances. However, they do not necessarily signify mode
detections but are sometimes simply due to changes of the resonance frequency
of the resonator fundamental mode ($m=0$) caused by impedance variations of
the SIS junction. Consequently, conducting two-tone Josephson spectroscopy
with this type of junction is challenging.

Figure 10:  Two-tone Josephson spectroscopy performed using a tunnel Josephson
junction as an emitter and a $7$ GHz resonator as a detector. We interpret the
features at $100$ GHz and $200$ GHz as excitation of quasi-particles.

## References

  * Schoelkopf and Girvin [2008] R. J. Schoelkopf and S. M. Girvin, Nature 451, 664 (2008).
  * Blais  _et al._ [2020] A. Blais, S. M. Girvin, and W. D. Oliver, Nat. Phys. 16, 247 (2020).
  * Clerk  _et al._ [2020] A. A. Clerk, K. W. Lehnert, P. Bertet, J. R. Petta, and Y. Nakamura, Nat. Phys. 16, 257 (2020).
  * Huang  _et al._ [2021] S. Huang, B. Lienhard, G. Calusine, A. Vepsäläinen, J. Braumüller, D. K. Kim, A. J. Melville, B. M. Niedzielski, J. L. Yoder, B. Kannan, T. P. Orlando, S. Gustavsson, and W. D. Oliver, PRX Quantum 2, 1 (2021), arXiv:2012.01438 .
  * Anferov  _et al._ [2024] A. Anferov, S. P. Harvey, F. Wan, J. Simon, and D. I. Schuster, arXiv:2402.03031 , 1 (2024), arXiv:2402.03031 .
  * Josephson [1964] B. Josephson, Rev. Mod. Phys. 1918, 216 (1964).
  * Likharev [1986] K. Likharev,  _Dynamics of Josephson Junctions and Circuits_, edited by C. Press (1986) p. 614.
  * Yanson  _et al._ [1965] I. Yanson, V. Svistunov, and I. Dmitrenko, Sov. Phys. JETP 21, 650 (1965).
  * Langenberg  _et al._ [1966] D. N. Langenberg, D. J. Scalapino, and B. N. Taylor, Proc. IEEE 54, 560 (1966).
  * Dayem and Grimes [1966] A. H. Dayem and C. C. Grimes, Appl. Phys. Lett. 9, 47 (1966).
  * Varmazis  _et al._ [1977] C. Varmazis, J. E. Lukens, and T. F. Finnegan, Appl. Phys. Lett. 30, 660 (1977).
  * Crozat  _et al._ [1978] P. Crozat, G. Vernet, and R. Adde, Appl. Phys. Lett. 32, 506 (1978).
  * Hansen  _et al._ [1979] J. B. Hansen, M. T. Levinsen, P. E. Lindelof, B. Dueholm, J. Mygind, N. F. Pedersen, and O. H. Soerensen, Appl. Phys. Lett. 35, 28 (1979).
  * Deacon  _et al._ [2017] R. S. Deacon, J. Wiedenmann, E. Bocquillon, F. Domínguez, T. M. Klapwijk, P. Leubner, C. Brüne, E. M. Hankiewicz, S. Tarucha, K. Ishibashi, H. Buhmann, and L. W. Molenkamp, Phys. Rev. X 7, 1 (2017), arXiv:1603.09611 .
  * Cassidy  _et al._ [2017] M. C. Cassidy, A. Bruno, S. Rubbert, M. Irfan, J. Kammhuber, R. N. Schouten, A. R. Akhmerov, and L. P. Kouwenhoven, Science (80-. ). 355, 939 (2017).
  * Yan  _et al._ [2021] C. Yan, J. Hassel, V. Vesterinen, J. Zhang, J. Ikonen, L. Grönberg, J. Goetz, and M. Möttönen, Nat. Electron. 4, 885 (2021), arXiv:2103.07617 .
  * Devoret  _et al._ [1990] M. Devoret, D. Esteve, H. Grabert, and G. Ingold, Phys. Rev. Lett. 64, 1824 (1990).
  * Ingold and Nazarov [1992] G.-L. Ingold and Y. Nazarov, in  _Single Charg. tunneling_ , Vol. 294, edited by M. D. H. Grabert (Plenum Press, New York, 1992) pp. 21–107, arXiv:0508728v1 [arXiv:cond-mat] .
  * Silver and Zimmerman [1967] A. H. Silver and J. E. Zimmerman, Appl. Phys. Lett. 10, 142 (1967).
  * Pedersen  _et al._ [1972] N. F. Pedersen, T. F. Finnegan, and D. N. Langenberg, Phys. Rev. B 6, 4151 (1972).
  * Pedersen  _et al._ [1976] N. F. Pedersen, O. H. Soerensen, J. Mygind, P. E. Lindelof, M. T. Levinsen, and T. D. Clark, Appl. Phys. Lett. 28, 562 (1976).
  * Edstam and Olsson [1994] J. Edstam and H. K. Olsson, Appl. Phys. Lett. 64, 2733 (1994).
  * Holst  _et al._ [1994] T. Holst, D. Esteve, C. Urbina, and M. Devoret, Phys. Rev. Lett. 73, 3455 (1994).
  * Griesmar  _et al._ [2021] J. Griesmar, R. H. Rodriguez, V. Benzoni, J. D. Pillet, J. L. Smirr, F. Lafont, and Girit, Phys. Rev. Res. 3, 1 (2021), arXiv:2106.02632 .
  * Lindell  _et al._ [2003] R. Lindell, J. Penttilä, M. Sillanpää, and P. Hakonen, Phys. Rev. B - Condens. Matter Mater. Phys. 68, 2 (2003).
  * Leppäkangas  _et al._ [2006] J. Leppäkangas, E. Thuneberg, R. Lindell, and P. Hakonen, Phys. Rev. B 74, 054504 (2006).
  * Billangeon  _et al._ [2007] P.-M. Billangeon, F. Pierre, H. Bouchiat, and R. Deblock, Phys. Rev. Lett. 98, 126802 (2007).
  * Basset  _et al._ [2010] J. Basset, H. Bouchiat, and R. Deblock, Phys. Rev. Lett. 105, 166801 (2010).
  * Bretheau  _et al._ [2013a] L. Bretheau, Ç. Ö. Girit, H. Pothier, D. Esteve, and C. Urbina, Nature 499, 312 (2013a), arXiv:1305.4091 .
  * Bretheau [2013] L. Bretheau,  _Localized Excitations in Superconducting Atomic Contacts : Probing the Andreev Doublet_ , Phd thesis (2013).
  * Bretheau  _et al._ [2013b] L. Bretheau, Ç. Ö. Girit, C. Urbina, D. Esteve, and H. Pothier, Phys. Rev. X 3, 041034 (2013b), arXiv:1311.3540 .
  * Van Woerkom  _et al._ [2017] D. J. Van Woerkom, A. Proutski, B. Van Heck, D. Bouman, J. I. Väyrynen, L. I. Glazman, P. Krogstrup, J. Nygård, L. P. Kouwenhoven, and A. Geresdi, Nat. Phys. 13, 876 (2017), arXiv:1609.00333 .
  * Safonova  _et al._ [2024] V. Y. Safonova, A. V. Gordeeva, A. V. Blagodatkin, D. A. Pimanov, A. A. Yablokov, O. L. Ermolaeva, and A. L. Pankratov, Materials (Basel). 17, 10.3390/ma17010222 (2024).
  * Courtois  _et al._ [2008] H. Courtois, M. Meschke, J. T. Peltonen, and J. P. Pekola, Phys. Rev. Lett. 101, 067002 (2008).
  * Basset  _et al._ [2019] J. Basset, M. Kuzmanović, P. Virtanen, T. T. Heikkilä, J. Estève, J. Gabelli, C. Strunk, and M. Aprili, Phys. Rev. Res. 1, 1 (2019), arXiv:1902.04898 .
  * Dassonneville  _et al._ [2018] B. Dassonneville, A. Murani, M. Ferrier, S. Guéron, and H. Bouchiat, Phys. Rev. B 97, 1 (2018), arXiv:1801.09273 .
  * Langenberg [1974] D. Langenberg, Rev. Phys. Appliquée 9, 35 (1974).
  * Poulsen [1974] U. K. Poulsen, Rev. Phys. Appliquée 9, 41 (1974).
  * Hofheinz  _et al._ [2011] M. Hofheinz, F. Portier, Q. Baudouin, P. Joyez, D. Vion, P. Bertet, P. Roche, and D. Esteve, Phys. Rev. Lett. 106, 217005 (2011).
  * Rolland  _et al._ [2019] C. Rolland, A. Peugeot, S. Dambach, M. Westig, B. Kubala, Y. Mukharsky, C. Altimiras, H. Le Sueur, P. Joyez, D. Vion, P. Roche, D. Esteve, J. Ankerhold, and F. Portier, Phys. Rev. Lett. 122, 186804 (2019), arXiv:1810.06217 .
  * Peugeot  _et al._ [2021] A. Peugeot, G. Ménard, S. Dambach, M. Westig, B. Kubala, Y. Mukharsky, C. Altimiras, P. Joyez, D. Vion, P. Roche, D. Esteve, P. Milman, J. Leppäkangas, G. Johansson, M. Hofheinz, J. Ankerhold, and F. Portier, Phys. Rev. X 11, 1 (2021), arXiv:2010.03376 .
  * Albert  _et al._ [2024a] R. Albert, J. Griesmar, F. Blanchet, U. Martel, N. Bourlet, and M. Hofheinz, Phys. Rev. X 14, 011011 (2024a).
  * Bumble  _et al._ [2009] B. Bumble, A. Fung, A. B. Kaul, A. W. Kleinsasser, G. L. Kerber, and P. Bunyk, IEEE Trans. Appl. Supercond. 19, 226 (2009).
  * Tolpygo  _et al._ [2015] S. K. Tolpygo, V. Bolkhovsky, T. J. Weir, L. M. Johnson, M. A. Gouker, and W. D. Oliver, IEEE Trans. Appl. Supercond. 25, 10.1109/TASC.2014.2374836 (2015).
  * Grimm  _et al._ [2017] A. Grimm, S. Jebari, D. Hazra, F. Blanchet, F. Gustavo, J. L. Thomassin, and M. Hofheinz, Supercond. Sci. Technol. 30, 10.1088/1361-6668/aa8007 (2017), arXiv:1705.05608 .
  * Balembois [2023] L. Balembois,  _Magnetic resonance of a single electron spin and its magnetic environment by photon counting_ , Phd thesis, Université Paris Saclay (2023).
  * Balembois  _et al._ [2024] L. Balembois, J. Travesedo, L. Pallegoix, A. May, E. Billaud, M. Villiers, D. Estève, D. Vion, P. Bertet, and E. Flurin, Phys. Rev. Appl. 21, 1 (2024).
  * Park  _et al._ [2020] S. Park, C. Metzger, L. Tosi, M. F. Goffman, C. Urbina, H. Pothier, and A. L. Yeyati, Phys. Rev. Lett. 125, 077701 (2020), arXiv:2007.05030 .
  * Wustmann and Shumeiko [2017] W. Wustmann and V. Shumeiko, Physical Review Applied 8, 024018 (2017), publisher: American Physical Society.
  * Pillet  _et al._ [2010] J.-D. Pillet, C. H. L. Quay, P. Morfin, C. Bena, A. Levy Yeyati, and P. Joyez, Nat. Phys. 6, 965 (2010).
  * Deacon  _et al._ [2010] R. S. Deacon, Y. Tanaka, A. Oiwa, R. Sakano, K. Yoshida, K. Shibata, K. Hirakawa, and S. Tarucha, Physical Review Letters 104, 076805 (2010), publisher: American Physical Society.
  * Fatemi  _et al._ [2021] V. Fatemi, A. R. Akhmerov, and L. Bretheau, Phys. Rev. Res. 013288, 1 (2021), arXiv:2008.13758 .
  * Peyruchat  _et al._ [2021] L. Peyruchat, J. Griesmar, J. D. Pillet, and Girit, Phys. Rev. Res. 3, 1 (2021), arXiv:2009.03291 .
  * Peyruchat  _et al._ [2024] L. Peyruchat, R. H. Rodriguez, J. L. Smirr, R. Leone, and Ç. Ö. Girit, arXiv:2401.10876 (2024), arXiv:2401.10876 .
  * Liu  _et al._ [2024] C.-H. Liu, D. C. Harrison, S. Patel, C. D. Wilen, O. Rafferty, A. Shearrow, A. Ballard, V. Iaia, J. Ku, B. L. T. Plourde, and R. McDermott, Phys. Rev. Lett. 132, 017001 (2024), arXiv:2203.06577 .
  * Fatemi  _et al._ [2018] V. Fatemi, S. Wu, Y. Cao, L. Bretheau, Q. D. Gibson, K. Watanabe, T. Taniguchi, R. J. Cava, and P. Jarillo-Herrero, Science 362, 926 (2018).
  * Cao  _et al._ [2018] Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018), arXiv:1803.02342 .
  * Hao  _et al._ [2021] Z. Hao, A. M. Zimmerman, P. Ledwith, E. Khalaf, D. H. Najafabadi, K. Watanabe, T. Taniguchi, A. Vishwanath, and P. Kim, Science (80-. ). 371, 1133 (2021).
  * Park  _et al._ [2022] J. M. Park, Y. Cao, L. Q. Xia, S. Sun, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Nat. Mater. 21, 877 (2022).
  * Jebari  _et al._ [2018] S. Jebari, F. Blanchet, A. Grimm, D. Hazra, R. Albert, P. Joyez, D. Vion, D. Estève, F. Portier, and M. Hofheinz, Nat. Electron. 1, 223 (2018), arXiv:1704.04432 .
  * Ménard  _et al._ [2022] G. C. Ménard, A. Peugeot, C. Padurariu, C. Rolland, B. Kubala, Y. Mukharsky, Z. Iftikhar, C. Altimiras, P. Roche, H. Le Sueur, P. Joyez, D. Vion, D. Esteve, J. Ankerhold, and F. Portier, Phys. Rev. X 12, 1 (2022), arXiv:2111.09604 .
  * Albert  _et al._ [2024b] R. Albert, J. Griesmar, F. Blanchet, U. Martel, N. Bourlet, and M. Hofheinz, Phys. Rev. X 14, 1 (2024b), arXiv:2303.03173 .
  * Leppäkangas  _et al._ [2015] J. Leppäkangas, M. Fogelström, A. Grimm, M. Hofheinz, M. Marthaler, and G. Johansson, Phys. Rev. Lett. 115, 027004 (2015), arXiv:1507.02885 .
  * Souquet and Clerk [2016] J. R. Souquet and A. A. Clerk, Phys. Rev. A 93, 1 (2016), arXiv:1601.05077 .
  * Leppäkangas and Marthaler [2018] J. Leppäkangas and M. Marthaler, Phys. Rev. B 98, 1 (2018), arXiv:1803.06654 .
  * Leppäkangas  _et al._ [2018] J. Leppäkangas, M. Marthaler, D. Hazra, S. Jebari, R. Albert, F. Blanchet, G. Johansson, and M. Hofheinz, Phys. Rev. A 97, 1 (2018), arXiv:1612.07098 .
  * Bengtsson  _et al._ [2018] A. Bengtsson, P. Krantz, M. Simoen, I.-M. Svensson, B. Schneider, V. Shumeiko, P. Delsing, and J. Bylander, Phys. Rev. B 97, 144502 (2018).
  * Marković  _et al._ [2019] D. Marković, J. D. Pillet, E. Flurin, N. Roch, and B. Huard, Phys. Rev. Appl. 12, 1 (2019), arXiv:1904.01380 .
  * Danner  _et al._ [2021] L. Danner, C. Padurariu, J. Ankerhold, and B. Kubala, Phys. Rev. B 104, 1 (2021), arXiv:2105.02564 .
  * Peugeot [2020] A. Peugeot,  _Quantum microwave sources from inelastic Cooper pair tunneling_ , phdthesis, Université Paris-Saclay (2020).
  * McCumber [1968] D. E. McCumber, Journal of Applied Physics 39, 3113 (1968).


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

The 10th edition of the PhD Symposium on FDIA, July 20, 2022, Lisbon,
Portugal.

[orcid=0000-0003-4392-2382<EMAIL_ADDRESS>]

# Text Information Retrieval in Tetun:
A Preliminary Study

Gabriel de Jesus  INESC TEC - Institute for Systems and Computer Engineering,
Technology and Science, Portugal  FEUP - Faculty of Engineering, University of
Porto, Portugal

(2022)

###### Abstract

Tetun is one of Timor-Leste’s official languages alongside Portuguese. It is a
low-resource language with over 932,400 speakers that started developing when
Timor-Leste restored its independence in 2002. The media mainly uses Tetun,
and more than ten national online newspapers actively broadcast news in Tetun
every day. However, since information retrieval-based solutions for Tetun do
not exist, finding Tetun information on the internet is challenging. This work
aims to investigate and develop solutions that can enable the application of
information retrieval techniques to develop search solutions for Tetun. We
present a preliminary result of an experiment conducted on the task of ad-hoc
retrieval in Tetun.

###### keywords:

Information retrieval Tetun Search Ad-hoc retrieval Low-resource language

##  1 Introduction and Motivation

Information retrieval deals with finding documents of unstructured-nature
(usually text) that satisfies an information need from within a large
collection [1]. Users of the retrieval system tend to search for information
on a topic of their interest and express it in many different ways, often in
very different words using natural language text [2]. They usually formulate
their information needs in a set of words in textual-based queries and expect
to get the documents relevant to their information needs. One important
information retrieval application is keyword-based web search, as showcased by
the Google search engine. Since specific information retrieval-based
approaches for Tetun do not exist, it becomes more challenging to find the
documents that satisfy information needs written in Tetun.

Tetun is the language spoken in Timor-Leste. There are two major varieties of
Tetun: Tetun Dili (referred to as Tetun) and Tetun Terik [3]. Tetun is the
most spoken language in the country, and Tetun Terik is one of the Timor-Leste
dialects. Most of the Tetun verb, noun, and adjective are Portuguese loanwords
[4, 5]. Klinken and Hajek [4] studied a selection of seven articles from
different newspapers in 2009 and stated that an average of 32% words are
Portuguese loans, and Greksáková [5] reported 35% of Portuguese loanwords in
analyzing 73,892 words from the interview transcripts corpus. As per the
census 2015 report [6], the population of Timor-Leste was 1.17 million and the
proportion of the Tetun speakers was 79.04%, including Portuguese (2.56%),
Indonesian (2.02%), and English (1.04%). Tetun became a dominant language in
public life when the Government of Timor-Leste designated it as one of the
Timor-Leste’s official languages in 2002. Then, the Government established the
National Institute of Linguistics (INL) [7] and produced the Standard
Orthography of Tetun Language (Tetun INL) [8].

Although Tetun INL was already in place, the Australian linguist Catharina v.
Klinken and her colleagues in the Tetun Department of the Dili Institute of
Technology (DIT) disputed the orthography and established their standard for
Tetun (Tetun DIT) [3]. There are some differences between Tetun INL and Tetun
DIT in writing structures, such as Tetun INL uses “ñ” and “ll”, but Tetun DIT
uses “nh” and “lh”, e.g., millaun (Tetun INL), milhaun (Tetun DIT).
Consequently, there is still a wide variation on the use of Tetun in official
writing and media. Moreover, Tetun is also influenced by Portuguese,
Indonesian, and English spoken in Timor-Leste. The query in Figure 1
illustrates a search performed by one of the Timor News visitors and captured
by the platform – [ opiniaun ba subsidi governu ba funsionario publiko ] –
“subsidi” is Indonesian (Tetun: subsídiu) and “funsionario publiko” are mixed
of Portuguese and Tetun (Tetun: funsionáriu públiku). We copied this query to
the Google search engine and executed it to observe the retrieved results. As
shown in note 1, Google suggested the opiniaun word in Tetun to English and
the funsionario in note 2 to Portuguese. This is the evidence that motivates
us to contribute to the development of information retrieval techniques for
Tetun.

Figure 1: Example of searching using Tetun in Google Search.

To tackle the aforementioned problem, this work aims to focus on developing
solutions for text information retrieval tasks for Tetun, including the
development of Tetun corpus. Based on the Google indicator, there are around
4.8 million documents available on the internet from the sites hosted under
“.tl” top-level domain. Therefore, we will study them in detail to identify
the text documents that are potentially relevant to our work. Considering that
online news has played a crucial role in promoting Tetun in the last five
years, from 2017 to 2022, as also indicated in the documents retrieved for the
query in Figure 1, we will use Timor News111https://www.timornews.tl as our
case study. Timor News will be used to test the search solutions prototype for
Tetun to evaluate further the algorithms’ performance. We conducted an initial
indexing and retrieval experiment using documents from Timor News, where our
preliminary result is in Section 5. Timor New is an online news agency based
in Dili, Timor-Leste, that only broadcasts national and international news in
Tetun, and as it is founded by the author of this paper, we have full access
to the platform (documents, search queries, etc.).

The remainder of this paper is organized as follows. Section 2 presents
related work. We describe our methodology in Section 3. Then, Sections 4 and 5
present research question and preliminary study results. Finally, Section 6
summarizes our conclusion and planned future work.

##  2 Related Work

Low-resource languages can be understood as languages that are less studied,
less computerized, low-density [9], that lack text corpora, or reduced
accessibility [10]. Developing information retrieval tools and techniques for
a low-resource language requires the availability of corpora and test
collections. Moreover, few algorithms support information retrieval for low-
resource languages, and considering linguistic diversity, some language-
specific algorithms, e.g., for text pre-processing and normalization, also
need to be developed.

Different strategies to build datasets for low-resource languages have been
studied. Artetxe et al. [11] built the Basque corpus using an ad-hoc crawling
approach — crawling 33 websites as the data sources — and stated that ad-hoc
crawling could be an effective alternative to obtain high-quality data for
low-resource languages. Issam and Mrini [12] and Mbonu et al. [13] built the
datasets for text summarization tasks for Moroccan Darija and Igbo and made
them publicly available. The first authors used GOUD.MA news website to build
the GOUD.MA dataset, and the latter used the Anambra Broadcasting Service
(ABS) website to build the IgboSum1500 dataset. Getao and Mariti [14] built
the Kiswahili corpus employing a query-based crawling. The queries were
manually created using bigram model and then crawled the documents for each
queries. Hoenen et al. [10] developed a manual for web corpus crawling of low-
resource languages with several data collection approaches. Moreover, the
practical tasks of web corpus construction described by Schäfer and Bildhauer
[15] can be used as the main reference. Some techniques for dataset
construction are presented in LABEL:tab:datasets.

Table 1: Dataset Construction Techniques. Technique | Data source | #Documents (or words) | Language
---|---|---|---
Ad-hoc crawling | 33 websites | 12.5M documents | Basque
Scraping | ABS website | 1.5k documents | Igbo
Scraping | News websites | 158k documents | Moroccan Darija
Query-based crawling | World Wide Web | 4.84M words | Kiswahili

For the text processing tasks, Jurish et al. [16] proposed a Hidden Markov
Model-based approach to segment automatically text documents into tokens and
sentences. Ferilli [17] presented a term-document frequency approach that
automatically detects stopwords from a small amount of the corpora and stated
that it was the most effective approach which outperformed the classic term
frequency [18, 2, 19] and the normalized inverse document frequency of Lo et
al. [20]. Trishala and Mamatha [21] proposed a rule-based Kannada stemmer
relying on an unsupervised approach using k-means algorithm, and Thangarasu
and Inbarani [22] presented an analogy removal stemmer that automatically stem
Tamil words from the text corpora. The state-of-art of query spelling
corrections explained by Chang et al. [23] are approaches applicable for low-
and high-resource languages.

The TREC-style approach based on the Cranfield paradigm is used in developing
test collections for low-resource languages. Chavula and Suleman [24] built a
test collection for Chichewa, Citumbuka, and Cinyanja, and the relevance
assessment was conducted by the five external evaluators. Esmaili et al. [25]
built a Pewan test collection for Sorani Kurdish and Aleahmad et al. [26]
created a Hamshahri test collection for Persian. The first test collection was
judged by the team members (native Sorani speakers), and Persian students
collaborated in the latter. Both test collections were built using news
datasets. The details are in LABEL:tab:test-collection.

Table 2: Test Collection Details. Name | #Documents | #Queries | #Run | #Pool documents | #Final Queries
---|---|---|---|---|---
C3222In the paper, the collection has no name assigned. We designated C3 to represent Chichewa, Citumbuka, and Cinyanja. | 11,778 | 387 | 4 | 38,700 | 387
Pewan | 115,340 | 42 | 5 | 2,500 | 22333For the reliability, queries with less than ten or more than 100 relevant documents were excluded.
Hamshahri | 166,774 | 65 | 7 | 6,500 | 62444Three queries from the collection that have less than ten or more than 90 relevant documents were omitted.

There are several existing search engines for low-resource languages, such as
the works of Malumba et al. [27] and Kyeyune [28] with search engines for
Bantu languages, IsiZulu and IsiXhosa. Both search engines are developed using
the same infrastructure — crawling the documents from the web and indexing and
retrieving them using the Solr platform.

Regarding work for Tetun text, to the best of our knowledge, no one has
conducted research in this area so far.

##  3 Methodology

In this section, we present the proposed steps for our PhD work focused on
text information retrieval in Tetun.

### Literature Review

Review the existing research on text information retrieval, specifically the
works for low-resource languages, to understand the concepts, methods, and
evidence that can be used to support our work. This step includes studying and
comparing the methods for document collection and characterization and text
pre-processing development.

### Documents Collection and Characterization

Build the collection of documents by collecting news articles and search
queries from the Timor News database. To enrich the collections, we will
develop a web crawler to collect text documents in Tetun from the sites hosted
under Timor-Leste top-level domain “.tl”. The Tumba!, a search engine that
involves crawling sites hosted under the “.pt” top-level domain, including
domains that contain a large set of Portuguese pages with incoming links from
web pages hosted under “.pt” [29] can be used as reference. The publicly
available legal documents and Wikipedia information in Tetun will also be
collected. After gathering and building the collection of documents, it will
be characterized according to each retrieval task.

### Retrieval System Development

Develop document and query processing methods for Tetun, such as tokenizer,
spelling correction, stopwords removal, and stemmer. Afterward, develop the
indexing and retrieval strategies and experiment with available indexing and
ranking tools, including the retrieval and ranking models.

### Building Test Collection

Build the TREC-style test collection for evaluation based on the Cranfield
approach [30]. The relevance judgments will require Timorese expertise in
different subjects, such as linguistics and computer science, and different
backgrounds, mainly from academia and industry, to compare their assessment
results.

### Evaluation

Asses retrieval system effectiveness using different evaluation measures and
then compare their results. Finally, identify the most effective solution that
will be used to integrate with the Timor News platform for further evaluation.

### Prototype Implementation

Implement the prototypes in the Timor News production environment to evaluate
the algorithms’ performance and the interaction mechanism with the user using
A/B tests.

##  4 Research Question

This research is focused on developing solutions for text information
retrieval in Tetun, therefore our main research questions are the following.

### Research Question 1

What text pre-processing techniques can influence the text retrieval results
in Tetun? According to the textbook of Baeza-Yates and Ribeiro-Neto [19, p.
8], an information retrieval system process comprise text transformation for
both document and query, query expansion and modification, document indexing,
retrieval, and ranking. The text transformation operations such as removing
stopwords and stemming are also applied for both document and query. For
queries, operations such as spelling correction and synonym expansion are also
enforced. Considering that text pre-processing methods for Tetun do not exist
and will be developed, this research question can extend our knowledge of
those influencing the text retrieval results for a given query written in
Tetun.

### Research Question 2

What indexing and retrieval strategies provide the most effective solutions
for Tetun text search? Text indexing is a central part of a retrieval system.
Different indexing tasks might have different results. Although our
preliminary experiment indicates that indexing the news title provides the
most relevant documents to the given queries, and is supported by the textbook
of Baeza-Yates and Ribeiro-Neto [19, p. 30], more experiments are needed to
prove this evidence. Moreover, since in recent years several works have
reported the promising results of applying neural language models in ad-hoc
retrieval tasks [31, 32, 33, 34], both traditional and neural-based approaches
will be adopted in the experiments.

### Research Question 3

Which query processing techniques improve the quality of the documents
retrieved? Since query processing involves parsing, expansion, and
modification steps, it is vital to understand in what condition it contributes
to improving the retrieval results. This research question broadens our
knowledge to analyze the retrieval solutions specifically to ensure that the
search solutions developed are explainable and understandable.

##  5 Preliminary Study Results

We report a preliminary result related to the experiment conducted on ad-hoc
retrieval in Tetun. We employed 442 documents and 5 search queries (or topics)
written in Tetun from the Timor News database. Documents are the news
articles, and queries are the search logs captured by the platform. The
retrieval system is developed using Apache Solr [35] as the indexing,
retrieval, and ranking platform.

As an initial experiment, we manually developed a list of stopwords for Tetun
by translating the English stopwords available in the
NLTK555https://www.nltk.org/ library into Tetun. Some of the stopwords are in
LABEL:tab:stopwords. Moreover, we developed a light rule-based stemmer for
Tetun by stemming to the root the noun words terminated with “saun”, “mentu”,
“dór’’, and “-teen”, e.g., selebrasaun (selebra: celebrate, selebrasaun:
celebration), juramentu (jura: promise, juramentu: promise), lohidór (lohi:
lie, lohidór: lier), and nauktén (nauk: steal, nauk-teen: thief). For the text
normalization, we developed a rule-based list, e.g., junho, junu $\rightarrow$
juñu (june), to correct the misspelled Tetun terms. The same rule-based list
is also used in constructing the lists of abbreviation and synonym expansions.
The misspelled, abbreviation expansion and synonym lists were built based on
the information extracted from the Timor News search query logs.

Table 3: Tetun Stopwords Translated from English. Tetun | English | Tetun | English
---|---|---|---
no | and | husi | from
atu | so that | ha’u | I
ne’e | this | sira | they
ne’ebé | which | tanba | because

For the experiments, we set up three configurations (default, with stopwords,
and without stemming) for query and document processing (referred to
LABEL:tab:config), and four indexing for documents. The elements of the
documents indexed and tested are the combination of (i) title (ii) title and
content, (iii) lead and content, and (iv) title, lead, and content. Thus, we
have 12 different retrieval strategies (runs). For each of the 12 retrieval
strategies and 5 queries, we produce top-30 ranked documents, therefore, we
have a pool of 1,800 documents (results) to be evaluated.

Table 4: Configuration of the Document and Query Processing. | Default | With stopwords | Without stemming
---|---|---|---
HTML removal | x | x | x
Case folding (lowercasing) | x | x | x
Tokenization | x | x | x
Text normalization | x | x | x
Stopwords removal | x |  | x
Abbreviation expansion | x | x | x
Synonym expansion | x | x | x
Light stemming | x | x |

To build the collection for evaluation, the aforementioned retrieval approach
is used, but the total number of ranked documents was reduced to ten (10),
then we have 120 documents for all retrieval strategies. We analyze and merge
10 documents out of 120 based on their ranked scores in each retrieval
strategy to be judged.

We developed relevance judgment guidelines for the evaluators to manually
judge the relevance of the documents for the given query. We adopt the
relevance scoring scale of TREC 2020 for the news track [36], with scores
ranging from 0 to 3 as in the following:

  * 0 \- the retrieved document provides no useful information (not relevant).

  * 1 \- the retrieved document provides some useful information or contextual information that would help the user understand the broader story context of the query article (less relevant).

  * 2 \- the retrieved document provides significant information (relevant).

  * 3 \- the retrieved document provides essential information (highly relevant).

Five queries and ten relevant documents for each of the queries are
incorporated into the guidelines and then distributed to five independent
Timoreses evaluators. The evaluators judge the relevance of the documents for
each query according to the relevance scoring scale. After evaluating, we
compile the judgment results and use majority voting to select the final score
for the documents retrieved for each query. Then, we produce 50 lists of
relevant documents (or qrels) used to evaluate the retrieval system’s
effectiveness.

We use the latest version of trec_eval666https://trec.nist.gov/trec_eval/ to
evaluate our retrieval system’s effectiveness. We perform 12 runs based on the
12 retrieval strategies. For each run, we have 150 documents (5 queries x 30
documents). We format results and qrels according to the TREC format, and then
use precision (P) at 5 and 10 (P@5, P@10), mean average precision (MAP), and
normalized discounted cumulative gain (nDCG) evaluation metrics to evaluate
the system’s effectiveness. Results show that, overall, without stemming with
title indexing outperformed all the other retrieval strategies, achieving the
nDCG of 0.79 and MAP of 0.73 (see LABEL:tab:evaluation).

Table 5: The Evaluation Results: Title (T), Lead (L), and Content (C). | MAP | P@5 | P@10 | nDCG
---|---|---|---|---
Default | T | 0.709 | 0.800 | 0.533 | 0.775
T+C | 0.662 | 0.667 | 0.567 | 0.715
L+C | 0.649 | 0.633 | 0.533 | 0.780
T+L+C | 0.652 | 0.633 | 0.517 | 0.732
With stopwords | T | 0.572 | 0.600 | 0.483 | 0.715
T+C | 0.612 | 0.600 | 0.533 | 0.705
L+C | 0.593 | 0.600 | 0.500 | 0.688
T+L+C | 0.648 | 0.667 | 0.517 | 0.729
Without Stemming | T | 0.731 | 0.800 | 0.550 | 0.790
T+C | 0.671 | 0.700 | 0.550 | 0.730
L+C | 0.650 | 0.700 | 0.517 | 0.719
T+L+C | 0.650 | 0.700 | 0.550 | 0.738

According to Baeza-Yates and Ribeiro-Neto [19, p 29-30], the page title is
usually shown prominently in web search, and a page title and combination of
the title containing query terms are one of the search attributes that leads a
search result to more clicks. This evidence reflects our result, the
effectiveness of indexing documents by the title. However, since Timor News
has established its news writing standard and designed it for the readers to
understand the news content quickly from the title, we believe that it also
contributes to the retrieval effectiveness result. Although without stemming
terms to the root achieved the best result, since the overall scores of the
default configuration (with stemming) is slightly lower, we assume that it can
be affected by the quality of stemming rules and the amount of the documents
and queries used in the experimentation. We take it into account to be
investigated further in the future.

##  6 Conclusion and Future Work

This paper presents our preliminary results from the first ad-hoc retrieval
experience using documents and search queries in Tetun collected from the
Timor News database. We combined different indexing and retrieval strategies
in the experiment to study the impact of each strategy on the quality of
retrieved documents for each query. The experiment result provides a
preliminary answer to the research questions 1 and 2 in Section 4. For future
work, we will follow the steps for developing text information retrieval for
Tetun outlined in Section 3.

##  7 Acknowledgment

This work is financed by National Funds through the Portuguese funding agency,
FCT - Fundação para a Ciência e a Tecnologia under the scholarship grant
reference number SFRH/BD/151437/2021.

## References

  * Manning et al. [2009] C. D. Manning, P. Raghavan, H. Schütze, An Introduction to Information Retrieval, Online edition, Cambridge UP, England, 2009. URL: http://www.informationretrieval.org/.
  * Croft et al. [2009] W. B. Croft, D. Metzler, T. Strohman, Search Engines - Information Retrieval in Practice, Pearson Education, 2009. URL: http://www.search-engines-book.com/.
  * Williams-van Klinken et al. [2002] C. Williams-van Klinken, J. Hajek, R. Nordlinger, Tetun Dili: a grammar of an East Timorese language, Pacific Linguistics, 2002. doi:10.15144/PL-528.FRONT.
  * Williams-van Klinken and Hajek [2018] C. Williams-van Klinken, J. Hajek,  Language contact and functional expansion in tetun dili: The evolution of a new press register,  Multilingua 37 (2018). doi:10.1515/multi-2017-0109.
  * Greksakova [2018] Z. Greksakova, Tetun in Timor-Leste: The role of language contact in its development, Ph.D. thesis, University of Coimbra, 2018. URL: http://hdl.handle.net/10316/80665.
  * de Estatística Timor-Leste [2016] D. G. de Estatística Timor-Leste, Timor-leste population and housing census 2015, 2016. URL: https://www.statistics.gov.tl/category/publications/census-publications/. Accessed July 10, 2022.
  * of Timor-Leste [2004] D. R. of Timor-Leste, Government decree no. 1/2004 of 14 april 2004: the standard orthography of the tetun language, 2004. URL: http://mj.gov.tl/jornal/lawsTL/RDTL-Law/RDTL-Gov-Decrees/Gov-Decree-2004-01.pdf. Accessed July 10, 2022.
  * of Linguistics [INL] N. I. of Linguistics (INL), The standard orthography of the tetum language: 115 years in the making, 2004. URL: https://archive.org/details/the-standard-orthography-of-the-tetum-language. Accessed July 10, 2022.
  * Magueresse et al. [2020] A. Magueresse, V. Carles, E. Heetderks,  Low-resource languages: A review of past work and future challenges,  CoRR abs/2006.07264 (2020). URL: https://arxiv.org/abs/2006.07264. arXiv:2006.07264.
  * Hoenen et al. [2020] A. Hoenen, C. Koc, M. D. Rahn,  A manual for web corpus crawling of low resource languages,  Umanistica Digitale 4 (2020) 342–344. URL: https://umanisticadigitale.unibo.it/article/view/9931. doi:10.6092/issn.2532-8816/9931.
  * Artetxe et al. [2022] M. Artetxe, I. Aldabe, R. Agerri, O. Perez-de-Viñaspre, A. Soroa,  Does corpus quality really matter for low-resource languages?,  CoRR abs/2203.08111 (2022). URL: https://doi.org/10.48550/arXiv.2203.08111. doi:10.48550/arXiv.2203.08111. arXiv:2203.08111.
  * Issam and Mrini [2021] A. Issam, K. Mrini,  Goud.ma: a news article dataset for summarization in moroccan darija,  in: 3rd Workshop on African Natural Language Processing, 2021. URL: https://openreview.net/forum?id=BMVq5MELb9.
  * Mbonu et al. [2021] C. E. Mbonu, C. I. Chukwuneke, R. U. Paul, I. Ezeani, I. Onyenwe,  IGBOSUM1500 - introducing the IGBO text summarization dataset,  in: 3rd Workshop on African Natural Language Processing, 2021. URL: https://openreview.net/forum?id=rMUccG4LZq.
  * Getao and Miriti [2005] K. W. Getao, E. K. Miriti,  Automatic construction of a kiswahili corpus from the World Wide Web,  Fountain Publisher, 2005. URL: http://erepository.uonbi.ac.ke/handle/11295/50524.
  * Schäfer and Bildhauer [2013] R. Schäfer, F. Bildhauer, Web Corpus Construction, Synthesis Lectures on Human Language Technologies, Morgan & Claypool Publishers, 2013. URL: https://doi.org/10.2200/S00508ED1V01Y201305HLT022. doi:10.2200/S00508ED1V01Y201305HLT022.
  * Jurish and Würzner [2013] B. Jurish, K. Würzner,  Word and sentence tokenization with hidden markov models,  J. Lang. Technol. Comput. Linguistics 28 (2013) 61–83. URL: http://www.jlcl.org/2013_Heft2/3Jurish.pdf.
  * Ferilli [2021] S. Ferilli,  Automatic multilingual stopwords identification from very small corpora,  Electronics 10 (2021). URL: https://www.mdpi.com/2079-9292/10/17/2169. doi:10.3390/electronics10172169.
  * Ferilli et al. [2020] S. Ferilli, G. L. Izzi, T. Franza,  Automatic stopwords identification from very small corpora,  in: M. Stettinger, G. Leitner, A. Felfernig, Z. W. Ras (Eds.), Intelligent Systems in Industrial Applications, 25th International Symposium, ISMIS 2020, Graz, Austria, September 23-25, 2020, Selected Papers from the Industrial Part, volume 949 of Studies in Computational Intelligence, Springer, 2020, pp. 31–46. URL: https://doi.org/10.1007/978-3-030-67148-8_3. doi:10.1007/978-3-030-67148-8_3.
  * Baeza-Yates and Ribeiro-Neto [2011] R. Baeza-Yates, B. A. Ribeiro-Neto, Modern Information Retrieval - the concepts and technology behind search, Second edition, Pearson Education Ltd., Harlow, England, 2011. URL: http://www.mir2ed.org/.
  * Lo et al. [2005] R. T. Lo, B. He, I. Ounis,  Automatically building a stopword list for an information retrieval system,  J. Digit. Inf. Manag. 3 (2005) 3–8. URL: http://www.dirf.org/jdim/abstractv3i1.htm#01.
  * Trishala and Mamatha [2021] G. Trishala, H. R. Mamatha,  Implementation of stemmer and lemmatizer for a low-resource language—kannada,  in: Proceedings of International Conference on Intelligent Computing, Information and Control Systems, Advances in Intelligent Systems and Computing 1272, 2021. URL: https://doi.org/10.1007/978-981-15-8443-5_28.
  * Mathan and Inbarani [2016] M. T. Mathan, H. Inbarani,  Analogy removal stemmer algorithm for tamil text corpora,  2016, pp. 70–81. doi:10.1007/978-981-10-3274-5_6.
  * Chang and Deng [2020] Y. Chang, H. Deng, Query Understanding for Search Engines, 2020. doi:10.1007/978-3-030-58334-7.
  * Chavula and Suleman [2021] C. Chavula, H. Suleman,  Ranking by language similarity for resource scarce southern bantu languages,  ICTIR ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 137–147. URL: https://doi.org/10.1145/3471158.3472251. doi:10.1145/3471158.3472251.
  * Sheykh Esmaili et al. [2013] K. Sheykh Esmaili, D. Elyasi, S. Salavati, P. Aliabadi, A. Mohammadi, S. Yosefi, S. Hakimi,  Building a test collection for sorani kurdish,  2013, pp. 1–7. doi:10.1109/AICCSA.2013.6616470.
  * Aleahmad et al. [2009] A. Aleahmad, H. Amiri, E. Darrudi, F. Oroumchian,  Hamshahri: A standard persian text collection,  Knowledge-Based Systems 22 (2009) 382–387. doi:10.1016/j.knosys.2009.05.002.
  * Malumba et al. [2015] N. Malumba, K. Moukangwe, H. Suleman,  Afriweb: A web search engine for a marginalized language,  2015, pp. 180–189. doi:10.1007/978-3-319-27974-9_18.
  * Kyeyune [2015] M. J. Kyeyune, Isixhosa search engine development report, 2015. Department of Computer Science, University of Cape Town. URL: https://pubs.cs.uct.ac.za/id/eprint/1035/. Accessed July 10, 2022.
  * Silva [2003] M. J. Silva,  Searching and archiving the web with tumba,  Conferência da Associação Portuguesa de Sistemas de Informação (2003). URL: http://hdl.handle.net/10451/14625.
  * Clough and Sanderson [2013] P. D. Clough, M. Sanderson,  Evaluating the performance of information retrieval systems using test collections,  Inf. Res. 18 (2013). URL: http://www.informationr.net/ir/18-2/paper582.html.
  * Nogueira et al. [2020] R. Nogueira, Z. Jiang, R. Pradeep, J. Lin,  Document ranking with a pretrained sequence-to-sequence model,  in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, Association for Computational Linguistics, 2020, pp. 708–718. URL: https://doi.org/10.18653/v1/2020.findings-emnlp.63. doi:10.18653/v1/2020.findings-emnlp.63.
  * Yang et al. [2019] W. Yang, H. Zhang, J. Lin,  Simple applications of BERT for ad hoc document retrieval,  CoRR abs/1903.10972 (2019). URL: http://arxiv.org/abs/1903.10972. arXiv:1903.10972.
  * Dai and Callan [2019] Z. Dai, J. Callan,  Deeper text understanding for IR with contextual neural language modeling,  in: B. Piwowarski, M. Chevalier, É. Gaussier, Y. Maarek, J. Nie, F. Scholer (Eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, ACM, 2019, pp. 985–988. URL: https://doi.org/10.1145/3331184.3331303. doi:10.1145/3331184.3331303.
  * Lin [2019] J. Lin,  The neural hype, justified!: a recantation,  SIGIR Forum 53 (2019) 88–93. URL: https://doi.org/10.1145/3458553.3458563. doi:10.1145/3458553.3458563.
  * Solr [2022] Solr, Apache solr platform, 2022. URL: https://solr.apache.org/. Accessed July 10, 2022.
  * Soboroff et al. [2020] I. Soboroff, S. Huang, D. Harman,  TREC 2020 news track overview,  in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 16-20, 2020, volume 1266 of NIST Special Publication, National Institute of Standards and Technology (NIST), 2020. URL: https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.N.pdf.