# Metrology of Rydberg states of the hydrogen atom

Simon Scheidegger  Departement of Chemistry and Applied Biosciences, ETH
Zurich, Zurich, Switzerland  Quantum Center, ETH Zurich, Zurich, Switzerland
Josef A. Agner  Departement of Chemistry and Applied Biosciences, ETH Zurich,
Zurich, Switzerland  Hansjürg Schmutz  Departement of Chemistry and Applied
Biosciences, ETH Zurich, Zurich, Switzerland  Frédéric Merkt
<EMAIL_ADDRESS>Departement of Chemistry and Applied Biosciences, ETH
Zurich, Zurich, Switzerland  Departement of Physics, ETH Zurich, Zurich,
Switzerland  Quantum Center, ETH Zurich, Zurich, Switzerland

###### Abstract

We present a method to precisly measure the frequencies of transitions to
high-$n$ Rydberg states of the hydrogen atom which are not subject to
uncontrolled systematic shifts caused by stray electric fields. The method
consists in recording Stark spectra of the field-insensitive $k=0$ Stark
states and the field-sensitive $k=\pm 2$ Stark states, which are used to
calibrate the electric field strength. We illustrate this method with
measurements of transitions from the 2s$(f=0\text{ and }1)$ hyperfine levels
in the presence of intentionally applied electric fields with strengths in the
range between 0.4 and $1.6\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$. The
slightly field-dependent $k=0$ level energies are corrected with a precisely
calculated shift to obtain the corresponding Bohr energies
$\left(-cR_{\mathrm{H}}/n^{2}\right)$. The energy difference between $n=20$
and $n=24$ obtained with our method agrees with Bohr’s formula within the 10
kHz experimental uncertainty. We also determined the hyperfine splitting of
the 2s state by taking the difference between transition frequencies from the
2s$(f=0\text{ and }1)$ levels to the $n=20,k=0$ Stark states. Our results
demonstrate the possibility of carrying out precision measurements in high-$n$
hydrogenic quantum states.

##  I Introduction

The hydrogen atom is a fundamental two-body quantum system. Studies of its
spectrum by experiment and theory have played a key role in the development of
the quantum theory Ångström (1868); Balmer (1885); Bohr (1913); Schrödinger
(1926); Dirac (1926) and of quantum electrodynamics Lamb and Retherford (1947,
1950); Bethe (1947); Bethe and Salpeter (1957). Spectroscopic measurements of
energy intervals between the quantum states of the hydrogen atom have reached
exceptional precision and the results can be exactly explained by first-
principles calculations and accurately known physical constants such as the
Rydberg constant $R_{\infty}$, the fine-structure constant $\alpha$ and the
proton charge radius ${r_{\rm p}}$. The theoretical treatment of the H atom by
relativistic quantum mechanics and quantum electrodynamics is indeed so
accurate that the comparison with the results of precision measurements in the
H atom can serve to determine the values of these constants Tiesinga  _et al._
(2021).

In the past years, a significant revision of the values of $R_{\infty}$ and
$r_{\rm p}$ became necessary after a new measurement of the Lamb shift in
muonic hydrogen Pohl  _et al._ (2010); Antognini  _et al._ (2013); Lensky  _et
al._ (2022) challenged earlier results from H-atom spectroscopy, a challenge
that was referred to as the proton-radius puzzle. This challenge essentially
results from the correlation between the $R_{\infty}$ and $r_{\rm p}$ values
which necessitates the combination of at least two transition frequencies in
the H atom to determine these constants. The latest CODATA values of
$R_{\infty}$ and $r_{\rm p}$ are based on a combination of multiple results,
in which the 1s–2s interval in H Parthey  _et al._ (2011); Matveev  _et al._
(2013) and the Lamb shift in muonic hydrogen Pohl  _et al._ (2010) play a
central role. Several recent precision measurements in H confirmed the revised
values Beyer  _et al._ (2017); Bezginov  _et al._ (2019) whereas others cover
the range between the old and the new values of $R_{\infty}$ and $r_{\rm p}$
Fleurbaey  _et al._ (2018); Grinin  _et al._ (2020); Brandt  _et al._ (2022).
Measurement of quantities that are only sensitive to either $r_{\rm p}$, such
as electron-scattering measurements Belushkin  _et al._ (2007); Bernauer  _et
al._ (2010); Sick (2012); Lee  _et al._ (2015); Higinbotham  _et al._ (2016);
Horbatsch  _et al._ (2017); Alarcón  _et al._ (2019); Xiong  _et al._ (2019);
Lin  _et al._ (2021), or $R_{\infty}$, such as measurements in non-penetrating
Rydberg series of H, have regained interest.

Early, remarkable experiments designed to determine $R_{\infty}$ from
transition frequencies between circular states, i.e., states with orbital-
angular-momentum quantum number $\ell=n-1$ and magnetic quantum number
$m_{\ell}=\pm\ell$, of high principal quantum numbers in the H atom were
carried out in the group of D. Kleppner at MIT Paine (1992); Lutwak (1997); De
Vries (2002), giving values of $R_{\infty}$ compatible with the recommended
CODATA values available at the time Mohr and Taylor (2000). In that work, the
frequencies of $\Delta n=1$ transitions between circular states of H were
measured with 2-3 Hz accuracy at $n$ values around 30. These transition
frequencies scale as $2R_{\infty}/n^{3}$ and are completely insensitive to the
proton size because the Rydberg electron does not penetrate in the core
region. The $2/n^{3}$ sensitivity factor to $R_{\infty}$ of these measurement
is only $\approx 1\times 10^{-4}$ for the transition between the $n=27,\
\ell=26,m_{\ell}=26$ and $n=28,\ \ell=27,m_{\ell}=27$, but this disadvantage
could be compensated by the fact that circular states are not sensitive to
stray electric fields to first order, and through the exceptional control of
all aspects of the millimeter-wave-spectroscopic experiments by the MIT team.
An $R_{\infty}$ value with an absolute uncertainty of 69 kHz and a relative
uncertainty of $2.1\times 10^{-11}$ was determined De Vries (2002), close to
the $R_{\infty}$ uncertainty value of $7.6\times 10^{-12}$ of the 1998 CODATA
adjustment. Since this pioneering work, circular Rydberg states of Rb have
been proposed as an alternative system to determine $R_{\infty}$ Ramos  _et
al._ (2017). The properties of circular Rydberg states of any atom or molecule
are indeed ideally suited to metrology, as illustrated by the use of such
states as ultrasensitive electric-field sensors Facon  _et al._ (2016).

If circular Rydberg states are excepted, high Rydberg states are usually not
considered to be suitable for precision measurements because of their high
sensitivity to stray electric fields (see discussion in, e.g., Refs. Garreau
_et al._ (1990); Beyer  _et al._ (2013)). In the context of metrology in the H
atom, this sensitivity has implied that almost all precision experiments
involving Rydberg states of H with $n\geq 3$ have targeted states with $n$
values below 12 de Beauvoir  _et al._ (1997); Beyer  _et al._ (2017);
Fleurbaey  _et al._ (2018) and that the measurements required a careful
evaluation of the Stark effect on the level structure induced by stray
electric fields.

We introduce here an alternative method to determine $R_{\infty}$ which relies
on measuring the spectra of $|m_{\ell}|=1$ Rydberg states of the H atom in the
presence of intentionally applied electric fields. Stark states of the H atom
exhibit shifts of $\approx 1.5a_{0}ekn\mathcal{F}$ that are linear in the
field strength $\mathcal{F}$ at low fields and proportional to the integer
difference $k=n_{1}-n_{2}$ between the quantum numbers $n_{1}$ and $n_{2}$
that arise in the solution of the Schrödinger equation in parabolic
coordinates ($k=0,\pm 1,\pm 2,\ldots,\pm(n-1-|m_{\ell}|)$, where $m_{\ell}$ is
the magnetic quantum number associated with the electron orbital motion) Bethe
and Salpeter (1957); Gallagher (1988). Consequently, even-$n$,
$k=0,|m_{\ell}|=1$ states are to first order field insensitive, as circular
Rydberg states. Their magnetic moments are, however, much smaller than for
circular states, which makes them less sensitive to Zeeman shifts by magnetic
stray fields. $|m_{\ell}|=1$ Stark states do not possess any $s$ character and
their $\ell$-mixed wavefunctions are dominated by nonpenetrating high-$\ell$
components; consequently, their spectral positions are also insensitive to the
proton size. Experimentally, we measure the frequencies of transitions from
the 2s($f=0$ and 1) states to $n=20,k=0,\pm 2,|m_{\ell}|$ Stark states and use
the separation between the $k=\pm 2$ states to precisely determine the value
of the applied field. We then extract the position of the $k=0$ state to
determine the Bohr energy $\left(-hcR_{\rm H}n^{-2}\right)$ after correcting
for the quadratic Stark shifts. To obtain a value of $R_{\infty}$ without
having to consider its correlation with $r_{\rm p}$, the position of the 2s
levels and the $n=20,k=0,\pm 2,|m_{\ell}|$ Stark states can be related to the
position of the 2p levels using the 2s Lamb shift determined by Bezginov et
al. Bezginov  _et al._ (2019). The sensitivity factor of the measurement to
$R_{\infty}$ is thus 1/4, i.e., more than 3000 times higher than for the
measurement based on circular states at $n\approx 30$. Consequently, an
accuracy of about 20 kHz would make this measurement competitive with the MIT
measurements and we believe that this is achievable. The price to pay for this
advantage is that the transition frequencies are in the UV range of the
electromagnetic spectrum rather than in the millimeter-wave range and,
therefore, compensation of the Doppler effect becomes much more critical.

In this article, we present several of the key aspects of this method of
determining $R_{\infty}$. We are still in the middle of the data-acquisition
process, and use subsets of the data to discuss systematic uncertainties in
the measurements of $nkm_{\ell}\leftarrow 2{\rm s}$ transition frequencies
originating from the Stark effect. We also present the determination of (i)
the $f=0-f=1$ hyperfine interval in the 2s state, which we obtain by combining
two sets of measurements, from 2s($f=0$) and 2s($f=1$) to $n=20$ Stark states,
and (ii) the difference between the $n=20$ and $n=24$ Bohr energies by
combining measurements from the 2s$(f=1)$ hyperfine state to $n=20$ and 24
Stark states. The article is structured as follows: Section II describes the
experimental setup and provides details on the laser systems used to prepare H
atoms selectively in the 2s($f=0$ and 1) hyperfine states and to record
spectra of the $nkm_{\ell}\leftarrow 2{\rm s}(f)$ transitions, as well as the
detection system and the procedure we follow to cancel the Doppler shifts.
Section III describes how we calculate the energies of the Stark states of H
and draws attention to the aspects that are most relevant for the
determination of the Bohr energies. Section IV illustrates the current status
of our measurements by using small data sets to compare spectra recorded at
different electric fields from the two hyperfine components of the 2s state
and to $n=20$ and 24 Stark states. The results we present here only concern
small energy intervals $(\sim$177\text{\,}\mathrm{MHz}$$ for the
2s($f=1\leftarrow f=0$) interval and $2.51\text{\,}\mathrm{THz}$ for the
difference between the Bohr energies at $n=20$ and 24) obtained by building
differences of (currently still blinded) UV laser frequencies. Absolute
transition frequencies will be reported when the analysis of the systematic
errors related to the Doppler effect is completed. In the last section, we
draw several conclusions concerning our new approach.

##  II Experimental setup

The experimental setup is presented schematically in Fig. 1. It consists of
(i) a differentially pumped set of vacuum chambers in which the H atoms are
produced and entrained in a pulsed supersonic beam and subsequently
photoexcited to Rydberg states via the metastable 2s state within a double-
layer mu-metal magnetic shield; (ii) a pulsed near-Fourier-transform-limited
laser system delivering radiation at $243\text{\,}\mathrm{nm}$ to drive the
$2{\rm s}\leftarrow 1{\rm s}$ transition; and (iii) an SI-traceable single-
mode continuous-wave (cw) UV laser to further excite the H atoms to Rydberg
states. The experiment is run in a pulsed mode at a repetition rate of
$25\text{\,}\mathrm{Hz}$.

The hydrogen atom source has been described in Ref. Scheidegger  _et al._
(2022) to which we refer for details.

Figure 1: Schematic representation of the experimental setup. Upper part:
laser system and geometry of the photoexcitation from the metastable 2s state
of H to Rydberg states. Lower part: vacuum chambers in which the supersonic
beam of H atoms is generated, these atoms are photoexcited to Rydberg states
and the Rydberg states are detected by pulsed field ionization Top right
inset: Configuration of laser and supersonic beams used for the determination
of Doppler-free frequencies. See text for details.

The hydrogen atoms are produced by dissociating molecular hydrogen in a
dielectric-barrier discharge near the orifice of a pulsed cryogenic valve and
are entrained in a supersonic beam of H2. The temperature ($T_{0}$) of the
valve can be adjusted between $45\text{\,}\mathrm{K}$ and
$160\text{\,}\mathrm{K}$ to vary the forward velocity of the supersonic
expansion between $970\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ and
$1800\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$. The final longitudinal
temperature of the supersonic beam ($\approx$$12\text{\,}\mathrm{mK}$) and its
forward velocity
($v_{\text{z}}\approx\sqrt{\nicefrac{{2k_{\text{B}}T_{0}\gamma}}{{m_{\text{H}_{2}}(\gamma-1)}}}$)
can be well approximated using the model of an adiabatic expansion Scoles
(1988). At valve temperatures below the characteristic rotational temperature
of the carrier gas H${}_{2}\,(\theta_{\text{rot}}\approx$
$90\text{\,}\mathrm{K}$), the heat capacity ratio $\gamma$ can be approximated
by the one of a monoatomic gas, i.e., $\gamma=\nicefrac{{5}}{{3}}$. The
central part of the supersonic beam is selected by two skimmers with diameters
of $2\text{\,}\mathrm{mm}$ and $3\text{\,}\mathrm{mm}$ placed at distances of
$45\text{\,}\mathrm{cm}$ and $135\text{\,}\mathrm{cm}$ from the nozzle
orifice, respectively.

The skimmed supersonic beam enters a magnetically shielded chamber in which
the H atoms are excited to Rydberg states in a sequential three-photon
absorption process. The $2\text{s}\leftarrow 1\text{s}$ transition is first
induced between two copper plates kept at the same electric potential of
4V${}_{\text{DC}}$ by the third-harmonic ($\lambda=$243\text{\,}\mathrm{nm}$$)
beam of a pulse-amplified near-Fourier-transform-limited Ti:Sa laser Seiler
_et al._ (2005) which crosses the supersonic beam at right angles. The
molecular beam then traverses a region with a weak homogeneous electric field
$\mathcal{F}_{\text{DC}}=\nicefrac{{V_{\text{DC}}}}{{\text{cm}^{-1}}}$, where
it intersects a single-mode cw UV laser
($\lambda\approx$368\text{\,}\mathrm{nm}$$) used to excite the metastable
H(2s) atoms to specific Rydberg-Stark states. These states are field ionized
by a large pulsed electric field (up to
$6\text{\,}\mathrm{kV}\text{\,}{\mathrm{cm}}^{-1}$) and the resulting protons
are accelerated towards a micro-channel-plate (MCP) detector. The different
components are discussed in more details in the following subsections. Spectra
of Rydberg-Stark states are recorded by monitoring the H+ field-ionization
yield as a function of the UV laser frequency.

###  II.1 Laser system for the 2s $\leftarrow$ 1s transition

The 243-nm radiation used to excite the H atoms to the 2s state by nonresonant
two-photon excitation is generated by amplification of the 120-ns-long chopped
output of a titanium-sapphire (Ti:Sa) seed laser at $729\text{\,}\mathrm{nm}$
using a Nd:YAG-pumped Ti:Sa multipass amplifier, as described in Ref. Seiler
_et al._ (2005). The output pulses, with pulse energies of
$\approx$15\text{\,}\mathrm{mJ}$$, are frequency tripled in two successive
$\beta$-barium-borate (BBO) crystals, resulting in 40-ns-long pulses at
$243\text{\,}\mathrm{nm}$ with typical pulse energies of
$800\text{\,}\mathrm{\SIUnitSymbolMicro J}$. The 243-nm-laser beam is focused
slightly beyond the supersonic beam using a 30-cm-focal-length lens. The use
of two skimmers reduces the Doppler width of the $2{\rm s}\leftarrow 1{\rm s}$
transition and enables the full resolution of the $f=0\leftarrow f=0$ and
$f=1\leftarrow f=1$ hyperfine components.

Because the 243-nm laser beam propagates along the x axis (see Fig. 1),
perpendicularly to both the supersonic beam and the cw UV laser, the focus
selects a narrow cylinder (diameter of $0.1\text{\,}\mathrm{mm}$) of H atoms
with a reduced velocity distribution along the y axis (see axis system in Fig.
1). This selection narrows down the Doppler width of the Rydberg-excitation
spectra from the 2s level. The photoexcitation only excites H(1s) atoms in a
very restricted longitudinal phase-space volume. Consequently, the H(2s)-atom
cloud remains compact and hardly expands as the beam propagates through the
4-cm-long distance separating the $2{\rm s}\leftarrow 1{\rm s}$ excitation
region from the $nkm\leftarrow 2{\rm s}$ excitation region. However, the
spatial and velocity selection can lead to a nonthermal velocity distribution,
potentially resulting in asymmetric Doppler profiles in the Rydberg-excitation
spectra.

The 243-nm laser unavoidably ionizes a significant fraction of the H(2s) atoms
Haas  _et al._ (2006). To avoid stray fields from the generated protons, they
are accelerated out of the H(2s) cloud by the electric field
$\mathcal{F}_{\text{DC}}$ resulting from the potentials applied between the
different electrodes within the mu-metal magnetic shield (see Fig. 1). To
eliminate line broadening caused by interactions between closely spaced
Rydberg atoms in the sample volume, the measurements are carried out in a
regime where at most one Rydberg atom is in the excitation volume and on
average much less than one field-ionization event is detected per experimental
cycle.

###  II.2 Laser system for the nkm$\leftarrow$2s excitation

The primary laser used for the precision spectroscopy of the $nkm\leftarrow
2{\rm s}$ transition is a commercial continuous-wave (cw) Ti:Sa ring laser
(Coherent, 899-21) pumped by a $12\text{\,}\mathrm{W}$ solid-state laser
(Coherent, Verdi V-12). The Ti:Sa ring laser is operated in the range
$729\\--$$736\text{\,}\mathrm{nm}$ and provides $1\text{\,}\mathrm{W}$ of
output power. In addition to the standard actuators of the ring laser, an
intra-cavity electro-optic modulator (EOM) (QUBIG, PS3D-BC) is used as a fast
actuator to maintain a phase lock to an ultrastable reference laser, as
discussed below. Around $98\text{\,}\mathrm{\char 37\relax}$ of the optical
power is sent to a home-built second-harmonic-generation enhancement cavity
(SHG) equipped with a 12-mm-long lithium-triborate (LBO) crystal cut at
Brewster’s angle. The SHG cavity is stabilized using a Hänsch-Couillaud scheme
Hänsch and Couillaud (1980). The typical conversion efficiency to the second
harmonic is $20\text{\,}\mathrm{\char 37\relax}$. The 368-nm output of the SHG
cavity is coupled into an optical fiber and guided to an actively stabilized
retroreflector (AFR) setup for Doppler-shift compensation (see below). The
forward-propagating and retroreflected laser beams cross the molecular beam at
right angles $4\text{\,}\mathrm{cm}$ downstream of the $2{\rm s}\leftarrow
1{\rm s}$ excitation spot.

The remaining $2\text{\,}\mathrm{\char 37\relax}$ of the fundamental laser
power is used for the frequency calibration and stabilization. The light is
tightly collimated and sent through an acousto-optic modulator (AOM) (Isomet,
M1260-T350L). The first-order diffraction is retro-reflected and its
polarization turned by $90\text{\,}\mathrm{\SIUnitSymbolDegree}$, as
illustrated in the upper left part of Fig. 1. The double-pass configuration
induces a shift of the fundamental frequency $\nu_{\text{L}}$ by
$2\nu_{\text{aom}}$ which can be adjusted up to $320\text{\,}\mathrm{MHz}$. A
polarizing beam splitter then deflects the frequency-shifted radiation and
sends it through an optical fiber to an amplified, spectrally broadened, and
frequency-doubled optically stabilized ultra-low-noise frequency comb
(MenloSystems, FC1500-ULN & M-VIS). The repetition rate of the frequency comb
is locked to an ultrastable laser, the frequency of which is referenced to an
SI-traceable frequency standard, as characterized in Ref. Husmann  _et al._
(2021). The output of the spectrally broadened frequency comb is dispersed
with a reflective grating and the spectral components around $\nu_{\text{L}}$
are selected and spatially overlapped with the laser. The beat, with frequency

$\nu_{b}=\nu_{c}-\nu_{\text{L}^{\prime}}$ (1)

between the shifted laser frequency
$\nu_{\text{L}^{\prime}}=\nu_{\text{L}}+2\nu_{\text{aom}}$ and the spectrally
closest frequency-comb tooth $\nu_{c}$ is recorded using a balanced photodiode
(Thorlabs, PDB425A-AC) and processed using the electronic circuit depicted in
Fig. 2. A bandpass filter centered at $60\text{\,}\mathrm{MHz}$ is used to
suppress beat frequencies originating from neighboring comb teeth. The RF beat
signal is amplified with an automatic-gain-control (AGC) amplifier and sent to
a frequency counter (K+K Messtechnik, FXM50). A fraction of the RF-signal is
used to establish a phase-lock of the Ti:Sa laser to the frequency comb. To
this end, the beat signal is amplified again and fed to a phase-frequency
detector (PFD) (Analog Devices, HMC403) where $\nu_{b}$ is compared to a
$60\text{\,}\mathrm{MHz}$ local oscillator. The error signal is transmitted to
the control box of the ring laser Haubrich and Wynands (1996) via an isolation
amplifier (IA). The frequency components in the range
$0\\--$$20\text{\,}\mathrm{MHz}$ are isolated with a diplexer, pre-amplified
and distributed to an inverting bipolar high-voltage amplifier (APEX
Microtechnology, PA90) and an amplifier (Comlinear, CLC103). The amplified
signals are applied to the intracavity EOM as shown in Fig. 2. This frequency-
offset-locking scheme provides a phase lock of the Ti:Sa ring laser to the
ultra-low-noise frequency comb and makes $\nu_{\text{L}}$ SI traceable.

Figure 2: Schematic electric-circuit diagram of the laser-stabilization
electronics (see text for details). Color-shaded inset: Spectral density (SD)
of the in-loop beat note $\nu_{b}$ recorded with a bandwidth of
$3\text{\,}\mathrm{kHz}$ with (black) and without (gray) active stabilization
using the intracavity EOM.

###  II.3 Detection of the nkm$\leftarrow$ 2s transition

The $nkm\leftarrow 2{\rm s}$ excitation is carried out in the center of two
electro-polished stainless-steel plates separated by
$\approx$2.1\text{\,}\mathrm{cm}$$ and designed for the application of
homogeneous electric fields. A ring electrode consisting of four segments is
inserted between the two plates to eliminate all line-of-sight trajectories of
charged particles to insulators. This measure effectively prevents
accumulation of charges near the excitation volume and is crucial to reduce
stray electric fields. The segmented geometry enables one to apply transverse
electric fields for stray-field compensation. A short plate distance of
$2.1\text{\,}\mathrm{cm}$ between the ion-repeller plate and the grounded
extraction plate was chosen to be able to generate electric fields up to
$6\text{\,}\mathrm{kV}\text{\,}{\mathrm{cm}}^{-1}$ in less than
$29\text{\,}\mathrm{ns}$ with a home-built $12.5\text{\,}\mathrm{kV}$ low-
noise high-voltage switch. With such fields, Rydberg states with principal
quantum number as low as 20 can be efficiently field ionized (see Fig. 7
below).

The electronic circuit was conceived to combine the high voltage pulse with
low-noise DC potentials (2V${}_{\text{DC}}$) on the repeller plate using a
20-bit digital-to-analogue low-noise voltage source. This enabled us to either
minimize stray-electric-field components or to apply well-defined electric
fields in the z direction. The only openings in the electrode structure
surrounding the photoexcitation region are 5-mm-diameter holes along the
molecular-beam axis and 9-mm-diameter holes for the UV laser beam.

###  II.4 Doppler-shift cancellation

The inset of Fig. 1 schematically describes the photoexcitation geometry,
where $\vec{v}$ is the H(2s)-atom velocity and $\vec{k}$ the wavevector of the
forward-propagating (blue) and reflected (red) UV radiation. Any deviation
$\delta\alpha$ from $90\text{\,}\mathrm{\SIUnitSymbolDegree}$ of the angle
between the laser beam and the supersonic beam leads to a first-order Doppler
shift. To cancel this shift, we choose $\delta\alpha$ to be large enough so
that the spectral lines from the forward-propagating and reflected UV laser
beams do not overlap. In addition, a
$180\text{\,}\mathrm{\SIUnitSymbolDegree}$ reflection angle is enforced
through an active-stabilization feedback system, based on a design introduced
in Refs. Beyer  _et al._ (2013, 2016); Wirthl  _et al._ (2021). This procedure
resulted in a mirror-symmetric double-line profile with center at the first-
order Doppler-free frequency Beyer  _et al._ (2018). Choosing $\delta\alpha$
as close to zero as possible, as advocated in Ref. Beyer  _et al._ (2016);
Wirthl  _et al._ (2021), turned out not to be practical in our case because
the nonthermal nature of the H(2s)-atom velocity distribution made it
challenging to extract the central frequency from the lineshapes under
conditions where the fine structure is not fully resolved.

An aberration-free set of four antireflection-coated lenses Lens assembly
with an effective focal length of $21.35\text{\,}\mathrm{mm}$ is used to
collimate the diverging beam emerging from a pure-silica-core, polarization-
maintaining, single-mode optical fiber (mode-field diameter
$2.3\text{\,}\mathrm{\SIUnitSymbolMicro m}$), resulting in a parallel beam
with a M2 value of $\approx$ 1.02. The focus of the resulting Gaussian beam is
located $\approx$ $20\text{\,}\mathrm{m}$ beyond the chamber. Consequently,
the reflected beam almost exactly retraces the incoming beam and the change of
wavefront curvature is negligible.

The active stabilization of the alignment of the
$180\text{\,}\mathrm{\SIUnitSymbolDegree}$ reflecting mirror is achieved by
dithering its tip and tilt angles by applying sinusoidal electric potentials
to piezo-electric elements installed at the back of the mirror holder (see
Fig. 1). The dithering leads to a modulation of the incoupling efficiency of
the reflected beam into the silica-core fiber beyond the lens system. These
modulations are detected with an auto-balanced photodiode (PD). The dithering
frequencies are selected to minimize cross talk between the motions of the tip
and tilt axes. The error signal used to correct the mirror position is
produced by lock-in amplifiers (LIA) (Femto, LIA-MVD-200L) connected to a
proportional-integral controller (PI). To compensate slow drifts, the time
constant of the feedback loop was chosen to be $0.1\text{\,}\mathrm{s}$.

##  III Theoretical description of Rydberg states of the H atom in electric
fields

The energy levels of the H atom in a static homogeneous electric field
$\vec{\mathcal{F}}=(0,0,\mathcal{F})$ are eigenvalues of the Hamiltonian

$\hat{\mathcal{H}}=\hat{\mathcal{H}}_{0}+e\mathcal{F}\hat{z},$ (2)

where $\hat{\mathcal{H}}_{0}$ is a diagonal matrix containing the field-free
energies of the $\left|nljfm_{f}\right>$ states with principal quantum number
$n$, orbital angular momentum quantum number $l$, total angular momentum
quantum number without nuclear spin $j$, total angular momentum quantum number
$f$, and associated magnetic quantum number $m_{f}$. The field-free hyperfine-
centroid energies, including terms arising from relativistic, quantum-
electrodynamics (QED) and finite-nuclear-size corrections, can be accurately
calculated using Eqs. $7\\--41$ of Ref. Tiesinga  _et al._ (2021) and the
latest recommended physical constants (2018 CODATA, see Ref. Tiesinga  _et
al._ (2002)). To obtain the field-free energy-level structure at high $n$
values, we used Bethe logarithms tabulated in Ref. Jentschura and Mohr (2005)
and included the hyperfine splittings using the analytical expressions
provided in Ref. Horbatsch and Hessels (2016). The calculated structure of the
$m_{l}=0$ levels at $n=20$ is depicted in the inset of Fig. 3b).

Figure 3: Stark effect in the $n=20,\,m_{f}=0$ manifold of the H atom. a)
Field dependence of the $k=0$ state revealing a quadratic shift below
$50\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-1}$ caused by the
intramanifold mixing of different orbital-angular-momentum components, and a
smaller quadratic shift at larger fields arising from the interaction between
different $n$ manifolds. b) Overview of the field dependence of all $m_{l}=0$
Stark states, which is essentially linear. c) Calculated spectra for different
electric-fields strengths and electric-field vectors $\vec{\mathcal{F}}$
pointing parallel or perpendicular to the laser polarization
$\vec{\epsilon}_{\mathrm{p}}$.

The operator $e\mathcal{F}\hat{z}$ in Eq. 2 describes the effect of the
external field. The perturbation can be treated in excellent approximation in
a nonrelativistic framework and relativistic corrections to the Stark effect
as discussed in Ref. Cordle (1974) become negligible as $n$ increases.
$e\mathcal{F}\hat{z}$ only contributes off-diagonal elements connecting zero-
field states differing in $l$ by $\pm 1$. These matrix elements can be
expressed in analytic form using standard angular-momentum algebra (see, e.g.,
Refs. Zare (1988); Sobelman (1992)) as

$\displaystyle\left<n^{\prime}l^{\prime}j^{\prime}f^{\prime}m_{f}^{\prime}\right|\hat{z}\left|nljfm_{f}\right>=(-1)^{\Delta
f+\Delta j+\Delta l-m_{f}^{\prime}+I+S}\times$
$\displaystyle\left(\begin{matrix}l^{\prime}&1&l\\\
0&0&0\end{matrix}\right)\left(\begin{matrix}f^{\prime}&1&f\\\
-m_{f}^{\prime}&0&m_{f}\end{matrix}\right)\left\\{\begin{matrix}j^{\prime}&f^{\prime}&I\\\
f&j&1\end{matrix}\right\\}\left\\{\begin{matrix}l^{\prime}&j^{\prime}&S\\\
j&l&1\end{matrix}\right\\}\times$
$\displaystyle\sqrt{\Theta(f^{\prime})\Theta(f)\Theta(j^{\prime})\Theta(j)\Theta(l^{\prime})\Theta(l)}\left<n^{\prime}l^{\prime}\right|r\left|nl\right>,$
(3)

where the expressions in parentheses and curly parentheses are Wigner 3j and
6j symbols, respectively, $\Theta(x)=2x+1$, $\Delta x=x^{\prime}-x$, and
$\left<n^{\prime}l^{\prime}\right|r\left|nl\right>$ are radial integrals
connecting the $r$-dependent parts of the solutions of the Schrödinger
equation of the H atom (see Eqs. 63.2 and 63.5 of Ref. Bethe and Salpeter
(1957)). Restricting the calculations of the Stark effect to a single $n$
value, one obtains an intra-manifold quadratic Stark effect at low fields and
a linear Stark effect at intermediate fields, as depicted in Fig. 3. The Stark
states are commonly labeled by the parabolic quantum numbers $n_{1}$ and
$n_{2}$ or by their difference $k=n_{1}-n_{2}$ Bethe and Salpeter (1957);
Dalgarno  _et al._ (1994). At intermediate field strengths, the states can
approximately be described by their $k$ and $m_{l}$ values. States of a given
value of $k$ form near degenerate groups with $m_{l}$ values ranging from
$-(n-|k|-1)$ to $(n-|k|-1)$ in steps of 2. The $k=0$ states, highlighted in
red in Fig. 3, are the only states retaining almost pure parity
$\left[(-1)^{n-1}\right]$. They have a zero electric dipole moment and are
insensitive to the field over a large range of fields, which makes them
attractive for precision measurements, except at fields very close to zero.
All other states exhibit a dipole moment in the field. At intermediate to high
field strengths, the coupling between states of different $n$ values induced
by the field becomes significant and the states start exhibiting an inter-
manifold quadratic Stark effect. This behavior is displayed on an enlarged
vertical scale for $m_{f}=0$ in Fig. 3a). To reliably calculate Stark shifts
in this field range, it is necessary to include basis states of neighboring
$n$ values until convergence with the size of the basis set is reached.

Figure 4: Expansion coefficients of the $k=0$, $|m_{f}|=0,1$ and 2 Rydberg-
Stark wavefunctions in the $\left|ljfm_{f}\right>$ angular-momentum basis as
labeled in the figure. Only basis states with odd orbital angular momentum
quantum number make significant contributions.

Figure 4 presents the decomposition of the $n=20,\,k=0$ Stark states with
$m_{f}=0-2$ in the $\left|ljfm_{f}\right>$ basis. For each $m_{f}$ value, the
eigenstates possess contributions from up to four hyperfine-structure
components, as indicated by the color labels. The intensity of transitions
from the 2s level corresponds to the coherent squared sum of the p characters
in the evaluation of electric-dipole-moment matrix elements.

Figure 3c depicts calculated intensity distributions in spectra of the
$n=20\leftarrow 2s$ transitions at field strength below
$1\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ and for laser polarizations
parallel and perpendicular to the DC electric field. At fields below
$20\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-1}$, corresponding to typical
stray fields, the center of gravity of the distribution depends on the
polarization and varies strongly and nonlinearly with the field strength,
making precision measurements prone to systematic uncertainties. This behavior
explains why high-$n$ Rydberg states are usually avoided in precision
measurements. However, in the linear regime of the Stark effect, i.e., above
$0.2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ at $n=20$, the spectra
regain a regular intensity pattern and the spacings between the Stark states
encode the field strength. When the polarization is parallel to the field
($\pi$ transitions), the intensity is strongest at the outer edges of the
manifold and vanishes at $k=0$, for even $n$ values because $k=0,m_{l}=0$
states do not exist, and for odd $n$ values because $k=0$ states have
vanishing p character. When the polarization is perpendicular to the field,
the opposite behavior is observed (see right panel of Fig. 3c).

Consideration of Fig. 3 leads to the following conclusions concerning
precision spectroscopy in high-$n$ states of hydrogen-like systems:

  * •

Because of the nontrivial field dependence of the line profiles, precision
measurements are not attractive in the region of the intra-manifold quadratic
Stark effect.

  * •

In the linear regime of the Stark effect, regular spectral pattern are
restored and the states with $|k|\geq 0$ form pairs of levels with Stark
shifts of opposite sign. The positions of the $k\neq 0$ states can be used for
the electric-field calibration, as will be demonstrated in Section IV.

  * •

If an easily calculable shift from the Bohr energy
$\left(-hcR_{\mathrm{H}}n^{-2}\right)$ arising from the quadratic Stark effect
is disregarded, the $k=0$ Stark states are essentially field-independent.
Consequently, spectra of $k=0$ Stark states in the linear regime are not
subject to broadening by inhomogeneous fields and their positions can be
converted into the Bohr energy by adding the calculated Stark shift (see red
curves in Fig. 3a)).

  * •

The linear Stark manifold is thus perfectly suited for metrological purposes,
in particular for precise determination of the Bohr energy. It has previously
been used to determine the binding energy of Rydberg states of H2 Hölsch  _et
al._ (2022).

Figure 5: Energy level structure of the eight $n=20,\,k=0$ Stark states with
$m_{l}=1$ character, calculated at an electric field strength $\mathcal{F}=$
$0.8\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$. These states split into
two groups of four states each separated by
$\approx$600\text{\,}\mathrm{kHz}$$. The Zeeman effect induced by a magnetic
field pointing along the quantization axis is schematically illustrated on the
right side and lifts all remaining degeneracies.

The wavefunctions of the Stark states can be used to estimate their magnetic
moments and systematic shifts arising from the Zeeman effect caused by
residual magnetic fields, as illustrated in Fig. 5 with the example of the
$k=0,|m_{l}|=1$ Stark states. In this case, the electric field splits the
structure into two $m_{f}=0$, two $m_{f}=1$ and one $m_{f}=2$ components and a
total of eight states. The magnetic moments are given by the relative
orientations of the electron orbital angular momentum, electron spin, and
nuclear spin vectors. A magnetic field parallel to the electric field further
splits these components according to their magnetic moments, as displayed
schematically on the right-hand side of Fig. 5. Because the Zeeman shifts are
symmetric and extremely small in a magnetically shielded environment (less
than $2.4\text{\,}\mathrm{kHz}$ for $\mu=2\mu_{\text{B}}$ and
$|$B$|\leq$$100\text{\,}\mathrm{nT}$), we conclude that the Zeeman effect in
low-$m_{l}$ states can be neglected in metrological applications relying on
Stark states in the linear regime. This is also the case for perpendicular
magnetic-field components because the corresponding Zeeman effect couples
states with $\Delta m_{l}=\pm 1$ which are located in different $k$ manifolds
and thus energetically too distant for significant mixing to occur.

As explained in Section II, the maximal electric-field strength we apply to
record Stark spectra is $2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$. The
applied fields also induce shifts of the 2s level energies, which need to be
considered when extracting the absolute positions of the Rydberg-Stark states.
The Stark shifts of the 2s levels can be calculated in the same manner as
explained above for higher n values.

Figure 6: Stark shifts of the metastable 2s levels of the H atom calculated
for electric fields in the range between 0 and
$2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$.

The calculated shifts are displayed in Fig. 6. They are positive and quadratic
for small electric fields because the dominant interactions are with the
2p${}_{\nicefrac{{1}}{{2}}}$ states, which are located energetically just
below the 2s states. When determining the absolute positions of the $nkm$
Rydberg-Stark states from spectra of the $nkm\leftarrow$ 2s transitions, the
2s Stark shifts must be added to the measured transition frequencies.

##  IV Results

Figure 7: PFI spectra of the $n=20$ Rydberg-Stark states of H recorded from
the 2s($f=1$) hyperfine component in an electric field
$\mathcal{F}_{\mathrm{DC}}\approx$200\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-1}$$.
The direction of the strong pulsed electric-field
($\mathcal{F}_{\text{PFI}}=$5.7\text{\,}\mathrm{kV}\text{\,}{\mathrm{cm}}^{-1}$$)
used for ionization was set parallel to
$\mathcal{F}_{\mathrm{DC}}^{\uparrow\uparrow}$ to record the upper spectrum
and antiparallel $\mathcal{F}_{\mathrm{DC}}^{\uparrow\downarrow}$ to record
the lower, inverted spectrum. The red and blue stick spectra represent the
calculated intensity distributions. Inset: The alignment of the two fields
leads to ionization without change of the field polarity (red) or to
ionization after a diabatic state inversion upon reversal of the field
polarity.

Figure 7 displays pulse-field-ionization (PFI) spectra of the $n=20$ Stark
manifold recorded from the 2s($f=1$) hyperfine level using laser radiation
polarized linearly in the direction orthogonal to the applied DC electric
field $\mathcal{F}_{\text{DC}}$. The upper (lower) trace was recorded by field
ionizing the Rydberg states with a pulsed field $\mathcal{F}_{\text{PFI}}$
pointing in the same (opposite) direction as the DC field
$\left[\mathcal{F}_{\text{PFI}}=$5.7\text{\,}\mathrm{kV}\text{\,}{\mathrm{cm}}^{-1}$,\mathcal{F}_{\text{DC}}=$0.2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$($-0.2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$)\right]$.
The orthogonal laser-polarization arrangement led to the observation of
dominant transitions to Stark states of even $k$ values, as assigned at the
top of the figure. The intensity distributions in both spectra are very
similar, except at the edges of the manifold. Whereas the intensities of the
transitions to the highest $k$ states ($k\geq 14$) are strongly depleted in
the upper spectrum, the lowest $k$ states ($k\leq-14$) are depleted in the
lower spectrum. The reason for the disappearance of the intensities at the
edges of the Stark manifold are twofold: First, the transition dipole moment
gradually decreases with increasing $|k|$ value. Second, the ionization rates
of the Stark states that are shifted to higher energies by the pulsed field
rapidly decrease with increasing $k$ value. In the case of the upper spectrum,
these states are those observed at the highest frequencies. For the lower
spectrum, they are observed at the lowest frequencies because of the reversal
of the sign of $k$ when the field polarity changes upon application of the
pulsed field, which diabatically inverts the Stark manifold, as schematically
illustrated in the inset. This interpretation is fully supported by
calculations of the spectral intensities, as depicted in the red and blue
stick spectra in Fig. 7. These intensities were obtained by multiplying the
squared transition dipole moments calculated as explained in Section III with
the field-ionization probabilities over the 80-ns-long detection window
calculated using the analytical expressions reported by Damburg and Kolosov
Damburg and Kolosov (1979).

Figure 8: a) Typical experimental (dots) and fitted (blue) spectra of the
three ($k=0,\pm 2$) Rydberg-Stark states near the center of the $n=20$ Stark
manifold of H, each exhibiting two Doppler components. b) Weighted residuals
(see text for details).

Before recording the lower spectrum in Fig. 7, the transverse stray fields
were carefully compensated. Consequently, the laser polarization was almost
perfectly parallel to the DC field. Under these conditions, transitions to
Stark states of odd $k$ values have zero intensity. In the case of the upper
spectrum, a weak transverse stray field made the Stark states with odd $k$
values optically accessible. Transitions to these states are strongest at the
edges of the manifold and weakest at the center. The calculated intensities of
transitions to odd $k$ states in the presence of the transverse stray field
($\sim$10\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-1}$$) are depicted as
gray sticks in Fig. 7. They are only observable at the low-frequency edge of
the Stark manifold because the Stark states at the high-frequency edge are not
efficiently ionized by the pulsed field, as explained above. The good
agreement between measured and calculated intensity distributions enables us
to conclude that the Rydberg-Stark states located near the center of the
$n=20$ manifold are fully ionized by the
$5.7\text{\,}\mathrm{kV}\text{\,}{\mathrm{cm}}^{-1}$ pulsed field used in the
experiments.

Figure 8 displays a typical spectrum of transitions to the $k=0,\pm 2$ Stark
states of the $n=20$ manifold recorded from the H(2s,$f=1$) state using laser
radiation with linear polarization orthogonal to the
$0.8\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ DC field. The spectrum was
recorded at an angle deviation $\delta\alpha=$1.1\text{\,}\mathrm{mrad}$$ from
exact orthogonality between the H-atom beam and the laser beam, leading to two
Doppler components per $k$ state, separated by $6.28\text{\,}\mathrm{MHz}$.
The two Doppler components are slightly asymmetric with mirror-symmetric
lineshapes (opposite sign of $\gamma$ in Eq. 4 below). To optimize the data
acquisition rate when recording the Stark spectra, the frequency was scanned
in steps of $400\text{\,}\mathrm{kHz}$ within the line profiles and of
$2\text{\,}\mathrm{MHz}$ between the lines. In addition, the data points
within the spectral lines were obtained by averaging over 500 experimental
cycles (i.e., over $20\text{\,}\mathrm{s}$) whereas only 100 cycles were
averaged for data points between the lines. The central frequency, the
electric field strength and additional parameters were determined in a least-
squares fit to the experimental data (black dots) based on the following line
profile for each $k$ value

$\displaystyle
g_{k}(\nu)=\sum_{i=1}^{2}\sum_{m_{f}=-2}^{2}\mathrm{I}^{i}\mathrm{I}^{m_{f}}(\mathcal{F})\exp\left\\{\frac{-\left[\nu-\nu_{0}^{i,m_{f}}(\mathcal{F},\gamma)\right]^{2}}{2\left(\sigma_{\mathrm{D}}^{2}+|k|\sigma_{\mathrm{S}}^{2}\right)}\right\\}\times\left[1+\mathrm{erf}\left((-1)^{i}\gamma\frac{\left(\nu-\nu_{0}^{i,m_{f}}(\mathcal{F},\gamma)\right)}{\sqrt{2}\sigma_{\mathrm{D}}}\right)\right],$
(4)

with

$\displaystyle\nu_{0}^{i,m_{f}}(\mathcal{F},\gamma)=\nu_{0}+\nu_{\mathrm{S}}^{m_{f}}(\mathcal{F})+(-1)^{i}\left\\{\nu_{\mathrm{D}}-\delta\nu(\gamma)\right\\}.$
(5)

In Eqs. 4 and 5, $i\,(=1,2)$ is an index specifying the Doppler component,
$\nu_{0}$ is the transition frequency to the reference position
$\left(-\nicefrac{{cR_{\text{H}}}}{{n^{2}}}\right)$ of the calculated Stark
map of the $n=20$ levels (see Fig. 3), $\nu_{\mathrm{S}}^{m_{f}}(\mathcal{F})$
is the field-dependent Stark shift of the $m_{f}$ level, $\nu_{\mathrm{D}}$ is
the Doppler shift arising from the angle deviation $\delta\alpha$, and
$\delta\nu(\gamma)$ is a frequency offset used to compensate the shift of the
intensity maximum of the asymmetric line profiles from the centers of the
hypothetical symmetric profiles. This shift is introduced to reduce the
correlation between the asymmetry parameter $\gamma$ and $\nu_{\mathrm{D}}$ in
the least-squares fit. $\sigma_{\mathrm{D}}$ is the Doppler width and
$\sigma_{\mathrm{S}}$ accounts for the broadening of the $|k|=2$ lines arising
from weak field inhomogeneities in the photoexcitation volume. As mentioned in
Section II.1, the asymmetry of the line profiles originate from the nonthermal
velocity distribution caused by the 2s $\leftarrow$ 1s excitation.

Table 1: Fit results obtained in the least-squares fit of the line profiles based on Equations 4 and 5. | value
---|---
$\nicefrac{{\nu_{0}}}{{\text{kHz}}}$ |  $0(26)$ (blinded)
$\nicefrac{{\mathcal{F}}}{{\text{V\,cm${}^{-1}$}}}$ | $0.8076(3)$
$\nicefrac{{\nu_{\mathrm{D}}}}{{\text{MHz}}}$ | $3.16(5)$
$\nicefrac{{\sigma_{\mathrm{D}}}}{{\text{MHz}}}$ | $1.56(7)$
$\nicefrac{{\sigma_{\mathrm{S}}}}{{\text{MHz}}}$ | $0.27(6)$
$\gamma$ | $0.65(18)$

The fit of the line profiles depicted in Fig. 8 resulted in the parameters
listed in Table 1. These parameters are helpful in characterizing the
experimental conditions. For instance, the homogeneous component of the field
is found to correspond closely to the
$0.8\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ applied experimentally
with an uncertainty of only $0.4\permil$ or
$300\text{\,}\mathrm{\SIUnitSymbolMicro V}\text{\,}{\mathrm{cm}}^{-1}$. The
electric field inhomogeneity leads to a broadening of the $k=\pm 2$ Stark
components and is well represented by a field gradient of
$12(3)\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-2}$, which corresponds to a
field change of $2.4(6)\text{\,}\mathrm{mV}\text{\,}{\mathrm{cm}}^{-1}$ over
the $2\text{\,}\mathrm{mm}$ diameter of the UV laser. The Doppler shift
$\nu_{\mathrm{D}}$ reflects the deviation angle $\delta\alpha$ which, in this
case, is $1.1\text{\,}\mathrm{mrad}$. $\sigma_{\mathrm{D}}$ is a measure of
the transversal velocity distribution, which in the present case corresponds
to a temperature of $40\text{\,}\mathrm{\SIUnitSymbolMicro K}$ and is the
result of the geometric constraints along the supersonic beam imposed by the
skimmers and the $2\mathrm{s}\leftarrow 1\mathrm{s}$ excitation. The asymmetry
parameter is alignment specific and typically varied between -2 and 4. The
central frequency was arbitrary set to zero because the absolute frequency
determination is still in a blinded phase. The weights used for the least-
squares fits are determined in an iterative procedure to approach a normal
distribution of the residuals.

Figure 9: Spectra of the $n=20,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow
2\text{s}(f=1)$ transitions measured when applying nominal electric fields of
0.4, 0.8, 1.2 and $1.6\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$,
respectively. Each spectrum represents the sum of three independent scans as
described in Section II. Right: Relative positions of the line center
$\nu_{0}$ with respect to the line center measured at a nominal field strength
of $0.4\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$. The error bars
represent 1$\sigma$ uncertainties.

The overall data set collected so far involves more than 500 individual
spectra of transitions recorded from the initial 2s$(f=1)$ and 113 from the
2s$(f=0)$ hyperfine state to $n=20$ Rydberg states and 35 spectra from the
2s$(f=1)$ to $n=24$ Rydberg states. These spectra were recorded for different
valve temperatures, electric-field strengths and deviation angles
$\delta\alpha$ to investigate possible sources systematic uncertainties.
The main objective of the study presented here was to verify that the central
frequencies extracted from the spectra do not depend on the strength of the
applied electric field. A typical set of four measurements recorded at nominal
field strengths of 0.4, 0.8, 1.2 and
$1.6\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ under otherwise identical
experimental conditions (beam velocity of
$1060\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ and deviation angle
$\delta\alpha$ of $1.1\text{\,}\mathrm{mrad}$) is presented in Fig. 9. At the
scale of the figure, the Stark effect appears essentially linear. Table 2
summarizes the relevant lineshape parameters (see Eqs. 4 and 5) extracted from
the fits of the lineshapes to the experimental data.

Table 2: Lineshape parameters extracted from fits to the measured spectra of the $n=20,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow 2\text{s}(f=1)$ transitions measured when applying a nominal electric field of 0.4, 0.8, 1.2 and $1.6\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$, respectively.  | $0.4\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ | $0.8\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ | $1.2\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$ | $1.6\text{\,}\mathrm{V}\text{\,}{\mathrm{cm}}^{-1}$
---|---|---|---|---
$\nicefrac{{\nu_{0}}}{{\mathrm{kHz}}}$ | 0(21) | 21(18) | -20(21) | -0(20)
$\nicefrac{{\mathcal{F}}}{{\mathrm{Vcm}^{-1}}}$ | 0.4012(4) | 0.7990(3) | 1.1882(3) | 1.5794(3)
$\nicefrac{{\sigma_{\mathrm{S}}}}{{\mathrm{MHz}}}$ | 0.31(10) | 0.22(9) | 0.17(14) | 0.23(6)
$\nicefrac{{\nu_{\mathrm{D}}}}{{\mathrm{MHz}}}$ | 4.26(5) | 4.61(4) | 5.20(5) | 5.18(3)
$\nicefrac{{\sigma_{\mathrm{D}}}}{{\mathrm{MHz}}}$ | 2.02(10) | 1.78(5) | 2.12(5) | 1.81(5)

The central frequencies corrected for the Stark shift of the 2s state agree
within the combined uncertainties and do not reveal any systematic dependence
on the field strength within the $20\text{\,}\mathrm{kHz}$ accuracy of the
measurements. The field strength corresponds to the applied electric potential
within the expected uncertainties resulting from the geometry of the electrode
plates and the electronic circuits used to apply the potentials. The field-
dependent line broadening does not reveal a significant dependence on the
applied field strength, which suggests that the applied field distribution
does not contribute to the observed field inhomogeneity. The slight variations
in the values of $\nu_{\mathrm{D}}$ and $\sigma_{\mathrm{D}}$ reflect small
changes in the day-to-day alignments of the beams and the supersonic-beam
properties.

Figure 10: Spectra of the $n=20,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow
2\text{s}(f=1)$ (blue) and $n=20,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow
2\text{s}(f=0)$ (red). The difference of the two central frequencies $\nu_{0}$
corresponds to the hyperfine interval of the 2s state.

The data set collected so far was used to determine the hyperfine splitting in
the 2s level as well as the difference between the Bohr energies of the $n=20$
and $n=24$ Rydberg states. Figure 10 presents spectra of the transitions to
the $n=20$, $k=0,\pm 2$ Stark states recorded from the 2s$(f=0)$ (red) and
2s$(f=1)$ (blue) states as illustration. Taking the difference in the central
frequencies $\nu_{0}$ (see Eq. 5) for the two sets of data (197 spectra and 50
spectra for $f=1$ and $f=0$, respectively) yields a value of
$177.546(11)\text{\,}\mathrm{MHz}$ for the 2s hyperfine splitting, which
corresponds within the 1$\sigma$ uncertainty to the much more precise value of
177.55683887(85) MHz determined by Ramsey spectroscopy in the $n=2$ manifold
Bullis  _et al._ (2023).
The difference in the Bohr energies of the $n=20$ and 24 Rydberg states was
determined in an analogous manner from spectra of the $n=20$ and 24 Stark
states recorded from the 2s$(f=1)$ state as illustrated in Fig. 11.

Figure 11: Spectra of the $n=20,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow
2\text{s}(f=1)$ (top) and $n=24,k=0,\,\pm 2,\,|m_{l}|=1\leftarrow
2\text{s}(f=1)$ (bottom). The energy scale has its origin at the $n=20$ Bohr
energy which is located $\approx$2.2\text{\,}\mathrm{MHz}$$ above the
$n=20,k=0$ Stark states (see Fig. 3). The Bohr energies are indicated by the
dashed lines in the two panels.

The difference of the two $\nu_{0}$ values is
$2\,511\,705.793(10)\text{\,}\mathrm{MHz}$ and also agrees within the
experimental uncertainty with the value
$cR_{\mathrm{H}}\left(\nicefrac{{1}}{{20^{2}}}-\nicefrac{{1}}{{24^{2}}}\right)=$2\,511\,705.802\text{\,}\mathrm{MHz}$$.
The uncertainty of $10\text{\,}\mathrm{kHz}$ results from the addition in
quadrature of the $7\text{\,}\mathrm{kHz}$ uncertainties of the blinded
$\nu_{0}$ values extracted from the experimental data.

##  V Conclusion

In this article, we have outlined an experimental approach to determine
$R_{\infty}$ from $k=0,\,\pm 2,\,|m_{l}|=1$ Rydberg-Stark spectra of H. We
have demonstrated that systematic errors resulting from the Stark effect are
insignificant within the $\sim$11\text{\,}\mathrm{kHz}$$ precision of the four
data sets used as illustrations (see Fig. 9). We have also demonstrated that
the differences between the Bohr energy at $n=20$ and the positions of the
$f=0$ and 1 hyperfine components of the 2s state are consistent within the
$11\text{\,}\mathrm{kHz}$ statistical uncertainty of the present determination
with the more precise value of the 2s $(f=0)\\--(f=1)$ interval determined
recently by Ramsey microwave spectroscopy Bullis  _et al._ (2023).
Finally, we have determined the difference between the Bohr energies at $n=20$
and 24 and found the results to agree with Bohr’s formula using the CODATA
2018 recommended value for $R_{\mathrm{H}}$ Tiesinga  _et al._ (2021). The
data presented in this article was collected over a period of several months
with frequent realignment of the optical system and supersonic beam. We did
not observe inconsistencies in any of the relative frequencies determined for
this article over this time.
The 2s$(f=0)\\--(f=1)$ and $\nu_{0}(n=24)\\--\nu_{0}(20)$ intervals presented
in this article correspond to differences of large frequencies, and systematic
errors largely cancel out when building the differences. The main potential
source of systematic errors in our method originates from the Doppler effect
and a possible imperfect cancellation of the Doppler shifts. To characterize
such uncertainties, measurements of absolute frequencies are underway, in
which we systematically vary the velocity of the supersonic beam and the
deviation angle $\delta\alpha$. Absolute transition frequencies will be
reported when these measurements are completed.

###### Acknowledgements.

We thank Dominik Husmann (METAS, Bern) for his help in maintaining the SI-
traceable frequency dissemination network and Gloria Clausen for helpful
discussions. We also thank Prof. Klaus Ensslin and Peter Märki for the low-
noise DC voltage source used in the measurements of the Stark spectra.
This work was supported by the Swiss National Science Foundation through the
Sinergia-program (Grant No. CRSII5-183579) and a single-investigator grant
(Grant No. 200020B-200478).

## References

  * Ångström (1868) A. J. Ångström,  _Recherches sur le spectre solaire: Spectre normal du soleil: Atlas de six planches_ (W. Schultz, Imprimeur de l’université, Uppsala, 1868).
  * Balmer (1885) J. J. Balmer, Ann. Phys. Chem. 261, 80 (1885).
  * Bohr (1913) N. Bohr, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 26, 1 (1913).
  * Schrödinger (1926) E. Schrödinger, Ann. Phys. 384, 361 (1926).
  * Dirac (1926) P. A. M. Dirac, Proc. R. Soc. London A 110, 561 (1926).
  * Lamb and Retherford (1947) W. E. Lamb, Jr. and R. C. Retherford, Phys. Rev. 72, 241 (1947).
  * Lamb and Retherford (1950) W. E. Lamb, Jr. and R. C. Retherford, Phys. Rev. 79, 549 (1950).
  * Bethe (1947) H. A. Bethe, Phys. Rev. 72, 339 (1947).
  * Bethe and Salpeter (1957) H. A. Bethe and E. E. Salpeter,  _Quantum Mechanics of One- and Two-Electron Atoms_ (Springer, Berlin, 1957).
  * Tiesinga  _et al._ (2021) E. Tiesinga, P. J. Mohr, D. B. Newell, and B. N. Taylor, Rev. Mod. Phys. 93, 025010 (2021).
  * Pohl  _et al._ (2010) R. Pohl, A. Antognini, F. Nez, F. D. Amaro, F. Biraben, J. M. R. Cardoso, D. S. Covita, A. Dax, S. Dhawan, L. M. P. Fernandes, A. Giesen, T. Graf, T. W. Hänsch, P. Indelicato, L. Julien, C.-Y. Kao, P. Knowles, E.-O. Le Bigot, Y.-W. Liu, J. A. M. Lopes, L. Ludhova, C. M. B. Monteiro, F. Mulhauser, T. Nebel, P. Rabinowitz, J. M. F. dos Santos, L. A. Schaller, K. Schuhmann, C. Schwob, D. Taqqu, J. F. C. A. Veloso, and F. Kottmann, Nature 466, 213 (2010).
  * Antognini  _et al._ (2013) A. Antognini, F. Nez, K. Schuhmann, F. D. Amaro, F. Biraben, J. M. R. Cardoso, D. S. Covita, A. Dax, S. Dhawan, M. Diepold, L. M. P. Fernandes, A. Giesen, A. L. Gouvea, T. Graf, T. W. Hänsch, P. Indelicato, L. Julien, C.-Y. Kao, P. Knowles, F. Kottmann, E.-O. L. Bigot, Y.-W. Liu, J. A. M. Lopes, L. Ludhova, C. M. B. Monteiro, F. Mulhauser, T. Nebel, P. Rabinowitz, J. M. F. dos Santos, L. A. Schaller, C. Schwob, D. Taqqu, J. F. C. A. Veloso, J. Vogelsang, and R. Pohl, Science 339, 417 (2013).
  * Lensky  _et al._ (2022) V. Lensky, F. Hagelstein, and V. Pascalutsa, Phys. Lett. B 835, 137500 (2022).
  * Parthey  _et al._ (2011) C. G. Parthey, A. Matveev, J. Alnis, B. Bernhardt, A. Beyer, R. Holzwarth, A. Maistrou, R. Pohl, K. Predehl, T. Udem, T. Wilken, N. Kolachevsky, M. Abgrall, D. Rovera, C. Salomon, P. Laurent, and T. W. Hänsch, Phys. Rev. Lett. 107, 203001 (2011).
  * Matveev  _et al._ (2013) A. Matveev, C. G. Parthey, K. Predehl, J. Alnis, A. Beyer, R. Holzwarth, T. Udem, T. Wilken, N. Kolachevsky, M. Abgrall, D. Rovera, C. Salomon, P. Laurent, G. Grosche, O. Terra, T. Legero, H. Schnatz, S. Weyers, B. Altschul, and T. W. Hänsch, Phys. Rev. Lett. 110, 230801 (2013).
  * Beyer  _et al._ (2017) A. Beyer, L. Maisenbacher, A. Matveev, R. Pohl, K. Khabarova, A. Grinin, T. Lamour, D. C. Yost, T. W. Hänsch, N. Kolachevsky, and T. Udem, Science 358, 79 (2017).
  * Bezginov  _et al._ (2019) N. Bezginov, T. Valdez, M. Horbatsch, A. Marsman, A. C. Vutha, and E. A. Hessels, Science 365, 1007 (2019).
  * Fleurbaey  _et al._ (2018) H. Fleurbaey, S. Galtier, S. Thomas, M. Bonnaud, L. Julien, F. Biraben, F. Nez, M. Abgrall, and J. Guéna, Phys. Rev. Lett. 120, 183001 (2018).
  * Grinin  _et al._ (2020) A. Grinin, A. Matveev, D. C. Yost, L. Maisenbacher, V. Wirthl, R. Pohl, T. W. Hänsch, and T. Udem, Science 370, 1061 (2020).
  * Brandt  _et al._ (2022) A. D. Brandt, S. F. Cooper, C. Rasor, Z. Burkley, A. Matveev, and D. C. Yost, Phys. Rev. Lett. 128, 023001 (2022).
  * Belushkin  _et al._ (2007) M. A. Belushkin, H.-W. Hammer, and U.-G. Meißner, Phys. Rev. C 75, 035202 (2007).
  * Bernauer  _et al._ (2010) J. C. Bernauer, P. Achenbach, C. A. Gayoso, R. Böhm, D. Bosnar, L. Debenjak, M. O. Distler, L. Doria, A. Esser, H. Fonvieille, J. M. Friedrich, J. Friedrich, M. G. R. de la Paz, M. Makek, H. Merkel, D. G. Middleton, U. Müller, L. Nungesser, J. Pochodzalla, M. Potokar, S. S. Majos, B. S. Schlimme, S. Širca, T. Walcher, and M. Weinriefer, Phys. Rev. Lett. 105, 242001 (2010).
  * Sick (2012) I. Sick, Prog. Part. Nucl. Phys. 67, 473 (2012).
  * Lee  _et al._ (2015) G. Lee, J. R. Arrington, and R. J. Hill, Phys. Rev. D 92, 013013 (2015).
  * Higinbotham  _et al._ (2016) D. W. Higinbotham, A. A. Kabir, V. Lin, D. Meekins, B. Norum, and B. Sawatzky, Phys. Rev. C 93, 055207 (2016).
  * Horbatsch  _et al._ (2017) M. Horbatsch, E. A. Hessels, and A. Pineda, Phys. Rev. C 95, 035203 (2017).
  * Alarcón  _et al._ (2019) J. M. Alarcón, D. W. Higinbotham, C. Weiss, and Z. Ye, Phys. Rev. C 99, 044303 (2019).
  * Xiong  _et al._ (2019) W. Xiong, A. Gasparian, H. Gao, D. Dutta, M. Khandaker, N. Liyanage, E. Pasyuk, C. Peng, X. Bai, L. Ye, K. Gnanvo, C. Gu, M. Levillain, X. Yan, D. W. Higinbotham, M. Meziane, Z. Ye, K. Adhikari, B. Aljawrneh, H. Bhatt, D. Bhetuwal, J. Brock, V. Burkert, C. Carlin, A. Deur, D. Di, J. Dunne, P. Ekanayaka, L. El-Fassi, B. Emmich, L. Gan, O. Glamazdin, M. L. Kabir, A. Karki, C. Keith, S. Kowalski, V. Lagerquist, I. Larin, T. Liu, A. Liyanage, J. Maxwell, D. Meekins, S. J. Nazeer, V. Nelyubin, H. Nguyen, R. Pedroni, C. Perdrisat, J. Pierce, V. Punjabi, M. Shabestari, A. Shahinyan, R. Silwal, S. Stepanyan, A. Subedi, V. V. Tarasov, N. Ton, Y. Zhang, and Z. W. Zhao, Nature 575, 147 (2019).
  * Lin  _et al._ (2021) Y.-H. Lin, H.-W. Hammer, and U.-G. Meißner, Phys. Lett. B 816, 136254 (2021).
  * Paine (1992) S. N. Paine,  _High precision millimeter-wave spectroscopy of atomic hydrogen_ , Ph.D. thesis, MIT (1992).
  * Lutwak (1997) R. Lutwak,  _Millimeter-wave studies of hydrogen Rydberg states_ , Ph.D. thesis, MIT (1997).
  * De Vries (2002) J. C. De Vries,  _A precision millimeter-wave measurement of the Rydberg frequency_ , Ph.D. thesis, MIT (2002).
  * Mohr and Taylor (2000) P. J. Mohr and B. N. Taylor, Rev. Mod. Phys. 72, 351 (2000).
  * Ramos  _et al._ (2017) A. Ramos, K. Moore, and G. Raithel, Phys. Rev. A 96, 032513 (2017).
  * Facon  _et al._ (2016) A. Facon, E.-K. Dietsche, D. Grosso, S. Haroche, J.-M. Raimond, M. Brune, and S. Gleyzes, Nature 535, 262 (2016).
  * Garreau  _et al._ (1990) J. C. Garreau, M. Allegrini, L. Julien, and F. Biraben, Journal de Physique 51, 2275 (1990).
  * Beyer  _et al._ (2013) A. Beyer, J. Alnis, K. Khabarova, A. Matveev, C. G. Parthey, D. C. Yost, R. Pohl, T. Udem, T. W. Hänsch, and N. Kolachevsky, Ann. Phys. 525, 671 (2013).
  * de Beauvoir  _et al._ (1997) B. de Beauvoir, F. Nez, L. Julien, B. Cagnac, F. Biraben, D. Touahri, L. Hilico, O. Acef, A. Clairon, and J. J. Zondy, Phys. Rev. Lett. 78, 440 (1997).
  * Gallagher (1988) T. F. Gallagher, Rep. Prog. Phys. 51, 143 (1988).
  * Scheidegger  _et al._ (2022) S. Scheidegger, D. Schlander, J. A. Agner, H. Schmutz, P. Jansen, and F. Merkt, J. Phys. B: At. Mol. Opt. Phys. 55, 155002 (2022).
  * Scoles (1988) G. Scoles, ed.,  _Atomic and molecular beam methods_ (Oxford University Press, New York, 1988) 2 vols.
  * Seiler  _et al._ (2005) R. Seiler, Th. Paul, M. Andrist, and F. Merkt, Rev. Sci. Instr. 76, 103103 (2005).
  * Haas  _et al._ (2006) M. Haas, U. D. Jentschura, C. H. Keitel, N. Kolachevsky, M. Herrmann, P. Fendel, M. Fischer, Th. Udem, R. Holzwarth, T. W. Hänsch, M. O. Scully, and G. S. Agarwal, Phys. Rev. A 73, 052501 (2006).
  * Hänsch and Couillaud (1980) T. W. Hänsch and B. Couillaud, Opt. Commun. 35, 441 (1980).
  * Husmann  _et al._ (2021) D. Husmann, L.-G. Bernier, M. Bertrand, D. Calonico, K. Chaloulos, G. Clausen, C. Clivati, J. Faist, E. Heiri, U. Hollenstein, A. Johnson, F. Mauchle, Z. Meir, F. Merkt, A. Mura, G. Scalari, S. Scheidegger, H. Schmutz, M. Sinhal, S. Willitsch, and J. Morel, Opt. Express 29, 24592 (2021).
  * Haubrich and Wynands (1996) D. Haubrich and R. Wynands, Opt. Commun. 123, 558 (1996).
  * Beyer  _et al._ (2016) A. Beyer, L. Maisenbacher, A. Matveev, R. Pohl, K. Khabarova, Y. Chang, A. Grinin, T. Lamour, T. Shi, D. C. Yost, Th. Udem, T. W. Hänsch, and N. Kolachevsky, Opt. Express 24, 17470 (2016).
  * Wirthl  _et al._ (2021) V. Wirthl, L. Maisenbacher, J. Weitenberg, A. Hertlein, A. Grinin, A. Matveev, R. Pohl, T. W. Hänsch, and T. Udem, Opt. Express 29, 7024 (2021).
  * Beyer  _et al._ (2018) M. Beyer, N. Hölsch, J. A. Agner, J. Deiglmayr, H. Schmutz, and F. Merkt, Phys. Rev. A 97, 012501 (2018).
  * (50) Lens assembly, $d$:distance of the first surface to the previous lens; $R_{1}$ and $R_{2}$: radius of the two surfaces; and $CT$: center thickness. L1=$\left[d=7.96,R_{1}=25.50,CT=5.06,R_{2}=13.27\right]$, L2=$\left[d=2.1,R_{1}=25.49,CT=5.1,R_{2}=13.28\right]$, L3=$\left[d=11.3,R_{1}=-36.46,CT=6.11,R_{2}=30.60\right]$, L4=$\left[d=4.2,R_{1}=17.21,CT=4.12,R_{2}=30.59\right]$. All dimensions are in mm.
  * Tiesinga  _et al._ (2002) E. Tiesinga, S. Kotochigova, and P. S. Julienne, Phys. Rev. A 65, 042722 (2002).
  * Jentschura and Mohr (2005) U. D. Jentschura and P. J. Mohr, Phys. Rev. A 72, 012110 (2005).
  * Horbatsch and Hessels (2016) M. Horbatsch and E. A. Hessels, Phys. Rev. A 93, 022513 (2016).
  * Cordle (1974) J. E. Cordle, J. Phys. B: At. Mol. Phys. 7, 1284 (1974).
  * Zare (1988) R. N. Zare,  _Angular momentum_ (John Wiley & Sons, New York, 1988).
  * Sobelman (1992) I. I. Sobelman,  _Atomic Spectra and Radiative Transitions_, 2nd ed., edited by J. P. Toennis, Springer Series on Atomic, Optical, and Plasma Physics, Vol. 12 (Springer, Berlin, 1992).
  * Dalgarno  _et al._ (1994) A. Dalgarno, P. L. Knight, F. H. Read, and R. N. Zare, eds.,  _Rydberg Atoms_, Cambridge Monographs on Atomic, Molecular and Chemical Physics, Vol. 3 (Cambridge University Press, Cambridge, 1994).
  * Hölsch  _et al._ (2022) N. Hölsch, I. Doran, M. Beyer, and F. Merkt, J. Mol. Spectrosc. 387, 111648 (2022).
  * Damburg and Kolosov (1979) R. J. Damburg and V. V. Kolosov, J. Phys. B: At. Mol. Phys. 12, 2637 (1979).
  * Bullis  _et al._ (2023) R. G. Bullis, C. Rasor, W. L. Tavis, S. A. Johnson, M. R. Weiss, and D. C. Yost, Phys. Rev. Lett. 130, 203001 (2023).


# Instruction-Following Speech Recognition

###### Abstract

Conventional end-to-end Automatic Speech Recognition (ASR) models primarily
focus on exact transcription tasks, lacking flexibility for nuanced user
interactions. With the advent of Large Language Models (LLMs) in speech
processing, more organic, text-prompt-based interactions have become possible.
However, the mechanisms behind these models’ speech understanding and
“reasoning” capabilities remain underexplored. To study this question from the
data perspective, we introduce instruction-following speech recognition,
training a Listen-Attend-Spell model to understand and execute a diverse set
of free-form text instructions. This enables a multitude of speech recognition
tasks – ranging from transcript manipulation to summarization – without
relying on predefined command sets. Remarkably, our model, trained from
scratch on Librispeech, interprets and executes simple instructions without
requiring LLMs or pre-trained speech modules. It also offers selective
transcription options based on instructions like “transcribe first half and
then turn off listening,” providing an additional layer of privacy and safety
compared to existing LLMs. Our findings highlight the significant potential of
instruction-following training to advance speech foundation models.

Index Terms—  Large Language Model, Speech Recognition, Speech Foundation
Model, Instruction-Following

##  1 Introduction

The successes of Large Language Models (LLMs) in natural language tasks have
prompted the speech community to develop speech foundation models that are
able to process, reason, and generate interleaving speech, audio, and text. It
could be immensely useful for digital assistant, because speech foundation
models provide a versatile user interface that is natural, flexible, and
powerful. For instance, to perform the task of Please transcribe the audio,
speech foundation models can accomplish it by decoding condition on the text
query, instead of first processing the Natural Language Understanding query
followed by an ASR as the target action [1].

Speech LLMs. We use the term “Speech LLM” to denote models that integrate LLMs
for speech and audio tasks [2, 3, 4, 5, 6]. We can think of the development of
these models as equipping existing LLMs with additional perception
capabilities. The underlying assumption of this new modeling paradigm is that
pre-trained LLMs can enable new capabilities to speech and audio processing
tasks that were previously unattainable: reasoning over speech perception,
open-ended multi-modal response generation, and zero-shot task generalization
via in-context learning. These models generally consist of three main
components: (i) an encoder or discrete units tokenizer for speech perception,
(ii) a pre-trained autoregressive language model as a decoder, and (iii) a
fine-tuning stage focused on speech instructions, formulated as
$\\{\text{speech, text instruction, model outputs}\\}$. Notably, they have
demonstrated the ability for understanding, or “reasoning”, over the speech
and audio recording via text instructions [2]. This raises the question of how
each component contributes to this remarkable capability.

Fig. 1: Instruction-trained speech recognizer reasons over free-from text
instructions and performs the desired ASR-related actions.

A Motivating Example. Consider the simple text query: “Ignore speech.” This is
a straightforward command that should be easy for a speech foundation model to
process, considering it merely requires the model to output an end-of-sentence
([EOS]) token. However, our experiments with opensourced models like Whisper
[7] and LTU v2 [2] revealed that they fail to execute such simple commands,
despite their impressive recognition and translation capabilities. This
suggests the importance of (iii) instruction-following task constructions. In
other words, however advanced “reasoning” capabilities these speech foundation
models possess, it is unlikely they can execute unseen actions or tasks that
were not present in training distributions.

This observation led us to develop a new kind of speech recognition model, one
that is instruction-following by design. Conditioned on the speech recording,
our model aims to understand and execute a wide range of ASR-related tasks
based on free-form text instructions, all without degrading the default ASR
capabilities. See Figure 1 for an illustration. Surprisingly, we find that a
224M parameter model without pre-trained speech or text foundation models, the
aforementioned (i) and (ii), can achieve these capabilities.

Related Work. Beyond Speech LLMs, there is a growing body of research
integrating visual perception into text LLMs [8, 9, 10, 11, 12]. For speech
prompting, WavPrompt [13], SpeechPrompt [14, 15], and WhisperPrompt [16]
leveraged pre-trained autoregressive models—namely GPT-2, GSLM [17], and
Whisper—for task-specific prompting. Instruction-based training has gained
traction in NLP [18, 19, 20, 21]. Different from them, we present an
instruction-trained speech recognizer that does not rely on any pre-trained
components.

Paper Organization. We begin by formulating the tasks illustrated in Figure 1
as a set of “skills,” which we define in Section 2.1. Next, we outline the
method for constructing instruction prompts for these skills in Section 2.2.
In Section 3, we describe an implementation of an instruction-following ASR
model that acquires these skills from scratch. Section 4 presents the
performance of our model in instruction-following tasks. We provide concluding
remarks in Section 5.

Fig. 2: Our model follows “Listen-Attend-Spell” (LAS) architecture: an
acoustic encoder and an autoregressive decoder with cross-attention to the
encoder latents. The training objective is next token prediction over sampled
prefix instructions and targeted text transcriptions. An end-of-turn ([EOT])
token is introduced to separate the two. In test-time decoding, prefix
instructions are specified by users.

##  2 Skills and Instructions

###  2.1 Defining Skills

We identify a set of ASR-related “skills” that our instruction-following
models should collectively master: (1) speech transcription, (2) ignoring
speech, (3) word replacement, (4) transcript manipulation, and (5)
summarization or keyword spotting. The term “speech transcription” refers to
standard ASR functionality, while “ignoring speech” implies that the model
should output an [EOS] token without considering the audio. For “word
replacement,” the model is tasked to pinpoint targeted words and replace them
as specified. We categorize this into sub-tasks: common word replacement
(e.g., replace ‘the’ with ‘a’), out-of-distribution (OOD) word replacement
(e.g., replace ‘the’ with ‘quokka’), and word deletion (e.g., remove ‘the’).
For “transcript manipulation,” the model should perform actions like deletion
or repetition in the transcript while preserving its accuracy. This is broken
down into sub-tasks such as repetition, transcribing the first half, and
transcribing the second half. Finally, “summarization and keyword spotting”
require the model to convey the essence of the speech concisely, possibly
reordering sentence structures. A comprehensive summary of these skills is
provided in Table 1.

Although these skills may appear simple in comparison to open-ended dialogue
generation, they encompass a broad range of functionalities in ASR: deletion
(skills 2 and 4), substitution (skill 3), insertion (skill 4), and a
combination (skill 5). Mastering these skills and dynamically switching
between them while maintaining precise ASR is non-trivial. Please refer to
Table 5 for examples of expected model outputs of each skill.

Skills | Description / Sub-tasks | Num. of | Error Type
---|---|---|---
Instructions
Speech Transcribe | Standard ASR | 500 | N/A
Ignore Speech | Outputs [EOS] token directly | 500 | Deletion
Word Replacement |  (3a) Common word (“the”$\rightarrow$ “a”) |  |
(3b) OOD word (“the”$\rightarrow$ “quokka”) | 600 | Substitution
(3c) Word deletion (“the”$\rightarrow$ N/A) |  |
Manipulation | (4a) Repetition |  | Insertion
(4b) Transcribe first half | 300 | Deletion
(4c) Transcribe second half |  | Deletion
Summarization / | Extract key idea and phrases | 100 | Mix
Keyword Spotting

Table 1: Summary of ASR-related Skills

###  2.2 Skill Constructions

Dataset We build our instruction-following templates on the Librispeech 960h
training set [22] and evaluate the model’s instruction-following performance
on its dev and test sets.

Constructing Instructions For each skill or sub-task, we generate a set of
diverse instructions, ranging from 100 to 600 prompts, via prompting GPT-4. We
constructed the initial GPT-4 prompt based on task description generation
prompts specified in SpeechGPT [3]. After careful inspection, we iteratively
refined the GPT-4 prompts to improve instruction diversity. An example of this
iterative process is listed in Table 2. This approach not only ensures diverse
instructions but also narrows the scope of out-of-distribution instructions
during inference, compelling the model to reason over the text query rather
than memorizing it directly. Some examples of text instructions for each skill
are highlighted in blue in Table 5.

Constructing Targets For skill (1), speech transcription, the original ASR
transcript serves as the target. For skills (2) ignoring speech, (3) word
replacement, and (4) transcript manipulation, we generate the target outputs
through rule-based processing. For skill (5), summarization and keyword
spotting, we use GPT-4 and GPT-3.5 to generate target summaries111We found the
following prompt works well for prompting GPTs for summarization / keyword
spotting: “Find the keyword in the following sentence: {original text}. Find
the most important 3-5 keywords. Make sure to rephrase it *very succinctly*.
Ideally less than 5.”. We found that selecting the shortest response from
multiple GPT runs produces more robust target responses.

GPT-4 Prompts Used | Num. of Instructions
---|---
Generated in each Iteration
You are asked to come up with a set of | 100
100 diverse task instructions about automatic
speech recognition. Here are the requirements… [3]
Based on the above instructions, generate similar | 50
ASR instructions with the following two rules:
1\. Succinct (1-4 words) 2. Contains similar vocab.
Generate another set of ASR instructions | 50
but with 5-8 words instead.
Generate commands that “enable the speech recognition | 50
capability”. Be creative and be succinct 1-5 words.

Table 2: Iterative GPT-4 prompting for generating diverse skill instructions.
Here we show the GPT-4 prompts adopted for retrieving the first 250 speech
transcription skill instructions.

##  3 Instruction-Following ASR

Model Our model is based on the Listen, Attend, Spell (LAS) architecture
[23]222LAS offers two main advantages as the backbone architecture. First, its
cross-attention mechanism allows the model to attend to the entire speech
recording, while selective cross-attention can aid tasks like word replacement
or keyword spotting. Second, we aim for the model to condition on a set of
prefix instruction tokens during inference, making an autoregressive decoder a
natural choice. This would be challenging for alignment-based models like
RNN-T or CTC.. The LAS encoder employs a Conformer-L architecture and takes an
80-dimensional log-Mel filterbank as input, with SpecAugment applied [24]. The
LAS decoder is a 12-layer Transformer LM decoder with cross-attention to the
encoder context vectors.

Training During training, for each utterance, we randomly sample a text
instruction and apply the corresponding operation to its transcript, such as
deletion, repetition, substitution, or paraphrasing, as outlined in Section
2.2. Specifically, skill sampling is performed according to Equation 1, using
weight ratios $\alpha$ for speech transcription and $\beta$ for
summarization/keyword spotting. After selecting the sampled skill $S$, we
choose an instruction $I$ from $S$’s corresponding list of instructions. The
original transcript is then modified to produce $T$, which is concatenated
(denoted as $\oplus$) with $I$ and an End of Turn [EOT] token to form the
training sample $Y$. The model is subsequently trained on $Y$ using next-token
prediction with teacher-forcing.

Sample Skill: $\displaystyle
S\sim\begin{cases}\text{Transcribe,}&P=\frac{\alpha}{\alpha+3+\beta}\\\
\text{Ignore Speech,}&P=\frac{1}{\alpha+3+\beta}\\\ \text{Word
Replace,}&P=\frac{1}{\alpha+3+\beta}\\\
\text{Manipulate,}&P=\frac{1}{\alpha+3+\beta}\\\
\text{Summarize,}&P=\frac{\beta}{\alpha+3+\beta}\end{cases}$ (1) Sample
Instruction: $\displaystyle I\sim\text{Skill Instruction List}(S)$ (2) Target
Transcript: $\displaystyle T=\text{Modify according to }S$ (3) Training Data:
$\displaystyle Y=I\oplus\text{[EOT]}\oplus T$ (4)

Formally, let tokenized text instruction be $I$, tokenized target transcript
be $T$, and $A$ be the referenced speech. The training objective is

$\max_{\theta}\hskip 2.84526pt\mathbb{E}\big{[}\log P(T\mid I,A;\hskip
2.84526pt\theta)\big{]}$ (5)

where $\theta$ represents the model parameters.

Decoding During test time, instructions concatenated with an [EOT] token serve
as prefixes to the LAS decoder. As all our tasks relate to ASR, we found that
beam-search decoding produced better results than sampled decoding. Define a
score function $\bm{s}(O,A,I^{\prime})$, where $I^{\prime}$ is the test-time
instruction,

$\vspace{-1mm}\bm{s}(O,A,I^{\prime})\hskip 2.84526pt=\hskip 2.84526pt\log
P(O\mid I^{\prime},A;\hskip 2.84526pt\theta)\hskip 2.84526pt/\hskip
2.84526pt\text{lp}(O)$ (6)

with an length normalization term $\text{lp}(O)$ included333We referenced [25]
for length-normalized beam search, setting $\text{lp}(O)=(5+\mid
O\mid)^{0.8}/(5+1)^{0.8}.$. The most probably output sequence $O^{*}$ given
instruction $I^{\prime}$ and the trained LAS model is then,
$O^{*}=\operatorname*{arg\,max}_{O}\hskip 2.84526pt\bm{s}(O,A,I^{\prime})$.
Moreover, we perform model selection on the dev set with its speech
transcription skill (model’s ASR capability) only, where speech transcription
instructions are randomly drawn and the model outputs are evaluated in WERs.

Implementations For instruction sampling, we configure $\alpha=56$ and
$\beta=4$. We employ a 1024-token word-piece model, which includes special
tokens ([BOS], [EOT], [EOS]), constructed from the LibriSpeech transcripts.
LAS decoder dropout rate is set to 0.1. The learning rate is linearly warmup-
ed to 0.2 in 10k steps and exponentially decayed for another 90k steps. Models
were implemented in Jax and trained on 256 v4-TPUs. We set the beam to 10 for
all decoding results presented.

##  4 Results and Discussions

###  4.1 Librispeech ASR Benchmark

We evaluate the model’s ASR performance on the LibriSpeech dev and test sets.
The primary aim is to demonstrate that instruction-based training enables
high-precision speech recognition when the model is prompted for speech
transcription. Note that our models were trained from scratch on Librispeech
960h only and no LM fusion was used.

Table 3: Instruction-based training doesn’t compromise the model’s ASR
performance. Prompt “Please transcribe the speech” was used.

Model | Param | test-clean | test-other | Instruction
---|---|---|---|---
Following
Whisper Large V2 | 1550M | 2.7 | 5.2 | ✗
Google’s LAS [26] | - | 2.8 | 6.8 | ✗
Our vanilla LAS | 224M | 3.1 | 6.0 | ✗
Instruction-Trained LAS | 224M | 2.6 | 5.6 | ✓

Initially, we trained a baseline LAS model without instruction-based training,
achieving reasonable WERs relative to Whisper [7] and Google’s LAS [26] (see
Table 3). Subsequently, we trained an identical LAS model incorporating
instruction-based training. We used the text prompt “Please transcribe the
speech” as the default instruction for ASR evaluation. The results suggest
that instruction-based training does not degrade ASR performance (see Table 3,
row 4). This finding also implies that the acquisition of other skills does
not negatively affects model’s ASR capabilities.

Table 4: Beam-search decoding results given unseen instructions.

Samples of Unseen Instructions | Expected Behavior | Model Behavior
---|---|---
Prompt voice to text translation | ASR | ASR
Forsake Voice Interaction | ignore speech | ignore speech
Annihilate Voice Interpretation System | ignore speech | ASR
Before you have your lunch, convert | ASR | ASR
the following speech into text. Then
make sure the windows are closed.
Ignore the audio; but rather, | ASR | ASR
note the essential words

Table 5: Sampled text instruction prompts on the left (blue) and instruction-
trained ASR model outputs on the right (grey). Skill 1: Speech Transcribe
---
Decode the content of this audio. |
Start recognizing audio! |
Transcribe the following spoken words: | the influence with the timaeus has exercised upon posterity is due partly to a misunderstanding.
Analyze dialogue recording. |
Listen and jot down the speech content. |
Skill 2: Ignore Speech
Ignore the audio in this clip. |
Avoid interaction with this conversation. |
Omit the dialogue from this audio | N/A
Mute Recognition |
Overlook any notation of this conversation. |
Skill 3: Word Replacement
Replace ’the’ with ’a’ as you listen. | a influence with a timaeus has exercised upon posterity is due partly to a misunderstanding.
Let us switch all ’the’ to ’a’, shall we? | a influence with a timaeus has exercised upon posterity is due partly to a misunderstanding.
Transcribe, making ’the’ into ’quokka’ in speech. | quokka influence with quokka timaeus has exercised upon posterity is due partly to a misunderstanding.
Substitute ’the’ with ’quokka’ like a chef’s secret. | quokka influence with quokka timaeus has exercised upon posterity is due partly to a misunderstanding.
Overlook ’the’ in the transcription. | influence with timaeus has exercised upon posterity is due partly to a misunderstanding.
Like a magician. Make ’the’ vanish! | influence with timaeus has exercised upon posterity is due partly to a misunderstanding.
Skill 4: Manipulation
Repeat the transcription. | the influence with the timaeus has exercised upon posterity is due partly to a misunderstanding the influence
with the timaeus has exercised upon posterity is due partly to a
misunderstanding.
Transcribe the speech and then create two copies. | the influence with the timaeus has exercised upon posterity is due partly to a misunderstanding the influence
with the timaeus has exercised upon posterity is due partly to a
misunderstanding.
Only write the first half. Delete the rest. | the influence with the timaeus has exercised upon
Capture beginning. | the influence with the timaeus has exercised upon
Omit first half. | posterity is due partly to a misunderstanding
Write from halfway to end. | posterity is due partly to a misunderstanding
Skill 5: Summarization / Keyword Spotting
Provide a concise summary of the audio. |
Craft a short audio rephrase. |
Identify pivotal audio keywords. | timaeus influence due to misunderstanding.
Summarize the audio. |
Provide a brief speech summary. |

###  4.2 Instruction-Following Capabilities

To probe the capabilities of the instruction-trained LAS effectively, we
employed a test suite comprising 10 randomly sampled seen and 10 constructed
_unseen_ instruction prompts for each skill. We executed these instructions on
the first 500 utterances from the test-clean set. Our key observations are as
follows:

High Fidelity to Seen Instructions. The model reliably discerns the target
skill specified in seen instructions, executing them with 100% accuracy. Given
our instruction-following dataset’s complexity, featuring 2000 distinct seen
instructions with a broad range of vocabulary and intricate sentence
structures, the consistent performance is notable. Our shallow LAS decoder
further suggest that this ability extends beyond mere prompt memorization.
Refer to Table 5 for example outputs across varied instruction prompts.

Generalization to Unseen Instructions. The model executes unseen instructions
with $\sim 80\%$ accuracy, providing direct evidence of its instruction
understanding, even without pre-trained LLMs. The model’s performance on
unseen instructions is illustrated in Table 4.

Implicit Speech Understanding. Although the execution of skills (1) to (4)
does not necessitate any form of understanding of the speech, the ability to
summarize and identify keywords suggests otherwise. The model likely first
implicitly understands the audio before autoregressively decide which key
phrases are representative enough to be decoded. Examples are in Table 6.

Table 6: Summarization / Keyword Spotting examples via instruction “Rephrase
or summarize the audio”.

Original Transcript | Model Outputs
---|---
there’s a heavy storm coming on I | heavy storm cried pointing towards horizon
cried pointing towards the horizon
in the court yard some of the merry | married children danced christmas glad sight
children were playing who had danced
at christmas round the fir tree and
were so glad at the sight of him
if you dressed in silk and gold from top | dressed in silk gold not nicer than red cap
to toe you could not look any nicer
than in your little red cap

###  4.3 Skill Evolution

To understand the co-existence and evolution of multiple skills within the
instruction-trained LAS, we observed the model’s performance across different
phases of training. At 31k steps, the model exhibits a preliminary capability
in speech transcription, reflected by a 16% WER, while showing negligible
skills in either ignoring or summarizing speech. At this point, the model can
perform simple manipulations, such as repetition, albeit repeating more than
the specified number of times in the instruction. Intriguingly, the model
initially grasps the concept of repetition before refining the number of
repetitions over training time. Speech transcription errors also propagate to
other skills. For example, in Table 6’s middle row, the model’s summarization
incorrectly includes the word “married” due to a mistake in its ASR, which
should have decoded the word as “merry.” Lastly, we found a model trained only
on Skill (4) diverged, whereas one trained on both Skill (1) and (4) did not,
suggesting that speech transcription serves as a foundational skill from which
others evolve. This indicates that skills are not likely learned in isolation;
rather, the model seems to first acquire a proficiency in speech
transcription, which then acts as a basis for the development of other, more
specialized text manipulation skills.

###  4.4 Discussions

The training of Speech LLMs typically involves three key components: (i) a
pre-trained speech encoder or discrete units, (ii) a pre-trained text LLM like
LLaMA [27] or PaLM [28], and (iii) speech instruction fine-tuning. The core
contribution of this paper is to demonstrate that a relatively small model can
effectively follow instructions without requiring components (i) or (ii). The
skill set selection detailed in Section 2.1 serves to establish a clean and
workable framework that emphasizes the importance of (iii). We now delineate
some limitations of our current implementation of instruction-following LAS:

  * •

Instruction Generalization: The model’s ability to follow instructions is
tightly linked to the training data, struggling with OOD instructions, such as
those with unfamiliar vocabulary or sentence structures. For instance, in
Table 4 row 3, the word “annihilate” was not seen in training and thus the
wrong skill interpretation. Conversely, our model capably handles contrastive
imperatives, a feature that stems from our instruction design in Section 2.2.

  * •

Task Generalization: The model is limited to predefined tasks, Skills (1) to
(5), and does not generalize well. For example, it cannot perform word
replacement of “the” with “car”.

  * •

Dialogue Engagement: The model’s responses are close-ended, lacking support
for open-ended and multi-turn conversations, or clarification follow-up if
instructions are ambiguous.

One way to address task generalization is to train the model on a more diverse
set of instruction data. For example, exposure to numerous word replacement
tasks could improve its ability to generalize to unseen target words. Both
task and instruction generalization could benefit from the integration of text
LLMs; for instance, initializing the LAS decoder with LLaMA could mitigate the
challenges posed by OOD instructions and enhance instruction generalization.
Training our instruction-following models on open-ended speech instructions,
such as those in LTU [2], should also prompt the model to pick up dialogue
engagement ability.

Another significant advantage of instruction-based speech recognition is its
potential for enhancing safety and privacy. Our model is capable of
selectively replacing target words or executing partial transcriptions. A
natural extension of this capability would be to filter out sensitive
information, such as profanities or personal names, a task not easily
achievable via traditional ASR and LLM cascading pipelines.

##  5 Conclusion

This paper illustrates the viability and importance of instruction-based
training [18, 19, 20, 21] for speech models, offering a straightforward
framework for skill execution based on natural language prompts. Utilizing a
small encoder-decoder model trained from scratch on Librispeech, we prove that
understanding and executing free-form text instructions is feasible. Our
carefully designed instructions elicited five key skills: speech
transcription, ignoring speech, word replacement, manipulation, and
summarization/keyword spotting. Evaluations indicate robust performance on
both familiar and novel instructions without compromising ASR capabilities.
Our study demonstrates the effectiveness of instruction-based speech
recognition via well-crafted instruction templates.

## References

  * [1] R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu, et al.,  “Audiogpt: Understanding and generating speech, music, sound, and talking head,”  arXiv, 2023.
  * [2] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass,  “Listen, think, and understand,”  arXiv, 2023.
  * [3] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu,  “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,”  arXiv, 2023.
  * [4] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang,  “Pengi: An audio language model for audio tasks,”  arXiv, 2023.
  * [5] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, et al.,  “Audiopalm: A large language model that can speak and listen,”  arXiv, 2023.
  * [6] Y. Fathullah, C. Wu, E. Lakomkin, J. Jia, Y. Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli, et al.,  “Prompting large language models with speech recognition abilities,”  arXiv, 2023.
  * [7] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever,  “Robust speech recognition via large-scale weak supervision,”  2023\.
  * [8] OpenAI,  “Gpt-4 technical report,”  arXiv, 2023.
  * [9] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, Q. Liu, et al.,  “Language is not all you need: Aligning perception with language models,”  arXiv, 2023.
  * [10] J. Li, D. Li, S. Savarese, and S. Hoi,  “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,”  arXiv, 2023.
  * [11] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al.,  “Palm-e: An embodied multimodal language model,”  arXiv, 2023.
  * [12] H. Liu, C. Li, Q. Wu, and Y. J. Lee,  “Visual instruction tuning,”  arXiv, 2023.
  * [13] H. Gao, J. Ni, K. Qian, Y. Zhang, S. Chang, and M. Hasegawa-Johnson,  “Wavprompt: Towards few-shot spoken language understanding with frozen language models,”  arXiv, 2022.
  * [14] K.-W. Chang, W.-C. Tseng, S.-W. Li, and H.-y. Lee,  “Speechprompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks,”  arXiv, 2022.
  * [15] K.-W. Chang, Y.-K. Wang, H. Shen, I.-t. Kang, W.-C. Tseng, S.-W. Li, and H.-y. Lee,  “Speechprompt v2: Prompt tuning for speech classification tasks,”  arXiv, 2023.
  * [16] P. Peng, B. Yan, S. Watanabe, and D. Harwath,  “Prompting the hidden talent of web-scale speech models for zero-shot task generalization,”  arXiv, 2023.
  * [17] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, et al.,  “On generative spoken language modeling from raw audio,”  TACL, 2021.
  * [18] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.,  “Training language models to follow instructions with human feedback,”  NeurIPS, 2022.
  * [19] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al.,  “Multitask prompted training enables zero-shot task generalization,”  ICLR, 2022.
  * [20] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le,  “Finetuned language models are zero-shot learners,”  ICLR, 2022.
  * [21] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al.,  “Scaling instruction-finetuned language models,”  arXiv, 2022.
  * [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  2015\.
  * [23] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals,  “Listen, attend and spell,”  ICASSP, 2016.
  * [24] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu,  “Pushing the limits of semi-supervised learning for automatic speech recognition,”  NeurIPS SAS workshop, 2020.
  * [25] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al.,  “Google’s neural machine translation system: Bridging the gap between human and machine translation,”  arXiv preprint arXiv:1609.08144, 2016.
  * [26] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  Interspeech, 2019.
  * [27] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.,  “Llama: Open and efficient foundation language models,”  arXiv, 2023.
  * [28] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.,  “Palm: Scaling language modeling with pathways,”  arXiv, 2022.