# Dephasingless two-color terahertz generation

Tanner T. Simpson<EMAIL_ADDRESS>Laboratory for Laser Energetics,
University of Rochester, Rochester, New York 14623-1299 USA  Jeremy J. Pigeon
Laboratory for Laser Energetics, University of Rochester, Rochester, New York
14623-1299 USA  Kyle G. Miller  Laboratory for Laser Energetics, University of
Rochester, Rochester, New York 14623-1299 USA  Dillon Ramsey  Laboratory for
Laser Energetics, University of Rochester, Rochester, New York 14623-1299 USA
Dustin H. Froula  Laboratory for Laser Energetics, University of Rochester,
Rochester, New York 14623-1299 USA  John P. Palastro<EMAIL_ADDRESS>Laboratory for Laser Energetics, University of Rochester, Rochester, New York
14623-1299 USA

###### Abstract

A laser pulse composed of a fundamental and an appropriately phased second
harmonic can drive a time-dependent current of photoionized electrons that
generates broadband THz radiation. Over the propagation distances relevant to
many experiments, dispersion causes the relative phase between the harmonics
to evolve. This “dephasing” slows the accumulation of THz energy and results
in a multi-cycle THz pulse with significant angular dispersion. Here, we
introduce a novel optical configuration that compensates the relative phase
evolution, allowing for the formation of a half-cycle THz pulse with almost no
angular dispersion. The configuration uses the spherical aberration of an
axilens to map a prescribed radial phase variation in the near field to a
desired longitudinal phase variation in the far field. Simulations that
combine this configuration with an ultrashort flying focus demonstrate the
formation of a half-cycle THz pulse with a controlled emission angle and 1/4
the angular divergence of the multi-cycle pulse created by a conventional
optical configuration.

## Introduction

The laser–matter interactions at the foundation of all nonlinear optical
applications occur in dispersive media. These applications benefit from, and
in some cases rely on, tailoring the wavefronts of a laser pulse to counteract
dispersion. This is accomplished either through the structure of the medium or
by preliminary structuring of the laser pulse. The former has become a
mainstay of nonlinear optics: the different refractive indices along
orthogonal crystal axes is regularly used for phase matching of sum-and-
difference frequency generation Midwinter and Warner (1965); Maker  _et al._
(1962); Boyd (2019), while periodically poled lithium niobate allows for
quasi-phase matching of the same processes Fejer  _et al._ (1992); Yamada  _et
al._ (1993). Structuring the laser pulse, although not as widespread, has
found utility in optical rectification, where tilting the phase fronts with
respect to the pulse front enables velocity matching to terahertz (THz)
radiation Hebling  _et al._ (2002); Wu  _et al._ (2023). The recent
development of more-advanced techniques for space–time structuring presents
new opportunities to overcome the limitations imposed by dispersion and
rethink the optimization of nonlinear optical applications Kondakci and
Abouraddy (2017); Sainte-Marie  _et al._ (2017); Froula  _et al._ (2018);
Kondakci and Abouraddy (2019); Palastro  _et al._ (2020); Caizergues  _et al._
(2020); Li and Kawanaka (2020); Jolly  _et al._ (2020); Yessenov  _et al._
(2022); Liang  _et al._ (2023); Ambat  _et al._ (2023); Pigeon  _et al._
(2024); Liberman  _et al._ (2024).

A subset of space–time structuring techniques known as the “flying focus”
modify the focal time and location of each frequency, temporal slice, or
annulus of a pulse to produce an intensity peak that moves independently of
dispersionFroula  _et al._ (2018); Palastro  _et al._ (2020); Jolly  _et al._
(2020); Simpson  _et al._ (2020, 2022); Li  _et al._ (2024). With the ability
to control the trajectory of an intensity peak, flying-focus techniques are
predicted to enhance a wide range of applications and fundamental
measurements, including advanced accelerators Palastro  _et al._ (2020);
Ramsey  _et al._ (2020); Palastro  _et al._ (2021); Miller  _et al._ (2023),
radiation sources Howard  _et al._ (2019); Franke  _et al._ (2021); Ramsey
_et al._ (2022); Kabacinski  _et al._ (2023), and signatures of quantum
electrodynamical phenomena Di Piazza (2021); Formanek  _et al._ (2022, 2024).
Nevertheless, the flying-focus techniques that have been proposed Simpson  _et
al._ (2020, 2022); Li  _et al._ (2024) and experimentally demonstrated Froula
_et al._ (2018); Jolly  _et al._ (2020); Pigeon  _et al._ (2024) thus far are
limited to control over the intensity of a laser pulse. Control over the
phase, which would substantially broaden the potential to enhance
applications, has remained elusive and constrained by the medium.

“Two-color” THz generation in gases is a promising approach to producing high-
quality, broadband THz radiation Kim  _et al._ (2007, 2008); Babushkin  _et
al._ (2010); You  _et al._ (2012); Bergé  _et al._ (2013); Johnson  _et al._
(2013); Zhang  _et al._ (2018); Yoo  _et al._ (2019); Koulouklidis  _et al._
(2020); Buldt  _et al._ (2021); Simpson  _et al._ (2024) that depends
sensitively on both the intensity and phase of a laser pulse. In this
approach, a laser pulse composed of a fundamental and second harmonic ionizes
a gas and drives a time-dependent current of freed electrons, which provides
the source for the THz radiation. The strength of the current (and rate of THz
generation) is proportional to the number of photoionized electrons and the
drift velocity acquired at the instant of their birth, both of which are
sensitive to the waveform (i.e., intensity and relative phase) of the two-
color electric field Kim  _et al._ (2007). Over short propagation distances,
the relative phase between the harmonics is preserved, the strength and sign
of the current are approximately constant, and the THz radiation
constructively interferes at a Cherenkov angle determined by the THz phase
velocity and ionization front velocity Johnson  _et al._ (2013); Simpson  _et
al._ (2024). Over longer distances, dispersion causes the relative phase to
evolve. This “dephasing” between the harmonics produces an oscillating current
that disrupts the constructive interference, slows the rate of THz generation,
and angularly disperses the THz bandwidth You  _et al._ (2012).

Here we introduce a novel optical configuration that overcomes dephasing in
two-color THz generation by simultaneously controlling the intensity and
relative phase of a two-color laser pulse. When the relative phase is
precompensated to counteract dispersion, the two-color pulse drives a non-
alternating current of photoionized electrons that produces a half-cycle THz
pulse nearly devoid of angular dispersion. This “dephasingless” two-color THz
generation is demonstrated by simulations that model the optical
configuration, nonlinear propagation of the two-color pulse through gas, and
the generation and propagation of the THz radiation. The optical configuration
consists of three elements: an axilens to extend the focal range; a radially
stepped echelon to control the focal time at each point within the focal
range; and a diffractive lens, or “phaser,” to apply a weak radially varying
phase to the second harmonic, which the axilens maps to a longitudinally
varying phase (Fig. 1). The first two elements describe the ultrashort flying
focus and provide control over the velocity of the intensity peak and emission
angle of the THz Simpson  _et al._ (2024), while the novel phaser element
provides the phase control. Aside from two-color THz generation, the intensity
and phase control afforded by this configuration may benefit other nonlinear
optical processes that rely on phase matching between multiple frequencies,
such as harmonic generation and optical parametric amplification.

Figure 1: Comparison of THz pulses generated by two-color laser pulses
prepared with varying degrees of phase and intensity control. The
spatiotemporal profiles for frequencies $<30$ THz are displayed to the right
of each optical configuration at three longitudinal locations in the far
field. The circles below each configuration illustrate the evolution of the
relative phase between the harmonics of the two-color pulse $\Delta\theta$.
(a) A phased-flying-focus pulse drives the conical emission of a half-cycle
THz pulse over multiple dephasing lengths. The THz pulse is emitted into a
single angle and has almost no angular dispersion. (b) A standard ultrashort-
flying-focus pulse drives the conical emission of a multi-cycle THz pulse.
Dephasing results in constructive interference at two characteristic angles
and causes significant angular dispersion, which manifests in the near field
as two diffuse rings. (c) A two-color laser pulse focused by an ideal lens
also drives a multi-cycle THz pulse. In this case, the THz pulse evolves into
a single diffuse ring in the near field, indicating both dephasing and complex
evolution of the two-color laser pulse and resulting photocurrent. The near-
field transverse profiles (red circles) are shown 1 m beyond the edge of the
gas, and the white dashed lines in (a) and (b) mark the predictions of Eqs.
(2) and (3). For ease of visualization, the optical configurations are
displayed in transmission. In practice, only one harmonic is modified by the
phaser.

## Results

A laser pulse composed of a fundamental and second harmonic produces the
largest current of photoionized electrons and maximizes the rate of THz
generation when the relative phase between the harmonics is approximately
$\pi/2$. While a relative phase of $\pi/2$ can occur locally, the frequency
dependence of the refractive index due to the presence of gas and plasma
causes the relative phase to evolve along the propagation path. During linear
propagation, the relative phase advances continuously, reaching integer
multiples of $\pi$ after every dephasing length

$L_{\pi}=\frac{\lambda_{1}}{4}\frac{1}{|n(\lambda_{1})-n(\lambda_{2})|},$ (1)

where $\lambda_{1}$ and $\lambda_{2}$ are the vacuum wavelengths of the two
harmonics and $n$ is the linear refractive index of the gas. Structuring the
phase and intensity of the two-color laser pulse to precompensate the
evolution of the relative phase eliminates dephasing and ensures the formation
of a half-cycle THz pulse devoid of angular dispersion (Fig. 1a).

To demonstrate the advantage of phase and intensity control in two-color THz
generation, simulations were performed using the unidirectional pulse
propagation equationKolesik  _et al._ (2002); Kolesik and Moloney (2004);
Couairon  _et al._ (2011) initialized with pulses prepared by the optical
configurations displayed in Fig. 1 (see Methods). In each simulation, a laser
pulse composed of a fundamental and second harmonic was focused by the optical
configuration in the near field to the entrance of the gas in the far field.
The pulses propagated through and photoionized the gas, producing a current
that generated THz radiation. The physical parameters for the simulations are
provided in Table 1. The parameters chosen for the laser pulses and gas were
motivated by commercially available high-repetition-rate Ti:Sapphire laser
systems KMLabs  and typical two-color THz generation experiments You  _et al._
(2012); Kim  _et al._ (2008); Yoo  _et al._ (2019); Koulouklidis  _et al._
(2020).

Figure 1 compares the THz pulses resulting from three optical configurations,
each of which provides a different degree of control over the waveform of the
two-color laser pulse. The novel, phased-flying-focus design introduced in
this work (Fig. 1a) precompensates the relative phase of the two harmonics to
counteract dispersion and produces an intensity peak that travels at a
specified velocity $v_{f}$. By counteracting dispersion, the two-color pulse
drives the conical emission of a half-cycle THz pulse over multiple dephasing
lengths. The pulse is emitted into an angle determined by the Cherenkov
condition

$\phi=\arccos(v_{T}/v_{f}),$ (2)

where $v_{T}$ is the phase velocity of the THz radiation. The emission angle
$\phi$ has only a weak dependence on frequency through the phase velocity
$v_{T}$. As a result, all of the generated THz frequencies are emitted into
nearly the same angle, which can be controlled through the focal velocity
$v_{f}$. The minimal angular dispersion results in a near-field fluence
profile characterized by a single, narrow ring.

The standard flying focus and ideal lens configurations do not precompensate
the relative phase of the two harmonics (Figs. 1b and c). The standard flying
focus does, however, offer control over the velocity of the intensity peak. In
this case (Fig. 1b), the two-color pulse drives the conical emission of a
multi-cycle THz pulse characterized by two emission angles You  _et al._
(2012); Johnson  _et al._ (2013)

$\phi_{\pm}=\arccos\left(\frac{v_{T}}{v_{f}}\pm\frac{\pi
v_{T}}{\omega_{T}L_{\pi}}\right),$ (3)

where $\omega_{T}$ is the THz frequency. Compared to $\phi$ [Eq. (2)], the
emission angles $\phi_{\pm}$ have a stronger dependence on the THz
frequency—the second term on the right-hand side of Eq. (3). This additional
frequency dependence is a direct consequence of dephasing and causes
significant angular dispersion. The resulting near-field profile exhibits two
rings corresponding to $\phi_{\pm}$, each of which are broadened by the
angular dispersion. With neither phase nor intensity control, a two-color
laser pulse focused by the ideal lens drives a multi-cycle THz pulse with a
complex and diffuse angular emission pattern that is not accurately predicted
by Eqs. (2) or (3) (Fig. 1c). The diffuse near-field emission pattern
indicates both dephasing and intensity evolution due to the complex interplay
between linear focusing, self-focusing, and plasma refraction.

Table 1: Simulation parameters for the two-color laser pulse, optical configurations, and gas.  Laser Parameters | Value
---|---
$\lambda_{1}(\mu\mathrm{m}$) | $0.8$
$\lambda_{1}$ Power (GW) | 100
$\lambda_{1}$ Duration (fs) | 30
$\lambda_{2}(\mu\mathrm{m})$ | $0.4$
$\lambda_{2}$ Power (GW) | 20
$\lambda_{2}$ Duration (fs) | $15\sqrt{2}$
Total energy (mJ) | 3.5
Optics Parameters | Value
Axiparabola $f/\\#$ | 15
Axiparabola radius $R$ (cm) | 5
Focal range $L$ (cm) | 12
Phaser length $L_{P}$ (cm) | 2.5
Focal velocity $v_{f}$ (c) | 1.001
Ideal lens $f/\\#$ | 250
Ideal lens confocal parameter (cm) | 12
Medium Parameters | Value
Species | Ar
Density ($\mathrm{cm}^{-3}$) | $2.7{\times}10^{19}$
Nominal dephasing length $L_{\pi}$ (cm) | 2.9
Electron-neutral collision frequency (THz) | 10
Nonlinear refractive index ($\mathrm{cm}^{2}/\mathrm{W}$) Zahedpour  _et al._ (2015) | $1{\times}10^{-19}$

The optical configuration for the phased flying focus consists of three
elements: an axiparabola, a radially stepped echelon, and a weak diffractive
lens or “phaser.” The axiparabola, a specific type of axilens, is designed to
produce a uniform, on-axis intensity over an extended distance when
illuminated by a laser pulse with a flattop transverse profile Sochacki  _et
al._ (1992); Smartsev  _et al._ (2019); Palastro  _et al._ (2020); Oubrerie
_et al._ (2022); Ambat  _et al._ (2023). Light rays incident at different
radii $r$ on the axiparabola are focused to different longitudinal locations
$z=f(r)$, where

$f(r)=f_{0}+\left(\frac{r}{R}\right)^{2}L,$ (4)

$f_{0}$ is the nominal focal length, $L$ is the focal range, and $R$ is the
maximum radius of the optic. The radially stepped echelon applies a radially
dependent delay $\tau(r)$ that controls the arrival time of the light rays at
their focus Palastro  _et al._ (2020); Ambat  _et al._ (2023). For a desired
velocity $v_{f}$,

$\tau(r)=\frac{R^{2}}{2cL}\left[\frac{f(r)}{f_{0}}-1+\ln\Bigl{(}\frac{f(r)}{f_{0}}\Bigr{)}\right]+\frac{L}{cR^{2}}\left(1-\frac{v_{f}}{c}\right)r^{2}.$
(5)

To avoid distorting the phase front applied by the axiparabola, the echelon
applies this delay in discrete concentric rings with varying width and half-
integer-wavelength depth. Together, the axiparabola and echelon produce an
ultrashort moving focus, which travels at a velocity $v_{f}$ that is
independent of dispersion Palastro  _et al._ (2020); Ambat  _et al._ (2023);
Pigeon  _et al._ (2024).

The weak diffractive lens, or phaser, imparts the necessary phase to
precompensate the relative phase advance of the first and second harmonic due
to dispersion in the gas. To design such an optic, recall that the axiparabola
maps each radius in the near field $r$ to a different on-axis location in the
far field $z=f(r)$. As a result, a desired longitudinal phase variation in the
far field $\psi(z)$ can be mapped back to a required radial phase variation
$\psi[z(r)]$ in the near field. A longitudinal phase variation of

$\psi_{P}(z)=\pi\frac{z}{L_{\pi}},$ (6)

applied to one of the harmonics would result in their relative phase advancing
by $\pi$ after every dephasing length, which is exactly the advance required
to cancel the phase evolution due to dispersion. Using $f(r)$, Eq. (6) can be
reexpressed as the near-field radial phase variation

$\psi_{P}[z(r)]=\pi\left(\frac{r}{R}\right)^{2}\frac{L}{L_{\pi}},$ (7)

where constant phase terms have been dropped. This phase is equivalent to that
applied by a diffractive lens with a focal length
$f_{P}=R^{2}L_{\pi}/\lambda_{2}L$. In practice, the actual dephasing length
depends on the detailed propagation and includes contributions from the
nonlinear refractive index of the gas and plasma formation. Thus, in order to
optimize the phaser optic for THz generation, it is useful to replace the
nominal dephasing length $L_{\pi}$ in Eq. (7) or $f_{P}$ with an adjustable
length $L_{P}$. For the parameters considered here, the formation of a half-
cycle THz pulse was not sensitive to the precise value of $L_{P}$ and setting
$L_{P}=L_{\pi}$ yielded similar, albeit slightly less optimized, results (see
Methods).

Figure 2: The accumulation of THz energy ($<30$THz) with distance depends on
the degree of phase and intensity control and nonlinear propagation of the
two-color pulse. (a) THz energy as a function of distance for the phased
flying focus (blue, solid), standard ultrashort flying focus (red, dotted),
and a conventional pulse focused by an ideal lens (black, dashed). (b,c) The
electric field amplitudes of the first (top) and second harmonic (bottom) for
(b) the phased flying focus and (c) conventional pulse at $\Delta
z=6\;\mathrm{cm}$. The flying-focus pulses are relatively resistant to
nonlinear propagation, which keeps the first and second harmonic overlapped in
space and time. This allows the THz energy to grows continuously when
dephasing is eliminated with the phaser element (blue, solid) or periodically
when dephasing is present (red, dotted). The first harmonic of the
conventional pulse undergoes rapid self-focusing, which causes a loss of
spatiotemporal overlap between the first and second harmonic. As a result, the
THz energy grows over a short distance and saturates.

Figure 2a displays the energy of the THz pulses produced by the three optical
configurations as a function of distance. When both the relative phase and
intensity of the two-color pulse are controlled using the phased flying focus
(Fig. 1a), the THz energy increases at a near-constant rate over $\sim$four
dephasing lengths. This steady increase in the THz energy indicates the
absence of dephasing. When only the intensity is controlled using the
ultrashort flying focus without the phaser element (Fig. 1b), the energy grows
where the relative phase between the two harmonics is favorable for driving a
net current and plateaus where the relative phase is unfavorable. This energy
growth is consistent with a simple theoretical picture of dephasing in two-
color THz generation You  _et al._ (2012). When neither phase nor intensity
control are applied, the THz energy grows rapidly over a short distance and
then saturates (Fig. 1c).

The rapid growth and saturation of the THz energy observed with the
conventional pulse is a result of nonlinear propagation. In each
configuration, the power of the two-color pulse exceeds the critical power for
self-focusing: $P/P_{c}=10$ and $8$ for the fundamental and second harmonic.
Upon entering the gas, the fundamental of the conventional pulse rapidly self-
focuses, which increases its intensity. Once the earlier time slices of the
fundamental reach sufficient intensity to ionize the gas, they create a
plasma, which refracts subsequent time slices. This refraction hollows out the
back of the fundamental (Fig. 2c). The second harmonic, in contrast, is less
susceptible to plasma refraction and approximately maintains its
spatiotemporal profile. The loss of overlap between the two harmonics within
the ionization front terminates the THz generation.

The flying-focus pulses are relatively resistant to nonlinear propagation.
This can be understood by considering two properties of ultrashort-flying-
focus pulses. First, the transverse profile of these pulses features a central
maximum surrounded by multiple rings. Thus, even though the total power is
high, the power contained within the central maximum is low, which mitigates
self-focusing Miller  _et al._ (2023) (a similar property leads to larger
effective critical powers for higher-order Laguerre Gaussian beams and
partially incoherent beams Sá and Vieira (2019); Bang  _et al._ (1999)).
Second, the $f/\\#$ of the flying-focus configurations is much smaller, so
that each annulus of the pulse comes in and out of focus relatively quickly
and does not have sufficient propagation path at high intensity to accumulate
a significant time-dependent, nonlinear phase.

Figure 3: Evolution of the differential current densities $\partial_{t}J_{p}$
and current densities $J_{p}$ produced by the (a,b) phased flying focus, (c,d)
standard ultrashort flying focus, and (e,f) ideal lens configurations. The
phased flying focus drives a differential current with a near-constant
temporal profile that advances at the prescribed focal velocity $v_{f}=1.001$
over $\sim$four dephasing lengths. The near-constant temporal profile of the
differential current yields a current that does not oscillate with distance.
The differential current densities have been low-pass filtered at 30 THz, and
the current density $J_{p}$ is shown $500$ fs after the maximum of
$\partial_{t}J_{p}$. Note that in the moving frame $t-z/v_{g}$, the focus
follows the trajectory $\Delta z=v_{f}(1-v_{f}/v_{g})^{-1}(t-z/v_{g})$.

Figure 3 illustrates the effects of propagation and dephasing on the
photocurrent responsible for THz generation. The differential current density
$\partial_{t}J_{p}$ is the direct source of the THz field (Figs. 3a, 3c, and
3e), while the net current density $J_{p}$ behind the two-color pulse
indicates whether or not THz was generated (Figs. 3b, 3d, and 3f). The
differential current density depends on the drift velocity acquired by ionized
electrons at the time of their birth $v_{d}$ and the ionization rate:
$\partial_{t}J_{p}\sim ev_{d}\partial_{t}n_{e}$, where $n_{e}$ is the electron
density. The duration over which the two-color pulse ionizes the gas
determines the temporal envelope, or width $\tau$, of $\partial_{t}J_{p}$. The
two-color pulse produces a net current with a bandwidth $\sim$$1/\tau$
sufficient to drive THz radiation when the integral of $\partial_{t}J_{p}$
over the duration of the pulse (i.e., $J_{p}$) is nonzero.

The evolution of $\partial_{t}J_{p}$ and $J_{p}$, both in time and along the
propagation path $\Delta z$, determine the structure of the THz pulse. The
differential current driven by the phased flying focus has a near-constant
temporal profile that advances at the prescribed focal velocity $v_{f}=1.001c$
over the entire interaction length $L\sim 4$$L_{\pi}$ (Fig. 3a). The constant
amplitude of $\partial_{t}J_{p}$ along lines $\Delta z=v_{f}t+z_{0}$
demonstrates the absence of dephasing. The near-constant profile and absence
of dephasing yield a non-alternating current $J_{p}$ (Fig. 3b) that
facilitates the constructive interference of the THz radiation into a half-
cycle burst emitted at the Cherenkov angle [Eq. (2) and Fig. 1a].

The differential current driven by the standard flying focus evolves as it
advances at the focal velocity (Fig. 3c). In this case, the amplitude
oscillates along lines $\Delta z=v_{f}t+z_{0}$ at a spatial period
approximately equal to the nominal dephasing length $L_{\pi}$. This yields an
alternating current $J_{p}$ (Fig. 3d) that generates a multi-cycle THz pulse
characterized by two angles and significant angular dispersion [Eq. (3) and
Fig. 1b]. The differential current driven by the conventional pulse also
evolves across the interaction region (Fig. 3e). Here, the amplitude has a
complex temporal dependence along lines $\Delta z=v_{g}t+z_{0}$ due to
dephasing and nonlinear propagation of the two-color pulse. Changes in the
local velocity of the current density result in the emission of a multi-cycle
THz pulse into range of angles, each of which are broadened by angular
dispersion (Fig. 1c).

For both the phased and standard flying focus, the amplitude of the current
density $J_{p}$ grows with distance within the focal range (Figs. 3b and 3d).
This is primarily due to the spherical aberration of the axiparabola, which
causes the incidence angle to increase with distance within the focal range.
Rays incident on the plasma at a steeper angle penetrate further into the
plasma before defocusing. As a result, the pulse can reach a higher intensity,
ionize more electrons, and produce a larger photocurrent.

## Discussion

While the THz pulses resulting from the phased flying focus and ideal lens
configurations had comparable energy (Fig. 2), further optimization of the
phased flying focus is possible. The energy of the THz pulse produced by the
phased flying focus can be increased by either (i) extending the focal range
or (ii) using an axiparabola with a smaller $f/\\#$. Extending the focal range
requires increasing the energy of the two-color pulse to maintain a fixed peak
intensity and mitigating the loss of spatiotemporal overlap between the two
harmonics due to group velocity dispersion (GVD). For the parameters simulated
here, GVD causes the first and second harmonic to walk off from one another at
a rate of $\approx$70 fs/m or $\approx$8.4 fs over $L=12\;\mathrm{cm}$, which
is already comparable to the duration of the second harmonic
$\tau_{2}=21\;\mathrm{fs}$. This walkoff can be mitigated by increasing the
duration of the first harmonic. The increased duration would ensure that the
harmonics remain overlapped over a longer distance without affecting the rise
time of the ionization front responsible for THz generation, which is set by
the duration of the second harmonic.

The axiparabola used for the phased and standard flying focus had a full-
aperture $f/\\#=15$. Simulations (not shown) were also conducted for an
$f/\\#=20$ axiparabola with all other parameters held constant. Qualitatively,
the results were nearly identical: The phased flying focus produced a single-
cycle THz pulse with almost no angular dispersion, and the standard flying
focus produced a multi-cycle pulse with a near-field profile that exhibited
two diffuse rings. Quantitatively, the $f/\\#=15$ configuration resulted in
${\sim}2\times$ greater THz energy than the $f/\\#=20$ configuration. This is
due to two effects. First, as shown in Ref. Ambat  _et al._ (2023), smaller
$f/\\#$’s mitigate the elongation of the pulse duration and accompanying drop
in intensity caused by the chromatic aberration imparted by the echelon.
Second, a smaller $f/\\#$ increases the incidence angle of the two-color
pulse, which allows the pulse to penetrate further into the plasma and reach a
higher intensity. The shorter duration results in more broadband THz
generation, while the higher peak intensity increases the rate of ionization,
thereby enhancing the THz-generating current. As a result, use of an
$f/\\#<15$ is expected to further enhance the THz energy.

The weak diffractive lens or phaser can compensate the relative phase
evolution of the first and second harmonic over many dephasing lengths, but
there is a limit: if the applied phase becomes too large, it can modify the
focal velocity and lengthen the duration of the affected harmonic. In the case
of the second harmonic, the phaser extends the focal range by a distance
$L_{D}=(\Delta\lambda/\lambda_{2})f_{0}^{2}/f_{P}$ and imparts a nonideal
radial delay $\tau_{D}(r)=-r^{2}/2cf_{P}$, where $\Delta\lambda$ is the
bandwidth of the second harmonic and $f_{0}\ll f_{P}$ has been assumed. The
phaser will have a negligible effect on the velocity of the focus and temporal
duration $\tau_{2}$ when $L_{D}\ll L$ and $|\tau_{D}|\ll|\tau|$ [see Eqs. (4)
and (5)]. For the parameters in Table 1 where $L\sim 4$ dephasing lengths,
these conditions are readily satisfied: $L_{D}=2\times 10^{-3}\;\mathrm{cm}$
and $|\tau_{D}(R)|=3\;\mathrm{fs}$, while $L=12\;\mathrm{cm}$ and
$|\tau(R)|=5\;\mathrm{ps}$.

A novel optical configuration that enables simultaneous control over the
intensity and relative phase of a two-color laser pulse can eliminate
dephasing in two-color THz generation. Simulations that modelled this
configuration, the nonlinear propagation of the two-color laser pulse through
gas, and the generation and propagation of THz radiation demonstrated the
formation of a single-cycle THz pulse with a controlled emission angle and
almost no angular dispersion. This contrasts the multi-cycle, angularly
dispersed THz pulse that formed when the two-color pulse was focused by an
ideal lens. The novel configuration combines the axiparabola and echelon of an
ultrashort flying focus Palastro  _et al._ (2020); Ambat  _et al._ (2023);
Pigeon  _et al._ (2024) with a weak diffractive lens (or equivalent chromatic
assembly). The axiparabola and echelon provide control over the motion of the
far-field intensity, while the axiparabola and diffractive lens provide
control over the relative phase evolution. The near absence of angular
dispersion and controlled emission angle of the THz pulses produced by this
phased flying focus can facilitate collection and refocusing of the radiation
for subsequent applications. Future work will explore whether the intensity
and phase control afforded by this configuration can be applied to other
nonlinear optical processes that rely on the interaction of multiple
frequencies and whether coatings or a redesign of the echelon can be used to
impart an equivalent phase difference $\psi_{P}$ between the two harmonics.
This would eliminate the need for an additional optic and allow the first and
second harmonic to travel a common path.

## Methods

The optical configuration, pulse propagation, and two-color THz generation
were simulated as follows. The transverse electric field of the laser pulse
was initialized in the near field with two carrier frequencies—a first and
second harmonic, each with the same transverse profile. The analytical phases
of the optical configurations illustrated in Fig. 1 were then applied to the
initial electric fields of each harmonic in the frequency domain. With the
phases applied, the two-color laser pulses were propagated in vacuum from the
exit of the optical configuration to the beginning of the gas using the
frequency-domain Fresnel integral.Palastro  _et al._ (2018); Simpson  _et al._
(2024) The electric fields resulting from the Fresnel integral provided the
initial condition for the unidirectional pulse propagation equation
(UPPE).Kolesik  _et al._ (2002); Kolesik and Moloney (2004); Couairon  _et
al._ (2011) The UPPE was used to simulate the nonlinear propagation of the
two-color laser pulse through the gas and the generation and propagation of
the THz radiation.

For the phased-flying-focus configuration, the analytical phase applied in the
near field included terms for the phaser, echelon, and axiparabola:
$\psi=\psi_{P}+\psi_{E}+\psi_{AP}$. The contribution from the phaser
$\psi_{P}$ is given by Eq. (7) with $L_{\pi}=L_{P}$ for the second harmonic
and $\psi_{P}=0$ for the first harmonic. The contributions from the echelon
and axiparabola are given by

$\psi_{E}=-\frac{\omega}{c}\frac{\lambda_{1}}{2}\left(\mathrm{ceil}\left[\frac{c\tau(r)}{\lambda_{1}}\right]+\mathrm{floor}\left[\frac{c\tau(r)}{\lambda_{1}}\right]\right),$
(8)

where $\tau(r)$ is found in Eq. (5), and

$\psi_{AP}=-\frac{\omega}{c}\frac{R^{2}}{2L}\ln\left(1+\frac{Lr^{2}}{f_{0}R^{2}}\right).$
(9)

A single echelon can be and was used for both harmonics because their
wavelengths are integer multiples $\lambda_{1}=2\lambda_{2}$. The standard
flying-focus configuration used the same $\psi_{E}$ and $\psi_{AP}$, but did
not include the phaser element (i.e., $\psi_{P}=0$ for both harmonics). For
the ideal lens configuration, the analytical phase included a single term for
the lens: $\psi=\psi_{l}$, where

$\psi_{l}=-\frac{\omega}{c}\frac{r^{2}}{2f_{l}}$ (10)

and $f_{l}$ is the focal length.

The parameters describing the optical configurations and the initial field
profiles are provided in Table 1. For the ideal lens configuration, the
confocal parameter of the lens was set equal to the focal range of the flying
focus to ensure a comparable length of high-intensity propagation and THz
generation. The ideal lens $f/\\#=250$ was acheived using a
$f_{l}=5\;\mathrm{m}$ focal length and initial spot size
$w_{i}=1\;\mathrm{cm}$. The transverse profiles of the pulses were Gaussian
and flattop for the ideal lens and flying-focus configurations, respectively.
The laser parameters were chosen based on commercially available, high-
repetition-rate Ti:sapphire laser systems. Second harmonic generation in
crystals like beta barium borate results in a second harmonic pulse with a
duration that is $\sim$$\sqrt{2}/2$ shorter than the first harmonic. This was
accounted for when setting the initial duration of the second harmonic.

In the figures, $\Delta z=0$ corresponds to the nominal focal point of the
axiparabola $z=f_{0}$ for the flying focus pulses and $z=f_{l}-4.5L_{\pi}$ for
the conventional pulse. The shift of $4.5L_{\pi}$ optimized the THz generation
for the conventional pulse. The initial relative phase and phaser length
$L_{P}$ were tuned over multiple simulations to produce the maximum THz
energy. This occurred for $L_{P}=2.5\,\mathrm{cm}$, which matched the observed
dephasing length. However, setting the phaser length equal to the nominal
dephasing length determined solely by the dispersion of the background gas
(i.e., $L_{P}=L_{\pi}=2.9\,\mathrm{cm}$) was sufficient to significantly
mitigate dephasing and produce a near-half cycle THz pulse. In a series of
simulations with lower energy but otherwise equivalent two-color pulses (not
shown), the observed dephasing length approached the nominal dephasing length
$L_{\pi}$ as the pulse energy was decreased: the lower-energy pulses ionized
fewer electrons, which reduced the plasma contribution to the dispersion. In
an experiment, the gas pressure could be adjusted to match a specific phaser
design or to compensate other uncertainties in experimental conditions.

The Fresnel integral was used to propagate from the optical configuration to
the start of the interaction region because it can have a lower computational
cost than the UPPE when the $f/\\#$ is small. More specifically, the Fresnel
integral can take advantage of different radial grids in the near field
(optics plane) and far field (start of the interaction region) without a
significant increase in implementation complexity. The UPPE captures linear
dispersion to all orders and can simulate arbitrarily large frequency ranges,
making it ideal for modeling nonlinear propagation of electromagnetic waves
with diverse frequency content. As implemented here, the UPPE included source
terms for the third-order nonlinear polarization density due to bound
electrons, an effective current density that accounts for ionization energy
losses, and the current density of the photoionized electrons with damping
from electron-neutral collisions (see Ref. Simpson  _et al._ (2024)). The
linear dispersion was calculated using the Sellmeier equation found in Ref.
Peck and Fisher (1964) and extrapolated to THz frequencies.

The UPPE was solved in a cylindrically symmetric, 2D+$t$ geometry with a
second-order predictor-corrector method for the nonlinear source terms. The
temporal resolution and domain sizes were $\Delta t=70\,\mathrm{as}$ and
$T=2.4\,\mathrm{ps}$, corresponding to a spectral resolution of $\Delta
f=0.5\,\mathrm{THz}$. The minimum radial resolution was $\Delta
r=3\,\mu\mathrm{m}$ with a radial domain size
$r_{\mathrm{max}}=1\,\mathrm{cm}$. The axial step size was $\Delta
z=20\;\mu\mathrm{m}$. The resolution was adjusted to ensure convergence, and
the maximum bounds on $t$ and $r$ were chosen to mitigate aliasing and
spurious reflections, respectively.

## References

  * Midwinter and Warner (1965) J. Midwinter and J. Warner, British Journal of Applied Physics 16, 1135 (1965).
  * Maker  _et al._ (1962) P. Maker, R. Terhune, M. Nisenoff, and C. Savage, Physical review letters 8, 21 (1962).
  * Boyd (2019) R. W. Boyd,  _Nonlinear optics_ (Academic Press, 2019).
  * Fejer  _et al._ (1992) M. M. Fejer, G. Magel, D. H. Jundt, and R. L. Byer, IEEE Journal of quantum electronics 28, 2631 (1992).
  * Yamada  _et al._ (1993) M. Yamada, N. Nada, M. Saitoh, and K. Watanabe, Applied Physics Letters 62, 435 (1993).
  * Hebling  _et al._ (2002) J. Hebling, G. Almasi, I. Z. Kozma, and J. Kuhl, Optics express 10, 1161 (2002).
  * Wu  _et al._ (2023) X. Wu, D. Kong, S. Hao, Y. Zeng, X. Yu, B. Zhang, M. Dai, S. Liu, J. Wang, Z. Ren, _et al._ , Advanced Materials 35, 2208947 (2023).
  * Kondakci and Abouraddy (2017) H. E. Kondakci and A. F. Abouraddy, Nature Photonics 11, 733 (2017).
  * Sainte-Marie  _et al._ (2017) A. Sainte-Marie, O. Gobert, and F. Quere, Optica 4, 1298 (2017).
  * Froula  _et al._ (2018) D. H. Froula, D. Turnbull, A. S. Davies, T. J. Kessler, D. Haberberger, J. P. Palastro, S.-W. Bahk, I. A. Begishev, R. Boni, S. Bucht, _et al._ , Nature Photonics 12, 262 (2018).
  * Kondakci and Abouraddy (2019) H. E. Kondakci and A. F. Abouraddy, Nature communications 10, 929 (2019).
  * Palastro  _et al._ (2020) J. P. Palastro, J. L. Shaw, P. Franke, D. Ramsey, T. T. Simpson, and D. H. Froula, Physical Review Letters 124, 134802 (2020).
  * Caizergues  _et al._ (2020) C. Caizergues, S. Smartsev, V. Malka, and C. Thaury, Nature Photonics 14, 475 (2020).
  * Li and Kawanaka (2020) Z. Li and J. Kawanaka, Communications Physics 3, 211 (2020).
  * Jolly  _et al._ (2020) S. W. Jolly, O. Gobert, A. Jeandet, and F. Quéré, Opt. Express 28, 4888 (2020).
  * Yessenov  _et al._ (2022) M. Yessenov, J. Free, Z. Chen, E. G. Johnson, M. P. J. Lavery, M. A. Alonso, and A. F. Abouraddy, Nature Communications 13, 4573 (2022).
  * Liang  _et al._ (2023) Z. Liang, Y. Liu, Y. Luo, H. Chen, and D. Deng, Optics Letters 48, 2543 (2023).
  * Ambat  _et al._ (2023) M. Ambat, J. Shaw, J. Pigeon, K. Miller, T. Simpson, D. Froula, and J. Palastro, Optics Express 31, 31354 (2023).
  * Pigeon  _et al._ (2024) J. Pigeon, P. Franke, M. L. P. Chong, J. Katz, R. Boni, C. Dorrer, J. Palastro, and D. Froula, Optics Express 32, 576 (2024).
  * Liberman  _et al._ (2024) A. Liberman, R. Lahaye, S. Smartsev, S. Tata, S. Benracassa, A. Golovanov, E. Levine, C. Thaury, and V. Malka, Optics Letters 49, 814 (2024).
  * Simpson  _et al._ (2020) T. T. Simpson, D. Ramsey, P. Franke, N. Vafaei-Najafabadi, D. Turnbull, D. H. Froula, and J. P. Palastro, Optics Express 28, 38516 (2020).
  * Simpson  _et al._ (2022) T. T. Simpson, D. Ramsey, P. Franke, K. Weichman, M. V. Ambat, D. Turnbull, D. H. Froula, and J. P. Palastro, Optics Express 30, 9878 (2022).
  * Li  _et al._ (2024) D. Li, K. Miller, J. Pierce, W. Mori, A. Thomas, and J. Palastro, Physical Review Research 6, 013272 (2024).
  * Ramsey  _et al._ (2020) D. Ramsey, P. Franke, T. T. Simpson, D. H. Froula, and J. P. Palastro, Physical Review E 102, 043207 (2020).
  * Palastro  _et al._ (2021) J. Palastro, B. Malaca, J. Vieira, D. Ramsey, T. Simpson, P. Franke, J. Shaw, and D. Froula, Physics of Plasmas 28 (2021).
  * Miller  _et al._ (2023) K. G. Miller, J. R. Pierce, M. V. Ambat, J. L. Shaw, K. Weichman, W. B. Mori, D. H. Froula, and J. P. Palastro, Scientific Reports 13, 21306 (2023).
  * Howard  _et al._ (2019) A. J. Howard, D. Turnbull, A. S. Davies, P. Franke, D. H. Froula, and J. P. Palastro, Physical Review Letters 123, 124801 (2019).
  * Franke  _et al._ (2021) P. Franke, D. Ramsey, T. T. Simpson, D. Turnbull, D. H. Froula, and J. P. Palastro, Physical Review A 104, 043520 (2021).
  * Ramsey  _et al._ (2022) D. Ramsey, B. Malaca, A. Di Piazza, M. Formanek, P. Franke, D. H. Froula, M. Pardal, T. T. Simpson, J. Vieira, K. Weichman, _et al._ , Physical Review E 105, 065201 (2022).
  * Kabacinski  _et al._ (2023) A. Kabacinski, E. Oliva, F. Tissandier, J. Gautier, M. Kozlová, J.-P. Goddet, I. A. Andriyash, C. Thaury, P. Zeitoun, and S. Sebban, Nature Photonics , 1 (2023).
  * Di Piazza (2021) A. Di Piazza, Physical Review A 103, 012215 (2021).
  * Formanek  _et al._ (2022) M. Formanek, D. Ramsey, J. P. Palastro, and A. Di Piazza, Physical Review A 105, L020203 (2022).
  * Formanek  _et al._ (2024) M. Formanek, J. P. Palastro, D. Ramsey, S. Weber, and A. Di Piazza, Physical Review D 109, 056009 (2024).
  * Kim  _et al._ (2007) K.-Y. Kim, J. H. Glownia, A. J. Taylor, and G. Rodriguez, Optics Express 15, 4577 (2007).
  * Kim  _et al._ (2008) K. Y. Kim, A. J. Taylor, J. H. Glownia, and G. Rodriguez, Nature Photonics 2, 605 (2008).
  * Babushkin  _et al._ (2010) I. Babushkin, W. Kuehn, C. Koehler, S. Skupin, L. Berge, K. Reimann, M. Woerner, J. Herrmann, and T. Elsaesser, Physical Review Letters 105, 053903 (2010).
  * You  _et al._ (2012) Y. S. You, T. I. Oh, and K. Y. Kim, Physical Review Letters 109, 183902 (2012).
  * Bergé  _et al._ (2013) L. Bergé, S. Skupin, C. Köhler, I. Babushkin, and J. Herrmann, Physical Review Letters 110, 073901 (2013).
  * Johnson  _et al._ (2013) L. A. Johnson, J. P. Palastro, T. M. Antonsen, and K. Y. Kim, Physical Review A 88, 063804 (2013).
  * Zhang  _et al._ (2018) Z. Zhang, Y. Chen, S. Cui, F. He, M. Chen, Z. Zhang, J. Yu, L. Chen, Z. Sheng, and J. Zhang, Nature Photonics 12, 554 (2018).
  * Yoo  _et al._ (2019) Y.-J. Yoo, D. Jang, and K. Y. Kim, Optics Express 27, 22663 (2019).
  * Koulouklidis  _et al._ (2020) A. D. Koulouklidis, C. Gollner, V. Shumakova, V. Y. Fedorov, A. Pugžlys, A. Baltuška, and S. Tzortzakis, Nature Communications 11, 292 (2020).
  * Buldt  _et al._ (2021) J. Buldt, H. Stark, M. Müller, C. Grebing, C. Jauregui, and J. Limpert, Optics Letters 46, 5256 (2021).
  * Simpson  _et al._ (2024) T. T. Simpson, J. J. Pigeon, M. V. Ambat, K. G. Miller, D. Ramsey, K. Weichman, D. H. Froula, and J. P. Palastro, Physical Review Research 6, 013041 (2024).
  * Kolesik  _et al._ (2002) M. Kolesik, J. Moloney, and M. Mlejnek, Physical Review Letters 89, 283902 (2002).
  * Kolesik and Moloney (2004) M. Kolesik and J. V. Moloney, Physical Review E 70, 036604 (2004).
  * Couairon  _et al._ (2011) A. Couairon, E. Brambilla, T. Corti, D. Majus, O. de J. Ramírez-Góngora, and M. Kolesik, The European Physical Journal Special Topics 199, 5 (2011).
  * (48) KMLabs, “Raea ultrafast amplifier,” https://www.kmlabs.com/raea-ultrafast-amplifier.
  * Zahedpour  _et al._ (2015) S. Zahedpour, J. Wahlstrand, and H. Milchberg, Optics letters 40, 5794 (2015).
  * Sochacki  _et al._ (1992) J. Sochacki, A. Kołodziejczyk, Z. Jaroszewicz, and S. Bara, Applied optics 31, 5326 (1992).
  * Smartsev  _et al._ (2019) S. Smartsev, C. Caizergues, K. Oubrerie, J. Gautier, J.-P. Goddet, A. Tafzi, K. T. Phuoc, V. Malka, and C. Thaury, Optics Letters 44, 3414 (2019).
  * Oubrerie  _et al._ (2022) K. Oubrerie, I. A. Andriyash, R. Lahaye, S. Smartsev, V. Malka, and C. Thaury, Journal of Optics 24, 045503 (2022).
  * Sá and Vieira (2019) L. Sá and J. Vieira, Phys. Rev. A 100, 013836 (2019).
  * Bang  _et al._ (1999) O. Bang, D. Edmundson, and W. Królikowski, Phys. Rev. Lett. 83, 5479 (1999).
  * Palastro  _et al._ (2018) J. P. Palastro, D. Turnbull, S. W. Bahk, R. K. Follett, J. L. Shaw, D. Haberberger, J. Bromage, and D. H. Froula, Physical Review A 97, 033835 (2018).
  * Peck and Fisher (1964) E. R. Peck and D. J. Fisher, JOSA 54, 1362 (1964).

## Acknowledgements

The authors would like to thank M. V. Ambat, A. Elliot, D. Gitlin, L. Mack, H.
Markland, and M. L. P. Chong for productive discussions.

This report was prepared as an account of work sponsored by an agency of the
U.S. Government. Neither the U.S. Government nor any agency thereof, nor any
of their employees, makes any warranty, express or implied, or assumes any
legal liability or responsibility for the accuracy, completeness, or
usefulness of any information, apparatus, product, or process disclosed, or
represents that its use would not infringe privately owned rights. Reference
herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply
its endorsement, recommendation, or favoring by the U.S. Government or any
agency thereof. The views and opinions of authors expressed herein do not
necessarily state or reflect those of the U.S. Government or any agency
thereof.

This material is based upon work supported by the Office of Fusion Energy
Sciences under Award Numbers DE-SC0021057, the Department of Energy National
Nuclear Security Administration under Award Number DE-NA0004144, the
University of Rochester, and the New York State Energy Research and
Development Authority.

## Author contributions statement

T.T.S., D.H.F., and J.P.P. conceived the idea of phase control for coherent
THz generation. T.T.S. performed and analyzed the simulations. T.T.S., K.G.M.,
D.R., and J.P.P. developed the theoretical and simulation techniques. J.J.P.
and D.H.F. provided experimental insights. T.T.S. and J.P.P. wrote the
manuscript. All authors reviewed and edited the manuscript.

## Additional information

The authors declare no competing interests.


# Enhancing Clinical Information Extraction with
Transferred Contextual Embeddings

Zimin Wan1, Chenchen Xu12, Hanna Suominen1234  1The Australian National
University (ANU) / Canberra, ACT, Australia  2Data61, Commonwealth Scientific
and Industrial Research Organization (CSIRO) / Canberra, ACT, Australia
3University of Canberra / Canberra, ACT, Australia  4University of Turku /
Turku, Finland<EMAIL_ADDRESS>
###### Abstract

The Bidirectional Encoder Representations from Transformers (BERT) model has
achieved the state-of-the-art performance for many natural language processing
(NLP) tasks. Yet, limited research has been contributed to studying its
effectiveness when the target domain is shifted from the pre-training corpora,
for example, for biomedical or clinical NLP applications. In this paper, we
applied it to a widely studied a hospital information extraction (IE) task and
analyzed its performance under the transfer learning setting. Our application
became the new state-of-the-art result by a clear margin, compared with a
range of existing IE models. Specifically, on this nursing handover data set,
the macro-average F1 score from our model was 0.438, whilst the previous best
deep learning models had 0.416. In conclusion, we showed that BERT based pre-
training models can be transferred to health-related documents under mild
conditions and with a proper fine-tuning process.

###### Index Terms:

Information extraction, Natural language processing, Medical data analysis,
Transfer learning

##  I Introduction

_Electronic Health_ (eHealth) solutions support patients and healthcare
workers in accessing, authoring, and sharing information. CLEF eHealth
evaluation lab has introduced tasks as part of the _Conference and Labs of the
Evaluation Forum_ to construct a system that converts verbal clinical handover
information into electronic structured handover forms automatically [1, 2].
This processing cascades voice recording, _speech recognition_ (SR),
_information extraction_ (IE), and information visualization with text editing
and document sign-off by nurses [3].

Figure 1: Filling out a handover form using information extraction. Revised
from [3].

In this study, we focus on this IE sub-task [2], which is to pre-fill a shift-
change nursing handover form by automatically extracting relevant text spans
for each slot in the form (Figure 1). Although many _machine learning_ (ML)
and _natural language processing_ (NLP) approaches have been applied to it,
the performance is not always satisfactory [3]. The main reasons for this gap
are the differing vocabularies and context between clinical sublanguages
(e.g., nursing vs. medical wards vs. medical-ward nursing) and generic
English, along with the ongoing problem of having a limited amount of
healthcare-specific data available for training and evaluation; data for real-
world healthcare scenarios for research and development are low-resourced [4,
5, 6]. Common methods in (deep) ML do not achieve a satisfying generalization
to unseen data on this clinical task and consequently, the current state-of-
the-art approach uses _transfer learning_ (TL) by adapting knowledge from
general via medical to clinical English to overcome data insufficiency [7].
However, this remedy relies on the selection of the source domain and requires
a large amount of time to pre-train the model on the source domain.

Consequently, our aim is to analyze the adaptability to a domain shift of the
timely _Bidirectional Encoder Representations from Transformers_ (BERT) method
in the nursing handover IE task. BERT has caused a stir in the ML-based NLP
community by obtaining new state-of-the-art results in various tasks [8]. BERT
applies the bidirectional training of the Transformer model from deep learning
to language modelling. By incorporating in-sentence word-to-word semantic
relations and the cross-sentence context knowledge, it addresses the gap in
linguistic understanding that limited data in typical biomedical and clinical
NLP data sets are unable to provide. The subsequent fine-tuning process then
provides extra domain specific knowledge in these health-related data.

Our main contributions are twofold:

  * •

Our application of BERT under a TL framework to the clinical IE task achieves
the new state-of-the-art result.

  * •

Conducted analysis to verify our framework’s capability to cope with a domain
shift from the pre-training corpora to the target task with scarce data
available for training and evaluation.

##  II Background

IE is a common task in NLP that aims to automatically extract structured
information from free-form or semi-structured text. Template filling is a
subtask in IE which is to fill out a structured form by extracting text spans.
It could be seen as an ML task called sequence tagging or labelling. Its major
difficulty is to build a good language model to understand language
representation. ML can learn features and representations from big data, but
rich, ambiguous human language is a difficult data modality to understand. A
word embedding is a distributed representation of words that models a given
language by taking the word context into account and embeds words into vectors
[9].

A _conditional random field_ (CRF) is a probabilistic sequence model, which
can learn the feature representation of the words, while taking the context of
words into account [10]. Linear chain CRFs are popular in NLP to solve a
sequence tagging tasks, and hence, their CRF++ implementation [11] is used as
the benchmark method in the handover IE task with both unigrams and bigrams
(which combine predicted label for the previous word and features of the
current word) and the following features [3, 2]: the previous, current, and
next location; pairwise correlations of the previous and current location and
the current and next location; and the combination of all features in the
current location.

TL is a (deep) ML method that can learn knowledge from solving one domain and
apply it to a different problem. It is becoming popular as a way to use pre-
trained models as a starting point towards more task-specific solutions in NLP
[12] and CV [13]. As in deep learning, many hidden layers exist between the
input and output layer; through the data feed-forward, the lower layers tend
to extract common and non-domain features, while the higher layers are likely
to concatenate learned features and generate higher-level concepts [14, 15].
Thus, the learned weights can be reused if the structure of the higher layers
are consistent.

Figure 2: Transformer architectures. Revised from [16].

In the handover IE task, TL has been applied to first train a linear chain CRF
on the source domain as a starting point, then, the learnt weights of the
source model have been used to train a 2-layer CRF to predict a target domain
label, and finally, the parameters of the target domain model have been
initialized by using the product of the second layer weights and the train
source model weights [7]. Besides, the baseline model of TL uses different
corpora as the source domain, and so far, the model with the best performance
has used a general domain source corpus, called _Bolt, Beranek and Newman_
(BBN) [17].

##  III Materials

This study used the public NICTA Synthetic Nursing Handover Data Set, which
was developed for clinical SR and IE related to nursing shift-change handover
from 2012 to 2016 [3]. The data set included three mutually exclusive subsets
and 301 synthetic patient records in total (training, validation, and test set
with 101, 100, and 100 records, respectively) . In this study, our goal was to
extract relevant text spans from the written free-form text documents in order
to generate the written structured documents. The structured documents
included annotations of five classes,111i.e., PATIENT INTRODUCTION, MY SHIFT,
APPOINTMENTS, MEDICATION, and FUTURE CARE and each class was further divided
into 38 subclasses and an additional category of _not applicable_ (N.A.) for
irrelevant information.

The IE task was challenged by the nursing notes not always being consistent
with formal general written English. For example, the data contained clinical
jargon and (ambiguous ad-hoc) shorthand (yrs for years and BP for Bachelor of
Pharmacy, bedpan, before present, birthplace, or blood pressure). Text spans
relevant to the same subclass were also sometimes scattered across the text
document. Finally, for some labels, as few as two or three instances were
present in the training set.222e.g., Clinician’s Last Name and Ward In fact,
not all documents included relevant information for all form categories and
some documents contained variants of the desired information multiple
times.333e.g., for a patient whose given name was Timothy, also his nickname
of Tim was supposed to be extracted to populate PATIENT INTRODUCTION: Given
names/initials

##  IV Processing Methods

Attention and Transformer mechanisms are of the significant breakthroughs in
language modelling, which have greatly promoted the development of NLP [16]
and CV [18]: The former was introduced to solve the bottleneck of information
pass caused by the conversion of long sequences to fixed-length vectors with
recurrent neural networks (RNNs). The latter uses self-attention mechanisms
and completely abandons the architecture of RNNs, which can model long term
dependencies among tokens in a temporal sequence more effectively (Figure 2).

As Transformer contains no recurrence and convolution layers, the position
information should be injected to enable the model to take advantage of the
sequence order. Thus, its input is the sum of input embeddings and the
positional encodings.

Transformer can model long term dependencies of a sequence and implement
parallel computing to increase training speed by using a self-attention
mechanism: In the encoder stack, each of the $N=6$ identical layers consists
of a multi-head self-attention mechanism and a position-wise fully connected
feed-forward network. Besides, each sub-layer uses the residual connection
followed by a normalization layer at its output. The decoder stack is similar
to the encoder stack, with $N=6$ identical layers. It inserts a multi-head
attention sub-layer over the output of the encoder. Similarly, each sub-layer
uses the residual connection followed by a normalization layer at its output.

The BERT model is designed to pre-train deep bidirectional language
representations from unlabeled text. It has become the new state-of-the-art in
at least eleven NLP tasks [8], and hence, in this study, we have used it as
the language representation model. Transformer is used in BERT to learn
contextual relations between words in a text sequence. Because BERT aims to
generate a language representation model, it only takes use of Transformer’s
encoder stack. BERT pre-trains deep bidirectional representations from
unlabeled text by jointly conditioning on both left and right context in all
layers.

The following two sizes of the pre-trained BERT model exist [8]:

  1. 1.

BERT${}_{\textrm{BASE}}$ with the number of transformer blocks $L=12$, hidden
layer size $H=768$, attention heads $A=12$, and number of total parameters
$P=110$ million and

  2. 2.

BERT${}_{\textrm{LARGE}}$ with $L=24$, $H=1,024$, $A=16$ and $P=340$ million.
In this study, we have used the BERT${}_{\textrm{BASE}}$ model.

Generally, BERT takes either one or two sentences as an input, uses WordPiece
embeddings [19] to convert the sentence into a token list, and expects special
tokens [CLS] and [SEP] to identify the sentences. WordPiece embeddings perform
textual preprocessing to convert the free-form text to a list of tokens in
order to balance the flexibility of characters and efficiency of words. One of
the main implementations of WordPiece is the _Byte Pair Encoding_ (BPE)
double-byte encoding. The general training process of BPE is first to divide
the word into characters, second, to count the number of occurrences of the
character pair within the range of the word, and third, to iteratively add the
most frequent character combinations to the vocabulary. Thus, BPE can split a
word further so that the vocabulary will be streamlined and the meaning of
each word will be clearer. For a given token, BERT’s input representation is
the sum of the corresponding token, segmentation, and position embeddings as
illustrated in Figure 3. The token embeddings are generated by WordPiece, the
segment embeddings are the sentence identifiers, and the position embeddings
are used to show the token position within the sequence.

Figure 3: BERT input representation Figure 4: BERT with a multi-class
classifier architecture.

The output representation is the word embeddings generated by the pre-trained
networks. By using BERT, they will be 768-dimensional vectors, which can be
then used as features in ML-based NLP models.

In IE and sequence labelling tasks, the model will receive a text sequence,
and label each token to one of the various classes. A sequence model can be
simply added on top of BERT by connecting BERT’s hidden-states output with a
token classifier (Figure 4). It is trainable by feeding the output
representation of each token into the classification layer to predict the
corresponding label. We add a multi-class classification layer followed by
sequential cross entropy loss as the optimization target. Fine-tuning all the
parameters end-to-end to model the IE downstream task with BERT is
straightforward. Most hyperparameters for the fine-tuning process are the same
as in pre-training; and only learning rate, batch size, and the number of
training epochs are chosen as according to the validation set [8].

##  V Evaluation Methods

We have evaluated models using the Precision, Recall, and F1, together with
confusion matrices. The macro-average is computed over the multiple classes to
emphasize the significance of performing well in all classes, including also
those with relatively rare occurrence frequency, due to the risk averse nature
of clinical judgement.

Because some categories in the training set do not appear in the test set and
vice versa,444e.g., City vs. Hospital or Clinician’s Title under APPOINTMENTS
we have calculated the performance of proposed model on the test set based on
the categories that occur in the training set. This follows the conventions of
the earlier studies [3, 2, 7].

TABLE I: Comparison to baseline models.
Because of not having access to the system outputs of the second best method [7], assessing statistical significance was not possible. Random and Majority refer to assigning class labels randomly and using the most common class of FUTURE: TaskToBeCompleted / ExpectedOutcome for every instance. Acronyms: _Macro-averaged_ (MAd) Method | MAd Precision | MAd Recall | MAd F1
---|---|---|---
BERT-based | 0.485 | 0.477 | 0.438
TL using BBN | 0.498 | 0.419 | 0.416
CRF | 0.435 | 0.233 | 0.246
Random | 0.018 | 0.028 | 0.019
Majority | 0.000 | 0.029 | 0.001
TABLE II: Top performing subclasses with F1 score $>$ 0.8. Acronyms: PatientIntroduction (PI) Subclasses | F1
---|---
PI_CurrentRoom | 1.0000
PI_Lastname | 0.9950
PI_GivenNames/Initials | 0.9900
PI_CurrentBed | 0.9645
PI_Ageinyears | 0.9444
MyShift_Input/Diet | 0.8618
MyShift_ActivitiesOfDailyLiving | 0.8604
N.A. | 0.8177
PI_UnderDr_Lastname | 0.8127

##  VI Results

TABLE III: The confusion matrix of BERT-based model on different categories. CATEGORY |  | Number of Words
---
in Test Set
| Number of
---
True Positive
| Number of
---
False Positive
| Number of
---
False Negative
A. PATIENT INTRODUCTION | 1, 611 | 1, 134 | 579 | 477
1\. Given Names/ Initials | 100 | 99 | 1 | 1
2\. Last Name | 101 | 100 | 0 | 1
3\. Age in Years | 281 | 280 | 32 | 1
4\. Gender | 178 | 62 | 0 | 116
5\. Current Room | 100 | 100 | 0 | 0
6\. Current Bed | 198 | 190 | 6 | 8
7\. Under Dr: Given Names/ Initials | 53 | 0 | 0 | 53
8\. Under Dr: Last Name | 128 | 128 | 59 | 0
9\. Admission Reason/ Diagnosis | 153 | 146 | 341 | 7
10\. Allergy | 38 | 18 | 1 | 20
11\. Chronic Condition | 73 | 4 | 14 | 69
12\. Disease/ Problem History | 206 | 5 | 110 | 201
13\. Care Plan | 2 | 2 | 15 | 0
B. MY SHIFT | 1, 033 | 717 | 606 | 316
14\. Status | 285 | 234 | 123 | 51
15\. Contraption | 43 | 7 | 172 | 36
16\. Input/ Diet | 107 | 106 | 33 | 1
17\. Output/ Diuresis/ Bowel Movement | 50 | 26 | 32 | 24
18\. Wounds/ Skin | 25 | 20 | 34 | 5
19\. Activities of Daily Living | 295 | 262 | 52 | 33
20\. Risk Management | 40 | 0 | 4 | 40
21\. Other Observation | 188 | 62 | 156 | 126
C. APPOINTMENTS | 472 | 143 | 227 | 329
22\. Status | 79 | 19 | 147 | 60
23\. Description | 313 | 91 | 44 | 222
24\. Clinician: Given Names/ Initials | 1 | 0 | 0 | 1
25\. Clinician: Last name | 1 | 0 | 1 | 1
26\. Date and Time: Day | 34 | 14 | 22 | 20
27\. Date and Time: Time | 39 | 17 | 12 | 22
28\. Date and Time: City | 0 | 0 | 1 | 0
29\. Date and Time: Ward | 5 | 2 | 0 | 3
D. MEDICATION | 493 | 138 | 95 | 355
30\. Medicine | 296 | 120 | 65 | 176
31\. Dosage | 52 | 8 | 18 | 44
32\. Status | 145 | 10 | 12 | 135
E. FUTURE CARE | 252 | 88 | 319 | 164
33\. Alert/ Warning/ Abnormal Result | 28 | 8 | 8 | 20
34\. Goal/ Task To Be Completed/ Expected Outcome | 103 | 19 | 279 | 84
35\. Discharge/ Transfer Plan | 121 | 61 | 32 | 60
F. N.A. | 2, 652 | 2, 090 | 370 | 562
36\. N.A. | 2, 652 | 2, 090 | 370 | 562
TABLE IV: Performance on extracted subclasses with F1 score $>$ 0.8. Acronyms:
PatientIntroduction (PI)

A. PATIENT INTRODUCTION | 0.662 | 0.7039 | 0.6823
---|---|---|---
1\. Given Names/Initials | 0.99 | 0.99 | 0.99
2\. Last Name | 1.0 | 0.9901 | 0.995
3\. Age in Years | 0.8974 | 0.9964 | 0.9444
4\. Gender | 1.0 | 0.3483 | 0.5167
5\. Current Room | 1.0 | 1.0 | 1.0
6\. Current Bed | 0.9694 | 0.9596 | 0.9645
7\. Under Dr: Given Names/Initials | 0.0 | 0.0 | 0.0
8\. Under Dr: Given Last Name | 0.6845 | 1.0 | 0.8127
9\. Admission Reason/ Diagnosis | 0.2998 | 0.9542 | 0.4562
10\. Allergy | 0.9474 | 0.4737 | 0.6316
11\. Chronic Condition | 0.2222 | 0.0548 | 0.0879
12\. Disease/ Problem History | 0.0435 | 0.0243 | 0.0312
13\. Care Plan | 0.1174 | 1.0 | 0.2105
B. MY SHIFT | 0.542 | 0.6941 | 0.6087
14\. Status | 0.6555 | 0.8211 | 0.729
15\. Contraption | 0.0391 | 0.1628 | 0.0631
16\. Input/ Diet | 0.7626 | 0.9907 | 0.8618
17\. Output/ Diuresis/ Bowel Movement | 0.4483 | 0.52 | 0.4815
18\. Wounds/ Skin | 0.3704 | 0.8 | 0.5063
19\. Activities of Daily Living | 0.8344 | 0.8881 | 0.8604
20\. Risk Management | 0.0 | 0.0 | 0.0
21\. Other Observation | 0.2844 | 0.3298 | 0.3054

C. APPOINTMENTS | 0.3865 | 0.303 | 0.3397
---|---|---|---
22\. Status | 0.1145 | 0.2405 | 0.1551
23\. Description | 0.6741 | 0.2907 | 0.4062
24\. Clinician: Given Names/ Initials | 0.0 | 0.0 | 0.0
25\. Clinician: Last Name | 0.0 | 0.0 | 0.0
26\. Day | 0.3889 | 0.4118 | 0.4
27\. Time | 0.5862 | 0.4359 | 0.5
28\. City | 0.0 | 0.0 | 0.0
29\. Ward | 1.0 | 0.4 | 0.5714
D. MEDICATION | 0.5923 | 0.2799 | 0.3802
30\. Medicine | 0.6486 | 0.4054 | 0.499
31\. Dosage | 0.3077 | 0.1538 | 0.2051
32\. Status | 0.4545 | 0.06897 | 0.1198
E. FUTURE CARE | 0.2162 | 0.3492 | 0.2671
33\. Alert/ Warning/ Abnormal Result | 0.6586 | 0.4054 | 0.499
34\. Goal/ Task To Be | 0.3077 | 0.1538 | 0.2051
Completed/ Expected Outcome |  |  |
35\. Discharge/ Transfer Plan | 0.4545 | 0.06897 | 0.1198
F. N.A. | 0.8496 | 0.7881 | 0.8177
36\. N.A. | 0.8496 | 0.7881 | 0.8177
Total | 0.4850 | 0.4770 | 0.4380

Overall, our fined-tuned model had the macro-average precision, recall, and F1
score of 48.5%, 47.7%, and 43.8%, respectively (Table I). It excelled in nine
subclasses out of the 36 sub-classes with F${}_{1}\geq 0.8$ (Table II). Its
performance was good for the main categories of PATIENT INTRODUCTION
(F${}_{1}=68.2\%$) and MY SHIFT (F${}_{1}=60.9\%$). Detailed performance
result per category is shown in Figure IV and our proposed model is able to
achieve promising result even in minor categories which only contain minimal
number of examples in the training set. Table III demonstrates the confusion
matrix result from our model.

When benchmarked against the previous works, our fine-tuned resulting model
outperformed the previous state-of-the-art TL method using BNN as a source
domain and CRF based method. It outperformed the second best method by 2.2%
for Macro $F_{1}$ score, with both promising precision and recall rate (Table
I).

##  VII Conclusion

The BERT-based model became the new state-of-the-art in the clinical handover
task. It performed well in IE of names, numbers, and some behaviour of the
patient with their own specialized, distinguishable vocabulary555e.g., diet
and daily activities but failed with complex descriptive content.666e.g., the
disease and problem history and chronic condition, which are rare (common) in
the generic English (clinical), were difficult to classify, because BERT’s
pre-training data. TL could partially overcome the training problem of CRF
with small data by learning features and reusing the weights pre-trained from
other domains (i.e., general English) to adapt for the current task [7].

Our fine-tuned model outperforming the prior state-of-the-art illustrates the
usefulness of TL of language representations for biomedical NLP applications.
The biomedical corpora embeds unique word distributions and different
semantics that are mostly not visible in the general corpora used to train
initial BERT model. We show that the domain shift can be properly mitigated by
a fine-tuning process, giving superior performance without any handcrafting.

However, we believe there are still spaces left to explore on the model
architecture to reduce the potential noises introduced by the pre-trained
language model. For example, a CRF could be applied at the top layer of BERT
instead of a fully-connected layer. This would make it possible to plug in
other available clinical features, while in this study, we only used text
data. In the meantime, we also propose to validate the capability of TL such
models to more diverse applications in bioinformatics, medicine, health
sciences, healthcare, and other health-related fields.

## References

  * [1] H. Suominen, L. Hanlen, L. Goeuriot, L. Kelly, and G. J. Jones, “Task 1a of the clef ehealth evaluation lab 2015: Clinical speech recognition,” in _CLEF 2015 Working notes._ , 2015.
  * [2] H. Suominen, L. Zhou, L. Goeuriot, and L. Kelly, “Task 1 of the CLEF eHealth Evaluation Lab 2016: Handover information extraction.” in _Working Notes of Conference and Labs of the Evaluation Forum (CLEF)_ , ser. CLEF2016 Working Notes, K. Balog, L. Cappellato, N. Ferro, and C. Macdonald, Eds., vol. 1609. Online: CEUR Workshop Proceedings, 2016, pp. 1–14.
  * [3] H. Suominen, L. Zhou, L. Hanlen, and G. Ferraro, “Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations,” _JMIR medical informatics_ , vol. 3, no. 2, p. e19, 2015.
  * [4] S. Velupillai, H. Suominen, M. Liakata, A. Roberts, A. D. Shah, K. Morley, D. Osborn, J. Hayes, R. Stewart, J. Downs, W. Chapman, and R. Dutta, “Using clinical natural language processing for health outcomes research: Overview and actionable suggestions for future advances,” _Journal of Biomedical Informatics_ , vol. 88, pp. 11–19, 2018.
  * [5] D. Li, C. Rodriguez, X. Yu, and H. Li, “Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 1459–1469.
  * [6] B. Zoph, D. Yuret, J. May, and K. Knight, “Transfer learning for low-resource neural machine translation,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2016, pp. 1568–1575.
  * [7] L. Zhou, H. Suominen, and T. Gedeon, “Adapting state-of-the-art deep language models to clinical information extraction systems: Potentials, challenges, and solutions,” _JMIR Medical Informatics_ , vol. 7, no. 2, p. e11499, 2019\.
  * [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 4171–4186.
  * [9] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in Neural Information Processing Systems_ , 2013, pp. 3111–3119.
  * [10] J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in _Proceedings of the Eighteenth International Conference on Machine Learning_ , ser. ICML ’01. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2001, pp. 282–289.
  * [11] T. Kudo, “CRF++: Yet another CRF toolkit (2005),” _Available under LGPL from the following URL: http://crfpp. sourceforge.net_ , 2015.
  * [12] S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf, “Transfer learning in natural language processing,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 15–18. [Online]. Available: https://www.aclweb.org/anthology/N19-5004
  * [13] D. Li, X. Yu, C. Xu, L. Petersson, and H. Li, “Transferring cross-domain knowledge for video sign language recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020\.
  * [14] J. Schmidhuber, “Deep learning in neural networks: An overview,” _Neural Networks_ , vol. 61, pp. 85–117, 2015.
  * [15] D. Li, C. Xu, X. Yu, K. Zhang, B. Swift, H. Suominen, and H. Li, “Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation,” in _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [17] R. Weischedel and A. Brunstein, “BBN pronoun coreference and entity type corpus,” _Linguistic Data Consortium, Philadelphia_ , vol. 112, 2005. [Online]. Available: https://catalog.ldc.upenn.edu/LDC2005T33
  * [18] D. Li, C. Xu, K. Zhang, X. Yu, Y. Zhong, W. Ren, H. Suominen, and H. Li, “Arvo: Learning all-range volumetric correspondence for video deblurring,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 7721–7731.
  * [19] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey _et al._ , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _arXiv preprint arXiv:1609.08144_ , 2016.