# Above-threshold ionization by polarization-crafted pulses

Camilo Granados<EMAIL_ADDRESS>Department of Physics, Guangdong
Technion - Israel Institute of Technology, 241 Daxue Road, Shantou, Guangdong,
China, 515063  Technion – Israel Institute of Technology, Haifa, 32000, Israel
Guangdong Provincial Key Laboratory of Materials and Technologies for Energy
Conversion, Guangdong Technion - Israel Institute of Technology, 241 Daxue
Road, Shantou, Guangdong, China, 515063  Enrique G. Neyra  Instituto Balseiro
(Universidad Nacional de Cuyo and Comisión Nacional de Energía Atómica) and
CONICET CCT Patagonia Norte. Av. Bustillo 9500, Bariloche 8400 (RN),
Argentina.  Lorena Rebón  Instituto de Física de La Plata, CONICET - CCT La
Plata, Diag. 119 e/ 63 y 64, La Plata 1900, Buenos Aires, Argentina
Departamento de Física, Facultad de Ciencias Exactas, Universidad Nacional de
La Plata, La Plata, Buenos Aires, Argentina  Marcelo F. Ciappina
<EMAIL_ADDRESS>Department of Physics, Guangdong Technion -
Israel Institute of Technology, 241 Daxue Road, Shantou, Guangdong, China,
515063  Technion – Israel Institute of Technology, Haifa, 32000, Israel
Guangdong Provincial Key Laboratory of Materials and Technologies for Energy
Conversion, Guangdong Technion - Israel Institute of Technology, 241 Daxue
Road, Shantou, Guangdong, China, 515063

###### Abstract

Coherent light has revolutionized scientific research, spanning biology,
chemistry, and physics. To delve into ultrafast phenomena, the development of
high-energy, high-tunable light sources is instrumental. Here, the photo-
electric effect is a pivotal tool for dissecting electron correlations and
system structures. Particularly, above-threshold ionization (ATI),
characterized by simultaneous multi-photon absorption, has been widely
explored, both theoretical and experimentally. ATI decouples laser field
effects from the structural information carried by photo-electrons,
particularly when utilizing ultra-short pulses. In this contribution we study
ATI driven by polarization-crafted (PC) pulses, which offer precise control
over the electron emission directions, through an accurate change of the
polarization state. PC pulses enable the manipulation of electron
trajectories, opening up new avenues for understanding and harnessing coherent
light. Our work explores how structured light could allow a high degree of
control of the emitted photo-electrons.

##  I Introduction

Coherent light has guided, for the last decades, countless advances in
different areas of sciences, ranging from biology to chemistry and physics [1,
2, 3, 4, 5]. It is now customary, in many optics laboratories, to explore the
frontiers of natural processes at temporal scales close to the electron motion
in atoms, molecules and larger structures [3]. This landscape of research is
only possible thanks to the development of ultrashort laser pulses with a
duration of the order of attoseconds [6, 7, 8]. Advances in this technology
towards the creation of table-top setups of higher energy attosecond light
sources, which would allow us to investigate nature on unexplored energy and
time scales, is one of the cornerstones of attosecond science.

One of the typical uses of coherent light is the photo-electric effect. Here,
both single or multi-photon absorption are key to understand, for example,
electron correlations or structural properties of the systems under
investigation. The ionization of atoms by the simultaneous absorption of
several photons arises as one of the fundamental non-linear effects, together
with high-harmonic generation (HHG) [9, 10, 11], where a complete new chapter
of the light-matter interaction was necessary to fully describe the seminal
experimental observations [9, 12, 10, 13]. The simultaneous absorption of
several photons, known as above-threshold-ionization (ATI), has been
investigated during several decades both theoretically and experimentally [12,
14]. Understanding photo-absorption processes allows us to decouple the
influence of the strong laser field in photoelectron dynamics from the
structural information that photoelectrons could carry, especially when very
short pulses are used [15]. In this sense, ATI is a well established
phenomenon to study structural properties of the target systems, either atomic
or molecular, since it inherits information on the bound-to-free and re-
scattering continuum-continuum electronic transitions. With the advent of
carrier-envelope-phase (CEP) stabilization mechanisms, the ATI played a
paramount role in demonstrating CEP effects in the observed photo-electron
spectra. This was proven experimentally by measuring correlation between
photo-electrons emitted in opposite directions [16]. Without the CEP
stabilization, the effect on the photo-electron spectra normally averages out
because the random phase of the electromagnetic field that each pulse carries.
In another interesting application, the diffraction pattern generated by re-
scattered electrons was used to infer changes in the molecular bond lengths of
oxygen and nitrogen molecules [17]. This self-imaging of molecules provides a
powerful technique by which the changes of the molecular bond lengths can be
followed in real time [17, 18]. In addition, with the experimental development
of laser light carrying angular momentum [19], it is now possible to decouple
the multi-scattering nature of ATI from the inter-pulses interference effects.
New pulse synthesis thus open the door to investigate, understand, and
manipulate coherent light with unexplored degrees of freedom [5, 20].

From the theoretical point of view, ATI and photo-electron angular
distributions can be calculated by solving the time-dependent Schrödinger
equation (TDSE) [21, 22, 23] or by semi-classical approaches [24]. The main
difference between both methods lies on the possibility, when using the
latter, of studying the different contributions giving rise to the final
photo-electron distributions. However, the solution of the TDSE for all
degrees of freedom involved in the problem is often difficult or even
impossible to find. By using the strong-field approximation (SFA), Suárez et.
al. [10], were able to characterise the ionization process in terms of direct
and re-scattered mechanisms. This is an step forward to fully decouple the
distinct contributions to the ionization process and opens a way to extract
structural properties of both atomic and molecular targets. More recently, in
Ref. [25] it was demonstrated that, by using amplitude-polarization (AP)
attosecond pulses, the emitted electron angular momentum could be controlled.
The possibility of tailoring the attosecond pulse polarization has the
potential to unlock the investigation of complex physical systems.

In this work, we investigate the ATI phenomenon driven by polarization-crafted
(PC) pulses, for which AP pulses are a special case. The use of PC pulses
allows us to control the electron emission direction and presents an approach
to control the degree of collimation in the photo-electron emission direction.
The complete understanding of the highly focused laser-generated electron
beams could be useful for extracting information for both the laser pulse
itself and the atomic or molecular target.

The manuscript is organized as follows: In section II we first define PC laser
fields and show how to use the time delay, phase and amplitude ratios for
crafting their polarization state. In section III we present atomic-ATI
calculations, obtained from the the solution of the three-dimensional
(3D)-TDSE, using two short laser fields with variable time delay and phase
between them. Here we show how the different electron emission trajectories
are defined by the PC laser field. In section IV we present the same results
as in the previous section, but here using long PC laser pulses and modifying
the laser field amplitudes’ ratio. Next, in section V, we show the angular
dependence for different PC laser fields. Finally, the conclusions are
included in section VI.

##  II Results

We first revise the concept of PC pulses and, then, we show how they can be
used to control the electron dynamics. For this purpose, we compute the photo-
electron spectra of an hydrogen atom, when interacting with a laser field
synthesized from two identical 800-nm ultrafast laser fields with variable
relative phase and time delay between them. Our study is divided in two
regimes, depending on the temporal duration of the laser pulses, i.e. we
analyse the use of short and long laser pulses. For calculating the photo-
electron ATI spectra we used the Q-prop 3D-TDSE solver [23].

###  II.1 Polarization-crafted pulses

A PC pulse results from the coherent superposition of two linearly polarized
orthogonal electromagnetic fields, i.e.,
$E(t)=E_{x}(t)\hat{x}+E_{y}(t)\hat{y}$, with no restriction on their relative
phase, time delay and relative amplitude. This is in contrast to the AP
pulses, for which the relative phase and amplitude of each component is the
same. Thus, each of the electric field components, $E_{x}(t)$ and $E_{y}(t)$,
can be written as follows:

$\displaystyle E_{x}(t)$ $\displaystyle=$ $\displaystyle
E_{0,x}\sin^{2}\left(\frac{\omega_{0}t}{2n_{c}}\right)\sin(\omega_{0}t+\phi_{x}),$
(1a) $\displaystyle E_{y}(t)$ $\displaystyle=$ $\displaystyle
E_{0,y}\sin^{2}\left(\frac{\omega_{0}(t-\delta
t)}{2n_{c}}\right)\sin(\omega_{0}(t-\delta t)+\phi_{y}),$ (1b)

where $E_{0,x}$ ($E_{0,y}$) is the laser field peak amplitude in the $x$
($y$)-direction, $\omega_{0}$ the laser field central frequency, $n_{c}$ the
number of total cycles and $\delta t$ and $\phi_{x}$ ($\phi_{y}$) are the time
delay and carrier envelope phase (CEP) of the $x$ ($y$) laser field component,
respectively. We also define the global phase $\phi=\phi_{y}-\phi_{x}$. This
quantity is instrumental for the control of the laser field shape, as we will
see in the next sections. Notice that by forcing $\phi_{x}=\phi_{y}$ and
$E_{0,x}=E_{0,y}$ one recovers the well-known AP pulses described, for
instance, in Refs. [5, 20]. Here, however, we explore all the possible ways to
control the polarization state of the driving laser field. In Ref. [5], the
pulse temporal width, used to investigate HHG with Gaussian-shaped AP pulses,
was set to $2\Delta t$ and $5\Delta t$ ($\Delta t$ is the full-width at half
maximum), corresponding to $\approx$ 6 and 15 total optical cycles, $n_{c}$.
We use a similar set of temporal parameters in the present contribution.

####  II.1.1 Effect of the laser electric field amplitudes’ ratio

By changing the ratio between the laser field amplitude components in Eqs.
(1a) and (1b), for a fixed time delay $\delta t$, it is possible to change the
angle of the total laser field with respect to the plane $E_{y}=0$, i.e. we
obtain a linear-like Lissajous curve. Furthermore, for certain values of
$\delta t$ (see next section), we get a circular-like Lissajous curve. Here,
the change in the ratio shrinks the total laser field in the direction of the
component with smaller amplitude. We show these general features in Figs. 1(a)
and 1(b),

Figure 1:  Polarization-crafted (PC) pulses for a laser pulse with $n_{c}=16$.
(a) Lissajous curves for $\delta t=T/2$ and ratios $E_{0,y}/E_{0,x}$ = 1, 1/2
and 1/4 in red, blue and black, respectively. (b) same as panel (a) but with
$\delta t=T$. (c) Lissajous curves for $E_{0,y}/E_{0,x}$ = 1 and 1/4 with
$\delta t=T/4$. In all cases, it is $\phi=0$.

where we depict the Lissajous curves for 3 different laser field amplitudes’
ratios $E_{0,y}/E_{0,x}=$ 1, 1/2 and 1/4 (red, blue and black curves in Figs.
1(a) and (b), respectively), and two different time delays $\delta t=T/2$ in
Fig. 1(a) ($T=2\pi/\omega_{0}$ is the optical period) and $\delta t=T$ in Fig.
1(b), keeping the global phase at the value $\phi=0$. Finally, in Fig. 1(c) we
show the resulting Lissajous figures for a time delay corresponding to $\delta
t=T/4$ and two different amplitudes’ ratios, $E_{0,y}/E_{0,x}=$ 1 and 1/4, in
red and black, respectively. Experimentally, changing the ratio between the
amplitudes of the laser field components corresponds to change the angle on
the initial incident laser field before entering the birefringent media used
to set their relative time delay [26, 5]. This initial incident angle
corresponds, in our current description, to the global angle $\theta$. For
$E_{0,y}/E_{0,x}=$ 1, $\theta$ takes the values $-\pi/4$ or $\pi/4$, for
$\delta t=T/2$ and $T$, respectively. When the ratio is $E_{0,y}/E_{0,x}=1/2$
($E_{0,y}/E_{0,x}=1/4$), $\theta$ takes the value $0.35\pi$ ($0.45\pi$). In
the next section we show how to analytically calculate $\theta$.

####  II.1.2 Effect of the time delay

By changing the time delay $\delta t$ in Eq. (1b), it is also possible to
modify the shape of the Lissajous curve formed by the two orthogonal laser
fields. For example, if the delay is set to $\delta t=nT/2$, with $n$ an
integer, the Lissajous curves have a distinct line-like shape. However, if
$\delta t=(2n-1)T/4$ the Lissajous curves change to a circular-like shape.
Thus, the latter value of $\delta t$ defines a circular polarized laser field
with time-dependent ellipticity. This time-dependent ellipticity steers the
electron trajectories in such a way that the electron re-scattering with the
parent ion is avoided. For a fixed laser electric field amplitudes’ ratio,
$E_{0,y}/E_{0,x}$ it is possible to obtain the evolution of the global angle,
$\theta$, as a function of the time delay, $\delta t$. Here $\theta$ is
considered as the polarization angle with respect to the plane $E_{x}=0$. To
calculate $\theta$ we use the expression $\tan(\theta)=A\sin(\omega(t-\delta
t))/\sin(\omega t)$, where we have defined $A=E_{0,y}/E_{0,x}$. Then,

$\theta=\arctan\Bigg(A\frac{\sin(\omega(t-\delta t))}{\sin(\omega
t)}\Bigg{missing})=\begin{cases}-\arctan\Big(A\cot(\omega
t)\Big{missing}),&\text{for }\delta t=T/4\\\ -A\pi/4,&\text{for }\delta
t=T/2\\\ \arctan\Big(A\cot(\omega t)\Big{missing}),&\text{for }\delta
t=3T/4\\\ A\pi/4,&\text{for }\delta t=T.\end{cases}$ (2)

The different cases of Eq. (2) are presented in Fig. 2. Here, we depict in
red, blue and black, the results of Eq. (2) with a time delay of $\delta
t=T/2$ and for $E_{0,y}/E_{0,x}=1,1/2$ and 1/4, respectively. The dashed red
and dashed black lines show the case for $E_{0,y}/E_{0,x}=1$ and $\delta
t=T/4$ and $\delta t=3T/4$, respectively. We can see that (i) $\tan(\theta)$
is constant during one full laser cycle for $\delta t=nT/2$, and (ii)
$\tan(\theta)$ varies with time for $\delta t=(n+1)T/4$.

Figure 2:  Temporal evolution of the global angle, $\theta$, calculated from
Eq. (2). The red, blue and black solid curves are the $\theta$ values for the
parameters used in Fig. 1(a). In the same plot, the dashed red and dashed
black curves are for the case of $E_{0,y}/E_{0,x}=1$ and $\delta t=T/4$ and
$\delta t=3T/4$, respectively.

From Eq. (2) we can define an AP laser field with constant polarization, e.g.
setting $\theta=\pi/4$. Likewise, Eq. (2) includes AP laser fields with
variable polarization as well, when $\theta=\arctan\big(\cot(\omega
t)\big{missing})$. Notice that AP pulses are only possible if
$\phi_{x}=\phi_{y}=0$ and $A=1$. From PC pulses, changes in time delays,
$\delta t$, and amplitude ratios, $E_{0,y}/E_{0,x}$, allow us to tailor the
global angle $\theta$ and polarization state of the AP laser fields. Thus, we
could have a more precise control of the electron trajectories.

Figure 3:  Electron trajectories for (a) $\delta t=T/2$ and (b) $\delta t=T$
(red curves in Fig. 1(a)) calculated by integrating the two-dimensional
classical equations of motion of an electron in the laser electric fields
given by Eqs. (1a) and (1b).

To show this possibility, we calculate the resulting electron trajectories by
solving the two-dimensional classical equations of motion for one free
electron in the laser electric fields defined by Eqs. (1a) and (1b). Here, we
set the time delays between the pulses as $\delta t=T/2$ and $\delta t=T$. The
results are presented in Figs. 3(a) and 3(b), respectively. Our calculations
show good agreement with the parametric plot of the fields presented in Figs.
1(a) and 1(b). It is important to mention that the classical trajectories show
a larger angular dispersion as the delay between the pulses increases. This
will have a clear impact on the ATI spectra, as we show in the next section.

####  II.1.3 Angular dispersion of the electron trajectories

The angle between successive laser field oscillations is useful to investigate
the angular dispersion of the electron trajectories. For simplicity we call
these oscillations “petals”. To calculate such an angle, we compute the ratio
between the laser electric field components envelopes. By expanding the ratio
and taking only the linear terms in $t$ and $\delta t$ we arrive to:

$\alpha=\arctan\Bigg(A\frac{\sin^{2}(\omega(t-\delta
t)/(2n_{c}))}{\sin^{2}(\omega
t/(2n_{c}))}\Bigg{missing})-A\frac{\pi}{4}\approx\arctan\Bigg(\frac{2}{3}A\frac{\omega^{2}t}{n_{c}^{2}}\delta
t\Bigg{missing})-A\frac{\pi}{4}+O(\delta t^{2}).$ (3)

For a laser field with $n_{c}=6$ cycles, $\omega=$ 800 nm, $\delta t=T/2$ and
$t=T/2$, the angle between petals is $\alpha\approx 10^{\text{o}}$. For the
same laser parameters, but with $n_{c}=15$ cycles, Eq. (3) predicts a smaller
angle, $\alpha\approx 1^{\text{o}}$. Thus, by increasing the number of cycles
we can control the collimation of the electron trajectories when PC laser
fields are used. Comparing Eq. (3) with the results presented in [5], we see a
smaller increase in the angle as the time delay increases. However, the
functionality of Eq. (3) is more complete, considering we have kept the ratio
between the orthogonal laser field amplitudes. This ratio plays a fundamental
role on the angular dispersion of the Lissajous curves. Overall, from Eq. (3),
we can conclude that the angle between successive petals increases with the
time delay, making the angular dispersion of the petals larger. Contrarily,
for a fixed time delay, by increasing the number of cycles, the electron
trajectories become more collimated around the global angle, $\theta$.
Likewise, we can use the ratio between the laser fields amplitudes to change
the angular dispersion of the electron trajectories and the angle $\alpha$, as
well as the global angle $\theta$ of the Lissajous curve, as mentioned
previously. This means that we can vary the amplitudes ratio to select the
electron emission angle, as already shown in Fig. 1(a). This is consistent
with the fact that, reducing the amplitude of one of the laser fields
components, the excursion of the electron in such a direction gets reduced as
well. It is important to remark that the calculation of the angular dispersion
dependence with the number of cycles and the time delay between the laser
electric field components is only valid for time delays that are multiple
integers of $T/2$. For time delays that are multiple integers of $T/4$, a
constant dispersion angle has not meaning since it changes with time, as shown
by Eq. (2).

####  II.1.4 Effect of the global phase

Figure 4:  Parametric plots for different values of the laser field global
phase, $\phi$. In red and blue we plot the field’s parametric plot with
$\phi=\pi/2$ and$\phi=0$ rad. The time delay for both cases is the same,
$\delta t=0$ and T/2. Notice that for the case of AP pulses, a global phase
change of $\pi$ will only change the position of the petals and not the shape
of the total laser field. This is shown by the black curve for which we used
$\phi=0$ but with each laser field phase given by $\phi=\pi/2$ and a time
delay between the pulses of $\delta t=T/2$.

Until now, we have demonstrated control over electron trajectories by tuning
the laser electric field components amplitudes ratio, $E_{0,y}/E_{0,x}$ and
its relative time delay, $\delta t$. In addition, by modifying the CEPs
$\phi_{x}$ and $\phi_{y}$ of each laser electric field components, we can
change the Lissajous curve from a circular-like to a line-like shape. Using
the global phase $\phi$ we have then, three different methods to control the
electron dynamics. The difference is, however, that by changing the Lissajous
curve from the time delay $\delta t$, the angular dispersion of the electron
trajectories changes. The same modification in the Lissajous curves is
obtained by varying the global phase $\phi$, with the difference that this
method leaves the dispersion angle unchanged.

We show in Fig. 4 the parametric plots of the laser field components for a
constant time delay $\delta t=T/2$, and two different global phases, namely
$\phi=0$ (red solid line) and $\phi=\pi/2$ (blue solid line). Here, the most
remarkable thing is that the Lissajous curve has a well defined shape for a
fixed time delay, $\delta t$ and a fixed global phase $\phi$ (solid black
line). Experimentally, this opens the possibility of characterizing the CEP:
for example, by measuring the photo-electrons produced by ultrashort PC laser
fields one could extract variations in the CEP from pulse to pulse if the time
delay between the pulses is fixed. This is because the time delay, $\delta t$,
together with the amplitude ratio of the laser filed, $E_{0,y}/E_{0,x}$, and
the global phase $\phi$ define a unique Lissajous curve, which consequently
gives rise to a unique photo-electron ATI spectra. This technique, however,
would be limited to the degree of time delay stabilization. More importantly,
the PC laser fields could provide a novel way to characterize the CEP for
pulses with many cycles, where the direct CEP measurement is difficult.

##  III ATI with short PC fields

In the previous sections, we demonstrated how the manipulation of the time
delay, $\delta t$, relative phase $\phi$, and field’s components amplitude
ratio $E_{0,y}/E_{0,x}$, allowed us to control the electron trajectories.
Additionally, we illustrated that specific time delays can result in a laser
pulse exhibiting time-dependent ellipticity. This occurs when $\delta
t=(2n-1)T/4$. Furthermore, the time delay between the laser electric field
components determines whether the electron undergoes multiple rescatterings
with the parent ion or not. For example, a delay of $\delta t=T/4$ leads to
the total absence of rescattering. Conversely, for $\delta t=T/2$, there are
multiple rescattering events as the electron’s trajectory intersects the ion
position repeatedly. It is then instrumental to distinguish between these
features to understand if the observed photo-electron spectra is formed by
multiple rescattering events and/or by intra-cycle interferences [23]. This
distinction is particularly important for molecular structure characterization
[17].

Figure 5:  Parametric plots of the laser field components and generated photo-
electron spectra for the case of short PC pulses. The relative time delay
between the laser fields components is $\delta t=T/4$ for (a) and (b), $\delta
t=T/2$ for (c) and (d) and $\delta t=T$ for (e) and (f). It is clear from (d)
and (f) that the angular dispersion of the emitted electrons increases with
the time delay.

In order to study how the different PC-pulse configurations tailor the photo-
electron spectra, we drive an hydrogen atom with short and long laser PC
pulses and compute the 2D photo-electron momentum distributions, i.e. the
probability to emit electrons parallel and perpendicular to the laser electric
field polarization, using the 3D-TDSE [23]. For the case of short PC pulses,
we set the number of cycles $n_{c}=6$ (total duration $\approx 16$ fs) and the
laser field central wavelength $\lambda=$ 800 nm ($\omega=0.057$ a.u.).
Furthermore, the laser field amplitudes of each component are set to
$E_{0,x}=0.0534$ a.u. and $E_{0,y}=0.0534$ a.u. and their CEPs as
$\phi_{x}=\phi_{y}=0$. The calculated ATI 2D photo-electron momentum
distributions and laser electric field components are presented in Fig. 5.
Figure 5(a) shows the parametric plot of the laser field components with a
time delay $\delta t=T/4$. The resulting ATI spectrum is depicted in Fig.
5(b). Here, the shape of the 2D photo-electron momentum distribution is
typical for atoms driven by circularly polarized fields [27]. In Figs. 5(c)
and (d) we show the corresponding parametric plot of the laser fields and the
resulting 2D photo-electron momentum distribution, respectively, for the case
of a time delay $\delta t=T/2$. Finally, Figs. 5(e) and (f) present the
parametric plot of the laser fields and the resulting 2D photo-electron
momentum distribution, respectively, for $\delta t=T$. Several characteristics
predicted by Eq. (3) are clear in the computed ATI spectra, namely 1) The
electron trajectories closely follow the polarization direction of the PC
pulses, 2) it is evident in all the plots (Figs. 5(c) to (f)) that, for the
specific time delay used and an amplitudes ratio $E_{0,x}/E_{0,y}=1$, the
global angle is $\pm\pi/4$ and 3) the electron trajectories angular dispersion
angle, $\alpha$, increases with the time delay between the laser electric
field components. This results in a larger angular dispersion in the 2D photo-
electron momentum distribution, as shown in Figs. 5(d) and (f).

##  IV ATI with long PC fields

We also investigate the 2D photo-electron momentum distributions for long PC
pulses. This case is of great interest since it can provide a way to
characterize the CEP for long laser pulses. Here, we use $n_{c}=$ 16 cycles
($\approx 43$ fs) and $\phi_{x}=\phi_{y}=0$. We show, in Fig. 6, the
parametric plots for the laser field with different laser field amplitude
ratios, namely (a) $E_{0,y}/E_{0,x}=1$, (c) $E_{0,y}/E_{0,x}=2$ and (e)
$E_{0,y}/E_{0,x}=4$. The resulting 2D photo-electron momentum distributions
are presented in panels (b), (d) and (f), respectively. In all these cases we
have set $\delta t=T/2$. Once again, as predicted by Eq. (3), we can see the
control of the photoelectrons global angle by changing the laser fields
amplitudes ratio. Note also the changes in the amplitudes in the differential
ionization probability for different global angles (see the color bars). As
mentioned before, experimentally, this scenario corresponds to change the
initial angle of the fundamental field.

Figure 6:  Parametric plots of the laser field components and generated 2D
photoelectron spectra for the case of long PC pulses. Here $n_{c}$ = 16 and
the time delay $\delta t=T/2$. (a) and (b) $E_{0,y}/E_{0,x}=1$, (c) and (d)
$E_{0,y}/E_{0,x}=2$ and (e) and (f) and $E_{0,y}/E_{0,x}=4$.

##  V Angular distribution

As shown in the previous section, the photo-emission spectra generated with PC
pulses (see Figs. 5 and 6) follows closely the laser electric field. It is,
however, important to characterize the angular distribution of the
differential ionization probability, in order to quantify up to what order we
can control the angle $\theta$ and, in the case of AP pulses, $\alpha$. We
compare this latter case with the resulting photo-electron spectra from a
laser field with variable polarization (corresponding to $\delta
t=(2n-1)T/4)$).

Figure 7:  Angular dependency of the ATI spectra. Angular dispersion (a)-(b)
and differential ionization probability (c)-(d) extracted from the ATI spectra
when a) $\delta t=T/2$ and b) $\delta t=T/4$. The lower plots show the
differential ionization probability for the angles $\phi=2.3$ (blue line) and
3 rad (red line). The ratio between the maximum values for the red and blue
curves in (a) is $\approx 5$ and in (b) is $\approx$ 22.

To understand the differences in the angular dispersion between the particular
cases of AP and polarization dependent pulses, in Fig. 7 we show contour plots
with the differential ionization probability as a function of the global
angle, $\theta$, and the photo-electron momentum, $k$. The ATI spectra were
computed using different values for time delay, namely (a) $\delta t=T/2$ and
(b) $\delta t=T/4$. The photo-electron distribution for $\delta t=T/2$ shows a
collimated distribution around the angles $3\pi/4$ and $7\pi/4$. Likewise, for
$\delta t=T/4$ the electron distribution maxima are larger around the same
angles, but they exhibit a collimation in the electron momentum as well (see
the “islands” around to $k\approx 0.6-0.8$ a. u.). In the same plots, it can
be seen that the differential ionization probability is larger for $\delta
t=T/2$ (see color bar scale in Figs. 7 (a) and (b)). As discussed in Ref.
[22], fields with variable polarization, as the one used to obtain Fig. 7(b),
drive the electron away from the parent ion, avoiding re-scattering. There
exists, however, cases where multiple re-scattering events are present (see
Fig. 7(a)).

In Figs. 7(c) and (d) we show, in blue, the differential ionization
probability as a function of the electron momentum for the same cases shown in
(a) and (b) and the angle, $\phi=2.3$ rad. The red curves show the
differential ionization probability for $\phi=$ 3 rad. While the differential
ionization probability in Fig. 7 (c) shows a large decrease when
$\phi\rightarrow 2\pi$, in Fig. 7(d) the change in the differential ionization
probability for the same global angle values is smaller. To quantify this, we
calculate the ratio between the differential ionization probability maximum
values for the two different angles. In the case of $\delta t=T/2$ the ratio
corresponds to $\approx$ 22 while for $\delta t=T/4$ the ratio is $\approx$
5\. This clearly proves that it is possible to control the photoelectron
angular distribution with different PC pulses.

Figure 8:  Photoelectrons angular dispersion. In panels (a) and (b) the case
for short pulses, while the case for long pulses it is shown in panels (c) and
(d). Here, we have used two different delays, i.e., $\delta t=T/2$ for (a) and
(b) and $\delta t=T$ for (d) and (d).

We now compare the change of the photoelectron angular distribution as a
function of the time delay and number of cycles. In Fig. 8 we show the
resulting photoelectron distributions. In (a) and (b) we use short PC pulses
and in (c) and (d) long PC pulses. In all the cases we set $\delta t=T/2$. As
described by Eq. (3), by increasing the number of cycles, we observe a smaller
angular dispersion (compare (a) and (c) and (b) and (d)). Also, let us notice
that the effect of the time delay on the angular dispersion is larger for
short pulses than for the long ones. This is also observed in the 2D
photoelectron momentum distributions shown in Figs. 5 and 6.

Figure 9:  Photo-electrons angular dispersion for $\delta t=(2n-1)$T/4. In a)
and b) short pulses and c) and d) long pulses. Here, we have used two
different delays, i.e., $\delta t=T/4$ (a) and (c) and $\delta t=3T/4$ (b) and
(d).

The plots in Fig. 8, contrast with the photo-electron angular distribution for
the laser fields with variable polarization, as seen in Fig. 9. While the
angular distribution is collimated around the global angle (given by the laser
field components’ amplitude ratio) for the AP pulses, when variable
polarization pulses are used, the photo-electron signal is distributed around
larger angles. This angle increases with the number of cycles and the time
delay. In Figs. 9(a) and (b) we plot the photo-electron angular distribution
for $\delta t=T/4$ and $3T/4$ and $n_{c}=6$ cycles, meanwhile in Figs. 9(c)
and (d) we set $\delta t=T/4$ and $3T/4$ and $n_{c}=15$ cycles. It is clear
from these figures that by increasing number of pulses both the angular and
momentum dispersion increase.

##  VI Conclusions

We demonstrated control over the electron emission trajectories by solving the
3D-TDSE for above-threshold ionization (ATI) with short and long polarization-
crafted (PC) laser fields. The uniqueness of these laser fields lies in the
possibility to finely control the laser field polarization using three
different parameters: the relative phase, the time delay, and the laser
field’s amplitude ratio between the orthogonal components. These various
control parameters are crucial for altering the global angle of the electron
trajectories and the angle between the oscillations of the total laser field.
Since the electron trajectories are well characterized by these parameters,
additional information is available in the resulting ATI spectra. By fixing
the time delay and the amplitude ratio, one can characterize the CEP of the
pulse independently, whether the laser fields are short or long. The latter
case is particularly significant, as retrieving long pulse CEP information is
exceptionally challenging. Similarly, by fixing any two parameters, one can
extract clear information about the remaining one.

The level of characterization and control achieved with the ATI spectra is
fundamental for fully understanding the structural information that ATI could
provide in more complex target experiments. Furthermore, while the
experimental realization of different polarization-crafted laser fields may be
challenging, it is within reach for today’s ultrafast laser laboratories

## Acknowledgements

The present work is supported by the National Key Research and Development
Program of China (Grant No. 2023YFA1407100) and Guangdong Province Science and
Technology Major Project (Future functional materials under extreme conditions
- 2021B0301030005). M. F. C. and C. G. acknowledge financial support from the
Guangdong Natural Science Foundation (General Program project No.
2023A1515010871). E. G. N. and L. R. acknowledge to Consejo Nacional de
Investigaciones Científicas y Técnicas (CONICET).

## References

  * Ikonnikov  _et al._ [2023] E. Ikonnikov, M. Paolino, J. C. Garcia-Alvarez, Y. Orozco-Gonzalez, C. Granados, A. Röder, J. Léonard, M. Olivucci, S. Haacke, O. Kornilov, and S. Gozem, Photoelectron spectroscopy of oppositely charged molecular switches in the aqueous phase: Theory and experiment, J. Phys. Chem. Lett 14, 6061 (2023).
  * Volarić  _et al._ [2021] J. Volarić, W. Szymanski, N. A. Simeth, and B. L. Feringa, Molecular photoswitches in aqueous environments, Chem. Soc. Rev. 50, 12377 (2021).
  * Krausz and Ivanov [2009] F. Krausz and M. Ivanov, Attosecond physics, Rev. Mod. Phys. 81, 163 (2009).
  * Zewail [1992] A. Zewail, Femtosecond chemistry, in  _Eighth International Conference on Ultrafast Phenomena_ (Optica Publishing Group, 1992) p. FD1.
  * Neyra  _et al._ [2023] E. Neyra, F. Videla, D. Biasetti, M. F. Ciappina, and L. Rebón, Twisting attosecond pulse trains by amplitude-polarization IR pulses, Phys. Rev. A 108, 043102 (2023).
  * Isinger  _et al._ [2017] M. Isinger, R. J. Squibb, D. Busto, S. Zhong, A. Harth, D. Kroon, S. Nandi, C. L. Arnold, M. Miranda, J. M. Dahlström, E. Lindroth, R. Feifel, M. Gisselbrecht, and A. L’Huillier, Photoionization in the time and frequency domain, Science 358, 893 (2017).
  * Paul  _et al._ [2001] P. M. Paul, E. S. Toma, P. Breger, G. Mullot, F. Augé, P. Balcou, H. G. Muller, and P. Agostini, Observation of a train of attosecond pulses from high harmonic generation, Science 292, 1689 (2001).
  * Hentschel  _et al._ [2001] M. Hentschel, R. Kienberger, C. Spielmann, G. A. Reider, N. Milosevic, T. Brabec, P. Corkum, M. D. U. Heinzmann, and F. Krausz, Attosecond metrology, Nature 414, 509–513 (2001).
  * Lewenstein  _et al._ [1995] M. Lewenstein, K. C. Kulander, K. J. Schafer, and P. H. Bucksbaum, Rings in above-threshold ionization: A quasiclassical analysis, Phys. Rev. A 51, 1495 (1995).
  * Suárez  _et al._ [2015] N. Suárez, A. Chacón, M. F. Ciappina, J. Biegert, and M. Lewenstein, Above-threshold ionization and photoelectron spectra in atomic systems driven by strong laser fields, Phys. Rev. A 92, 063421 (2015).
  * Lewenstein  _et al._ [1994] M. Lewenstein, P. Balcou, M. Y. Ivanov, A. L’Huillier, and P. B. Corkum, Theory of high-harmonic generation by low-frequency laser fields, Phys. Rev. A 49, 2117 (1994).
  * Kulander [1987] K. C. Kulander, Multiphoton ionization of hydrogen: A time-dependent theory, Phys. Rev. A 35, 445 (1987).
  * Hoerner  _et al._ [2021] P. Hoerner, W. Li, and H. B. Schlegel, Sequential double ionization of molecules by strong laser fields simulated with time-dependent configuration interaction, J. Chem. Phys. 155, 114103 (2021).
  * Muller  _et al._ [1986] H. G. Muller, H. B. van Linden van den Heuvell, and M. J. van der Wiel, Experiments on “Above-Threshold Ionization” of atomic hydrogen, Phys. Rev. A 34, 236 (1986).
  * Milošević [2022] D. B. Milošević, High-order above-threshold ionization by a few-cycle laser pulse in the presence of a terahertz pulse, Phys. Rev. A 105, 053111 (2022).
  * Paulus  _et al._ [2001] G. G. Paulus, F. Grasbon, H. Walther, P. Villoresi, M. Nisoli, S. Stagira, E. Priori, and S. D. Silvestri, Absolute-phase phenomena in photoionization with few-cycle laser pulses, Nature 414, 182–184 (2001).
  * Blaga  _et al._ [2012] C. I. Blaga, J. Xu, A. D. DiChiara, E. Sistrunk, K. Zhang, P. Agostini, T. A. Miller, L. F. DiMauro, and C. D. Lin, Imaging ultrafast molecular dynamics with laser-induced electron diffraction, Nature 483, 194 (2012).
  * Pullen  _et al._ [2015] M. Pullen, B. Wolter, A.-T. Le, M. Baudisch, M. Hemmer, A. Senftleben, C. D. Schröter, J. Ullrich, R. Moshammer, C. D. Lin, and J. Biegert, Imaging an aligned polyatomic molecule with laser-induced electron diffraction, Nat. Commun. 6, 7262 (2015).
  * Mirhosseini  _et al._ [2013] M. Mirhosseini, O. S. Magaña-Loaiza, C. Chen, B. Rodenburg, M. Malik, and R. W. Boyd, Rapid generation of light beams carrying orbital angular momentum, Opt. Exp. 21, 30196 (2013).
  * Fleischer  _et al._ [2017] A. Fleischer, E. Bordo, O. Kfir, P. Sidorenko, and O. Cohen, Polarization-fan high-order harmonics, J. Phys. B 10.1088/1361-6455/aa507c (2017).
  * Koval and Bauer [2006] P. Koval and D. Bauer, Qprop: A schrödinger-solver for intense laser–atom interaction, Comp. Phys. Commun. 174, 396 (2006).
  * Mosert and Bauer [2016] V. Mosert and D. Bauer, Photoelectron spectra with qprop and t-SURFF, Comp. Phys. Commun. 207, 452 (2016).
  * Tulsky and Bauer [2020] V. Tulsky and D. Bauer, Qprop with faster calculation of photoelectron spectra, Comp. Phys. Commun. 251, 107098 (2020).
  * Amini  _et al._ [2019] K. Amini, J. Biegert, F. Calegari, A. Chacón, M. F. Ciappina, A. Dauphin, D. K. Efimov, C. F. de Morisson Faria, K. Giergiel, P. Gniewek, _et al._ , Symphony on strong field approximation, Rep. Prog. Phys. 82, 116001 (2019).
  * Li and He [2021] W. Li and F. He, Generation and application of a polarization-skewed attosecond pulse train, Phys. Rev. A 104, 063114 (2021).
  * Karras  _et al._ [2015] G. Karras, M. Ndong, E. Hertz, D. Sugny, F. Billard, B. Lavorel, and O. Faucher, Polarization shaping for unidirectional rotational motion of molecules, Phys. Rev. Lett. 114, 103001 (2015).
  * Milošević  _et al._ [2006] D. B. Milošević, G. G. Paulus, D. Bauer, and W. Becker, Above-threshold ionization by few-cycle pulses, J. Phys. B: At. Mol. Opt. Phys. 39 (2006).


# Symmetric Dot-Product Attention for Efficient Training
of BERT Language Models

Martin Courtois, Malte Ostendorff, Leonhard Hennig, Georg Rehm
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI), Berlin,
Germany
Corresponding author<EMAIL_ADDRESS>

###### Abstract

Initially introduced as a machine translation model, the Transformer
architecture has now become the foundation for modern deep learning
architecture, with applications in a wide range of fields, from computer
vision to natural language processing. Nowadays, to tackle increasingly more
complex tasks, Transformer-based models are stretched to enormous sizes,
requiring increasingly larger training datasets, and unsustainable amount of
compute resources. The ubiquitous nature of the Transformer and its core
component, the attention mechanism, are thus prime targets for efficiency
research. In this work, we propose an alternative compatibility function for
the self-attention mechanism introduced by the Transformer architecture. This
compatibility function exploits an overlap in the learned representation of
the traditional scaled dot-product attention, leading to a symmetric with
pairwise coefficient dot-product attention. When applied to the pre-training
of BERT-like models, this new symmetric attention mechanism reaches a score of
$79.36$ on the GLUE benchmark against $78.74$ for the traditional
implementation, leads to a reduction of $6\%$ in the number of trainable
parameters, and reduces the number of training steps required before
convergence by half.

Symmetric Dot-Product Attention for Efficient Training
of BERT Language Models

Martin Courtois, Malte Ostendorff, Leonhard Hennig, Georg Rehm Deutsches
Forschungszentrum für Künstliche Intelligenz GmbH (DFKI), Berlin, Germany
Corresponding author<EMAIL_ADDRESS>

##  1 Introduction

Since its introduction in 2017, the Transformer architecture powered by its
scaled dot-product attention mechanism (Vaswani et al., 2017) has become the
core component of modern deep-learning architectures and has enabled
researchers to achieve breakthroughs in both natural language processing (NLP)
and computer vision tasks such as language modelling (Brown et al., 2020),
machine translation (Raffel et al., 2019), speech processing (Radford et al.,
2022), and image recognition (Dosovitskiy et al., 2020). One of the many
successes of the Transformer lies in its ability to operate and learn in an
unsupervised setting from unstructured textual data, as well as its ability to
handle complex and varied structures such as graphs, images, and sentences by
increasing the model’s number of layers. However, this trend has led to the
emergence of machine learning models so enormous that the gap between the
amount of compute resources available to many research groups and the amount
needed to stay competitive is increasing year after year (Togelius and
Yannakakis, 2023), and by training larger and larger models, brought deep-
learning’s energy consumption to unsustainable amounts (Thompson et al.,
2021).

Efficient Transformer implementations are a popular area of research with many
recent contributions on encoding and dense representation of tokens (Su et
al., 2021), hardware-optimized implementation of attention (Dao et al., 2022),
or implementations for long document processing (Beltagy et al., 2020). While
the attention mechanism itself has been studied extensively (Niu et al.,
2021), and several improvements to its computational complexity have been
achieved (Kitaev et al., 2020; Zhou et al., 2020), it is still primarily
computed via the dot-product between a _query_ and a _key_ (see Figure 1).
Vaswani et al. (2017) highlight the difficulty of determining a proper
compatibility function, and suggest that a more sophisticated compatibility
function than dot product may be beneficial.

In this work, we propose alternative compatibility functions for the attention
mechanism, i. e., the scaled dot-product attention mechanism. With this
approach, we aim to improve the training efficiency of Transformer-based
models and to reduce their resource consumption. We especially focus on the
_self-attention_ mechanism of BERT (Devlin et al., 2018), a Transformer-based
encoder model.

Our contributions can be summarized as follows:

  * •

We introduce an alternative formula to replace the scaled dot-product
attention (Section 2) that takes advantage of the underlying symmetric
structure of attention, in order to reduce the number of parameters and
improve the computational efficiency of the model.

  * •

We benchmark our approach by training several BERT models on three attention
mechanism setups as well as two different model sizes (Section 3).

We demonstrate that our new attention formula reduces the number of parameters
of the model by $6\%$, and achieves a reduction of the number of training
steps required for model convergence by $50\%$ without sacrificing accuracy
(Section 4). Finally, we discuss the effects of our proposed compatibility
function on training efficiency, and situate our approach within the context
of research on efficient Transformer-based models (Section 5).

##  2 Improving the Attention Mechanism

Modern Transformer-based models are neural networks that rely on the scaled
dot-product attention mechanism introduced by Vaswani et al. (2017). We
propose two variations of this mechanism: a symmetric dot-product and a
symmetric with pairwise factors dot-product, that lead to a reduction in the
number of parameters of the self-attention layer.

###  2.1 Scaled Dot-Product Attention

Figure 1: Scaled Dot-Product Attention (Vaswani et al., 2017)

The scaled dot-product attention given by the following equation (Equation 1)
is an operator on three input matrices, queries $Q$, keys $K$ and values $V$.
We focus on the dot product $QK^{T}$ between queries $Q$ and keys $K$, which
is responsible for measuring the compatibility between tokens. The
compatibility $\mathbf{A}$ is an operator on two input tokens
($x,y\in\mathbb{R}^{h}$), that computes the dot-product of the projections of
$x$ and $y$ respectively through the operators $\mathbf{Q}$ and $\mathbf{K}$:

$Attn(Q,K,V)=\operatorname{softmax}\left(\frac{\mathbf{QK^{T}}}{\sqrt{d}}\right)\mathbf{V}$
(1)

Given two linear operators $\mathbf{Q}:\mathbb{R}^{h}\to\mathbb{R}^{d}$ and
$\mathbf{K}:\mathbb{R}^{h}\to\mathbb{R}^{d}$, we define a compatibility
operator $\mathbf{A}:\mathbb{R}^{h}\times\mathbb{R}^{h}\to\mathbb{R}$, such
that:

$\mathbf{A}(x,y)=\mathbf{Q}(x)\cdot\mathbf{K}(y)^{T}$ (2)

We challenge the necessity of using two different operators to compute the
affinity of the self-attention encoder layer of the Transformer block. Since
both $\mathbf{Q}$ and $\mathbf{K}$ are operators on the same token space, it
is reasonable to assume that the representations they learn share some
features. In that case, since the original expression (Equation 1) does not
enforce any feature sharing, it may possess redundant parameters that will
need to be learned twice.

We attempt to make this feature sharing property explicit in the compatibility
operator expression, in order to remove redundant parameters, reduce overall
model size, and improve convergence rate.

###  2.2 Symmetric Dot-Product attention

A simple way to make the feature sharing property explicit, is to enforce the
following relation $\mathbf{Q}=\mathbf{K}$ between the two operators. This
ensures that $\mathbf{Q}$ and $\mathbf{K}$ share features and results in the
symmetric compatibility operator:

$\mathbf{A}_{sym}(x,y)=\mathbf{Q}(x)\cdot\mathbf{Q}(y)^{T}$ (3)

###  2.3 Pairwise Dot-Product Attention

One aspect that needs to be considered is the amount of features shared
between the two operators. Complete overlap in terms of features may be
detrimental to the overall performance of the attention mechanism, e. g., it
could prevent the model to learn asymmetric relationships. Thus, we suggest
the following compatibility operator where the amount of feature sharing is
learned during training. To achieve this, we start with an operator
$\mathbf{L}$ that will be shared, and we define operators $\mathbf{Q}$ and
$\mathbf{K}$ as a composition of $\mathbf{L}$ with a base change, resulting in
the following compatibility operator (Equation 5):

Given a linear operator $\mathbf{L}:\mathbb{R}^{h}\to\mathbb{R}^{d}$ and two
square matrices $W_{q},W_{k}\in\mathbb{R}^{d\times d}$, we define two linear
operators $\mathbf{Q}:\mathbb{R}^{h}\to\mathbb{R}^{d}$ and
$\mathbf{K}:\mathbb{R}^{h}\to\mathbb{R}^{d}$, such that:

$\begin{split}\mathbf{Q}(x)&=\mathbf{L}(x)\cdot W_{q}\\\
\mathbf{K}(x)&=\mathbf{L}(x)\cdot W_{k}\\\ \end{split}$ (4)

Let $S\in\mathbb{R}^{d\times d}$ be the product $S=W_{q}\cdot W_{k}^{T}$, we
define a compatibility operator
$\mathbf{A}:\mathbb{R}^{h}\times\mathbb{R}^{h}\to\mathbb{R}$, such that:

$\begin{split}\mathbf{A}(x,y)&=\mathbf{Q}(x)\cdot\mathbf{K}(y)^{T}\\\
\mathbf{A}(x,y)&=\mathbf{L}(x)\cdot W_{q}\cdot
W_{k}^{T}\cdot{\mathbf{L}(y)}^{T}\\\ \mathbf{A}(x,y)&=\mathbf{L}(x)\cdot
S\cdot{\mathbf{L}(y)}^{T}\\\ \end{split}$ (5)

This operator can be interpreted as a weighted dot-product whose weights are
stored in $S$, a matrix of pairwise factors. To make the expression consistent
with the previously established expressions (Equation 2 and Equation 3), we
relabel the $\mathbf{L}$ operator with the letter $\mathbf{Q}$, resulting in
the following pairwise compatibility operator (Equation 6):

$\mathbf{A}_{pair}(x,y)=\mathbf{Q}(x)\cdot S\cdot{\mathbf{Q}(y)}^{T}$ (6)

###  2.4 Parameter Count

Function | Expression | Parameters
---|---|---
original | $\mathbf{Q}(x)\mathbf{K}(y)^{T}$ | $\mathcal{O}(3h^{2})$
symmetric | $\mathbf{Q}(x)\mathbf{Q}(y)^{T}$ | $\mathcal{O}(2h^{2})$
pairwise | $\mathbf{Q}(x)S\mathbf{Q}(y)^{T}$ | $\mathcal{O}(2h^{2}+h^{2}/n)$
Table 1: Parameter count of the attention layer per compatibility function.

For a Transformer block of $n$ heads, with input size $h$ and attention size
$d$, we give the parameter count formula for a complete block (with parameters
from $Q$, $K$ and $V$). We note that most Transformer implementations impose
$d=h/n$.

As shown in Table 1, the symmetric compatibility operator uses two thirds of
the original number of parameters. For the pairwise compatibility operator,
the parameter count also depends on the number of attention heads, it
converges towards $2/3$ of the original number of parameters as the number of
attention heads increases.

In this section, we introduced two alternative compatibility functions for the
attention mechanism, a symmetric dot-product operator and a symmetric with
pairwise factors dot-product operator. In the following sections we will refer
to them respectively as the _symmetric operator_ and the _pairwise operator_ ,
we will refer to the traditional scaled dot-product operator as the _original
operator_.

##  3 Experiments

Config | Operator | Parameters
---|---|---
$BERT_{small}$ | original | 28,795,194
symmetric | 27,744,570 (3.65%)
pairwise | 27,875,642 (3.19%)
$BERT_{base}$ | original | 109,514,298
symmetric | 102,427,194 (6.47%)
pairwise | 103,017,018 (5.93%)
Table 2: Parameter count per model configuration and compatibility function
(relative amount of parameters saved compared to the original).
$Bert_{small}$: nlayers: 4, nheads: 8, hidden size: 512, intermediate size:
2048. $Bert_{base}$: nlayers: 12, nheads: 12, hidden size: 768, intermediate
size: 3072.

To evaluate the symmetric and pairwise operators against the original
operator, we train and evaluate several Transformer-based encoder models, each
using a different compatibility operator as part of the self-attention
mechanism. The models are trained under the same conditions. First, we pre-
train the models, because we want to measure the evaluation loss during
training to see if our modifications have an impact on the training efficiency
and the accuracy of the model. Then, we evaluate each model on the GLUE
benchmark (Wang et al., 2019b) to evaluate the model’s accuracy on relevant
downstream tasks, such as, sentence acceptability (Warstadt et al., 2018),
sentiment analysis (Socher et al., 2013), sentence similarity (Cer et al.,
2017), and natural language inference (Williams et al., 2018; Rajpurkar et
al., 2016). Finally, we select model checkpoints during training and evaluate
those checkpoints on GLUE to measure the models’ accuracy on downstream tasks
during training.

###  3.1 Pre-Training Dataset

To pre-train our models, we select a subset of 30 million English documents
from the OSCAR corpus (Abadji et al., 2022; Jansen et al., 2022) by applying
content quality filters (see Appendix A). Using OSCAR data instead of the
BookCorpus (Zhu et al., 2015) and Wikipedia dumps is recommended for training
BERT models (Geiping and Goldstein, 2023) and ensures that the amount of
documents is large enough for single epoch training.

This training dataset is tokenized using the pre-trained _bert-base-uncased_
tokenizer Devlin et al. (2018) and sentences are aggregated into groups of 512
tokens. After tokenization, the resulting dataset contains 137 million
training samples, 70 billion tokens and 10,000 test samples.

###  3.2 Model Architectures

We prepare three variations of the BERT model (Devlin et al., 2018) using the
original, the symmetric and the pairwise operators. We also train on two model
sizes, _bert-small_ and _bert-base_.

As shown in Table 2, the symmetric and pairwise operators lead to significant
reduction in the number of parameters, $3.65\%$ and $3.19\%$ for the _bert-
small_ model, $6.47\%$ and $5.93\%$ for the _bert-base_ model.

In the following sections, we refer to a _bert-base_ model as $BERT_{base}$
when it uses the original operator, $BERT_{base,sym}$ or $BERT_{base,pair}$
when it uses the symmetric or pairwise operator respectively.

###  3.3 Pre-Training Setup

We follow the pre-training setup described by Devlin et al. (2018). The models
are trained on a pure masked language modeling task with masking probability
of 0.15 and batch size of 256 samples per training steps. Models are trained
on 200,000 steps with a linear learning rate of $10^{-4}$ and learning rate
warm-up during the first 10,000 steps. For the optimizer, we use Adam (Kingma
and Ba, 2014) with weight decay, $\beta_{1}=0.9$, $\beta_{2}=0.999$,
$\epsilon=10^{-12}$, resulting in models pre-trained on 26 billion tokens. We
measure evaluation cross-entropy loss during training to assess the training
efficiency of our models.

###  3.4 Benchmark Fine-Tuning Setup

After pre-training, the models are fine-tuned and benchmarked on the GLUE
dataset (Wang et al., 2019b) to assess their natural language understanding
(NLU) capabilities. Each model is fine-tuned on the provided downstream task
training dataset for 5 epochs, with a batch size of 16 and a linear learning
rate of $1\cdot 10^{-5}$. This benchmarking step is repeated on 5 downstream
trials with different seeds. We measure individual task’s scores, benchmark
average and standard deviation across all trials. For each model, we measure:
the combined F1 and accuracy on the Microsoft Research Paraphrase Corpus
_mrcp_ (Dolan and Brockett, 2005), Matthews correlation on the Corpus of
Linguistic Acceptability _cola_ (Warstadt et al., 2018), matched and mis-
matched accuracy on the Multi-Genre Natural Language Inference Corpus _mnli_
(Williams et al., 2018), accuracy on the Quora Question Pairs dataset _qqp_
111https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs, accuracy
on the Recognizing Textual Entailment dataset _rte_ (Dagan et al., 2006; Haim
et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), the combined
Pearson and Spearman correlation on the Semantic Textual Similarity Benchmark
_stsb_ (Cer et al., 2017), accuracy on the Stanford Question Answering Dataset
_qnli_ (Rajpurkar et al., 2016), and accuracy on the Stanford Sentiment
Treebank _sst2_ (Socher et al., 2013). The Winograd schema challenge _wnli_
task has been excluded from the evaluation following the recommendation of
Devlin et al. (2018).

Compared to the original BERT setup or more recent compute optimized fine-
tuning setups (Geiping and Goldstein, 2023), we choose to fine-tune for a
longer time (5 epochs instead of 3) and with a lower learning rate ($1\cdot
10^{-5}$ instead of $4\cdot 10^{-5}$), to have a more stable fine-tuning
experience and reduce the risk of lucky seeding. With this choice, we aim to
have a fairer evaluation of the models.

###  3.5 Checkpoint Benchmarking

We want to evaluate how downstream accuracy evolves during pre-training. We
extract checkpoints during training and evaluate them on the GLUE benchmark.
Each checkpoint is fine-tuned and evaluated on GLUE using the previously
established fine-tuning setup.

##  4 Results

In this section, we present the results of our experiments, the pre-training
of our three variants (Figure 2), the scores they reach on the GLUE benchmark
(Table 3) once fully trained and the evolution of the GLUE score during
training (Figure 3).

###  4.1 Pre-Training Experiment

(a) $BERT_{small}$

(b) $BERT_{base}$

Figure 2: BERT pre-training evaluation loss. Models are trained for 200,000
steps, the evaluation loss is the cross-entropy loss. We observe that the
models using the symmetric and pairwise operators converge faster than the
original model.

Figure 2(b) shows that the symmetric and pairwise variant converge much faster
than the original variant for the $BERT_{base}$ model. The evaluation loss of
the original variant remains on the initial plateau until step 25,000, when it
sharply decreases. The symmetric variant remains on the initial plateau until
step 13,500 and the pairwise variant until step 12,000. We also note that the
original and pairwise variants will eventually reach the same evaluation loss
plateau, while the symmetric variant remains above the two other variants with
an additional absolute error of $0.1$.

Comparing Figures 2(a) and 2(b), we observe the impact of model size on
training efficiency. When the model size increases, the original variant’s
initial plateau is expanded from step 12,000 to step 25,000, while the
symmetric and pairwise variant were almost unaffected.

###  4.2 GLUE Benchmark Fine-Tuning

Table 3 shows that the pairwise variant performs better than the original
variant with an increase of $0.6$ points on the average GLUE score for both
model sizes. The symmetric variant, however, is outperformed by the original
variant in both cases, with a drop of $4$ points on the average GLUE score. We
also observe that both proposed variants have a lower standard deviation on
the _bert-base_ model.

###  4.3 GLUE Benchmarking Along Training Steps

Figure 3 shows that the improved training efficiency observed during pre-
training translates to a faster convergence rate on the GLUE benchmark as
well. The pairwise and original variants both reach a final average GLUE score
of approximately $79$. The pairwise variant achieves $95\%$ (a score of $75$)
of its final value after 30,000 steps, the original variant reaches the same
score after 65,000 steps.

We also observe a smoother evolution of the accuracy for the pairwise variant
compared to the original variant. The experiment also highlights the
performance drop of the symmetric variant when compared to the original
variant.

Figure 3: Average GLUE score over training steps. Checkpoints are sampled
during training and evaluated on the GLUE benchmark. The red dashed line
correspond to $95\%$ of the final GLUE average score.

##  5 Discussion

Model | GLUE Score | mrpc | cola | mnli(m/mm) | qqp | rte | stsb | qnli | sst2
---|---|---|---|---|---|---|---|---|---
$BERT_{small}$ | 72.72 (0.07) | 81.04 | 21.39 | 77.16/77.76 | 86.08 | 54.87 | 82.20 | 85.45 | 88.49
$BERT_{small,sym}$ | 69.61 (0.32) | 76.81 | 10.29 | 75.25/75.72 | 85.13 | 55.74 | 77.83 | 82.79 | 86.90
$BERT_{small,pair}$ | 73.38 (0.37) | 82.34 | 24.21 | 76.37/76.89 | 86.67 | 56.25 | 84.13 | 84.97 | 88.60
$BERT_{base}$ | 78.74 (0.63) | 85.30 | 44.35 | 81.66/82.07 | 88.86 | 59.42 | 87.30 | 88.76 | 90.92
$BERT_{base,sym}$ | 74.82 (0.36) | 78.36 | 35.22 | 78.66/79.05 | 87.70 | 53.43 | 84.47 | 86.90 | 89.56
$BERT_{base,pair}$ | 79.36 (0.37) | 87.83 | 46.91 | 81.60/82.02 | 88.89 | 60.58 | 86.88 | 88.78 | 90.78

Table 3: Average GLUE scores over the GLUE benchmark per model with individual
task breakdown. $BERT_{base,pair}$ achieves the best GLUE Score of 79.36 with
a standard deviation of 0.37, in comparison to $BERT_{base}$ which achieve a
GLUE Score of 78.74 with a standard deviation of 0.63.

###  5.1 Pre-training Efficiency

During the pre-training experiment, we observed that both variants
$BERT_{base,sym}$ and $BERT_{base,pair}$ outperformed the original variant
$BERT_{base}$ in terms of convergence rate (they initiated the learning and
reached their respective plateau faster), for a _bert-base_ model the
convergence rate seems to be two times faster. However, $BERT_{base}$ and
$BERT_{base,pair}$ ultimately met around the same evaluation loss, while
$BERT_{base,sym}$ performed a little worse.

One obvious explanation for the improved convergence rate can be found in the
reuse of the $\mathbf{Q}$ operator, this can impact convergence rate in three
way:

  * •

The accumulation of two loss gradients per forward/backward pass instead of a
single one, resulting in an effect similar (but not exactly equivalent) to
doubling the learning rate for the parameters of the $\mathbf{Q}$ operator.

  * •

The reduction in the number of parameters.

  * •

Sharing representation for both $\mathbf{Q}$ and $\mathbf{K}$ operators. If
they do learn a subset of the same features, then enforcing a shared
representation for both of them will reduce the amount of learning required.

These effects explain why both $BERT_{base,sym}$ and $BERT_{base,pair}$
converge much faster than $BERT_{base}$.

While converging faster than $BERT_{base}$, $BERT_{base,sym}$ did not reach
the same evaluation loss. It is fair to assume that this is a modelling issue
and not a size issue since $BERT_{base,pair}$ outperformed $BERT_{base,sym}$
with a similar number of parameters. Thus, we can conclude that symmetry is
not a desired property of the compatibility function of the attention
mechanism.

###  5.2 GLUE Benchmark

The evaluation of the three variants on the GLUE benchmark shows that
$BERT_{base,pair}$ is more accurate than $BERT_{base}$, reaching an average
score of $79.39$ against $78.74$ respectively. The evaluation also shows that
the standard deviation of the average score across five trials is lower for
both $BERT_{base,pair}$ and $BERT_{base,sym}$, with a standard deviation of
$0.37$ and $0.36$ against $0.63$ for $BERT_{base}$.

This confirms that the training efficiency improvement observed on the pre-
training task translates to the fine-tuning task and leads to improvement on
the downstream task’s accuracy. With the added benefit of making the fine-
tuning task more stable, as shown by the lower standard deviation.

We also note that the fairly small $0.1$ difference in evaluation loss during
training for $BERT_{base,sym}$ has translated to a $4$ points accuracy drop on
the evaluation benchmark, echoing our remark on the need to model asymmetric
relationships.

With these results, we experimentally prove that our pairwise operator
improves the training efficiency of Transformer-based models, leading to a
faster convergence rate and overall lower training loss. These improvements
also translate to downstream task benchmarks. Models using the pairwise
compatibility operator are indeed more accurate than the ones using the
original compatibility operator.

###  5.3 GLUE Evaluation During Pre-Training

Running the benchmark evaluation on our three models at several steps of the
pre-training experiment shows that the training efficiency we observed
translates well into downstream accuracy. Our $BERT_{base,sym}$ and
$BERT_{base,pair}$ converge faster towards their respective final values,
similarly to the training loss observed on the pre-training task.
$BERT_{base}$ reaches $95\%$ of its final value after 65,000 steps and
$BERT_{base,pair}$ after 30,000 steps. While $BERT_{base}$ eventually catches
up and improves on $BERT_{base,sym}$, $BERT_{base,pair}$ is consistently the
better model.

This final experiment highlights the improved training efficiency induced by
the pairwise compatibility operator. The faster convergence rate observed
during pre-training is also observed on the downstream task evaluation,
confirming the convergence rate improvement by a factor of two for the
$BERT_{base,pair}$ model.

##  6 Related Work

While the Transformer architecture (Vaswani et al., 2017) popularized the use
of the attention mechanism, and contributed to its adoption in the field of
NLP, the attention mechanism was first introduced to NLP with recurrent neural
networks applied to machine translation (Bahdanau et al., 2016). In this
setting, the compatibility operator is a simple multi-layer perceptron with
non-linear activation operating on the concatenation of inputs encoded by the
recurrent neural network. This definition of the attention mechanism was then
extended to other compatibility operator: Luong et al. (2015) mention the use
of the dot-product between the recurrent neural network’s hidden state,
propose to explicitly integrate token positions into the compatibility
operator, and even suggest the use of a general dot-product operator
$score(h_{t},\overline{h_{s}})=h_{t}^{T}W\overline{h_{s}}$. Those initial
influences have also been documented by Galassi et al. (2020) and Niu et al.
(2021), where the general dot-product appears as a weighted dot-product
between query and keys $f(q,K)=q^{T}WK$. Thus the pairwise compatibility
operator we introduce is an evolution of the general dot-product, where we
constrain it to a single and shared linear operator $\mathbf{Q}$ before
applying the bilinear form of matrix $S$, resulting in the following operator
$\mathbf{A}(x,y)=\mathbf{Q}(x)S\mathbf{Q}(y)^{T}$.

To the best of our knowledge, our work is the first application of the general
dot-product with enforced symmetry to the self-attention mechanism of the
Transformer architecture. While we focused on the compatibility operator,
recent improvements have been made on other parts of the attention mechanism.
Namely, He and Hofmann (2023) proposed to simplify the entire Transformer
block by carefully removing components and achieved an impressive $15\%$
weight reduction, while still relying on the traditional scaled dot-product.

##  7 Conclusions

In this work, we revisited the traditional scaled dot-product used in the
Transformer self-attention mechanism. We challenged the use of two distinct
operators to compute the dot-product between queries and keys, in favor of
single shared operator and a weighted dot-product with pairwise factors. By
doing so, we enforced a symmetric structure to the compatibility operator of
the attention mechanism, reducing the number of parameters used in the
Transformer layer by a third. As a result, when applied to BERT models, our
pairwise compatibility operator reduces the overall number of parameters of
the model by $6\%$, reduces the number of pre-training steps required by half
and improves accuracy on the GLUE benchmark, making Transformer-based encoders
more efficient, faster to train and lowering their resource requirements. We
believe our work can be applied to other Transformer architectures like
decoder and encoder-decoder models, as well as to other NLP tasks like machine
translation and language modeling. And, more generally, to the concept of
attention as a whole, where it would bring improvement in other fields such as
computer vision.

For future work, we plan to evaluate the pairwise dot-product attention
mechanism on larger models reaching into the billion parameters, and to
evaluate our attention mechanism on other benchmarks, like SuperGLUE (Wang et
al., 2019a) and SQuAD2.0 (Rajpurkar et al., 2018). We plan on implementing the
pairwise compatibility operator for the cross-attention mechanism, and
evaluating it on decoder and encoder-decoder tasks like language modeling and
machine translation. Finally, we want to evaluate our pairwise dot-product
attention not only on natural language processing tasks, but also on tasks
from other fields, computer vision, time series forecasting and reinforcement
learning.

## Limitations

Our work focuses only on the application and evaluation of alternative
compatibility functions for the self-attention mechanism of Transformer-based
encoder models, benchmarked on NLU tasks. While our work has shown positive
results on this specific use case, we cannot draw any conclusion on its
application to decoder models and pure language modeling tasks, or encoder-
decoder model and machine translation tasks. Those use cases rely on the
cross-attention mechanism for which the shared representation we exploit with
our pairwise compatibility operator may not be appropriate.

While we suggest that the $\mathbf{Q}$ and $\mathbf{K}$ operators learn a
shared representation, we did not perform any analysis of the original scaled-
dot product attention or of our pairwise dot-product attention. The parameter
redundancy of multi-head attention models has been covered in Bian et al.
(2021). However, to our knowledge the parameter redundancy between the query
and the key operator of a single head has not been studied.

While our work showed positive improvements on the training efficiency of
BERT-like models of fairly small sizes (100 million parameters), it is not
enough to draw conclusions on its efficiency on very large models (e.g., 10
billion parameters).

We decided to benchmark our models on GLUE, as it is the most popular
benchmark for NLU evaluation. However, this benchmark as been largely
surpassed by modern machine learning models. For that reason, new benchmarks
have been introduced, such as SuperGLUE (Wang et al., 2019a) or SQuAD2.0
(Rajpurkar et al., 2018).

## Reproducibility Statement

All software related to our experiments with the attention mechanism is
available online222https://github.com/mcrts/ACL2024-SymmetricAttentionBert. It
uses the PyTorch (Paszke et al., 2017) and Hugging Face Transformer (Wolf et
al., 2020) frameworks. The necessary steps to recreate the training dataset
are
documented333https://github.com/mcrts/ACL2024-SymmetricAttentionBert/tree/main/data,
the dataset used for training is available on Hugging
Face444https://huggingface.co/datasets/mcrts/OSCAR-2301_en_30M.

## Acknowledgements

The work presented in this paper has received funding from the German Federal
Ministry for Economic Affairs and Climate Action (BMWK) through the project
OpenGPT-X (project no. 68GX21007D), and has been supported by the German
Federal Ministry of Education and Research as part of the project TRAILS.

## References

  * Abadji et al. (2022) Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4344–4355, Marseille, France. European Language Resources Association.
  * Bahdanau et al. (2016) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016.  Neural machine translation by jointly learning to align and translate.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The fifth pascal recognizing textual entailment challenge.  _TAC_ , 7(8):1.
  * Bian et al. (2021) Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. 2021.  On attention redundancy: A comprehensive study.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 930–945, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The PASCAL recognising textual entailment challenge.  In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_ , pages 177–190. Springer.
  * Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the International Workshop on Paraphrasing_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929.
  * Galassi et al. (2020) Andrea Galassi, Marco Lippi, and Paolo Torroni. 2020.  Attention in natural language processing.  _IEEE transactions on neural networks and learning systems_ , 32(10):4291–4308.
  * Geiping and Goldstein (2023) Jonas Geiping and Tom Goldstein. 2023.  Cramming: Training a language model on a single gpu in one day.  In _International Conference on Machine Learning_ , pages 11117–11143. PMLR.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9. Association for Computational Linguistics.
  * Haim et al. (2006) R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second pascal recognising textual entailment challenge.  In _Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment_ , volume 7, pages 785–794.
  * He and Hofmann (2023) Bobby He and Thomas Hofmann. 2023.  Simplifying transformer blocks.  _arXiv preprint arXiv:2311.01906_.
  * Jansen et al. (2022) Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. 2022.  Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data.  _arXiv e-prints_ , page arXiv:2212.10440.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  _CoRR_ , abs/1508.04025.
  * Niu et al. (2021) Zhaoyang Niu, Guoqiang Zhong, and Hui Yu. 2021.  A review on the attention mechanism of deep learning.  _Neurocomputing_ , 452:48–62.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch.  In _NIPS 2017 Workshop on Autodiff_.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _CoRR_ , abs/1806.03822.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021.  Roformer: Enhanced transformer with rotary position embedding.  _CoRR_ , abs/2104.09864.
  * Thompson et al. (2021) Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. 2021.  Deep learning’s diminishing returns: The cost of improvement is becoming unsustainable.  _IEEE Spectrum_ , 58(10):50–55.
  * Togelius and Yannakakis (2023) Julian Togelius and Georgios N. Yannakakis. 2023.  Choose your weapon: Survival strategies for depressed ai academics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _CoRR_ , abs/1905.00537.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018.  Neural network acceptability judgments.  _CoRR_ , abs/1805.12471.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zhou et al. (2020) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2020.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  _CoRR_ , abs/2012.07436.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_.

##  Appendix A OSCAR Filters

To ensure high quality of our training dataset, we filter OSCAR dumps with the
following rules:

  * •

From the UT1 Blocklists project555http://dsi.ut-
capitole.fr/blacklists/index_en.php, we exclude the following categories:

    * –

“agressif”

    * –

“adult”

    * –

“cryptojacking”

    * –

“dangerous_material”

    * –

“phishing”

    * –

“warez”

    * –

“ddos”

    * –

“hacking”

    * –

“malware”

    * –

“mixed_adult”

    * –

“sect”

  * •

We exclude documents whose _harmful perplexity score_ is below 5.0 and above
100,000.

  * •

Following recommendation from Abadji et al. (2022), we exclude documents which
have been flagged with quality warnings.