# Quiescent Solar Wind Regions in the Near-Sun Environment: Properties and
Radial Evolution

Benjamin Short Department of Physics, University of Colorado, Boulder, CO, USA
Laboratory for Atmospheric and Space Physics, University of Colorado, Boulder,
CO, USA  David M. Malaspina Laboratory for Atmospheric and Space Physics,
University of Colorado, Boulder, CO, USA  Astrophysical and Planetary Sciences
Department, University of Colorado, Boulder, CO, USA  Alexandros Chasapis
Laboratory for Atmospheric and Space Physics, University of Colorado, Boulder,
CO, USA  Jaye L. Verniero NASA Goddard Space Flight Center, Greenbelt, MD, USA

###### Abstract

Regions of magnetic field with near-radial, Parker Spiral-like geometry known
as quiescent regions have been observed in Parker Solar Probe data. These
regions have very low $\delta B/\langle|B|\rangle$ compared to non-quiescent
solar wind at the same heliocentric distances. Quiescent regions are observed
to have lower solar wind bulk speeds, lower proton temperatures, and lower
proton density, consistent with properties of the slow solar wind. Inside of
15 Rs, identified quiescent regions show distinct thermal properties, having
higher proton temperature anisotropies and lower parallel plasma betas
compared to switchback patches observed at the same heliocentric distances.
When placed on $\mathcal{R}$ vs $\beta_{\parallel p}$ plots (where
$\mathcal{R}$ is the proton temperature anisotropy), quiescent region solar
wind is shown to be more stable to proton cyclotron and firehose instabilities
than non-quiescent solar wind at the same heliocentric distances. It is shown
that quiescent regions evolve similarly to the surrounding non-quiescent solar
wind, but quiescent solar wind begins at a different location in the
$\mathcal{R}$ vs $\beta_{\parallel p}$ parameter space, suggesting that these
regions have separate origins than the more turbulent non-quiescent solar
wind. Namely, enhanced temperature anisotropies and enhanced magnetic field
strength may be consistent with magnetic field lines which have undergone less
magnetic field expansion compared to non-quiescent wind at the same
heliocentric distances.

Solar wind(1534) — Interplanetary Turbulence(830) — Space Plasmas(1544) —
Solar Magnetic Fields(1503) — Heliosphere(711)

##  1 Introduction

Observed inside of 65 ${R_{\odot}}$ are regions of solar wind with magnetic
field geometries which are closely aligned with a Parker spiral interspersed
between patches of sharp magnetic deflections called “switchbacks” (Bale et
al., 2019; Kasper et al., 2019; Dudok de Wit et al., 2020; Horbury et al.,
2020). Quiescent regions appear to contain low levels of magnetic field
fluctuations (Section 2.4) and regions observed during Parker Solar Probe’s
(PSP) first close encounter exhibited magnetic turbulence power spectral
density with a spectral break point at frequencies lower than more general
solar wind (Dudok de Wit et al., 2020), suggesting that the turbulence of
these regions is less evolved than the surrounding solar wind. If these
regions display measurable differences in their bulk properties, this
difference in Alfvénic turbulence may allow us to make estimates of the role
that turbulence plays in modifying the solar wind plasma parameters and
through which estimate the role of turbulence in solar wind heating and
acceleration. Some quiescent regions persist for hours in time series data
(Section 2) and quiescent regions are measured as far out as 50 ${R_{\odot}}$
(Short et al., 2022). This raises the following questions: how long do
quiescent regions persist in the solar wind, from where do they originate, and
how are they related to switchbacks if at all? Recent studies have shown that
switchback patches are modulated on the scale of supergranulation networks
boundaries (Bale et al., 2021) at the solar surface. Bale et al. (2023)
demonstrates using a Potential Field Source Surface model that switchback
patches in encounter 10 are associated with open flux originating from two
coronal holes. The authors of that study argue that the magnetic fields at the
base of the corona undergo interchange reconnection near the edges of
supergranulation network boundaries. The structure of the network boundaries
then becomes embedded into the solar wind plasma which is measured by PSP at
higher altitudes.

###  1.1 Quiescent Region Origins

Quiescent regions appear to be distributed between switchback patches (Figure
1; Bale et al. (2019, 2021)). Because quiescent regions are intermingled with
switchback patches, it may also be the case that some quiescent regions share
an association with supergranulation network boundaries. However, if quiescent
regions and switchback patches share this supergranulation network
association, why then is it that they possess different turbulence properties
such as differing spectral break points (Dudok de Wit et al., 2020) and
different magnetic field fluctuation amplitudes (Figure 4)? Results from Dudok
de Wit et al. (2020) suggest that quiescent regions have properties of less
evolved solar wind. These statements can be used to generate hypotheses:
hypothesis 1 says that quiescent regions and switchback patches originate from
similar locations on the solar surface and at the same altitude. If this
hypothesis is true then the location from which quiescent regions originate
must have properties that inhibit the development of large scale magnetic
field fluctuations. Hypotheses 2 and 3 suggest that switchback patches do not
share an origin location with switchback patches. Because many quiescent
regions are distributed between switchback patches, it is expected that at
least some quiescent regions originate along the similar field lines that
switchback patches do, but it is not necessary that they originate from the
same altitude. Hypothesis 2 says that quiescent regions originate from deeper
within the corona than switchback patches do and the pristine properties of
the younger solar wind pass through the magnetic field fluctuations induced by
interchange reconnection or other mechanisms. Hypothesis 3 states that
quiescent regions originate from higher altitudes than more turbulent solar
wind regions like switchback patches and develop in-situ.

If hypothesis 1 is correct, it indicates that the structure of the
supergranulation network boundaries and events on the solar surface such as
interchange reconnection (Bale et al., 2023) lead to the appearance of solar
wind regions which take on distinct characteristics. Hypothesis 2 suggests
that there are lanes through which pristine plasma from low in the corona can
penetrate the turbulence of the lower corona without becoming unstable to
switchback growth. Hypothesis 3 necessitates a mechanism by which distinct
turbulent properties develop between quiescent and non-quiescent solar wind,
only for those differences to be filled back in by the time the solar wind
reaches 1 AU. This study seeks to determine the properties of quiescent
regions measured at a range of solar altitudes and distinguish between these
possibilities.

Section 2 details the data used in this study and how they are selected. It
also details the data processing algorithm used to identify quiescent regions.
Section 3 details the results of this study. Section 4 discusses the possible
interpretations of the results and Section 5 summarizes the interpretations
and compares them to hypotheses above.

##  2 Data Set and Processing

###  2.1 Data Products

This study utilizes data collected from the FIELDS instrument suite (Bale et
al., 2016) as well as the Solar Wind Electron, Alphas, and Protons (SWEAP)
(Kasper et al., 2016; Livi et al., 2021) instrument suite. From FIELDS we use
survey data from the fluxgate magnetometers. From SWEAP we use measurements
from the Solar Probe Cup (SPC) (Case et al., 2020) and the Solar Probe
ANalyzers Ion (SPAN-Ion) (Livi et al., 2021) sensors. This study uses data
from encounters 1 through 16 and covers radial distances of 13.3 $R_{\odot}$
to 65 $R_{\odot}$. The fluxgate magnetometers are located on a 3.5 m boom on
the rear of the spacecraft opposite the Thermal Protection Shield (TPS). While
these magnetometers operate at a cadence of roughly 293 Sa/s (Bale et al.,
2016) this study uses a downsampled cadence of roughly 4.53 Sa/s which is
available in the L2 public data products. This study also uses particle
distributions and derivative quantities from the SWEAP instruments for protons
(Kasper et al., 2016; Livi et al., 2021). Using the publicly available L3
SPAN-I moments, we calculate the proton temperatures, densities, bulk
velocities, plasma betas, Alfvén velocities, and temperature anisotropies.

Quiescent regions have been defined in terms of the normalized deflection
angle from a theoretical Parker Spiral (Dudok de Wit et al., 2020). This
deflection angle, $z$, is defined as

$z=\frac{1}{2}(1-\cos(\theta_{def}))=\frac{1}{2}(1-\mathbf{\hat{b}_{meas}}\cdot\mathbf{\hat{b}_{Park}})$

where $\theta_{def}$ is the deflection angle of the measured magnetic field
away from a theoretical Parker Spiral and its cosine is the dot product of the
measured magnetic field unit vector $\mathbf{\hat{b}_{meas}}$ with the Parker
Spiral unit vector $\mathbf{\hat{b}_{Park}}$. The unit vector
$\mathbf{\hat{b}_{meas}}$ is measured by the fluxgate magnetometers and
reported in RTN coordinates. The unit vector for the Parker Spiral is derived
using a combination of the radial component of the solar wind velocity
measurements from SPAN-I and from SPC.

###  2.2 Quiescent Region Identification

This study seeks times where the solar wind matches the previously used (Dudok
de Wit et al., 2020) $z$ threshold of $z<0.05$ (or $z>0.95$ when PSP crosses
the Heliospheric Current Sheet and the global field polarity changes sign) but
differs from previous work by seeking times when the magnetic field angle is
stable for a time duration. The reason for selecting for magnetic field is so
that this study can distinguish between switchback patches and larger scale
quiescent wind. Previous work selected quiescent wind by using the $z$
threshold of $z<0.05$ and so captured quiescent wind between individual
switchbacks as well as streams of quiescent wind between larger scale
switchback patches.

To distinguish between streams quiescent wind and quiescent wind inside of
switchback patches, it is necessary to bin $z$ in time. This binning process
presents a challenge because the characteristic time lengths that should be
used identify quiescent regions are unknown. At present, no study has
quantified a characteristic quiescent region duration. Based on work on
characteristic switchback timescales by Dudok de Wit et al. (2020) and
occurrence rates of switchback patches by Jagarlamudi et al. (2023) it is not
clear that a typical duration does exist.

To accurately identify quiescent regions in time, we employ an algorithm which
places bins randomly in time across PSP encounters. In this algorithm and
analysis, solar encounters are defined as times when PSP approaches within
$65$ solar radii ($R_{\odot}$) of the sun. $z$ is calculated at the 4 Sa/cycle
cadence, or roughly 4.53 Sa/s; the same downsampled cadence which we use for
the magnetometer data, acknowledging that both the SPAN and SPC instruments
record survey data at lower cadences than the magnetometer. As such, each data
point from the magnetometer is assigned to its “nearest neighbor” in the
particle data to calculate $z$ for each point in the magnetometer data.

Once $z$ has been calculated for each encounter, 1000 points in $z$ are
selected randomly to become the edges of bins in time, the gaps between these
edges make up the width of the bin. The random selection helps eliminate
systematic biases that may arise from bin sizing and bin placement. For each
of these bins, the fraction of quiescent data points in each bin is
determined. Quiescent data points are those which have $z$ values of $<0.05$
or $>0.95$. If this fraction of quiescent data within the bin is $>0.95$ then
each time step in the bin is assigned a 1, if the fraction is less, it
receives a 0. A single iteration of this algorithm generates a line featuring
square pulses with height 1. This procedure is repeated $N$ times and the
average of each iteration $q_{n}$ (the first iteration being $q_{0}$) is taken
as a final quality flag $q_{rand}$ over time $\tau$ which will be used to
identify quiescent regions.

Figure 1 shows this algorithm acting on data from PSP encounter 11. Panel (a)
displays the radial component of magnetometer survey data (blue) and quiescent
regions identified by the final output of this algorithm (orange). Panels (b)
though (d) show the quality flag output by this algorithm at increasing number
of iterations. Panel (b) shows the result after one iteration, panel (c) after
five iterations, and panel (d) shows the final output at $N=300$ iterations.

$q_{rand}(\tau)=\displaystyle\frac{\sum_{n=0}^{N}q_{n}(\tau)}{N}$

Points in this quality flag with values $q_{rand}>0.5$ are labeled quiescent
solar wind and points with values $q_{rand}<0.5$ are labeled non-quiescent.
Using this metric, start and stop times are determined for each quiescent
region. In panel (d) of Figure 1, we see several portions of solar wind which
approach the $q_{rand}>0.5$ threshold, but do not reach it. A set of examples
are shown around 2022-02-25/06:00:00. This threshold of 0.5 is chosen in order
to set it equally between the possible range of $q_{rand}$ between 1 and 0 and
to better emulate the stepwise behavior of the initial $q_{0}$ iteration. This
choice of threshold minimizes the risk of flagging false positives in
quiescent identification, but throws away some portions of solar wind which
may be on the edge of being quiescent.

The number of quiescent regions identified using this algorithm for encounters
1 through 16 are 504 and cover radial distances from 13.3 $R_{\odot}$ to 65
$R_{\odot}$. The total amount of time covered by these identified quiescent
regions are nearly 494 hours of the total hours 5702 spent by PSP below 65
$R_{\odot}$. Quiescent regions thus make up $\sim 8.7\%$ of the total data
available below 65 $R_{\odot}$. For this analysis, the data are broken into
bins of radial distance. These radial bins are taken in 10 $R_{\odot}$
increments with the exception of the innermost bin. These bins are 13.3
$R_{\odot}$ to 15 $R_{\odot}$, 15 $R_{\odot}$ to 25 $R_{\odot}$, 25
$R_{\odot}$ to 35 $R_{\odot}$, and so on up to the 55$R_{\odot}$ to 65
$R_{\odot}$ radial bin. The statistics and data durations in hours are listed
in Table 1.

Figure 1: Quiescent region finding algorithm. Figure 1a is the magnetic field
radial component in time series for encounter 7 with quiescent times
highlighted in orange. Figure 1b is the result for quality flag $q_{rand}$
after one iteration of this identification algorithm. Figure 1c is the result
after 5 iterations of the algorithm. Figure 1d is the result after 300
iterations. The final result for $q_{rand}$ used in this study is taken after
300 iterations. For panel a, the grey dashed line represents 0 nT. For panels
b through d, the dashed grey line represents the 0.5 threshold required for
solar wind to be labeled quiescent.

Around 2022-02-25/12:00:00 is a Heliospheric current sheet (HCS) crossing
followed by a large quiescent region highlighted in orange. We can see during
this interval many switchbacks and switchback patches which are separated by
quiescent regions.

###  2.3 SPAN-Ion Field-of-View Compensation and Quasi-Thermal Noise (QTN)
Agreement

This investigation uses plasma moments derived from the Solar Wind Electrons
Alphas and Protons (SWEAP) instrument suite (Kasper et al., 2016). Because
Parker Solar Probe orbits so close to the Sun, it has a Thermal Protection
Shield (TPS), or heat shield, to protect the spacecraft bus and its
instruments from the intense light of the Sun (Fox et al., 2016). This heat
shield blocks light from the Sun, but also partially blocks incoming solar
wind particles which means we only measure partial plasma moments of solar
wind temperature, density, and velocity. SWEAP accounts for this by using two
different particle detectors in conjunction. On the forward end of the
spacecraft sits the Solar Probe Cup (SPC), which is a sun-pointing Faraday cup
that is limited to measuring particles in the radial direction, producing a
reduced 1D velocity distribution function (VDF) (Case et al., 2020). The Solar
Probe ANalyzer for ions (SPAN-I) sits on the ram of the spacecraft and can
measure a full 3-dimensional proton VDF. However, because of the instrument
positioning behind the heat shield, the core of the proton distribution is not
always within the SPAN-I field-of-view, leading to inaccuracies in determining
solar wind bulk properties under some circumstances (Livi et al., 2022). The
proton core drifts out of the SPANs field-of-view when the vector of the
proton flow velocity in the PSP frame is oriented more along the radial
direction than the tangential direction, and occurs more often when PSP is
moving slowest in the tangential direction relative to the Sun-PSP line.

In this study we investigate thermal properties which require the proton VDF
in 3-dimensions; namely, the solar wind temperature anisotropy and the plasma
beta parallel to the background magnetic field. To ensure that our results are
robust to field-of-view (FOV) changes in the SPAN-I data, we control for the
field-of-view in two ways: the first is by observing the energy partition in
SPANs anodes along its Sunward-Antisunward viewing directions. When the peak
of the energy flux is within the 3rd anode an accurate fit of the core is
possible (Livi et al., 2022). Thus times when the peak energy flux is within
SPANs 3rd anode from the heat shield are considered accurate and labeled with
a 2, while times where the peak flux is not within the 3rd anode are labeled
with a 0. For the purposes of ease of data analysis, these times are averaged
across using a 30 minute long sliding window. The 30 minute window is chosen
to select times when PSP enters large scale solar wind streams in which the
proton core is within SPANs FOV. This gives us a partial quality flag which we
call $q_{fov}$ which varies in a binary fashion between 2 and 0.

The second way in which SPAN data quality is controlled for is by comparing
the estimated proton density measured by SPAN (called $n_{p}$) to the value
for the electron density ($n_{e}$) inferred from quasi-thermal noise (QTN)
measurements of the upper hybrid resonance line. By comparing the difference
between the $n_{e}$ and $n_{p}$ and assuming quasi-neutrality, we get an
estimation on how accurately SPAN-I is measuring the proton distribution. An
accurate measurement of the proton density $n_{p}$ by SPAN-I is a strong
indication that SPAN-I is getting a accurate fit of the proton core and thus,
at those times, an accurate measurement of the solar wind velocity vector.
This comparison gives For each point in the QTN measurement there is an
uncertainty in the density which originates from the frequency resolution of
the FIELDS instrument suite. This uncertainty is $\delta n_{e}$. For times
when the QTN estimates are available, each point in the QTN data is assigned
to its nearest neighbor in the SPAN data using the same method described in
Section 2.2. The percent error $\delta n_{e}/n_{e}*100$ on the QTN density
$n_{e}$ has typical values of $\sim 8\%$ and thus $2\times\delta n_{e}$ is
around $\sim 16\%$ of the measured value. A SPAN density $n_{p}$ within
$2\times\delta n_{e}$ is considered most accurate and assigned a 2. If $n_{p}$
is within $3\times\delta n_{e}$ of $n_{e}$ it is considered somewhat accurate
and assigned a 1. Anything else, whether $n_{p}$ is outside of $3\times\delta
n_{e}$ or if the QTN data is unavailable, that time is considered inaccurate
and is assigned a 0. This gives us a partial quality flag which we call
$q_{qtn}$ which varies between 2, 1, and 0. Note that this QTN/SPAN density
comparison does not consider alpha abundance. So long as the alpha abundance
remains below $~{}4\%$ the alphas should not have an exceptionally large
effect on this analysis given the comparatively large window $2\times\delta
n_{e}$, which is typically around $\sim 16\%$ of $n_{e}$, allowed for this
comparison. However, it does mean that times with high alpha abundance, if the
proton core is not sufficiently within the field of view as determined by
$q_{fov}$, will be cut out of the analysis and may serve to bias this analysis
to low alpha solar wind.

These two methods described above are combined into a total quality flag for
SPAN. The values for $q_{fov}$ and $q_{qtn}$ are added to create a total
quality flag for SPAN which we call $q_{SPAN}=q_{fov}+q_{qtn}$. This total
quality flag varies between integer values 4, 3, 2, 1, and 0. To calculate the
Parker Spiral unit vector for $z$, SPAN-I is used whenever this quality flag
$q_{SPAN}$ is $\geq 1$ and SPC is used otherwise. This occurs when the SPAN
density is within $3\times\delta n_{e}$ or when the peak of the SPAN
distribution is within the 3rd anode in the Sunward-Antisunward viewing
directions or some combination of both situations. For all other measurements
which rely on SPAN in this study such as the bulk velocity, proton
temperature, proton density, proton temperature anisotropy $\mathcal{R}$,
proton plasma beta parallel $\beta_{\parallel p}$, and Alfvén mach number, a
stricter requirement of $q_{SPAN}\geq 2$ is required. This stricter
requirement ensures that either the proton core is well within the SPAN-I FOV
or that the SPAN proton density $n_{p}$ is within $2\times\delta
n_{e}/n_{e}*100\%$ of $n_{e}$ or some combination of both situations.

$\\#$ of Regions (Hours) |  $<15$ $R_{\odot}$ | 15-25 $R_{\odot}$ | 25-35 $R_{\odot}$ | 35-45 $R_{\odot}$ | 45-55 $R_{\odot}$ | 55-65 $R_{\odot}$
---|---|---|---|---|---|---
Raw Region Count | 44 (35.6) | 92 (109.3) | 107 (114.3) | 128 (107) | 67 (79.4) | 62 (48.7)
After SPAN Check | 44 (31.4) | 87 (72.7) | 96 (49.3) | 49 (15.7) | 30 (12) | 6 (1.1)
Table 1: Table of quiescent region counts and hours of data available for
different PSP radial bins. The number of regions are integers to the left
while the number of hours are contained in parenthesis on the right of each
cell. Top row is the counts of regions found at all radial distances in the
magnetic field data. Bottom row are the number of regions for which the SPAN-I
field of view is sufficient.

###  2.4 Magnetic Field Fluctuation Amplitudes: $\delta B/\langle|B|\rangle$

From previous studies, quiescent regions possess different turbulent
properties than switchbacks (Dudok de Wit et al., 2020) and the quiescent
regions have lower amplitude fluctuations but this property has not been fully
quantified to date. We can estimate the amplitude of the turbulent
fluctuations by employing a $\delta B/\langle|B|\rangle$ estimation. $\delta
B/\langle|B|\rangle$ is the ratio of the average amplitude of magnetic
fluctuations relative to the average background magnetic field. $\delta B$ is
defined as

$\delta
B=\sqrt{\langle\delta{B_{r}}^{2}+\delta{B_{t}}^{2}+\delta{B_{n}}^{2}\rangle}$

where the fluctuations of the i-th component are defined to be

$\delta B_{i}=B_{i}-\langle B_{i}\rangle$

For quiescent region, the above operation is used to return a single value.
Because it involves many averages, we must choose a scale over which to
evaluate that average. The scale by which we determine the average is the
scale of the quiescent region itself. For example, if a particular quiescent
region is 10 minutes long, the scale over which the average is determined is
10 minutes. Likewise, if the next region is 2 hours long, the scale of the
average for that region is 2 hours. For each quiescent region a value of
$\delta B/\langle|B|\rangle$ is determined. Then using the same set of
averaging scales, portions of the remaining solar wind not identified as
quiescent regions are selected at random and a value for $\delta
B/\langle|B|\rangle$ is calculated for non-quiescent wind in the same way as
the quiescent wind. This is done in order to compare quiescent and non-
quiescent magnetic field fluctuation amplitudes in a way that minimizes bias
in bin placement. For example, if we calculated $\delta B/\langle|B|\rangle$
for a quiescent region and, keeping the same averaging scale, calculated
$\delta B/\langle|B|\rangle$ for non-quiescent wind immediately following in
time series, it may be the case that the value of $\delta B/\langle|B|\rangle$
for both sets of solar wind are correlated by virtue of proximity. Randomly
selecting where to place the non-quiescent wind $\delta B/\langle|B|\rangle$
calculations helps capture the general trend of $\delta B/\langle|B|\rangle$
in non-quiescent wind more accurately. To increase the amount of statistics in
the non-quiescent distribution, this calculation is done as many times as non-
quiescent wind over an encounter is able to be divided. This allows for direct
comparisons of quiescent region turbulence amplitudes with non-quiescent solar
wind.

##  3 Results

###  3.1 Velocity Differences

Figure 2 shows solar wind bulk flow velocity data from the entirety of the
Parker Solar Probe mission up to encounter 16 plotted against PSPs
heliocentric distance from the sun. Figure 2 represents 2d histograms for
quiescent and non-quiescent wind over the course of PSPs mission. Quiescent
region data are pictured in orange and non-quiescent wind is depicted in blue.
The solid lines at the top and bottom of each shaded area represent the 1st
and 3rd quartiles for each distribution and the dotted line in the center of
each shaded curve is the median. From this figure, we can see that quiescent
times make up the slower end of the solar wind velocity while non-quiescent
times tend towards higher velocities. This observation becomes more clear when
comparing to the third column of Figure 3. Figure 3 is four sets of three
histograms for proton density, proton core temperature, bulk flow velocity,
and the magnetic field magnitude in order from left column to right column.
The rows of Figure 3 represent three radial bins which are $<15$ $R_{\odot}$,
$15$ $R_{\odot}$ to $25$ $R_{\odot}$, and $25$ $R_{\odot}$ to $35$ $R_{\odot}$
from top row to bottom row. This figure contains data for the innermost three
radial bins as outside of 35 $R_{\odot}$, the amount of SPAN data which
satisfies the quality flag in Section 2.3 drops off quickly.

At each radial distance, the average solar wind velocity for quiescent regions
is lower than the average of non-quiescent times. For Figure 3c1 the average
quiescent speed is 302 km/s and the average speed during non-quiescent times
is 333 km/s, which is a 31 km/s difference or $\sim 10\%$ change. For Figure
3c1, it is possible that the observed long tail for non-quiescent regions may
be a spacecraft observation effect: higher fluctuations in the perpendicular
directions can create larger uncertainties due to sampling geometry. This
change is outside the uncertainty for reported velocities from SPAN-I moments
which is $\sim 3\%$, assuming good SPAN-I FOV. However, this uncertainty
estimate does not include systematic uncertainties not yet quantified by the
instrument. At larger radial distances the averages begin to diverge. When
looking at the 15 to 25 $R_{\odot}$ radial bin in Figure 3c2 the average
quiescent speed becomes 198 km/s and the average speed during non-quiescent
times becomes 266 km/s. The difference in solar wind speeds expands to a 68
km/s difference, or a change of $\sim 34\%$ from the quiescent solar wind
speed. This change in the percent differences could indicate differential
acceleration between quiescent and non-quiescent wind populations across the
observed radial bins. However, this possibility warrants further investigation
because while the differences in solar wind speeds between the two populations
increase between radial distances the innermost bin also appears to have
higher solar wind speeds overall. This could be a result of selection biasing
due to the innermost bin being dominated by data at solar maximum. Possible
biasing due to changes in the solar cycle is discussed in Section 4.2.

Figure 2: Bulk Flow Velocity span across entire PSP mission inside of 55 Rs vs
radial distance. Each curve represents a 2d-histogram of the solar wind
velocity measured by either SPAN or SPC depending on quality of data as
described in Section 2.3. The top and bottom solid black lines bounding the
colored areas represent the 1st and 3rd quartiles in each radial bin, while
the dotted line in the center of each colored area is the median value for the
solar wind speed. Quiescent regions data is highlighted in orange where non-
quiescent wind is highlighted in blue. The velocity spread for quiescent
regions are consistently lower for most altitudes. Figure 3: Normalized
histograms of various solar wind properties. The rows of histograms represent
radial distances with the top row being the innermost radial bin with data
ranging from 13.3 $R_{\odot}$ to 15 $R_{\odot}$. The following two rows are in
10 $R_{\odot}$ increments, the center row being from 15 to 25 $R_{\odot}$ and
the bottom row from 25 to 35 $R_{\odot}$. The columns are each property at
those radial distances. Column (a) is the number of protons per cm3, column
(b) is the proton core temperature in eV, column (c) is the bulk velocity
magnitude in km/s, column (d) is the magnetic field magnitude in nT.

###  3.2 Temperature Differences

In addition to the differences in velocity, there is also a temperature
difference between quiescent and non-quiescent solar wind. In Figure 3b2,
quiescent regions make up the lower end of the temperature range. However, for
the other two radial bins, Figure 3b1 and Figure 3b3, the difference becomes
more ambiguous. For Figure 3b1, the average for both quiescent and non-
quiescent temperatures are 79.3 eVs. Then stepping out in radial distance to
Figure 3b2, a difference in temperatures emerges, with quiescent times
averaging 31.6 eV and non-quiescent solar wind averaging 47.2 eV. This
represents a $\sim 49\%$ increase in temperature, which is outside the SPAN
moments uncertainty in temperature which is $\sim 10\%$ to $\sim 15\%$, though
for non-quiescent solar wind this uncertainty may be higher due to increased
magnetic field fluctuations (Verscharen & Marsch, 2011). Then finally Figure
3b3, quiescent regions average 26.9 eV whereas non-quiescent wind averages
30.2 eV, which yields a $\sim 12\%$ increase in temperature. This increase is
within the SPAN-I uncertainty values for temperature. A difficulty with these
results is that the temperature in the lowest altitude radial bin is multi-
modal. This may indicate too little quiescent region data (44 quiescent
regions, 35.6 hours of data) in the range of altitudes that are being
investigated. While the number of individual data points per quiescent region
are high, the number of quiescent regions within this altitude range may not
be enough to capture the full nature of the temperature distribution for
quiescent regions at this altitude range. More orbits within 15 $R_{\odot}$
are needed to fill in this distribution.

Q-region , Non-Q | 13.3 $R_{\odot}$ \- 15 $R_{\odot}$ | 15 $R_{\odot}$ \- 25 $R_{\odot}$ | 25 $R_{\odot}$ \- 35 $R_{\odot}$
---|---|---|---
Density (cm-3) | 1288 ± 1057, 2207 ± 1732 | 1904 ± 1434 , 2029 ± 1421 | 776 ± 354, 800 ± 407
Temperature (eV) | 79.3 ± 46.9 , 79.3 ± 57.2 | 31.6 ± 23.5, 47.2 ± 57.4 | 26.9 ± 10.3 , 30.2 ± 16.6
Velocity (km/s) | 302 ± 70 , 333 ± 97 | 198 ± 54 , 266 ± 82 | 240 ± 42 , 259 ± 67
$|B|$ (nT) | 752.6 ± 84.9 , 701.9 ± 128.5 | 339.2 ± 107.8 , 335.8 ± 135.4 | 154.7 ± 65.8 , 149.0 ± 89.6
Table 2: Solar wind quantity averages for Figure 3 with standard deviations
included. Quiescent solar wind averages are on the left side of the comma in
each cell, non-quiescent solar wind is on the right. The rows represent the
solar wind properties, the columns represent the radial distance. Figure 4:
Non-normalized histograms of $\delta B/\langle|B|\rangle$ for each quiescent
region across all radial distances. The orange histogram is for quiescent
regions and the blue is for non-quiescent solar wind of the same scale.

###  3.3 Magnetic Field Differences

From Figure 3d1, the DC magnetic field magnitude is on average greater for
quiescent regions than non-quiescent wind at similar altitudes. The averages
are enumerated in Table 2. From both Table 2 and from each panel in Figure 3d
for each radial distance the magnetic field is greater on average than for
non-quiescent intervals. Moreover, from Figure 4 it is shown that the
turbulence amplitudes for quiescent regions relative to the background field
are lower than for non-quiescent solar wind. Figure 4 are a set of two
histograms, one for quiescent regions in orange, one for non-quiescent wind in
blue. The histograms represent the magnetic field fluctuation amplitudes
$\delta B/\langle|B|\rangle$ for quiescent and non-quiescent wind. The average
magnetic field fluctuation amplitudes $\delta B/\langle|B|\rangle$ for
quiescent regions are 0.15±0.05 whereas the value is 0.4±0.2 for non-quiescent
solar wind. This amounts to a $62.5\%$ decrease in $\delta
B/\langle|B|\rangle$ in quiescent regions compared to non-quiescent wind in
this study.

###  3.4 Density Differences

With the exception of the innermost bin, the density measured is roughly
comparable. From Figure 3a2 and 3a3, it is shown that the distributions for
the density measurements, while not identical, are similar in form and in
position between the quiescent and non-quiescent wind. However, when looking
at the 13.3 $R_{\odot}$ \- 15 $R_{\odot}$ radial bin, the distribution for
quiescent times peaks in an entirely different place when compared to non-
quiescent solar wind and its averages are lower. This bin shows average
densities of 1288 cm-3 for quiescent regions and 2207 cm-3 for non-quiescent
wind. However, like the temperature measurement, this difference may be a
result of low quiescent region counts in that particular radial bin. Further
orbits will help clarify the validity of this observation.

Figure 5: Scatter plots of proton $\mathcal{R}$ vs $\beta_{\parallel p}$ for
quiescent and non-quiescent solar wind at different radial distances. The
y-axes of the subplots are the proton temperature anisotropies and the x-axes
of the subplots are the plasma betas parallel to the background magnetic
field. The columns of this plot represents distributions for quiescent plots
on the left and non-quiescent plots on the rightmost column. The rows
represent radial bins with the topmost row being the innermost altitudes that
PSP reaches and lower rows represent higher altitudes. The color scale is the
Alfvén mach number and the contours represent point density.

Figure 6: Ion $\mathcal{R}$ vs $\beta_{\parallel p}$ plot for Figure 6 peaks.
The y-axis is the proton temperature anisotropy and the x-axis is the plasma
beta parallel to the background magnetic field. The peaks of each subplot in
Figure 6 are tracked through altitudes and plot relative to one another.
Theencounter 13 ICME wake is plotted in purple, this event is discussed in
Section 4.2. Solid lines are various instability thresholds past which solar
wind would become likely to grow waves. The black line is the ion cyclotron
instability, the blue is the mirror mode instability, and the green line is
the parallel firehose instability (Hellinger et al., 2006). These peaks are
based on 2d histograms of the data and the error bars are full width of the
distributions at half maximum.

###  3.5 Ion Temperature Anisotropy ($\mathcal{R}$) vs Plasma Beta
($\beta_{\parallel p}$) and Alfvénic Instabilities

This study also investigates proton plasma beta $\beta_{||p}$ and proton
temperature anisotropies $\mathcal{R}$. Figures 6 and 6 are log-scale scatter
plots of solar wind in $\mathcal{R}$ vs $\beta_{\parallel p}$ parameter space,
where $\mathcal{R}\equiv T_{\perp p}/T_{\parallel p}$ and $\beta_{\parallel
p}\equiv 2\mu_{0}nkT_{\parallel p}/B^{2}$, where the proton properties
$n,T_{\parallel p}$ are taken from proton measurements of SPAN-I. These plots
quantify the evolution of quiescent regions and non-quiescent wind and relate
these solar wind populations to plasma instabilities. Figures 6 and 6
represent different aspects of the same data. The left column of Figure 6 is a
set of $\mathcal{R}$ vs $\beta_{\parallel p}$ plots including every point of
data for which the proton core peak is within the SPAN FOV. These scattered
points are colored by the Alfvén mach number, the ratio of the solar wind
velocity magnitude in the Heliocentric RTN frame to the local Alfvén speed.
The contours overplotted onto the scatter plots represent normalized point
distributions where the contours bound regions in the parameter space where
points are most concentrated. Figure 6 on the right displays the positions of
the peaks of the $\mathcal{R}$ vs $\beta_{\parallel p}$ distributions as they
are measured at varying distances from the sun and tracks them through the
parameter space. From both plots, it is observed that both the quiescent and
non-quiescent solar wind are well-bounded by the ion cyclotron instability and
the parallel firehose instability thresholds (Hellinger et al., 2006). From
Figure 6, it is observed that the peaks of the quiescent regions distributions
are consistently farther into the stable portion of parameter space, to the
left of the non-quiescent solar wind. From Figure 6, it is observed that
quiescent region solar wind has a lower Alfvén Mach number than non-quiescent
solar wind at similar portions of the $\mathcal{R}$ vs $\beta_{\parallel p}$
parameter space. In both Figures 6 and 6 for the lowest altitude bin,
quiescent regions start from a different portion of parameter space but evolve
along a similar trajectory as non-quiescent wind. The distribution peaks in
Figure 6 are generated using a 2d-histogram of the scatter plot data in Figure
6. Because of the use of histograms, determined peak position can be affected
by where bins are placed and the size of those bins. The 2d histograms used to
generate this plot are NxN grids in log space and the number N chosen in
Figures 6 and 6 is 32. In order to ensure that the trends observed in Figure 6
are not a result of binning, this plot has been generated using varied bin
counts from N=10 to N=38. When changing the bin numbers the exact peak
positions and error bars do shift, but the general results shown in Figure 6
remain the same: quiescent wind begins at a different position in the
$\mathcal{R}$ vs $\beta_{\parallel p}$ space, with greater $\mathcal{R}$ and
lower $\beta_{\parallel p}$ and evolve along similar trajectories while
remaining in the more stable part of the space.

##  4 Discussion

###  4.1 Quiescent Region Turbulence and the Possibility of In-Situ Origins.

Figure 4 demonstrates that quiescent regions have magnetic field fluctuation
amplitudes $\delta B/\langle|B|\rangle$ lower than non-quiescent solar wind.
Along with lower amplitude field fluctuations, quiescent regions have lower
bulk flow velocity and cooler proton temperatures in the radial bins which
have the most statistics. This cooler temperature has implications for the
possible origins of quiescent regions. We suggest two distinct possibilities
exist for quiescent region origins: either they are generated in-situ as
plasma travels from the photosphere into the corona consistent with hypothesis
3 in Section 1.1, or they are launched from very close to the solar surface
and better retain their turbulent and bulk properties as they travel to the
spacecraft consistent with hypotheses 1 and 2 in Section 1.1.

If quiescent regions developed on the way to the spacecraft, this suggests the
existence of a baseline level of magnetic turbulence which, in quiescent
regions, is either damped, or is otherwise evacuated from the quiescent
region. In the case of turbulent damping, suppose there exists a property of
the solar wind in quiescent regions that causes Alfvénic fluctuations to be
damped, the damping of these fluctuations would presumably place some or all
of their wave energy into heating the protons in the quiescent regions leading
to an increase in proton temperature relative to non-quiescent wind or would
contribute to acceleration of the wind leading to an increase in velocity
relative to non-quiescent wind. However, from Figure 3 it is observed that the
opposite is the case and quiescent regions maintain a similar or cooler
temperature than non-quiescent wind and are slower than more turbulent
populations. Therefore we rule out this method of in-situ generation.

###  4.2 Region Evolution and Plasma Instabilities

From Figures 6 and 6, it is shown that quiescent regions occupy a more stable
portion of the $\mathcal{R}$ vs $\beta_{\parallel p}$ parameter space. With
decreasing radial distance, quiescent solar wind diverges from non-quiescent
wind in the parameter space. This may suggest that closer to the source of
quiescent regions the solar wind conditions are different. If these results
can be extrapolated further down into the solar corona, it may suggest that
quiescent regions emerge from field lines associated with low
$\beta_{\parallel p}$ and high $\mathcal{R}$. Despite their different
turbulent properties, both populations appear to evolve through the
$\mathcal{R}$ vs $\beta_{\parallel p}$ parameter space along similar
trajectories, suggesting that the same process dominates the evolution of
both. Figures 6 and 6 show that the trajectories of the quiescent and non-
quiescent plasma through the parameter space approach one another as Parker
Solar Probe moves radially outward, which could imply a second evolution
process driving the two populations together or that the two populations are
affected differently by the primary process.

However, these changes could also be a result of identifying different
populations of quiescent-like wind that occur at different radial distances.
For instance, during PSPs 13th orbit, a large Interplanetary Coronal Mass
Ejection (ICME) passed over PSP. In the following days, PSP observed a region
where the plasma density dropped significantly in the wake behind the path of
the ICME (Romeo et al., 2023). This region of low plasma density in the wake
of the ICME was marked as quiescent by the detection algorithm described in
Section 2.2. Because this plasma density dropout was clearly associated with
the ICME and its position in the $\mathcal{R}$ vs $\beta_{\parallel p}$
parameter space is substantially different from the rest of the quiescent
regions (shown in Figure 6), it was excluded from the dataset used in this
study. If multiple sources for quiescent wind exist; for example if some
quiescent wind emerges from between switchback patches as in Bale et al.
(2019, 2021, 2023) and others emerge from sources like the encounter 13 ICME,
the properties of those sources would be mixed in this study.

The apparent convergence of the trajectories of quiescent and non-quiescent
wind with radial distance seen in Figures 6 and 6 may also be an effect of
selection biasing with respect to the solar cycle. The innermost radial bin is
dominated by later solar encounters 10 through 16 which are more towards solar
maximum than earlier solar encounters 1 through 9. PSP flies through varied
solar wind streams as the Sun approaches solar maximum. PSP sampling these
different streams may be causing a difference in the properties of solar wind
which we observe as an increasing difference in the $\mathcal{R}$ vs
$\beta_{\parallel p}$ positions for quiescent and non-quiescent wind in the
innermost radial bin.

Figure 7: Radial evolution of quiescent region occurence. The vertical axis
represent the percentage of solar wind plasma which has been labeled as
quiescent out of the total plasma that PSP has flown through. The percentages
presented are ratios of the quiescent meters of plasma flown through over the
total meters of plasma flown through at that altitude. The fit exponential
falls off with a scale of $\sim 30.7$ $R_{\odot}$ indicating a rapid decay of
quiescent plasma with solar wind distance.

Quiescent regions are not reported in studies from missions which orbit
farther out in the solar system. In order to pull out the shape of the Parker
Spiral in measurements taken at 1 AU, one must average over some timescale
(Ness & Wilcox, 1964). However, with these regions, their fields are roughly
along the Parker Spiral direction without the need to average. The fact that
missions farther out do not observe quiescent regions suggests that quiescent
regions must decay, either by internally developing magnetic field
fluctuations or else merging with the surrounding ambient solar wind
turbulence. We attempt to quantify this decay rate by observing the meters of
plasma flown through by PSP as a function of radial distance. Figure 7 shows
the percentage of meters of plasma that quiescent regions make up at each
radial bin described in Table 1. The points in Figure 7 are placed at the
center of each radial bin. An exponential is fit to the curve to roughly
quantify a “scale height” for quiescent regions. This fit reveals an
exponential with a $30.7$ $R_{\odot}$ distance scale. This rapid decrease in
quiescent plasma helps explain why these regions have not been reported on
during previous missions. If the exponential fit in Figure 7 is extrapolated
out to $60$ and $65$ $R_{\odot}$, the altitudes of Solar Orbiter and the
Helios probes, the fit predicts $\sim 3.4\%$ of the solar wind should be
quiescent. With other missions like Bepi-Columbo orbiting at Mercury ($\sim
86$ $R_{\odot}$), missions orbiting at Venus ($\sim 155$ $R_{\odot}$), and
many at $\geq 1$ AU ($\sim 215$ $R_{\odot}$), it is unsurprising that these
regions have gone unreported by previous studies.

However, the mechanism by which quiescent regions decay remains unknown. One
possibility, based on the trajectories shown Figure 6, is that quiescent
regions may become unstable to plasma instabilities as the solar wind travels
outward. For example, the trajectory of both the quiescent and non-quiescent
solar wind in Figure 6 is towards the parallel and oblique firehose
instabilities. If these regions do become unstable as the solar wind moves
outward these instabilities would generate fluctuations in the plasma which
would eventually destroy the quiescent signatures of these solar wind regions
and may serve to homogenize their turbulent and thermal properties.
Alternatively, it may be the case that non-quiescent wind overtakes quiescent
wind, causing the two populations to merge. This alternative requires further
investigation.

Previous studies of the solar wind evolution such as Matteini et al. (2007)
show that the fast solar wind shows a similar trajectory to the trajectories
for quiescent and non-quiescent wind shown in Figure 6, but in that study the
fast solar wind evolves through the parameter space on a much larger radial
scale (0.3 AU to 1 AU). Matteini et al. (2007) also observed that there was no
obvious trend derivable from the slow solar wind except that the slow solar
wind tended to hover around the firehose instabilities. As such, the data
presented here may be consistent with observations from the Helios missions
which found that slow solar wind tended to approach the firehose instabilities
(Marsch et al., 2006; Matteini et al., 2007). This may indicate that the wind
observed in this study is consistent with that of the slow wind and may
indicate that slow solar wind evolves along similar trajectories as fast solar
wind, but on a shorter radial scale.

###  4.3 Quiescent Region Magnetic Field Origins

For the closest radial bins, quiescent regions have higher proton temperature
anisotropies $\mathcal{R}$ than non-quiescent wind at the same altitude above
the solar surface (Figures 6 and 6). The innermost radial bin is unique
because it is both narrow in radial distance (1.7 $R_{\odot}$ wide), and has a
high number of quiescent regions per $R_{\odot}$ (44 quiescent regions per 1.7
$R_{\odot}$) compared to the next radial bin (92 quiescent regions per 10
$R_{\odot}$). This bin has a well-defined peak in the $\mathcal{R}$ vs
$\beta_{\parallel p}$ space for both the quiescent solar wind as well as the
non-quiescent solar wind. This increase in temperature anisotropy is
consistent with a solar wind magnetic field that has not undergone as much
expansion as non-quiescent wind during its journey to the spacecraft.

Consider the case of two adjacent magnetic mirrors whose mirror ends are
placed into a source of thermal plasma like the solar surface and whose open
ends extend into the solar wind. As protons run up the diverging magnetic
field lines, they conserve their adiabatic invariant due to gyro motion, and
transfer perpendicular kinetic energy into parallel kinetic energy. If these
two magnetic mirrors have different mirror ratios, but comparable field
strengths at the base, then the protons which are in the magnetic field which
remains more compressed will retain more of their perpendicular kinetic
energy, this would manifest as higher temperature anisotropies in the higher
field streams. It is demonstrated in Figures 3d1,3d2, and 3d3 that the
magnetic field magnitude for quiescent regions is higher than non-quiescent
wind in each radial bin. This is consistent with low levels of magnetic field
expansion in quiescent regions as field line density is the definition of
magnetic field magnitude. Therefore we suggest that quiescent solar wind
regions may exist on magnetic field lines which have undergone less expansion,
compared to non-quiescent solar wind, at distances up to a few tens of solar
radii ($R_{\odot}$).

Alternatively, previous studies by Verniero et al. (2020, 2022) have shown
that certain cases of quiescent intervals are associated with the presence of
ion beams which are in turn associated with ion scale waves. If, for the
closest radial bins, quiescent regions are associated with ion scale waves
generated by ion beams, wave-particle interactions between protons and proton
cyclotron waves (Cranmer, 2000; Vech et al., 2021; Bowen et al., 2022, 2024)
may explain the enhanced perpendicular temperature observed by quiescent
regions in this radial bin. However, it has not been shown in this study that
quiescent regions are associated with ion beams generally and investigations
into this possibility are outside the scope of this study.

##  5 Conclusion

This study demonstrates that quiescent regions have statistically significant
differences in both turbulent and bulk properties and that the evolution of
quiescent and non-quiescent wind follows a similar path through instability
parameter space. The higher temperature anisotropies and higher magnetic field
strengths of quiescent regions in the lowest radial altitude are consistent
with magnetic field lines which remain compressed relative to magnetic field
lines of non-quiescent solar wind. This study has presented evidence against
hypothesis 3 detailed in Section 1.1. A version of hypothesis 3 which involves
damping of turbulent fluctuations is unlikely based on energy conservation of
the magnetic fluctuations. In quiescent regions, the proton temperature and
bulk flow velocity are shown to be depressed or comparable to non-quiescent
solar wind, inconsistent with damping of magnetic field fluctuations. This
study also estimates a scale height for quiescent regions which characterizes
the decay rate of quiescent regions in the solar wind. It is shown that
quiescent regions evolve similarly to non-quiescent solar wind but occupy
different portions of the $\mathcal{R}$ vs $\beta_{\parallel p}$ parameter
space. This may suggest that quiescent region magnetic field lines are
different at the respective points of origin for each solar wind population,
consistent with hypotheses 1 and 2 discussed in Section 1.1.

While not all possible variations of hypothesis 3 have been explored in this
work and has thus not been fully ruled out, we present strong evidence to rule
out one scenario and present evidence in favor of hypotheses 1 and 2. Further
work is needed to distinguish between hypotheses 1 and 2 as they are
variations of hypotheses which have quiescent sources located near the solar
surface. Work which distinguishes between hypotheses 1 and 2 will involve
dynamics of coronal plasma located very close to the source surface.

Parker Solar Probe was designed, built, and is now operated by the Johns
Hopkins Applied Physics Laboratory as part of NASA’s Living with a Star (LWS)
program (contract NNN06AA01C). Support from the LWS management and technical
team has played a critical role in the success of the Parker Solar Probe
mission. B.M.S. is funded by NASA’s Future Investigators in NASA Earth and
Space Science and Technology (FINESST) grant (grant number 80NSSC23K1627).
J.L.V. acknowledges support from NASA PSP-GI grant 80NSSC23K0208. Data
manipulation was performed using the Numpy (Harris et al., 2020) and Scipy
(Virtanen et al., 2020) Python packages. Figures in this work are produced
using the Python package Matplotlib (Hunter, 2007).

## References

  * Bale et al. (2016) Bale, S. D., Goetz, K., Harvey, P. R., et al. 2016, Space Science Reviews, 204, 49, doi: 10.1007/s11214-016-0244-5
  * Bale et al. (2019) Bale, S. D., Badman, S. T., Bonnell, J. W., et al. 2019, Nature, 576, 237, doi: 10.1038/s41586-019-1818-7
  * Bale et al. (2021) Bale, S. D., Horbury, T. S., Velli, M., et al. 2021, The Astrophysical Journal, 923, 174, doi: 10.3847/1538-4357/ac2d8c
  * Bale et al. (2023) Bale, S. D., Drake, J. F., McManus, M. D., et al. 2023, Nature, 618, 252, doi: 10.1038/s41586-023-05955-3
  * Bowen et al. (2022) Bowen, T. A., Chandran, B. D. G., Squire, J., et al. 2022, Phys. Rev. Lett., 129, 165101, doi: 10.1103/PhysRevLett.129.165101
  * Bowen et al. (2024) Bowen, T. A., Bale, S. D., Chandran, B. D. G., et al. 2024, Nature Astronomy, 8, 482, doi: 10.1038/s41550-023-02186-4
  * Case et al. (2020) Case, A. W., Kasper, J. C., Stevens, M. L., et al. 2020, The Astrophysical Journal Supplement, 246, 43, doi: 10.3847/1538-4365/ab5a7b
  * Cranmer (2000) Cranmer, S. R. 2000, ApJ, 532, 1197, doi: 10.1086/308620
  * Dudok de Wit et al. (2020) Dudok de Wit, T., Krasnoselskikh, V. V., Bale, S. D., et al. 2020, The Astrophysical Journal Supplement, 246, 39, doi: 10.3847/1538-4365/ab5853
  * Fox et al. (2016) Fox, N. J., Velli, M. C., Bale, S. D., et al. 2016, Space Science Reviews, 204, 7, doi: 10.1007/s11214-015-0211-6
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357–362, doi: 10.1038/s41586-020-2649-2
  * Hellinger et al. (2006) Hellinger, P., Trávníček, P., Kasper, J. C., & Lazarus, A. J. 2006, Geophys. Res. Lett., 33, L09101, doi: 10.1029/2006GL025925
  * Horbury et al. (2020) Horbury, T. S., Woolley, T., Laker, R., et al. 2020, The Astrophysical Journal Supplement, 246, 45, doi: 10.3847/1538-4365/ab5b15
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Jagarlamudi et al. (2023) Jagarlamudi, V. K., Raouafi, N. E., Bourouaine, S., et al. 2023, ApJ, 950, L7, doi: 10.3847/2041-8213/acd778
  * Kasper et al. (2016) Kasper, J. C., Abiad, R., Austin, G., et al. 2016, Space Science Reviews, 204, 131, doi: 10.1007/s11214-015-0206-3
  * Kasper et al. (2019) Kasper, J. C., Bale, S. D., Belcher, J. W., et al. 2019, Nature, 576, 228, doi: 10.1038/s41586-019-1813-z
  * Livi et al. (2021) Livi, R., Larson, D. E., Kasper, J. C., et al. 2021, Earth and Space Science Open Archive, 20, doi: 10.1002/essoar.10508651.1
  * Livi et al. (2022) Livi, R., Larson, D. E., Kasper, J. C., et al. 2022, ApJ, 938, 138, doi: 10.3847/1538-4357/ac93f5
  * Marsch et al. (2006) Marsch, E., Zhao, L., & Tu, C. Y. 2006, Annales Geophysicae, 24, 2057, doi: 10.5194/angeo-24-2057-2006
  * Matteini et al. (2007) Matteini, L., Landi, S., Hellinger, P., et al. 2007, Geophys. Res. Lett., 34, L20105, doi: 10.1029/2007GL030920
  * Ness & Wilcox (1964) Ness, N. F., & Wilcox, J. M. 1964, Phys. Rev. Lett., 13, 461, doi: 10.1103/PhysRevLett.13.461
  * Romeo et al. (2023) Romeo, O. M., Braga, C. R., Badman, S. T., et al. 2023, ApJ, 954, 168, doi: 10.3847/1538-4357/ace62e
  * Short et al. (2022) Short, B., Malaspina, D. M., Halekas, J., et al. 2022, The Astrophysical Journal, 940, 45, doi: 10.3847/1538-4357/ac97e4
  * Vech et al. (2021) Vech, D., Martinović, M. M., Klein, K. G., et al. 2021, A&A, 650, A10, doi: 10.1051/0004-6361/202039296
  * Verniero et al. (2020) Verniero, J. L., Larson, D. E., Livi, R., et al. 2020, ApJS, 248, 5, doi: 10.3847/1538-4365/ab86af
  * Verniero et al. (2022) Verniero, J. L., Chandran, B. D. G., Larson, D. E., et al. 2022, ApJ, 924, 112, doi: 10.3847/1538-4357/ac36d5
  * Verscharen & Marsch (2011) Verscharen, D., & Marsch, E. 2011, Annales Geophysicae, 29, 909, doi: 10.5194/angeo-29-909-2011
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2


[.25in].25in.15in.

# Masked Autoencoders As The Unified Learners For Pre-Trained Sentence
Representation

Alexander Liu†, Samuel Yang‡
${\dagger}$: The Hong Kong University of Science and Technology, Kowloon, Hong
Kong
${\ddagger}$: University of Science and Technology of China, Hefei, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite the progresses on pre-trained language models, there is a lack of
unified frameworks for pre-trained sentence representation. As such, it calls
for different pre-training methods for specific scenarios, and the pre-trained
models are likely to be limited by their universality and representation
quality. In this work, we extend the recently proposed MAE style pre-training
strategy, RetroMAE (Liu and Shao, 2022), such that it may effectively support
a wide variety of sentence representation tasks. The extended framework
consists of two stages, with RetroMAE conducted throughout the process. The
first stage performs RetroMAE over generic corpora, like Wikipedia,
BookCorpus, etc., from which the base model is learned. The second stage takes
place on domain-specific data, e.g., MS MARCO and NLI, where the base model is
continuingly trained based on RetroMAE and contrastive learning. The pre-
training outputs at the two stages may serve different applications, whose
effectiveness are verified with comprehensive experiments. Concretely, the
base model are proved to be effective for zero-shot retrieval, with remarkable
performances achieved on BEIR benchmark (Thakur et al., 2021b). The
continuingly pre-trained models further benefit more downstream tasks,
including the domain-specific dense retrieval on MS MARCO (Nguyen et al.,
2016), Natural Questions (Kwiatkowski et al., 2019), and the sentence
embeddings’ quality for standard STS (Agirre et al., 2016; Cer et al., 2017;
Marelli et al., 2014) and transfer tasks in SentEval (Conneau and Kiela,
2018). The empirical insights of this work may inspire the future design of
sentence representation pre-training. Our pre-trained models and source code
will be released to the public communities.

##  1 Introduction

Sentence representation is a fundamental issue in natural language processing.
It transforms each input sentence into its embedding, where semantically
similar instances can be located close to each other in the latent space.
Thanks to this property, sentence representation models are utilized in many
important applications, such as semantic matching, dense retrieval, and
recommender systems. With the development of deep learning, deep neural
networks are widely utilized for sentence representation tasks. However, the
training of deep models usually call for massive scales of labeled data, which
can be prohibitive in many real-world scenarios.

The pre-trained language models are regarded as a promising way to alleviate
the data scarcity problem. In recent years, the pre-trained language models
are specifically tailored for sentence representation purposes. For example,
there have been pre-training methods developed for dense retrievers (Chang et
al., 2020; Guu et al., 2020; Gao and Callan, 2021a, b; Lu et al., 2021);
besides, there have also been models pre-trained for tasks like semantic
matching (Reimers and Gurevych, 2019; Thakur et al., 2021a; Gao et al., 2021).
However, the existing pre-trained models rely on differentiated pre-training
workflow, where a pre-trained model for one specific task contributes little
to other scenarios. Such a problem is unfavorable to both universality and
representation quality; indeed, a unified pre-training framework will be
highly desirable for sentence representation.

The MAE-style algorithms become increasingly popular recently: it was
recognized to be a strong pre-trainer for various tasks in computer vision
communities (He et al., 2022; Feichtenhofer et al., 2022); later on, it was
also proved to be extremely effective for zero-shot dense retrieval in
RetroMAE (Liu and Shao, 2022). In this work, we demonstrate that RetroMAE may
serve as the unified pre-training framework for sentence representation, as it
can be easily extended to support various scenarios. The extension is inspired
by the previous practice on generic pre-training algorithms (Gururangan et
al., 2020), where a two-stage workflow is conducted. The first stage training
takes place on generic corpora, like Wikipedia and BookCorpus, where the base
model is learned through RetroMAE. The second stage leverages domain-specific
corpora, such as passages from MS MARCO (Nguyen et al., 2016) and sentence
pairs from NLI datasets (Bowman et al., 2015; Williams et al., 2017), where
the model is continuingly trained through RetroMAE and contrastive learning.
The extended pre-training workflow is conceptually simple and empirically
competitive, where the generated models may achieve remarkable performances in
different application scenarios. Particularly, the following empirical
insights are derived from our experiments.

  * •

The RetroMAE pre-training on generic corpora is critical, as it initializes
the model with a strong capacity on semantic encoding. Our reproduction of the
original RetroMAE remarkably achieves 0.452 on BEIR benchmark in total
average, which is way beyond any existing models with similar model sizes and
pre-training data.

  * •

The continued in-domain pre-training (also known as post pre-training) is
important to promote downstream tasks’ performances. For example, the
retrieval quality (with simple ANCE style fine-tuning) can be notably lifted
to 0.393 (MRR@10) and 0.985 (Recall@1000) for MS MARCO passage retrieval task,
compared with 0.378 (MRR@10) and 0.980 (Recall@1000) from the base model pre-
trained on generic corpora. These results are highly competitive on the
corresponding benchmark, which are even comparable to those based on much more
expensive knowledge distillation (Ren et al., 2021) and adversarial learning
(Zhang et al., 2021).

  * •

The contrastive learning’s impact is conditional. On one hand, it helps to
generate high quality sentence embeddings for standard STS (Agirre et al.,
2016; Cer et al., 2017; Marelli et al., 2014) and transfer tasks in SentEval
(Conneau and Kiela, 2018); on the other hand, it is found to bring limited
benefit to dense retrieval tasks. We conjecture that it’s the quality of
relevance label that makes the difference. Traditional self-contrastive
learning (Chang et al., 2020; Guu et al., 2020), which rely on data
augmentation to generate positive samples, are prone to inferior label
quality. Thus, they could be in vain at the presence of stronger pre-trainers,
like RetroMAE.

To conclude, the main points of this paper are summarized as follows. We
extend RetroMAE with continued in-domain pre-training and contrastive
learning, such that it may contribute to various sentence representation
tasks, including zero-shot retrieval, domain-specific retrieval, standard STS
and transfer tasks in SentEval. We perform comprehensive experimental studies,
whose results reveal the impacts introduced by RetroMAE, continued in-domain
pre-training, and contrastive learning. The empirical insights may inspire the
design of sentence representation pre-training in the future.

##  2 Methodology

In this section, we’ll present the preliminaries about RetroMAE in the first
place. Then, we’ll make discussions about our effort to extend it for
different downstream tasks.

###  2.1 Preliminaries

RetroMAE is to encode the masked sentence into its embedding, and then recover
the original sentence based on the masked input and the sentence embedding. It
is highlighted for two features: 1) the asymmetric model structures, and 2)
the asymmetric masking ratios about the encoder and decoder. The training
workflow of RetroMAE is shown as Figure 1. For the encoding operation, the
input sentence $X$ is masked as $\tilde{X}_{enc}$, with a small fraction of
its tokens randomly replaced by the special token [M] (the shadowed
rectangles). The masked input is transformed into sentence embedding
$\mathbf{h}_{\tilde{X}}$ based on the encoding network $\Phi^{enc}(\cdot)$:

$\mathbf{h}_{\tilde{X}}\leftarrow\Phi_{enc}(\tilde{X}_{enc}).$ (1)

Note that a relatively moderate masking ratio is applied to the encoding
operation, where 15$\sim$30% of the input tokens are sampled for masking. The
encoder is a BERT-base scale transformer so as to ensure the quality of
sentence representation.

Figure 1: The two-stage pre-training workflow with RetroMAE. The first stage
performs RetroMAE on generic data; the base model will effectively support
tasks like zero-shot retrieval. The second stage takes place on domain-
specific data; the base model is further trained on domain-specific data with
RetroMAE and contrastive learning, which benefits tasks like in-domain
retrieval and STS.

The decoding step is to recover the original sentence from the sentence
embedding and the masked input. Particularly, the input sentence $X$ is
polluted again for decoder as $\tilde{X}_{dec}$ with a different mask. The
masking ratio is chosen to be more aggressive than the one used by encoder,
with 50$\sim$70% of the input tokens replaced by the special token [M]. The
sentence embedding from the encoder and the masked input of the decoder are
combined into the following sequence:

$\mathbf{H}_{\tilde{X}_{dec}}\leftarrow[\mathbf{h}_{\tilde{X}},\mathbf{e}_{x_{1}}+\mathbf{p}_{1},...,\mathbf{e}_{x_{N}}+\mathbf{p}_{N}],$
(2)

where $\mathbf{e}_{x_{i}}$ stands for the embedding of $x_{i}$, to which an
extra position embedding $\mathbf{p}_{i}$ is added. The decoder $\Phi_{dec}$
is learned to recover the original sentence $X$ by minimizing the following
loss function:

$\mathcal{L}_{dec}=\sum_{x_{i}\in\text{masked}}\textit{cross-
entropy}(x_{i}|\Phi_{dec}(\mathbf{H}_{\tilde{X}_{dec}})).$ (3)

One notable feature about the decoding network is that it merely consists of a
one-layer transformer. For one thing, it may save a great deal of computation
cost; for another thing, it will ensure the decoding operation to be
challenging enough, which forces the generation of high-quality sentence
embedding from the encoder.

The basic decoding may only derive training signals (the cross-entropy losses)
from the masked tokens; besides, every masked token is always reconstructed
based on the same context, i.e., $\mathbf{H}_{\tilde{X}_{dec}}$. As a result,
it is prone to low sample efficiency. To mitigate this problem, the enhanced
decoding is introduced, where training signals can be derived from the entire
input sentence, and the prediction of the masked tokens can be performed based
on diversified contexts. Particularly, there are two input sequences:
$\mathbf{H}_{1}$ (query stream) and $\mathbf{H}_{2}$ (context stream),
generated for the decoding operation:

$\begin{gathered}\mathbf{H}_{1}\leftarrow[\mathbf{h}_{\tilde{X}}+\mathbf{p}_{0},...,\mathbf{h}_{\tilde{X}}+\mathbf{p}_{N}],\\\
\mathbf{H}_{2}\leftarrow[\mathbf{h}_{\tilde{X}},\mathbf{e}_{x_{1}}+\mathbf{p}_{1},...,\mathbf{e}_{x_{N}}+\mathbf{p}_{N}].\end{gathered}$
(4)

in which $\mathbf{h}_{\tilde{X}}$ is the sentence embedding from encoder,
$\mathbf{e}_{x_{i}}$ is the token embedding (all tokens are left non-masked in
the place), and $\mathbf{p}_{i}$ is the position embedding. Let
$\mathbf{M}\in\mathbb{R}^{L\times L}$ be the position-specific attention mask,
with which the self-attention operation is conducted as follows:

$\begin{gathered}\mathbf{Q}=\mathbf{H}_{1}\mathbf{W}^{Q},\mathbf{K}=\mathbf{H}_{2}\mathbf{W}^{K},\mathbf{V}=\mathbf{H}_{2}\mathbf{W}^{V};\\\
\mathbf{M}_{ij}=\begin{cases}0,~{}~{}~{}~{}~{}~{}\text{can be attended},\\\
-\infty,~{}\text{masked};\end{cases}\\\
\mathbf{H}^{o}_{dec}=\mathrm{softmax}(\frac{\mathbf{Q}^{T}\mathbf{K}}{\sqrt{d}}+\mathbf{M})\mathbf{V}.\end{gathered}$
(5)

The decoding output $\mathbf{H}^{o}_{dec}$ is used to reconstruct the original
sentence (operations like layer-norm, FFN, and residual connections are
omitted for simplicity). Finally, the decoder is learned to optimize the
following objective:

$\mathcal{L}_{dec}=\sum\nolimits_{x_{i}\in X}\textit{cross-
entropy}(x_{i}|\mathbf{H}^{o}_{dec}).$ (6)

Because the decoding network has one single layer, each token $x_{i}$ can only
be recovered from the context visible to the $i$-th row of the matrix
$\mathbf{M}$, which are generated by the following rules:

$\begin{gathered}\mathbf{M}_{ij}=\begin{cases}0,~{}~{}x_{j}\in s(X_{\neq
i}),~{}\text{or}~{}j_{{|i\neq 0}}=0\\\
-\infty,~{}~{}\text{otherwise}.\end{cases}\end{gathered}$ (7)

The sampled tokens, $s(X_{\neq i})$, and the first elements (except the first
row) will be visible when recovering $x_{i}$. All diagonal elements, i.e.,
$x_{i}$ in the $i$-th row, will be excluded, such that each token may not
attend to itself during decoding.

As mentioned, the first stage pre-training performs RetroMAE on generic data,
like Wikipedia and BookCorpus. It worth noting that the masked tokens for the
encoding network will also be reconstructed in the same way as the
conventional masked language modeling (MLM). The corresponding loss is denoted
as $\mathcal{L}_{enc}$. The encoder and decoder are jointly trained to
minimize the summation of $\mathcal{L}_{dec}$ and $\mathcal{L}_{enc}$.
Finally, the well-trained encoding network is utilized as the base model,
which will serve applications like zero-shot dense retrieval.

|  BERT |  RoBERTa |  SimCSE |  DiffCSE |  SEED |  Condenser |  SentMAE† |  GTR (base) |  GTR (large) |  GTR (xlarge) |  Contriever
---|---|---|---|---|---|---|---|---|---|---|---
TREC-COVID |  0.615 |  0.649 |  0.460 |  0.492 |  0.627 |  0.750 |  0.772 |  0.539 |  0.557 |  0.584 |  0.596
BioASQ |  0.253 |  0.279 |  0.263 |  0.258 |  0.308 |  0.322 |  0.421 |  0.271 |  0.320 |  0.317 |  –
NFCorpus |  0.260 |  0.243 |  0.260 |  0.259 |  0.278 |  0.277 |  0.308 |  0.308 |  0.329 |  0.343 |  0.328
NQ |  0.467 |  0.413 |  0.435 |  0.412 |  0.446 |  0.486 |  0.518 |  0.495 |  0.547 |  0.559 |  0.498
HotpotQA |  0.488 |  0.448 |  0.502 |  0.499 |  0.541 |  0.538 |  0.635 |  0.535 |  0.579 |  0.591 |  0.638
FiQA-2018 |  0.252 |  0.291 |  0.250 |  0.229 |  0.259 |  0.259 |  0.316 |  0.349 |  0.424 |  0.444 |  0.329
Signal-1M(RT) |  0.204 |  0.229 |  0.262 |  0.260 |  0.256 |  0.261 |  0.265 |  0.261 |  0.265 |  0.268 |  –
TREC-NEWS |  0.362 |  0.385 |  0.356 |  0.363 |  0.358 |  0.376 |  0.428 |  0.337 |  0.343 |  0.350 |  –
Robust04 |  0.351 |  0.384 |  0.330 |  0.343 |  0.365 |  0.349 |  0.447 |  0.437 |  0.470 |  0.479 |  –
ArguAna |  0.265 |  0.395 |  0.413 |  0.468 |  0.389 |  0.298 |  0.433 |  0.511 |  0.525 |  0.531 |  0.446
Touche-2020 |  0.259 |  0.299 |  0.159 |  0.168 |  0.225 |  0.248 |  0.237 |  0.205 |  0.219 |  0.230 |  0.230
CQADupStack |  0.282 |  0.278 |  0.290 |  0.305 |  0.290 |  0.347 |  0.317 |  0.357 |  0.384 |  0.388 |  0.345
Quora |  0.787 |  0.509 |  0.844 |  0.850 |  0.852 |  0.853 |  0.847 |  0.881 |  0.890 |  0.890 |  0.865
DBPedia |  0.314 |  0.275 |  0.314 |  0.303 |  0.330 |  0.339 |  0.390 |  0.347 |  0.391 |  0.396 |  0.413
SCIDOCS |  0.113 |  0.111 |  0.124 |  0.125 |  0.124 |  0.133 |  0.150 |  0.149 |  0.158 |  0.159 |  0.165
FEVER |  0.682 |  0.683 |  0.623 |  0.641 |  0.641 |  0.691 |  0.774 |  0.660 |  0.712 |  0.717 |  0.758
C-FEVER |  0.187 |  0.222 |  0.211 |  0.200 |  0.176 |  0.211 |  0.232 |  0.241 |  0.262 |  0.270 |  0.237
SciFact |  0.533 |  0.539 |  0.554 |  0.523 |  0.575 |  0.593 |  0.653 |  0.600 |  0.639 |  0.635 |  0.677
AVERAGE |  0.371 |  0.368 |  0.369 |  0.372 |  0.391 |  0.407 |  0.452 |  0.416 |  0.445 |  0.453 |  –
Table 1: Zero-shot dense retrieval performances on BEIR benchmark (measured by
NDCG@10). The methods on the left half are based on a BERT-base scale encoding
network and pre-trained on Wikipedia and BookCorpus; the methods on the right
half are based on enlarged model architectures and substantially increased
pre-training data. The highest performances are marked in bold; the highest
performances on the left half are underlined.

###  2.2 Two-stage Pre-Training

A two-stage pre-training workflow is conducted as Figure 1. In the first
stage, the model is trained by RetroMAE on generic corpus. We use English
Wikipedia and BookCorpus, which are also the pre-training data used by BERT
pre-training. We use the following hyper-parameter settings for the first
stage pre-training: the encoder is a BERT-base scale transformer and the
decoder is a single-layer transformer; the masking ratio is 0.3 for encoder
and 0.5 for decoder. Following the practice in BEIR benchmark (Thakur et al.,
2021b), the base model from the first stage is fine-tuned with MS MARCO
queries for zero-shot retrieval in other domains.

In the second stage, the base model continues to be trained to deal with more
specific downstream tasks. We consider two typical scenarios in this place:
in-domain retrieval and sentence embedding for tasks related to semantic
matching. The model is trained on MS MARCO corpus (Nguyen et al., 2016) and
Natural Questions (Kwiatkowski et al., 2019) corpus respectively, so as to
support the in-domain retrieval tasks on the corresponding datasets. Besides,
the model is trained with sentence pairs from NLI datasets (Bowman et al.,
2015; Williams et al., 2017) such that the generated sentence embeddings may
serve scenarios like semantic matching. For the second stage, we make use of
two different pre-training strategies: on one hand, RetroMAE may continue to
be conducted on the domain-specific corpus; on the other hand, the contrastive
learning may also be leveraged. The contrastive learning loss
$\mathcal{L}_{ctr}$ is formulated by the following equation:

$\mathcal{L}_{ctr}=-\sum\nolimits_{\mathbf{h}_{i}^{+}}\log\frac{e^{sim(\mathbf{h}_{i},\mathbf{h}_{i}^{+})/\tau}}{\sum_{\mathbf{h}_{i}^{-}}e^{sim(sim(\mathbf{h}_{i},\mathbf{h}_{i}^{-})/\tau}},$
(8)

where $\mathbf{h}_{i}^{+}$, $\mathbf{h}_{i}^{-}$ are the positive and negative
samples to $\mathbf{h}_{i}$, and $\tau$ is the temperature parameter. While
pre-training for in-domain retrieval, we leverage data augmentation to
generate the positive samples: following the approach in coCondenser (Gao and
Callan, 2021b), two sentences sampled from the same article are used as
$\mathbf{h}_{i}$ and $\mathbf{h}_{i}^{+}$. When targeting on the generation of
sentence embeddings, we directly leverage the sentence pairs from NLI datasets
as $\mathbf{h}_{i}$ and $\mathbf{h}_{i}^{+}$. For both scenarios, we use in-
batch negative samples; that’s to say, one training instance (e.g.,
$\mathbf{h}_{i}$) will use others’ positive samples within the same training
batch as its negative samples ($\\{\mathbf{h}^{+}_{j}\\}_{i\neq{j}}$).

We empirically examine the following combinations of RetroMAE and contrastive
learning. For in-domain retrieval, we consider using RetroMAE alone as Eq. 9
(1), and the summation of RetroMAE and contrastive learning as Eq. 9 (2):

$(1)~{}\underbrace{\mathcal{L}_{enc}+\mathcal{L}_{dec}}_{\text{RetroMAE}},~{}~{}(2)~{}\underbrace{(\mathcal{L}_{enc}+\mathcal{L}_{dec})}_{\text{RetroMAE}}+\mathcal{L}_{ctr}.$
(9)

The first option is the simple continuation of RetroMAE on domain-specific
corpus. The second option follows the same spirit as coCondenser, except that
the pre-training task is switched from Condenser to RetroMAE. While used for
the generation of sentence embeddings, the contrastive learning will always be
enabled.

##  3 Experiments

We conduct three sets of experimental studies to evaluate the pre-trained
models’ effectiveness. (1) For the base model pre-trained on the generic
corpus, we evaluate its zero-shot retrieval quality with BEIR benchmark
(Thakur et al., 2021b). (2) For the continuingly pre-trained model on domain-
specific data, we evaluate the in-domain retrieval quality on MS MARCO (Nguyen
et al., 2016) and Natural Questions (Kwiatkowski et al., 2019). (3) We also
evaluate the sentence embeddings’ quality based on the semantic textual
similarity (STS) (Agirre et al., 2016; Cer et al., 2017; Marelli et al., 2014)
and transfer tasks in SentEval (Conneau and Kiela, 2018).

---
| MS MARCO | Natural Questions
Methods |  MRR@10 |  MRR@100 |  R@10 |  R@100 |  R@1000 |  R@10 |  R@20 |  R@30 |  R@50 |  R@100
BERT |  0.346 |  0.357 |  0.622 |  0.873 |  0.964 |  0.788 |  0.823 |  0.846 |  0.860 |  0.878
RoBERTa |  0.343 |  0.354 |  0.613 |  0.871 |  0.964 |  0.763 |  0.805 |  0.828 |  0.845 |  0.870
DeBERTa |  0.340 |  0.351 |  0.602 |  0.872 |  0.967 |  0.765 |  0.810 |  0.829 |  0.848 |  0.870
SimCSE |  0.352 |  0.362 |  0.628 |  0.885 |  0.974 |  0.774 |  0.819 |  0.842 |  0.863 |  0.886
LaPraDoR |  0.346 |  0.356 |  0.613 |  0.876 |  0.964 |  0.780 |  0.825 |  0.842 |  0.859 |  0.877
DiffCSE |  0.346 |  0.357 |  0.622 |  0.875 |  0.965 |  0.785 |  0.825 |  0.841 |  0.862 |  0.878
SEED |  0.354 |  0.365 |  0.626 |  0.881 |  0.969 |  0.780 |  0.826 |  0.845 |  0.868 |  0.887
Condenser |  0.364 |  0.374 |  0.639 |  0.891 |  0.972 |  0.790 |  0.833 |  0.852 |  0.867 |  0.883
SentMAE† |  0.378 |  0.389 |  0.666 |  0.909 |  0.980 |  0.804 |  0.844 |  0.863 |  0.878 |  0.894
Co-Conderser |  0.382 |  – |  – |  – |  0.984 |  – |  0.843 |  – |  – |  0.890
SentMAE‡ |  0.393 |  0.403 |  0.675 |  0.918 |  0.985 |  0.805 |  0845 |  0.863 |  0.878 |  0.895
SentMAE${}^{\ddagger}_{co}$ |  0.389 |  0.398 |  0.674 |  0.915 |  0.985 |  – |  – |  – |  – |  –
Table 2: In-domain retrieval based on ANCE fine-tuning. The models in the
upper half are pre-trained with Wikipedia and BookCorpus; the models in the
lower half are further pre-trained with domain-specific data. SentMAE‡ purely
relies on RetroMAE during the second-stage pre-training; while
SentMAE${}^{{\ddagger}}_{co}$ jointly employs RetroMAE and contrastive
learning.

###  3.1 Zero-shot Retrieval

The zero-shot retrieval performances on BEIR benchmark are shown with Table 1.
The methods on the left half of the table follow similar settings: a BERT-base
scale transformer (110M model parameters) is used as the encoding network, and
the pre-training is performed mainly based on Wikipedia and BookCorpus (except
RoBERTa). Among these methods: BERT Devlin et al. (2019) and RoBERTa (Liu et
al., 2019) are the pre-trained models for generic NLU tasks; SimCSE (Gao et
al., 2021) and DiffCSE (Chuang et al., 2022) are two popular methods for pre-
trained sentence embedding; SEED (Lu et al., 2021) and Condenser (Gao and
Callan, 2021a) are competitive pre-trained models for dense retrieval.
SentMAE† stands for the reproduction of RetroMAE.

We can observe that SentMAE† achieves overwhelming advantages over the above
pre-trained models under the same settings. There’s a huge improvement: 0.452
over 0.407 (the best of baselines) in total average, and it leads to the best
retrieval quality on the majority of the datasets. Such an observation
verifies RetroMAE’s effectiveness for zero-shot retrieval. Besides, RoBERTa’s
performance is not better than BERT; and pre-trained models for sentence
embeddings, like SimCSE and DiffCSE, do not contribute to the zero-shot
retrieval.

A number of the competitive pre-trained models on BEIR benchmark are listed on
the right side of Table 1 for further comparison. These models take advantages
of enlarged encoding networks and substantially increased pre-training data to
enhance their cross-domain retrieval capability.

Specifically, GTR (Ni et al., 2021) is pre-trained based on contrastive
learning on a massive dataset of 2 billion community question-answer pairs,
and it employs transformers of different scales: GTR-base (110M), GTR-large
(335M), GTR-XL (1.24B), GTR-XXL (4.8B), as the encoding networks. Although
working with limited data and model size, RetroMAE still outperforms GTR-base
(both with 110M model parameters) by a big margin; and its overall performance
is on-par with GTR-xlarge, whose model size is more than 11$\times$ larger.

Another strong zero-shot retriever is Contriever (Izacard et al., 2021), whose
overall performance (0.466 in total average) is slightly lower than SentMAE†
(averagely 0.470 on the datasets used by Contriever). It is pre-trained via
self-contrastive learning, which takes place on Wikipedia and CC-Net (Wenzek
et al., 2020). The second dataset consists of about 700 millions high quality
documents, which is more than 100$\times$ larger than the data used for
SentMAE†’s pre-training. It also requires sophisticated data augmentation and
a considerable volume of negative samples (as many as 131,072) leveraging MoCo
(He et al., 2020). By comparison, SentMAE† is simple and cost-friendly, as it
uses much less pre-training data and requires no negative samples.

---
Dataset |  SentMAE† |  SentMAE‡ |  SentMAE§ |  Dataset |  SentMAE† |  SentMAE‡ |  SentMAE§
TREC-COVID |  0.772 |  0.652 |  0.673 |  Signal-1M(RT) |  0.265 |  0.258 |  0.279
Quora |  0.847 |  0.853 |  0.846 |  BioASQ |  0.421 |  0.367 |  0.373
TREC-NEWS |  0.428 |  0.348 |  0.365 |  DBPedia |  0.390 |  0.355 |  0.405
NFCorpus |  0.308 |  0.327 |  0.307 |  Robust04 |  0.447 |  0.392 |  0.411
SCIDOCS |  0.150 |  0.133 |  0.146 |  NQ |  0.518 |  0.483 |  0.529
ArguAna |  0.433 |  0.247 |  0.315 |  FEVER |  0.774 |  0.685 |  0.722
HotpotQA |  0.635 |  0.590 |  0.634 |  Touche-2020 |  0.347 |  0.258 |  0.209
C-FEVER |  0.232 |  0.171 |  0.209 |  FiQA-2018 |  0.316 |  0.289 |  0.306
CQADupStack |  0.317 |  0.310 |  0.322 |  SciFact |  0.653 |  0.634 |  0.653
Average |  0.452 |  0.408 |  0.428 |
Table 3: Comparisons of zero-shot retrieval between SentMAE†: the base model
pre-trained on generic corpus with RetroMAE, SentMAE‡: the continuingly pre-
trained model on MS MARCO corpus, and SentMAE§: the continuingly pre-trained
model on Natural Questions corpus (Wikipedia).

###  3.2 In-domain Retrieval

The evaluation of in-domain retrieval is shown as Table 2. The upper part of
the table demonstrates the performances from models pre-trained on generic
corpus; the lower part of the table indicates the performances from the models
pre-trained on domain-specific data. We can derive the following interesting
points from the demonstrated results. SentMAE‡ and
SentMAE${}^{{\ddagger}}_{co}$ indicate stand for RetroMAE only and RetroMAE
plus contrastive learning in the second stage pre-training.

First, the continued pre-training on domain-specific data is important to the
in-domain retrieval quality. Although the first-stage pre-training already
equips the model with a strong capability on dense retrieval, as SentMAE†
outperforms all baselines in the upper half, the continued in-domain pre-
training further brings substantially benefits. By keeping on the RetroMAE
pre-training on MS MARCO corpus, SentMAE‡ achieves significant improvements
over the base model SentMAE†, which is pre-trained on the generic data. By
comparison, the improvement is relatively marginal on Natural Questions. This
is probably because the NQ’s corpus, i.e., Wikipedia, is already included as a
part of the generic pre-training data. The corresponding in-domain retrieval
quality from SentMAE‡ is indeed competitive: the performances are almost the
same as the ones based on sophisticated knowledge distillation (Ren et al.,
2021) and adversarial learning (Zhang et al., 2021), and might be the
strongest performances with simple ANCE fine-tuning on both benchmarks.

Second, we explore which learning strategy, RetroMAE or self-contrastive
learning, leads to the improvement of retrieval quality. According to our
current experiments, the self-contrastive does not contribute to the in-domain
retrieval quality, as performances from SentMAE${}^{{\ddagger}}_{co}$ are no
better than SentMAE‡. To further confirm this point, we ablate the contrastive
learning part from coCondenser, and only preserve the in-domain pre-training.
The ablation leads to MRR@10 of 0.383 and Recall@1000 of 0.983. Both results
are no worse than the original results from coCondenser, which are consistent
with the foregoing observations.

We conjecture that it is the quality of relevance label that restricts the
performance of contrastive learning. Particularly, for supervised contrastive
learning, such as the one based on the query-passage pairs from MS MARCO or
the one with sentence pairs from NLI, the positive labels are reliable and
meaningful. Therefore, it will not only benefit the in-domain performance, but
also contribute to the transferability towards other domains. However, for
self-contrastive learning, the labeled relevance comes from data augmentation,
which is likely to be either over simplified to be discriminated, or introduce
potential false positives.

Third, the pre-training on domain-specific data can be harmful to the generic
retrieval quality in other domains. We explore the impacts introduced from
SentMAE‡ and SentMAE§, which are continued pre-trained models on MS MARCO and
NQ corpus, respectively. As shown in Table 3, there are notable decrements of
zero-shot retrieval quality for the two in-domain pre-trained models;
especially SentMAE‡, which performs continued pre-training on MS MARCO corpus.

It’s worth noting that the pre-trained models are fine-tuned with the query-
passage relevance from MS MARCO before they are tested on BEIR benchmark. In
other words, MS MARCO is used as the source domain and other datasets in BEIR
are treated as the target domains. Intuitively, people may expect that a model
will probably work better on target domains if its performance can be
substantially improved on the source domain. However, the comparison between
SentMAE† and SentMAE‡ is a clear contradiction to such an expectation. It
suggests that the retrieval model’s transferability might be more of the
result from pre-training, but less of the fine-tuned performance on the source
domain.

---
| Semantic textual similarity
Methods |  STS12 |  STS13 |  STS14 |  STS15 |  STS16 |  STS-B |  STS-R |  Average
SimCSE-BERT |  75.30 |  84.67 |  80.19 |  85.40 |  80.82 |  84.25 |  80.39 |  81.57
SentMAE¶ |  76.83 |  86.20 |  81.42 |  86.59 |  82.25 |  85.08 |  79.80 |  82.60
| SentEval Transfer Tasks
Methods |  MR |  CR |  SUBJ |  MPQA |  SST |  TREC |  MRPC |  Average
SimCSE-BERT |  82.69 |  89.25 |  94.81 |  89.59 |  87.31 |  88.40 |  73.51 |  86.51
SentMAE¶ |  82.29 |  88.88 |  94.64 |  90.29 |  86.88 |  92.00 |  75.07 |  87.15
Table 4: Evaluation of sentence embedding quality. The upper table indicates
the performances on semantic textual similarity; the lower table indicates
performances on the transfer tasks in SentEval.

###  3.3 Sentence Embeddings

The evaluation of sentence embedding quality is shown with Table 4, where
SentMAE¶ is compared with SimCSE-BERT (base) (Gao et al., 2021). The two
methods rely on the same contrastive learning process, i.e., the supervised
contrastive learning with sentence pairs from NLI datasets. However, SentMAE¶
leverages RetroMAE for pre-training, whose impact will be reflected from the
comparison.

It can be observed from the experiment results that the pre-training from
RetroMAE indeed contributes to the sentence embedding quality. It achieves
substantial empirical gains on most of the STS tasks, where the overall
performance can be improved from 81.57% to 82.60%. The average performance of
transfer tasks may also be improved from 86.51% to 87.15%, despite that it is
not as significant as the gain on the STS tasks. It is probably because the
improvement of sentence representation may not directly benefit the transfer
tasks, as discussed in (Gao et al., 2021).

Notice that the previous works seldom evaluate a pre-trained model
simultaneously on its capability as a dense retriever and the quality of its
sentence embeddings. This is probably because the two tasks are about
different intrinsic properties of the pre-trained models. In other words, a
pre-trained model which is effective for one type of task may become
ineffective for another task. For example, SimCSE brings no benefit to zero-
shot and in-domain retrieval according to our observations in Table 1 and 2,
despite its effectiveness in STS tasks. However, given what we have found in
this work, both tasks may benefit from the pre-training of RetroMAE.

###  3.4 Summarization

The major experimental findings in this work can be summarized into the
following points.

$\bullet$ The pre-training of RetroMAE on generic corpus contributes to all of
the tasks in experiments, including zero-shot retrieval, in-domain retrieval,
sentence embeddings for semantic textual similarity and transfer tasks in
SentEval.

$\bullet$ The continued pre-training on domain-specific data substantially
benefits the in-domain retrieval performances. However, it can be harmful to
the model’s transferability, as the zero-shot retrieval quality decreases for
the out-domain data.

$\bullet$ The contrastive learning’s impact is conditional in our experiments.
It helps to generate high-quality sentence embeddings for STS and transfer
tasks in SentEval; however, it turns out to be relatively limited by its
sample efficiency for zero-shot retrieval, and it does not bring extra benefit
to in-domain retrieval at the presence of RetroMAE.

##  4 Conclusion

In this paper, a unified framework is presented for the pre-training of
sentence representation. On top of the consecutive MAE style pre-training on
generic and domain-specific data, and with proper collaboration with
contrastive learning, the generated model may support a wide variety of
sentence representation tasks, including zero-shot retrieval, in-domain
retrieval, and sentence embeddings. The experiment studies demonstrate that
the proposed framework helps to achieve strong performances on benchmarks,
like BEIR, MS MARCO, Natural Questions, STS and transfer tasks in SentEval.
For future works, we’ll make extensions by scaling up the size of the encoding
network and having it pre-trained on more unsupervised data, which will push
to the empirical limit of the current method. We’ll also explore its
effectiveness in other languages other than English, and its impact to more
applications beyond dense retrieval and NLI.

## References

  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511._ ACL (Association for Computational Linguistics).
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017\.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020\.  Pre-training tasks for embedding-based large-scale retrieval.  _arXiv preprint arXiv:2002.03932_.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James R. Glass. 2022\.  Diffcse: Difference-based contrastive learning for sentence embeddings.  _CoRR_ , abs/2204.10298.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.  _arXiv preprint arXiv:1803.05449_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,_ , pages 4171–4186. Association for Computational Linguistics.
  * Feichtenhofer et al. (2022) Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. 2022.  Masked autoencoders as spatiotemporal learners.  _arXiv preprint arXiv:2205.09113_.
  * Gao and Callan (2021a) Luyu Gao and Jamie Callan. 2021a.  Condenser: a pre-training architecture for dense retrieval.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 981–993.
  * Gao and Callan (2021b) Luyu Gao and Jamie Callan. 2021b.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  _arXiv preprint arXiv:2108.05540_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Towards unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu and Shao (2022) Zheng Liu and Yingxia Shao. 2022.  Retromae: Pre-training retrieval-oriented transformers via masked auto-encoder.  _arXiv preprint arXiv:2205.12035_.
  * Lu et al. (2021) Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021.  Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2780–2791.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A sick cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@ NIPS_.
  * Ni et al. (2021) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021.  Large dual encoders are generalizable retrievers.  _arXiv preprint arXiv:2112.07899_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking.  _arXiv preprint arXiv:2110.07367_.
  * Thakur et al. (2021a) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a.  Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 296–310, Online. Association for Computational Linguistics.
  * Thakur et al. (2021b) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021b.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. 2020.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4003–4012.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Zhang et al. (2021) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021\.  Adversarial retriever-ranker for dense text retrieval.  _arXiv preprint arXiv:2110.03611_.