#  $\tau$SPECT: A spin-flip loaded magnetic ultracold neutron trap for a
determination of the neutron lifetime

J. Auler1, M. Engler2, K. Franz2, J. Kahlenberg1, J. Karch1, N. Pfeifer1, K.
Roß2, C.-F. Strid1, N. Yazdandoost2, E. Adamek1, S. Kaufmann2, Ch. Schmidt1,
P. Blümler1, M. Fertl1111Corresponding author<EMAIL_ADDRESS>W. Heil1,
and D. Ries2,3222Corresponding author<EMAIL_ADDRESS>1 Institute of
Physics, Johannes Gutenberg University Mainz, 55099 Mainz, Germany  2
Department of Chemistry - TRIGA site, Johannes Gutenberg University Mainz,
55099 Mainz, Germany  3 Paul Scherrer Institut, CH-5232 Villigen PSI,
Switzerland

###### Abstract

The confinement of ultracold neutrons (UCNs) in a three-dimensional magnetic
field gradient trap allows for a measurement of the free neutron lifetime with
superior control over spurious loss channels and can provide a large kinetic
energy acceptance to enhance statistical sensitivity. In this paper, we
present the first successful implementation of a pulsed spin-flip based
loading scheme for a three-dimensional magnetic UCN trap. The measurements
with the $\tau$SPECT experiment were performed at the pulsed UCN source of the
research reactor TRIGA Mainz. We report on detailed investigations of major
systematic effects influencing the neutron storage time, statistically limited
by the size of the recorded data set. The extracted neutron storage time
constant of $\tau=$859(16)\text{\,}\mathrm{s}$$ is compatible with, but not to
be interpreted as, a measurement of the free neutron lifetime.

Keywords Neutron, lifetime, ultracold, UCN, magnetic trap, spin-flip loading

##  1 Introduction

The free neutron lifetime is a fundamental parameter for particle and
astroparticle physics, as well as for cosmology. It is particularly important
in the understanding of the production of first light elements during Big Bang
Nucleosynthesis and the extraction of the element $V_{\mathrm{ud}}$ of the
Cabibbo–Kobayashi–Maskawa matrix without nuclear structure corrections. To
calculate the abundance of first light elements after Big Bang
Nucleosynthesis, the number of relativistic species, baryon-to-photon ratio,
neutron-proton mass difference and the neutron lifetime is needed. This
calculated abundance can be compared to measurements of the primordial
abundance to search for physics beyond the Standard Model of particle physics
[1]. Currently, $V_{\mathrm{ud}}$ is determined with the highest precision by
superallowed $0^{+}\rightarrow 0^{+}$ decays where nuclear structure
corrections are needed [2]. The neutron as the prototype of a $\beta$-decaying
system does not require nuclear structure corrections and therefore offers an
independent alternative in the determination of $V_{\mathrm{ud}}$. Here, the
only input parameters are the neutron lifetime $\tau_{n}$ and the ratio of
axial-vector and vector coupling $\lambda=\nicefrac{{g_{A}}}{{g_{V}}}$ [3]. In
order to become competitive to the results obtained from superallowed decays,
measurements of $\tau_{n}$ with a relative precision of $1\times 10^{-4}$ and
of $\lambda$ with a precision of $8\times 10^{-5}$ or better are required [2].
Two different methods for measuring the free neutron lifetime have been
pursued: Beam experiments count protons emerging from a cold neutron beam per
unit time and relate this number to the flux of the neutron beam [4]. Bottle
experiments store ultracold neutrons (UCNs) in traps and count the neutrons
which are left in the trap after fixed periods of time [5, 6]. The current
best measurement for the free neutron lifetime in a beam experiment is
$\tau_{\text{beam}}=$887.7(23)\text{\,}\mathrm{s}$$ [4], and in a magnetic
bottle experiment $\tau_{\text{bottle}}=$877.75(36)\text{\,}\mathrm{s}$$ [6].
These results deviate by $3.7\sigma$. In order to understand and resolve this
discrepancy, the search for unaccounted systematic effects and physics beyond
the Standard Model of particle physics is ongoing [7, 8]. Historically, the
main source of systematic uncertainties in material bottle type experiments
are wall losses. The often large corrections have in general resulted in
neutron lifetime values higher than the currently most precise measurement
result [6]. State of the art magnetically trapping of UCNs eliminates the
largest source of these systematic uncertainties. Neutrons with kinetic
energies below $350\text{\,}\mathrm{neV}$ are called UCNs. This energy
threshold is defined by the largest known neutron optical potential (Fermi
potential) ($V_{F}$) among materials. Neutrons with energies equal or below
the Fermi potential of a material are reflected under all angles of incidence.
Additionally, neutrons are affected by Earth’s gravitational potential
($\Delta
E_{\mathrm{grav}}=$102.4\text{\,}\mathrm{n}\mathrm{eV}\mathrm{/}\mathrm{m}$$),
on the same order of magnitude as the kinetic energy of UCNs. The interaction
with magnetic fields depends on the orientation of the neutrons magnetic
moment ($\mu=-$60.31\text{\,}\mathrm{n}\mathrm{eV}\mathrm{/}\mathrm{T}$$) [9].
Neutrons with their magnetic moment oriented antiparallel to a magnetic field
are decelerated by an increasing magnetic field. Therefore, they are called
low field seeking neutrons (LFS). Neutrons which are accelerated by an
increasing magnetic field are called high field seeking neutrons (HFS). The
$\tau$SPECT experiment is located at the pulsed research reactor TRIGA Mainz
and uses the ultracold neutron source installed at beam port D [10, 11].

##  2 Experimental setup

$\tau$SPECT is the first experiment to store UCNs in a three-dimensional
magnetic field gradient. A schematic drawing of the experiment is shown in
figure 1.

Figure 1: Schematic view of the $\tau$SPECT experiment. The cryostat (4) is
surrounded by an iron yoke (3) and houses the superconducting coils producing
the $B_{z}$ field (red line), the storage octupole (7) and the neutron
detector (8). $B_{z}$ can be divided into three regions, the field maxima in
regions I and III are used for longitudinal confinement of UCNs, the low field
in region II maintains the neutron spin polarization. UCNs enter the
experiment through a shutter (1) and are guided into the magnetic trap by
fused silica tubes (shown in pink). The spin flipping unit (SFU, 6) SF2
transforms high-field seeking neutrons into storable low-field seekers. An
additional spin flipper SF1 (5) was installed in an upgrade of the experiment.
Both SFU and detector are movable by translation stages (2) and (9) operated
by stepper motors. The pressure is monitored at positions $P_{1}$ and $P_{2}$,
the temperature inside the trap is measured at position T within the SFU.

The main components are the confinement magnets, which build up the magnetic
trap (Sec. 2.1), the spin flipping unit (SFU) consisting of an adiabatic fast
passage spin flipper (Sec. 2.2) surrounded by a Halbach octupole for local
field compensation (Sec. 2.1), the UCN detector (Sec. 2.3) as well as
additional components for monitoring purposes (Sec. 2.4).

###  2.1 Magnetic trapping fields

Longitudinal confinement of UCNs in the magnetic trap is achieved by the main
magnetic field $B_{z}$ along the horizontal $z$ axis. Its magnitude along the
z-axis is shown by the red line in figure 1. This field is produced by a
superconducting multi coil system, which along with its cryostat has been
repurposed from the $a$SPECT experiment [12, 13]. The specific shape of the
magnetic field arises from requirements as part of a MAC-E filter333MAC-E
filter stands for ’Magnetic Adiabatic Collimation in combination with an
Electrostatic Filter’[14]. The $B_{z}$ field can be divided into three regions
which are marked by I, II and III in figure 1. The high field maxima in
regions I and III are used as longitudinal confinement for UCNs in their low-
field seeking spin state. In between (region II), a low field is required to
maintain the neutron polarization during storage. The superconducting coils
are connected in series. Therefore, the general field shape cannot be changed,
but the field magnitude can be scaled by varying the current through the coils
in the range of $0\text{\,}\mathrm{A}100\text{\,}\mathrm{A}$. Radial
confinement of UCNs is provided by permanent magnets in a Halbach octupole
configuration [15]. It consists of 24 identical rings, each composed of $N=$
32 wedge-shaped segments with a height of $L=$57.5\text{\,}\mathrm{mm}$$ (see
figure 2a). All 768 pieces are made from
$\isotope[]{Sm}_{2}\isotope[]{Co}_{17}$ with a nominal remanence of
$B_{R}=$1.1\text{\,}\mathrm{T}$$ and relative permeability of $\mu_{r}=1.1$.
The whole structure is $1.38\text{\,}\mathrm{m}$ long, with an inner radius of
$R_{i}=$54.0\text{\,}\mathrm{mm}$$ and outermost radius of
$R_{o}=$84.1\text{\,}\mathrm{mm}$$. It was custom-made by Arnold Magnetic
Technologies AG, Lupfig, Switzerland. Details on the design of the Halbach
octupole can be found in [16]. Theoretically, the resulting magnetic flux
density inside an infinitely long multipole of polarity $2k$ (for an octupole
$k=$ 4) made from $N$ wedge-shaped elements is given in cylindrical
coordinates ($r,\theta$) by [15, 17]

$\vec{B}(r,\theta)=F_{geom}\;r^{\>(k-1)}\left[\begin{array}[]{c}{\cos[(k-1)\theta+k\alpha]}\\\
{-\sin[(k-1)\theta+k\alpha]}\\\ \end{array}\right],$ (1)

with

$\displaystyle F_{geom}=$
$\displaystyle\frac{kB_{R}}{(k-1)\sqrt{\mu_{r}}}\left(\frac{1}{R_{i}^{(k-1)}}-\frac{1}{R_{o}^{(k-1)}}\right)\cdot$
(2)
$\displaystyle\frac{\sin{((k+1)\pi/N)}}{(k+1)\pi/N}\frac{N\sin{(2\pi/N)}}{2\pi}$
(3)

The factor $F_{geom}$ is the scaling of the principal field due to the
particular geometry and materials. The angle $\alpha$ describes the rotation
angle of the cylinder around its axis. If such an infinitely long structure is
truncated to a length $L$, $F_{geom}$ has to be extended to give its central
field ($z=L/2$) by the following approximation by integrating magnetic dipoles
(for details see [17]):

$F^{\prime}_{geom}=F_{geom}\frac{\scriptstyle{L(L^{8}+18L^{6}R^{2}+126L^{4}R^{4}+420L^{2}R^{6}+630R^{8})}}{\scriptstyle{(4R^{2}+L^{2})^{9/2}}},$
(4)

with $R=\frac{R_{o}+R_{i}}{2}$ the central radius. Figure 2b shows excellent
agreement of this equation with finite elements (FEM) simulations (COMSOL
Multiphysics444https://www.comsol.com/) and some measurements with Hall-
probes. The magnetic flux density was measured at a distance of
$2.5\text{\,}\mathrm{mm}$ with respect to the octupole’s inner surface and was
extrapolated using the $r^{3}$ dependency in 1, resulting in an average
magnetic flux density of
$B(r=$53.5\text{\,}\mathrm{mm}$)=$0.902(26)$$\mathrm{T}$$. The minimum flux
density was found as
$B_{r}^{\text{min}}(r=$53.5\text{\,}\mathrm{mm}$)=$0.779\text{\,}\mathrm{T}$$.

Figure 2:  a) Schematic drawing of the two permanent magnet systems: The
storage octupole (in green) and the compensation octupole (in blue) are
coaxially aligned with the neutron guide (gray). The white arrows indicate the
magnetization direction of the permanent magnets. The eight poles are
accentuated using darker and brighter colors. b) Comparison of magnitudes of
the radial magnetic fields in the center of the storage (right) and
compensation (left) octupole magnets. The dark blue line shows the result of
4. For the storage octupole the arithmetic average
($R_{o}=$81.75\text{\,}\mathrm{mm}$$) of the 24 longer wedges with
$R_{o}=$84.1\text{\,}\mathrm{mm}$$ and the eight shorter ones with
$R_{o}=$74.7\text{\,}\mathrm{mm}$$ was used. The dashed yellow line shows the
results from the FEM-simulations, while the magenta circles indicate
measurements inside the constructed devices. The solid rectangles mark the
position of the magnets. A simulation of the resulting compensated field can
be found in figure 14.

The limited bandwidth of radio frequency (RF) excitation for flipping the
neutrons’ spin (see section 2.2) requires, however, relatively homogeneous
magnetic fields during this process. While the axial field shows sufficient
homogeneity, the strong radial component of the octupole certainly needs
compensation. This can be achieved by using a second octupole, coaxially
aligned and rotated by an angle
$\alpha=\pi/k=$45\text{\,}\mathrm{\SIUnitSymbolDegree}$$ to the first. If its
geometry is chosen such that $F^{\prime}_{geom}$ matches that of the other
octupole, the magnetic fields inside cancel each other (cf. 1) [18, 17].
Therefore, such a second, smaller octupole for compensation of the larger
octupole for storage (see figure 2a) is mounted together with the RF-coils of
the adiabatic fast passage (AFP) spin flipper [19] so that both can be moved
in and out of the trapping region (see section 2.2). To guarantee identical
behavior when the experiment is cooled down, the second octupole was made from
the same material with $N=$ 32, $R_{i}=$41.2\text{\,}\mathrm{mm}$$,
$R_{o}=$47.0\text{\,}\mathrm{mm}$$, and $L=$150\text{\,}\mathrm{mm}$$. These
dimensions were optimized using FEM-simulations, however 4 also gives
$F^{\prime}_{geom}=$6226\text{\,}\mathrm{T}\mathrm{/}\mathrm{{}^{3}}\mathrm{m}$$
compared to
$F^{\prime}_{geom}=$6032\text{\,}\mathrm{T}\mathrm{/}\mathrm{{}^{3}}\mathrm{m}$$
for the storage octupole (cf. figure 2b). The individual segments of the
octupole magnet were glued into a holding shell made from titanium
($V_{F}=$-48\text{\,}\mathrm{neV}$$). This material was chosen because it is
non-magnetic and allows for a small wall thickness while maintaining
structural stability.
In order to quantify the compensation, the radial field of the storage
octupole was measured at room temperature before the installation in
$\tau$SPECT with and without the compensation octupole [20]. The fields and
gradients were measured at a radius of $r=$25\text{\,}\mathrm{mm}$$,
corresponding to the outside surface of the neutron guide in the compensation
octupole. At this radius, the average magnetic field and gradient were reduced
from $B=$75(5)\text{\,}\mathrm{mT}$$ and
$|\nabla_{r}B|=$112(54)\text{\,}\mathrm{m}\mathrm{T}\mathrm{/}\mathrm{c}\mathrm{m}$$
to $B=$2.2(8)\text{\,}\mathrm{mT}$$ and
$|\nabla_{r}B|=$2.1(8)\text{\,}\mathrm{m}\mathrm{T}\mathrm{/}\mathrm{c}\mathrm{m}$$,
which is sufficiently low to allow for the spin flip efficiency to reach
$\epsilon_{\rm{SF}}\sim 1$. It must be taken into account that axial
misalignment leads to incomplete compensation of the magnetic field. The
alignment of the compensation octupole leading to the described compensation
was performed with an axial offset of only $\delta
y=$0.2\text{\,}\mathrm{mm}$$.
The superposition of the longitudinal and the radial field creates a central
region which is surrounded by increasing magnetic flux density, as
schematically shown in figure 3. While the longitudinal field can be changed
and thus the trap potential can be varied, this is not possible for the radial
field being made from permanent magnets. In order to maximize the trap volume,
the optimum current in the $B_{z}$ producing superconducting coils was
determined: On the one hand, both field maxima (regions I and III) should be
larger than the minimum flux density of the octupole
$B_{r}^{\text{min}}(r=$53.5\text{\,}\mathrm{mm}$)$ so that the trap depth is
limited by the fixed octupole field. On the other hand, the longitudinal
holding field in region II must be low, as otherwise the storable phase space
decreases (Sec. 2.2). The optimum current was determined as
$I=$33\text{\,}\mathrm{A}$$, leading to
$B_{z}^{\text{max}}=$1.023\text{\,}\mathrm{T}$$ in region I and
$B_{z}^{\text{max}}=$2.050\text{\,}\mathrm{T}$$ in region III, as well as a
holding field between $B_{z}=$202\text{\,}\mathrm{mT}$$ and
$208\text{\,}\mathrm{mT}$ in region II. The trap potential is then given, as
$V_{\text{magn}}=-\mu_{n}B_{r}^{\text{min}}(r=$53.5\text{\,}\mathrm{mm}$)=$47.0\text{\,}\mathrm{neV}$$
with $\mu_{n}=$-60.3\text{\,}\mathrm{n}\mathrm{eV}\mathrm{/}\mathrm{T}$$ the
neutron magnetic moment. The trap has a volume of up to
$\sim$10.7\text{\,}\mathrm{L}$$ for UCNs at maximum energies. Neutrons with
lower energies occupy a smaller volume. The compensation octupole and spin
flipper form the SFU, which along with the neutron guide is pulled out of the
trapping region during storage by a translation mechanics operated by a
stepper motor. The range of motion is between $z=$397\text{\,}\mathrm{mm}$$
and $1200\text{\,}\mathrm{mm}$ (see figure 17). Spring-mounted wheels are
attached to the front of the compensation octupole at
$45\text{\,}\mathrm{\SIUnitSymbolDegree}$ relative to the vertical axis to
reduce friction on the storage octupole surface. A copper shield
($V_{F}=$168\text{\,}\mathrm{neV}$$) on the front face reflects neutrons
during the filling process as long as the SFU is still inside the trap. A
dedicated holding construction prevents the compensation octupole from
rotation around the $z$-axis [21].

Figure 3: Magnetic field distribution forming the three-dimensional magnetic
trap. UCNs in the low-field seeking spin state are confined within the local
minimum of magnetic flux density created by the field superposition of the
superconducting coils and the permanent magnet octupole array. The contour
line (red) corresponds to a magnetic field of $B=$0.78\text{\,}\mathrm{T}$$
which translates to a UCN magnetic potential energy of
$V_{\text{magn}}=$47.0\text{\,}\mathrm{neV}$$.

###  2.2 Adiabatic fast passage spin flipper

Ultracold neutrons enter $\tau$SPECT through a telescopic neutron guide system
consisting of stainless steel tubes (inner diameter
$d_{i}=$66\text{\,}\mathrm{mm}$$) outside and a fused silica tube within the
cryostat ($d_{i}=$73\text{\,}\mathrm{mm}$$). An adiabatic fast passage (AFP)
spin flipper [19] is located at the end of the fused silica tube. To
accommodate the spin flipper, the tube diameter is reduced to
$d_{i}=$50\text{\,}\mathrm{mm}$$ (see figure 4).

Figure 4: The SFU as CAD rendering. The spin flipper with its center position
marked with $z_{\text{fill}}^{\text{SF}}$ is surrounded by the compensation
octupole. A ring with wheels made from PEEK reduces friction when moving the
SFU on the storage octupole surface. A copper shield
($V_{F}=$168\text{\,}\mathrm{neV}$$) reflects neutrons from the front face of
the SFU during the filling procedure. Due to spatial constraints, the neutron
guide’s inner radius is reduced from $r=$36.5\text{\,}\mathrm{mm}$$ to
$r=$25.0\text{\,}\mathrm{mm}$$. The stainless steel tube provides mechanical
support and is connected to the translation stage (see figure 1). A uniaxial
Hall probe is used to measure the longitudinal component of the magnetic
field.

As soon as the UCNs enter the high magnetic field in region I (see figure 1)
they are separated into high-field seekers (HFS) and low-field seekers (LFS).
Only HFS (as well as LFS with kinetic energies larger than the magnetic
potential in region I) are able to pass the field maximum. As high-field
seeking neutrons are not storable, the AFP spin flipper is used to rotate the
neutron spin by $180\text{\,}\mathrm{\SIUnitSymbolDegree}$ and thus to
transform HFS into storable LFS555The fraction of LFS that pass the field
maximum in region I are spin flipped to HFS which are not storable and leave
the trap.. An AFP spin flipper for neutrons has been proposed by V. Luschikov
[22] and several have been successfully employed in other experiments [23,
24]. Its application in the context of a vertical magneto-gravitational trap
for a neutron lifetime experiment is also discussed in [25].

Figure 5: Longitudinal magnetic field in the spin flipping region around the
nominal spin flip position, $z_{\text{fill}}^{\text{SF}}$ with the field
magnitude on the left axis and the required spin flipper angular frequency
$\omega_{\text{SF}}$ calculated using 5. The gray box marks the physical
extent of the spin flipper between $z_{1}$ and $z_{2}$. Figure 6: Sketch of
the spin flip of an initial HFS UCN. From the left, the UCN enters the spin
flipper (yellow) with its spin (green) antiparallel aligned to
$\vec{B}^{\prime}_{e}(z)$. $\vec{B}^{\prime}_{1}$ points along the
$\hat{x}^{\prime}$ axis and tilts $\vec{B}^{\prime}_{\text{red}}(z)$ by an
angle $\theta$. On the left side of the resonance at
$z_{\text{fill}}^{\text{SF}}$, $\vec{B}^{\prime}_{\text{red}}(z)$ points along
$z^{\prime}$ while on the right side it points to the opposite direction. The
neutron spin follows $\vec{B}^{\prime}_{e}(z)$ adiabatically and the UCN exits
the spin flipper as LFS.

The principle of operation is based on nuclear magnetic resonance and
visualized in figure 6. In a magnetic field $\vec{B}$, the neutron spin
precesses at the Larmor frequency $\omega_{0}=-\gamma|\vec{B}|$ with the
gyromagnetic ratio of the neutron
$\gamma=$-1.832\,471\,71(43)$\times${10}^{8}\text{\,}{\mathrm{T}}^{-1}\text{\,}{\mathrm{s}}^{-1}$$
[26]. In the spin flipping region around $z=$980\text{\,}\mathrm{mm}$$, the
longitudinal field has a gradient and therefore the Larmor frequency is
position dependent as shown in figure 5. In order to perform a spin flip, a
perpendicular ($xy$-plane) rotating magnetic field $B_{1}$ is required. With
the center of the spin flipper being located at a nominal position
$z_{\text{fill}}^{\text{SF}}$ (’resonance position’), this resonance frequency
is given by

$\omega_{\text{SF}}=\omega_{0}=-\gamma|\vec{B}(z_{\text{fill}}^{\text{SF}})|.$
(5)

In a reference frame which rotates at the spin flipper frequency (indicated by
primes) $\omega_{\text{SF}}$, the spin precesses at the so-called reduced
frequency $\omega_{\text{red}}^{\prime}=\omega_{0}-\omega_{\text{SF}}$ around
the locally reduced field

$\vec{B}_{\text{red}}^{\prime}(z)=\left(\frac{-\omega_{\text{red}}^{\prime}(z)}{\gamma}\right)\hat{z}^{\prime}=\vec{B}_{\text{z}}(z)+\left(\frac{\omega_{\text{SF}}}{\gamma}\right)\hat{z}^{\prime}.$
(6)

In this reference frame, $\hat{z}=\hat{z}^{\prime}$ and $\vec{B}_{1}^{\prime}$
is static and points along an arbitrarily chosen axis $\hat{x}^{\prime}$. The
superposition of $\vec{B}_{1}^{\prime}$ and $\vec{B}_{\text{red}}^{\prime}(z)$
gives the effective field the neutrons experience in the rotating reference
frame

$\vec{B}_{e}^{\prime}(z)=\vec{B}_{\text{red}}^{\prime}(z)+\vec{B}_{1}^{\prime}=\left(B_{z}(z)+\frac{\omega_{\text{SF}}}{\gamma}\right)\hat{z}^{\prime}+B_{1}\hat{x}^{\prime}.$
(7)

The angle between this effective field and the $\hat{z}^{\prime}$-axis is
given by

$\tan{(\theta)}=\frac{B_{1}^{\prime}}{\raisebox{-2.27621pt}{$B_{z}(z)+\frac{\omega_{\text{SF}}}{\gamma}$}}=\frac{\omega_{1}}{\raisebox{-2.27621pt}{$\omega_{\text{red}}^{\prime}(z)$}}.$
(8)

Due to the negative field gradient of $B_{z}$, $\omega_{\text{red}}>0$ for
$z<z_{\text{fill}}^{\text{SF}}$ and $\omega_{\text{red}}<0$ for
$z>z_{\text{fill}}^{\text{SF}}$. This results in a change of sign in $\theta$
and thus in the orientation of $\vec{B}_{e}^{\prime}$. For example, the spin
of an HFS neutron follows this field adiabatically thus changing its
orientation as shown in figure 6 and the neutron exits the spin flipper as
LFS.

Figure 7: Schematic of the $B_{1}$ field generation circuit (left) and the
finished spin flipper SF2 (right). The spin flipper consists of two saddle
coils (1) and (2) which are wound on a holding cylinder made from PEEK. Two
pick-up coils (3) and a temperature sensor (4) (not visible) can be used for
functionality and reproducibility checks.

The adiabatic fast passage spin flipper used for the $\tau$SPECT experiment
consists of two orthogonal saddle coils wound on a holding Polyether-ether-
ketone (PEEK) cylinder (see figure 7). The coils have an opening angle
$\phi=$120\text{\,}\mathrm{\SIUnitSymbolDegree}$$, half diameter
$d/2=$32.5\text{\,}\mathrm{mm}$$ and a length of $l=$60\text{\,}\mathrm{mm}$$.
A longitudinal offset of $10\text{\,}\mathrm{mm}$ between the coils is added
to reduce cross-talk, decreasing the spin flipper to an effective length of
$l_{\text{eff}}=$50\text{\,}\mathrm{mm}$$. An individual RF signal is fed into
each coil, resulting in a linearly polarized magnetic field (see figure 7). By
adjusting the phase between the two signals, the overall polarization of the
$B_{1}$ field can be set to circular ($\pm\sigma$), linear ($\pi$) or any
other elliptic polarization. The RF excitation signals are generated by the
two outputs of a frequency generator 666Siglent SDG2042X and amplified in two
steps 777First stage: K-N-E power amplifier 10W-QRP, second stage: RM Italy
HLA300 V plus. Frequency tuning and impedance matching of the excitation
circuits to $Z=$50\text{\,}\mathrm{\SIUnitSymbolOhm}$$ is done by radio tuners
888MFJ-969 Versa Tuner II outside the closed cryostat. Bidirectional couplers
within the signal lines are used to monitor and subsequently adjust the phase
between the excitation signals. A second very similar spin flipper has been
built to implement the double-spin flip loading scheme described in section
2.2.2. The spin flippers are named according to the order of passage by the
UCNs from the experiment entrance. The SF1 is located at a lower z-position
than SF2. Figure 8 shows the spin flip efficiency of SF2 as a function of the
irradiated RF power. $50\text{\,}\mathrm{s}$ storage measurements were used at
the spin flip position $z=$980\text{\,}\mathrm{mm}$$, without the spectral
cleaning procedure. The horizontal axis shows the RF power ($P$) which is
related to the amplitude $B_{1}$, as $B_{1}\propto\sqrt{P}$. The line is an
exponential function

$N=a\left(1-\exp\left(-\frac{P}{b}\right)\right)+c$ (9)

with the number of neutrons after a storage time of $50\text{\,}\mathrm{s}$
and the fit parameters $a$, $b$ and $c$ for scaling and consideration of an
offset. We find that the neutron density in the trap saturates from
$P\sim$50\text{\,}\mathrm{W}$$ onward, indicating a maximum spin flip
efficiency999The maximum spin flip efficiency does not automatically mean
$\varepsilon_{\text{SF}}=1$ as the absolute spin flip efficiency could not be
measured. from there on. The irradiated RF power in all measurements with
$\tau$SPECT is between $P=$80\text{\,}\mathrm{W}$$ and
$100\text{\,}\mathrm{W}$. The RF phase $\Delta\varphi$ between the excitation
signals of the saddle coils was scanned around the value of
$|\Delta\varphi|=$90\text{\,}\mathrm{\SIUnitSymbolDegree}$$ confirming the
expected phase setting [20].

Figure 8: Number of stored neutrons depending on the irradiated RF power at
$z_{\text{fill}}^{\text{SF}}=$980\text{\,}\mathrm{mm}$$ in SF2. The data show
a saturation behavior of the neutron counts beyond an RF power of
$50\text{\,}\mathrm{W}$. The line is an exponential function (9) plotted to
guide the eye.

####  2.2.1 Single spin flip energy acceptance

In order to maximize the UCN density per fill, the energy acceptance of the
trap should be as large as possible.

Figure 9: Magnetic potential as experienced by LFS (solid line) and HFS
(dashed line). Due to energy conservation, HFS (LFS) are accelerated
(decelerated) when they enter an increasing field and decelerated
(accelerated) in a decreasing field. The spin flip changes the sign of $\Delta
V_{\text{magn},0}$. The potential as seen from an initial HFS spin flipped to
an LFS is marked in orange.

We consider a neutron in the high-field seeking state
($\vec{\mu}_{n}\uparrow\uparrow\vec{B}$) with an initial kinetic energy
$E_{\text{kin},0}$, see figure 9. Before entering $\tau$SPECT, the neutron’s
potential energy due to the magnetic field is
$V_{\text{magn},0}=$0\text{\,}\mathrm{neV}$$. While moving into the high field
region, the HFS neutron gains the kinetic energy $\Delta
E_{\text{kin}}=|\mu_{n}|B_{z}^{\text{max,I}}=$61.7\text{\,}\mathrm{neV}$$.
After the field maximum, it is decelerated again until it reaches the position
of the spin flipper $z_{\text{fill}}^{\text{SF}}$. At that point, the HFS
neutron has gained kinetic energy of $\Delta
E_{\text{kin}}(z_{\text{fill}}^{\text{SF}})=-\Delta
V_{\text{mag},0}(z_{\text{fill}}^{\text{SF}})=|\mu_{n}|B_{z}(z_{\text{fill}}^{\text{SF}})$.
After the spin flip, the magnetic moment is aligned antiparallel with $B_{z}$
(LFS) and $\Delta V_{\text{magn},0}$ becomes positive. The total energy of the
now LFS neutron is thus given by

$\displaystyle E(z_{\text{fill}}^{\text{SF}})$ $\displaystyle=$ $\displaystyle
E_{\text{kin},0}+\Delta E_{\text{kin}}(z_{\text{fill}}^{\text{SF}})+\Delta
V_{\text{magn},0}(z_{\text{fill}}^{\text{SF}})$ (10) $\displaystyle=$
$\displaystyle E_{\text{kin},0}+2|\mu_{n}|B_{z}(z_{\text{fill}}^{\text{SF}})$
(11)

If its total energy is below the trap potential,
$V_{\text{magn}}=$47.0\text{\,}\mathrm{neV}$$ an LFS neutron is storable.
Therefore, the maximum allowed initial kinetic energy is

$E_{\text{kin},0}^{\text{max}}(z_{\text{fill}}^{\text{SF}})=$47.0\text{\,}\mathrm{neV}$-2|\mu_{n}|B_{z}(z_{\text{fill}}^{\text{SF}}).$
(12)

So far, we ignored gravity. To account for gravity, we need to consider the
height difference $s$ between the position of the spin flip and the lowest
point of the storage octupole surface. The concentrically aligned neutron
guide and storage octupole have different radii
($r_{\text{guide}}=$25\text{\,}\mathrm{mm}$$ compared to
$r_{\text{SO}}=$54\text{\,}\mathrm{mm}$$). Assuming the spin flip can take
place at any height within the neutron guide, $s$ can take values from
$29\text{\,}\mathrm{mm}$ to $79\text{\,}\mathrm{mm}$ given by the difference
and sum of the radii. The maximum allowed initial kinetic energy in 12 is
reduced by $\Delta
V_{\text{grav}}(s)=s\cdot$102.4\text{\,}\mathrm{n}\mathrm{eV}\mathrm{/}\mathrm{m}$$
and results in

$E_{\text{kin},0}^{\text{max}}(z_{\text{fill}}^{\text{SF}},s)=$47.0\text{\,}\mathrm{neV}$-2|\mu_{n}|B_{z}(z_{\text{fill}}^{\text{SF}})-\Delta
V_{\text{grav}}(s).$ (13)

For a nominal spin flip position
$z_{\text{fill}}^{\text{SF}}=$980\text{\,}\mathrm{mm}$$ of

$E_{\text{kin},0}^{\text{max}}($980\text{\,}\mathrm{mm}$,$29\text{\,}\mathrm{mm}$)=$18.9\text{\,}\mathrm{neV}$.$
(14)

The effect of an energy staggered vertical distribution will be considered in
the future.

####  2.2.2 Double spin flip energy acceptance

In order to increase the energy acceptance of $\tau$SPECT, an upgrade of the
experiment introduced a double AFP spin flip technique using a second spin
flipper (SF1) in the high field of region I. For the AFP spin flipper to work
we impose a longitudinal field gradient of
$\nabla_{z}B_{z}(z_{\text{SF1}})=$-0.48\text{\,}\mathrm{m}\mathrm{T}\mathrm{/}\mathrm{c}\mathrm{m}$$
using coils, named C3 and C5, inside the cryostat in anti-Helmholtz
configuration (see figure 10). Using this technique, LFS neutrons are at first
decelerated in the increasing $B_{z}$ field by
$|\mu_{n}|B_{z}(z_{\text{SF1}})=$61.5\text{\,}\mathrm{neV}$$ (see figure 11),
then the neutron spin state is flipped from LFS to HFS and the neutrons are
subsequently decelerated by
$|\mu_{n}|B_{z}(z_{\text{SF1}})-|\mu_{n}|B_{z}(z_{\text{SF2}})=$49.0\text{\,}\mathrm{neV}$$
on their way to the spin flipper in the low field region (SF2)
($|\mu_{n}|B_{z}(z_{\text{SF2}})=$12.5\text{\,}\mathrm{neV}$$). Spin flipper 2
converts HFS to storable LFS neutrons (Pos. III to IV).

Figure 10: Longitudinal magnetic field $B_{z}$ around the maximum of the field
in region I without (gray) and with the gradient coils C3 and C5 (black).
Solid lines refer to the magnetic field (left axis), dashed lines to the field
gradient (right axis). The shaded area shows the longitudinal range and the
field gradient covered by SF1. The gradient at
$z_{\text{SF1}}=$50\text{\,}\mathrm{mm}$$ is
$\nabla_{z}B_{z}(z_{\text{SF1}})=$-0.48\text{\,}\mathrm{m}\mathrm{T}\mathrm{/}\mathrm{c}\mathrm{m}$$
for $I_{\text{C3}}=$20\text{\,}\mathrm{A}$$ and
$I_{\text{C5}}=$-10\text{\,}\mathrm{A}$$. Figure 11: Magnetic potential as
experienced by LFS (solid line) and HFS (dashed line). The potential energy of
an initial LFS during the double spin flip is shown by the orange line. It is
decelerated as the potential energy rises. At SF1, it is converted to a HFS
with $\Delta V_{\text{magn},0}$ changing sign (from I to II). The HFS is
decelerated again until at SF2 $\Delta V_{\text{magn},0}$ changes sign again
(from III to IV). The neutron kinetic energy is reduced by
$110.5\text{\,}\mathrm{neV}$ at $z_{\text{SF2}}=$980\text{\,}\mathrm{mm}$$.
The potential energy at SF2 is $12.5\text{\,}\mathrm{neV}$.

The neutron energy after passage of SF2 is given by

$E=\left(E_{\text{kin},0}-$110.5\text{\,}\mathrm{neV}$\right)+V_{\text{magn}}(z_{\text{SF2}})+\Delta
V_{\text{grav}}(s).$ (15)

With the trapping condition $E\leq$47.0\text{\,}\mathrm{neV}$$ follows the
maximum allowed energy
$E_{\text{kin},0}^{\text{max}}(z_{\text{SF2}})=$142.0\text{\,}\mathrm{neV}$$
for an incoming neutron to be stored in the trap potential. Due to the
deceleration on their way into the trap, neutrons will be reflected if their
initial kinetic energy is less than
$E_{\text{kin},0}^{\text{min}}(z_{\text{SF2}})=$110.5\text{\,}\mathrm{neV}$$.
Therefore, the accepted initial energy interval using the double spin flip
technique is $[$110.5\text{\,}\mathrm{neV}$,$142.0\text{\,}\mathrm{neV}$]$.
This is an increase in interval width by $66\text{\,}\mathrm{\char 37\relax}$
compared to the single spin flip case. Combined with the $\sqrt{E}$-dependency
of the neutron density per unit energy $\varrho(E)$ at the exit of the source
[9] this results in a higher number of neutrons stored. Since the neutrons are
decelerated along $B_{z}$, only neutrons with a large enough longitudinal
velocity will reach the trap. To account for this, the minimum kinetic energy
can be rewritten as

$E_{\text{kin},0}^{\text{min}}(z_{\text{SF2}})=$110.5\text{\,}\mathrm{neV}$/\cos^{2}(\theta),$
(16)

where $\theta$ is the angle between the velocity vector and the longitudinal
field. For an angle $\theta=$28.1\text{\,}\mathrm{\SIUnitSymbolDegree}$$, the
minimum energy reaches
$E_{\text{kin},0}^{\text{min}}=$142.0\text{\,}\mathrm{neV}$$ and the kinematic
acceptance becomes zero. A comparison of both spin flip techniques has been
done using Monte Carlo simulations (see Sec. 3.2) Since the two spin flip
techniques work for different energy and therefore velocity ranges, a
combination of both techniques can lead to a higher number of neutrons in the
trap than one technique alone. This mixed filling technique is currently under
investigation.

###  2.3 In-situ UCN detector

Figure 12: CAD rendering of the UCN detector. For details, see main text.

The design of the magnetic trap and the environmental conditions within the
cryostat impose several constraints on the UCN detector.

  1. 1.

It must be suitable for high vacuum, cryogenic temperatures and high magnetic
fields of up to $2\text{\,}\mathrm{T}$.

  2. 2.

It has to be possible to move the detector inside $\tau$SPECT to different
positions.

  3. 3.

It must be able to detect UCNs efficiently under these conditions.

The construction of our custom UCN detector and its motion mechanism is
described in [21], the schematic setup is shown in figure 12. UCNs are
captured in an $80\text{\,}\mathrm{nm}$ thick $\isotope[10]{B}$ layer coated
on a silver-doped zinc sulphide (ZnS:Ag) scintillator of
$22.6\text{\,}\mathrm{\SIUnitSymbolMicro m}$ thickness. The
$\isotope[10]{B}$/$\isotope[11]{B}$ ratio was chosen to obtain
$V_{F}\approx$0\text{\,}\mathrm{neV}$$. The charged particles from the nuclear
$\isotope[10]{B}(n,\alpha)\isotope[7]{Li}$ reaction generate light in the
scintillator, which is guided towards a $4\times 4$ silicon photomultiplier
(SiPM) array (Hamamatsu S13361-6050AE-04) by a cone-shaped light guide made
from Poly-methyl-methacrylate (PMMA). The 16 channels are summed and amplified
in two steps inside and outside the experiment, and are subsequently digitized
by a 14-bit ADC with a sampling rate of
$10\text{\,}\mathrm{M}\mathrm{s}\mathrm{p}\mathrm{s}$.

Figure 13: Single event in the detector raw data (top) and the time derivative
of the same signal (bottom). If the time derivative exceeds the event
recognition threshold, the event is analyzed by the pattern recognition
algorithm with respect to the event maximum voltage $U_{\text{max}}$, the
integrated voltage $U_{\text{integral}}$, the rise time $\Delta
t_{\text{rise}}$, the fall time $\Delta t_{\text{fall}}$ and the overall event
length $\Delta t_{\text{event}}=\Delta t_{\text{rise}}+\Delta
t_{\text{fall}}$. The baseline voltage is defined as the voltage
$U_{\text{base}}$ when the event recognition threshold was exceeded.

Raw data is recorded continuously during each measurement run (see Sec. 5 for
the full sequence of each run), an offline pattern recognition algorithm
subsequently detects neutron events in the following way: A sharply rising
edge triggers a new event. Parameters like rise- and fall-times, peak voltage
with respect to baseline voltage, event length and voltage integral over the
event are evaluated and compared to empirical threshold values. An example
event is shown in figure 13, the algorithm is described in detail in [20]. The
performance of the custom UCN detector for $\tau$SPECT was compared to a
commercial CASCADE-U detector101010UCN detector by https://n-cdt.com at room
temperature: UCNs were stored for $t_{s}=$20\text{\,}\mathrm{s}$$ in an
aluminum storage bottle111111Using a storage bottle with aluminum walls leads
to a very soft energy spectrum of stored neutrons, similar to the expected
energy spectrum in $\tau$SPECT. ($V_{F}=$54\text{\,}\mathrm{neV}$$,
$V=$9.67(2)\text{\,}\mathrm{L}$$ [21]) and counted afterward. The relative
detection efficiency can then be expressed in terms of UCN counts $N$ and
results in

$\frac{N_{\text{$\tau$SPECT}}}{N_{\text{CASCADE}}}=1.06(7).$ (17)

In $\tau$SPECT, the detector is mounted on a translation stage to store the
detector outside or to move it inside the storage region to count UCNs.

###  2.4 Environmental monitoring

The environmental conditions in $\tau$SPECT are permanently monitored. Two
full range pressure sensors (positions marked in figure 1), one cold-cathode
sensor and one Pirani sensor, read the pressure in $\tau$SPECT. The
temperature is measured by various sensors, which is especially important for
the operation of the superconducting coils. The temperature inside the storage
octupole is approximately $170\text{\,}\mathrm{K}$. The detector and the SFU
are both equipped with a 1-axis Hall sensor, which can be used to measure the
$z$-component of the magnetic field for reproducibility purposes. A nuclear
magnetic resonance probe is mounted close to SF1. Two vibration sensors are
attached to the outside of the cryostat which can be used for systematic
studies on microphonic heating. To adjust the height of the experiment, the
iron yoke is placed on four heavy duty lifting columns. Distance sensors
monitor the height of the experimental setup.

##  3 Simulation of the trap filling process

The AFP spin flipping process is well understood in regions with small
transverse magnetic field gradients [24]. However, the additional transversal
field of the compensation octupole (if not fully compensated) makes an
analytic description of the spin flip difficult. Therefore, the filling
process was simulated using Monte Carlo methods and numerical
integration121212Python3 numerical integrator odeint
(https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html)
from the SciPy-library [27] of Bloch’s equation [28]

$\frac{d\vec{S}}{dt}=\gamma\left(\vec{S}\times\vec{B}\right)$ (18)

here $\vec{S}$ is the spin of a neutron in a magnetic field $\vec{B}$ and
$\gamma$ is its gyromagnetic ratio. The magnetic fields of the superconducting
solenoids, the storage octupole and the compensation octupole and the two spin
flippers were implemented as analytical approximations (see [20]).
Furthermore, the Stern-Gerlach force and gravity are included.

###  3.1 Single spin flipper efficiency

The influence of the residual transversal fields was investigated by
estimating the efficiency $\varepsilon_{\text{SF}}$ of spin flipping initial
HFS neutrons to the LFS state with SF2. The simulation was performed for two
different spin flip positions
$z_{\text{SF2},\text{a}}=$980\text{\,}\mathrm{mm}$$ and
$z_{\text{SF2},\text{b}}=$1020\text{\,}\mathrm{mm}$$ (see figure 14).

Figure 14: Analytic magnetic field approximations used for the Monte Carlo
simulation. The inset shows the compensated octupole field on a horizontal
line ($y=$0\text{\,}\mathrm{mm}$$). Two spin flip positions were simulated,
$z_{\text{SF2},\text{a}}=$980\text{\,}\mathrm{mm}$$ and
$z_{\text{SF2},\text{b}}=$1020\text{\,}\mathrm{mm}$$. The solid and dashed
areas mark the range covered by the spin flipper in each case.

Each simulation comprised a set of the same $N=4000$ neutrons with fixed but
random parameters: starting positions in the $x-y$ plane within the narrowed
neutron guide ($r=\sqrt{x^{2}+y^{2}}\leq$25\text{\,}\mathrm{mm}$$), equally
distributed kinetic energies between $0\text{\,}\mathrm{neV}$ and
$100\text{\,}\mathrm{neV}$, emission angles with respect to the $z$-axis up to
$\theta_{\text{max}}=$25\text{\,}\mathrm{\SIUnitSymbolDegree}$$ and the spin
$\vec{S}$ antiparallel to $\vec{B}_{z}$ (HFS state). The start position in
$z$-direction was set to $z=z_{\text{SF2}}-$162\text{\,}\mathrm{mm}$$ to save
computing time. The simulation of a single neutron is aborted as soon as it is
back reflected ($v_{z}<0$). At $z=$1100\text{\,}\mathrm{mm}$$, the simulation
is terminated and the number of successfully spin flipped neutrons
$N_{\text{LFS}}$ is counted. The spin flip efficiency is then calculated as

$\varepsilon_{\text{SF}}(\%)=\frac{N_{\text{LFS}}}{N}\cdot 100\,.$ (19)

The simulation was repeated with different $B_{1}$ amplitudes, the resulting
dependency is shown in figure 15. The lines are logistics functions

$\varepsilon_{\text{SF}}=\frac{\varepsilon_{\text{sat}}}{\raisebox{-2.27621pt}{$1+\exp(-b(B_{1}-B_{1}^{\text{mid}}))$}}$
(20)

with $\varepsilon_{\text{sat}}$ the spin flip efficiency in saturation, $b$
the steepness, and $B_{1}^{\text{mid}}$ the amplitude with
$50\text{\,}\mathrm{\char 37\relax}$ spin flip efficiency. Despite lower
amplitude requirements at $z_{\text{SF2},\text{a}}=$980\text{\,}\mathrm{mm}$$,
in both positions, a saturation of the spin flip efficiency is achieved. The
result of this simulation reinforces the experimentally determined spin flip
efficiency of SF2 (see section 2.2.2, figure 8).

Figure 15: Spin flip efficiency $\varepsilon_{\text{SF}}$ depending on the
irradiated $B_{1}$ amplitude as inferred from a Monte Carlo simulation at
$z_{\text{SF2},\text{a}}=$980\text{\,}\mathrm{mm}$$ and
$z_{\text{SF2},\text{b}}=$1020\text{\,}\mathrm{mm}$$. The efficiency was
calculated using 19, the lines are logistics functions (20) plotted to guide
the eye.

###  3.2 Comparison of trap filling with two spin flippers

The dependence of the effectiveness of the double spin flip technique on the
beam divergence $\theta_{\text{max}}$, i.e., the maximum angle up to which
neutron trajectories occur relative to the $z$-axis, was discussed in Sec.
2.2.2. In order to compare the single and double spin flip technique with
respect to $\theta_{\text{max}}$ simulations were performed with a set of
neutrons ($N=4000$) with randomized positions in the $xy$-plane, a
$\sqrt{E}$-dependent kinetic energy distribution between
$0\text{\,}\mathrm{neV}$ and $200\text{\,}\mathrm{neV}$ and randomized
emission angles between $0\text{\,}\mathrm{\SIUnitSymbolDegree}$ and
$\theta_{\text{max}}$ with respect to the $z$-axis. The simulations started in
front of the first field maximum of $B_{z}$ at
$z=$-1500\text{\,}\mathrm{mm}$$. Both spin flipper field amplitudes were set
to $B_{1}=$0.5\text{\,}\mathrm{mT}$$ which is sufficient for
$\varepsilon_{\text{SF}}=1$. For the simulation of the single spin flip
technique, SF1 was switched off ($B_{1}=$0\text{\,}\mathrm{mT}$$). The time
step size in the simulation was optimized in advance [20].

Figure 16: Simulation results obtained for the single and double spin flip
technique as fraction of storable neutrons $n_{\text{LFS}}$ depending on the
initial beam divergence $\theta_{\text{max}}$. The decline in $n_{\text{LFS}}$
with increasing $\theta_{\text{max}}$ is caused by a higher number of neutrons
being back-reflected from the increasing magnetic potential before reaching
SF2 when using the double spin flip technique.

Figure 16 shows the resulting storable fraction of the initial neutron
spectrum $n_{\text{LFS}}=N_{\text{LFS}}/N$ after passing the spin flippers.
The criterion was that the neutrons must exit SF2 as low-field seekers with a
total energy of $E\leq$47\text{\,}\mathrm{neV}$$. The gravitational potential
was inferred from the final $y$-position after the simulation terminated and
was included in the total energy. It shows that up to
$\theta_{\text{max}}\simeq$20\text{\,}\mathrm{\SIUnitSymbolDegree}$$ the
double spin flip is clearly superior to the single spin flip technique.
However, at higher beam divergences, both approaches become roughly
comparable. One reason for this might be that at larger emission angles, a
higher fraction of the neutrons are back reflected before they reach SF2 when
the double spin-flip technique is used.

##  4 Systematic effects on neutron storage time

The neutron storage time constant $\tau$ is related to the $\beta$-decay
lifetime $\tau_{\beta}$ via

$\frac{1}{\tau}=\frac{1}{\tau_{\beta}}+\sum\frac{1}{\tau_{\mathrm{X}}}$ (21)

where $\tau_{\mathrm{X}}$ characterizes losses through additional mechanisms.
These systematic effects of ”bottle”-type neutron lifetime experiments can be
separated into wall and non-wall related interactions. Earlier measurements
(see e.g. references 8-20 in [29]) have been regularly re-analyzed and
corrected for previously not known effects or shifts of effect sizes. Due to
the fact that $\tau$SPECT uses fully magnetic confinement of neutrons, the
first category is eliminated by design. Material wall interactions only happen
during the filling process, in preparation of the trapped neutron ensemble,
before the actual storage time is started. Known systematic changes to the
measured neutron lifetime not related to wall interactions can originate from

  1. 1.

interactions of the neutrons with residual gas molecules or atoms,

  2. 2.

loss of neutron polarization due to insufficient adiabaticity of magnetic
field changes,

  3. 3.

or the presence of marginally trapped neutrons in the trap after the filling
process.

1\. The superconducting coils and the permanent magnet array of $\tau$SPECT
are situated in a cold bore cryostat. Thus, all material surrounding the UCN
storage region is cooled to approximately $170\text{\,}\mathrm{K}$, the
cryostat walls even to approximately $50\text{\,}\mathrm{K}$, meaning that
many gases freeze on the walls, leading to good vacuum conditions. In steady
state, the remaining pressure inside $\tau$SPECT is of the order of
$2\text{\times}{10}^{-7}\text{\,}\mathrm{mbar}$. This is measured on the
neutron detector side of the vacuum chamber, while all pumps are connected at
the source side. Thus, the pressure inside the storage region can be assumed
to be even lower. In contrast to historical neutron lifetime experiments,
$\tau$SPECT does not use [3]He as a detection gas, which has a very large
neutron capture cross-section such that tiny partial pressures could lead to a
significant increase of neutron losses. To precisely determine a possible
small correction to the measured lifetime, a residual gas analyzer will be
connected in the future to the vacuum chamber during data taking such that
scattering and absorption probabilities for the identified gases can be
calculated. 2\. The magnetic field created by the superconducting solenoids
and the permanent octupole magnet in $\tau$SPECT is well understood. With the
strong holding field of approximately $200\text{\,}\mathrm{mT}$, a
depolarization time constant
$\tau_{\text{dep}}=$1.2\text{\times}{10}^{9}\text{\,}\mathrm{s}$\gg\tau_{n}$
was estimated in the worst case. This contribution is thus negligible. 3\.
Marginally trapped neutrons are neutrons on long-lived trajectories with total
energies above the trapping potential of the magnetic field configuration.
They can mimic a shorter neutron lifetime by leaving the trap at intermediate
times, such that they are not counted after long storage times. The measured
storage curve then shows a non-exponential behavior. In order to prepare the
UCNs inside the $\tau$SPECT trap such that no marginally trapped neutrons
remain when the actual storage period starts, the UCN detector is inserted
from its storage position $z_{\text{store}}^{\text{det}}$ to the cleaning
position $z_{\text{clean}}^{\text{det}}$ further inside the trap volume. UCNs
that can reach the boron layer of the detector at this position will be
absorbed, and thus the UCN energy spectrum is cut off below the trap edge
energy. After a cleaning time $t_{\text{clean}}$, the detector is moved back
to its storage position. Both $z_{\text{clean}}$ and $t_{\text{clean}}$ can be
optimized in order to effectively remove marginally trapped neutrons while
efficiently keeping as many storable neutrons in the trap as possible. In
order to make the cleaning process as effective as possible, $\tau$SPECT can
be tilted by a small angle towards the detector side, which breaks the
symmetry with respect to gravity, such that no closed orbits exist which never
intersect with the detector plane in cleaning position. Additional
measurements and simulations are currently carried out to understand and
improve the cleaning procedure.

##  5 Time structure of a single measurement

Figure 17: Positions of spin flipper SF2 and detector in the longitudinal
magnetic field (red line). The horizontal line (yellow) marks the trap limit
at $B=$0.779\text{\,}\mathrm{T}$$ imposed by
$B_{r}^{\text{min}}(r=$53.5\text{\,}\mathrm{mm}$)$. The accessible ranges of
spin flipper SF2 and detector are marked by the gray boxes and range from
$z=$397\text{\,}\mathrm{mm}$$ to $1200\text{\,}\mathrm{mm}$ for the spin
flipper and $z=$1599\text{\,}\mathrm{mm}$$ to $1934\text{\,}\mathrm{mm}$ for
the detector. The black lines indicate the positions of spin flipper and
detector during individual steps of the measurement sequence with
$z_{\text{store}}^{\text{SF}}=$400\text{\,}\mathrm{mm}$$ and
$z_{\text{fill}}^{\text{SF}}=$980\text{\,}\mathrm{mm}$$ for the spin flipper
as well as $z_{\text{count}}^{\text{det}}=$1604\text{\,}\mathrm{mm}$$,
$z_{\text{clean}}^{\text{det}}=$1854\text{\,}\mathrm{mm}$$ and
$z_{\text{store}}^{\text{det}}=$1932\text{\,}\mathrm{mm}$$ for the detector.

Each $\tau$SPECT measurement consists of four phases, which are controlled by
a timing device providing $5\text{\,}\mathrm{V}$ and $24\text{\,}\mathrm{V}$
logic outputs. Via these channels, all devices required to perform a
measurement can be either triggered to perform predefined actions or are
active as long as the corresponding output is switched on. Initially, the
detector is in its storage position
($z_{\text{store}}^{\text{det}}=$1932\text{\,}\mathrm{mm}$$) and the spin
flipping unit is at the filling position
($z_{\text{fill}}^{\text{SF}}=$980\text{\,}\mathrm{mm}$$) as in figure 17.
Filling phase: When the reactor pulse is triggered, the SFU is switched on. It
generates the rotating $B_{1}$ field and UCNs are filled into the trap. The
optimum active duration was found to be
$t_{\text{fill}}=$6\text{\,}\mathrm{s}$$. Subsequently, the SFU is retracted
from the trap volume to its storage position
($z_{\text{store}}^{\text{SF}}=$400\text{\,}\mathrm{mm}$$) in the high field.
Cleaning phase: The detector is moved to the fringe of the storage volume
($z_{\text{clean}}^{\text{det}}=$1854\text{\,}\mathrm{mm}$$) for
$200\text{\,}\mathrm{s}$ in order to remove marginally trapped neutrons from
the UCN spectrum (see Sec. 4). Afterward, it is retracted to its storage
position ($z_{\text{store}}^{\text{det}}$).
Storage phase: The logic signal causing the detector stepper motor to retract
the detector to its storage position marks the beginning of the nominal
storage time, which is typically varied between $20\text{\,}\mathrm{s}$ and
$2000\text{\,}\mathrm{s}$.
Counting phase: The nominal storage time ends with another logic signal to the
stepper motor of the detector, which then moves to the counting position
$z_{\text{count}}^{\text{det}}$ in the low field and counts the remaining UCNs
in the trap volume. Finally, SFU and detector are moved back to their initial
positions in preparation for the next measurement. With the detector moving at
a constant velocity of $10\text{\,}\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{s}$,
the storage time for each detected UCN varies. So instead of the nominal
storage time, we consider the effective storage time

$t_{s}^{\text{eff}}=t_{s}+t_{s}^{\prime}.$ (22)

Here, $t_{s}$ is the nominal storage time and
$t_{s}^{\prime}=$7.8\text{\,}\mathrm{s}$$ is the time in which the detector
moves from $z_{\text{clean}}^{\text{det}}$ to $z_{\text{store}}^{\text{det}}$.
For simplicity, however, we will refer to the nominal storage time for the
rest of this paper.
The detector raw data sent by the ADC (see Sec. 2.3) is written to hard drive
during the entire measurement sequence, which allows for offline waveform
analyses and e.g., retrospective optimization of the event detection
algorithm. Generally, each measurement follows a specific neutron counting
pattern, as is shown in figure 18. When the pulse is triggered, a short peak
caused by thermal neutrons is visible in the detector data (see inset in
figure 18), the timing of which is used as reference
$t_{0}=$0\text{\,}\mathrm{s}$$.

Figure 18: Typical neutron detection time distribution with
$200\text{\,}\mathrm{s}$ cleaning duration and $50\text{\,}\mathrm{s}$ storage
time at $2\text{\,}\mathrm{s}$ time binning. The origin of the time axis is
given by the pulse of thermal neutrons, which is visible as a sharp peak in
the inset ($100\text{\,}\mathrm{ms}$ binning) and marked as $t_{0}$ (red
line). This peak is not visible in the overview plot with larger time binning.
Cleaning starts $10\text{\,}\mathrm{s}$ after filling is finished, the yellow
region marks the cleaning interval. The counting interval in
$t=[265,340]\,\mathrm{s}$ is marked in green. The background interval
($t=[345,420]\,\mathrm{s}$) is shown in gray.

Many neutrons (UCNs and faster neutrons) guided along the beam tube which are
too fast to be stored in the trap reach the detector within the first few
seconds after the reactor pulse. Subsequently, ($t>$10\text{\,}\mathrm{s}$$),
after the detector has been moved to the cleaning position, marginally trapped
neutrons captured during the cleaning process appear as an additional peak in
the arrival spectrum. The interval of interest for measuring the storage curve
is the ’counting peak’ which follows after the predefined storage time. The
start time of this interval is calculated as
$t_{\text{clean}}^{\text{start}}+t_{\text{clean}}+t_{s}+$5\text{\,}\mathrm{s}$$
and it has a length of $75\text{\,}\mathrm{s}$. The length was estimated from
the duration when the neutron count rate drops back to background level. In
principle, the neutron lifetime needs to be factored in when integrating the
counting peak, as both detector and neutrons move at finite velocity. That is,
each of the neutrons experiences a different storage time and thus has a
different decay probability. By ensuring the counting procedure is identical
for every measurement, effects from differing storage times on the measured
lifetime are eliminated. The background count rate is obtained from an
interval with a length of $75\text{\,}\mathrm{s}$ starting
$5\text{\,}\mathrm{s}$ after the end of the nominal counting interval. On
average, the background count rate is $\sim$1.1\text{\,}\mathrm{Hz}$$. The
number of counted neutrons in the background interval is subsequently
subtracted from the number of counted neutrons in the counting interval.

##  6 First UCN storage curve

Figure 19: Storage curve measured in June/July 2021 at the research reactor
TRIGA Mainz with cleaning for $200\text{\,}\mathrm{s}$ at
$z_{\text{clean}}^{\text{det}}=$1854\text{\,}\mathrm{mm}$$. The lines are a
multidimensional fit (23) with each day assigned a separate amplitude but a
common decay constant $\tau=$859(16)\text{\,}\mathrm{s}$$. The remaining fit
parameters are listed in table 1.

After first parameter optimizations of the filling and cleaning procedures, a
storage curve was measured in June/July 2021 (see figure 19). Cleaning was
done at $z_{\text{clean}}^{\text{det}}=$1854\text{\,}\mathrm{mm}$$ for
$200\text{\,}\mathrm{s}$. During the beamtime strong daily fluctuations of the
UCN source were observed. To account for this observation the data were split
by days and a multidimensional fit of the form

$N(t)=N_{0}^{i}\exp(-t/\tau)$ (23)

was applied, allowing for varying initial numbers of neutrons, $N_{0}^{i}$,
but using the same decay constant $\tau$. The fit parameters are listed in
Table 1. Our first estimation for the neutron storage time constant is

$\tau=$859(16)\text{\,}\mathrm{s}$.$ (24)

This value is smaller than the current particle data group (PDG 2022) average
of $878.4(5)\text{\,}\mathrm{s}$ [30] which might indicate that not all
marginally trapped neutrons were cleaned during the $200\text{\,}\mathrm{s}$
cleaning interval (see Sec. 4). Therefore, measurements with varying cleaning
parameters will be necessary. To increase the counting statistics, we will
further investigate optimization of the filling process using the double spin
flip technique. Additionally, systematic investigations are ongoing, which
include Monte Carlo simulations of the cleaning process.

Table 1: Multidimensional fit results of a single exponential function (cf. 23) to the storage curve depicted in figure 19. Measurement day | Parameter | Value
---|---|---
June 28th | $N_{0}^{1}$ | $477(13)$
June 29th | $N_{0}^{2}$ | $557(11)$
June 30th | $N_{0}^{3}$ | $592(11)$
July 01st | $N_{0}^{4}$ | $574(10)$
July 02nd | $N_{0}^{5}$ | $605(15)$
July 05th | $N_{0}^{6}$ | $636(10)$
July 07th | $N_{0}^{7}$ | $628(11)$
July 08th | $N_{0}^{8}$ | $646(10)$
July 09th | $N_{0}^{9}$ | $650(13)$
| $\tau$ | $859(16)\text{\,}\mathrm{s}$
| $\chi^{2}/\text{ndf}$ | $1.14$

##  7 Summary

The $\tau$SPECT experiment has for the first time demonstrated the loading of
a three-dimensional magnetic field gradient trap with UCNs using a pulsed
spin-flip loading scheme. The implementation of a magnetic field gradient
compensation octupole in combination with adiabatic fast passage spin flippers
allows for trap loading with subsequent retraction of all material surfaces
from the storage volume. A sequential implementation of spin flippers
increases and shifts the kinetic energy acceptance interval of the $\tau$SPECT
experiment into a region of higher production rate from the UCN source. UCN
losses related to absorption on residual gas or due to spin-depolarization
have been shown to be negligible for a future measurement of the free neutron
lifetime. Our custom-built, SiPM-based in-situ UCN detector serves a dual
purpose. Partially inserted into the trapping volume, it absorbs marginally
trapped neutrons with kinetic energies above the nominal trap depth. The
parameter space for this spectral cleaning procedure was investigated within
the available statistics. After the preset neutron storage time, the detector
is fully inserted into the trap volume to count the remaining UCNs without
additional transport losses. Operation of $\tau$SPECT at the research reactor
TRIGA Mainz allowed to extract a neutron storage time constant of
$\tau=$859(16)\text{\,}\mathrm{s}$$, compatible with the most precise, but not
to be interpreted as a measurement of the free neutron lifetime. Relocation of
the $\tau$SPECT experiment to a more intense UCN source will allow the
investigation of all relevant systematic effects required to extract the free
neutron lifetime with an anticipated precision of $0.2\text{\,}\mathrm{s}$
before a large volume version of the setup can push the achievable precision
in the future.

We thank the administration of the research reactor TRIGA Mainz as well as the
reactor operators for providing us beamtime. We are grateful to the mechanical
and electrical workshops of the Department of Chemistry and of the Institute
of Physics at JGU Mainz. We thank S. Vanneste for his careful reading of the
text and helpful comments. The development of $\tau$SPECT has been supported
by the Cluster of Excellence “Precision Physics, Fundamental Interactions, and
Structure of Matter” (PRISMA+ EXC 2118/1) funded by the German Research
Foundation (DFG) within the German Excellence Strategy (Project ID 39083149).

## References

## References

  * [1] R.H. Cyburt, B.D. Fields, K.A. Olive, T.H. Yeh, Rev. Mod. Phys. 88, 015004 (2016).  DOI 10.1103/RevModPhys.88.015004
  * [2] J.C. Hardy, I.S. Towner, Phys. Rev. C 102, 045501 (2020).  DOI 10.1103/PhysRevC.102.045501.  URL https://link.aps.org/doi/10.1103/PhysRevC.102.045501
  * [3] B. Märkisch, H. Mest, H. Saul, et al., Phys. Rev. Lett. 122, 242501 (2019).  DOI 10.1103/PhysRevLett.122.242501.  URL https://link.aps.org/doi/10.1103/PhysRevLett.122.242501
  * [4] A.T. Yue, M.S. Dewey, D.M. Gilliam, et al., Phys. Rev. Lett. 111(22) (2013).  DOI 10.1103/physrevlett.111.222501.  URL http://dx.doi.org/10.1103/PhysRevLett.111.222501
  * [5] A.P. Serebrov, E.A. Kolomensky, A.K. Fomin, et al., Phys. Rev. C 97(5) (2018).  DOI 10.1103/physrevc.97.055503.  URL http://dx.doi.org/10.1103/PhysRevC.97.055503
  * [6] F.M. Gonzalez, E.M. Fries, C. Cude-Woods, et al., Phys. Rev. Lett. 127, 162501 (2021).  DOI 10.1103/PhysRevLett.127.162501.  URL https://link.aps.org/doi/10.1103/PhysRevLett.127.162501
  * [7] J. Byrne, D.L. Worcester, Journal of Physics G: Nuclear and Particle Physics 46(8), 085001 (2019).  DOI 10.1088/1361-6471/ab256b.  URL https://doi.org/10.1088/1361-6471/ab256b
  * [8] A.P. Serebrov, M.E. Chaikovskii, G.N. Klyushnikov, O.M. Zherebtsov, A.V. Chechkin, Phys. Rev. D 103, 074010 (2021).  DOI 10.1103/PhysRevD.103.074010.  URL https://link.aps.org/doi/10.1103/PhysRevD.103.074010
  * [9] R. Golub, D. Richardson, S. Lamoreaux, CRC Press (1991).  URL https://books.google.de/books?id=0yAuDwAAQBAJ
  * [10] J. Kahlenberg, D. Ries, K.U. Ross, C. Siemensen, M. Beck, C. Geppert, W. Heil, N. Hild, J. Karch, S. Karpuk, F. Kories, M. Kretschmer, B. Lauss, T. Reich, Y. Sobolev, N. Trautmann, The European Physical Journal A 53(11), 226 (2017).  DOI 10.1140/epja/i2017-12428-9
  * [11] G. Bison, M. Daum, K. Kirch, B. Lauss, D. Ries, P. Schmidt-Wellenburg, G. Zsigmond, T. Brenner, P. Geltenbort, T. Jenke, O. Zimmer, M. Beck, W. Heil, J. Kahlenberg, J. Karch, K. Ross, K. Eberhardt, C. Geppert, S. Karpuk, T. Reich, C. Siemensen, Y. Sobolev, N. Trautmann, Phys. Rev. C 95, 045503 (2017).  DOI 10.1103/PhysRevC.95.045503.  URL https://link.aps.org/doi/10.1103/PhysRevC.95.045503
  * [12] S. Baeßler, F. Ayala Guardia, M. Borg, F. Glück, W. Heil, G. Konrad, I. Konorov, R. Muñoz Horta, G. Petzoldt, D. Rich, M. Simson, Y. Sobolev, H.F. Wirth, O. Zimmer, The European Physical Journal A 38, 17 (2008).  DOI 10.1140/epja/i2008-10660-0.  URL https://doi.org/10.1140/epja/i2008-10660-0
  * [13] M. Beck, F. Ayala Guardia, M. Borg, J. Kahlenberg, R. Muñoz Horta, C. Schmidt, A. Wunderle, W. Heil, R. Maisonobe, M. Simson, T. Soldner, R. Virot, O. Zimmer, M. Klopf, G. Konrad, S. Baeßler, F. Glück, U. Schmidt, Phys. Rev. C 101, 055506 (2020).  DOI 10.1103/PhysRevC.101.055506.  URL https://link.aps.org/doi/10.1103/PhysRevC.101.055506
  * [14] Glück, F., Baeßler, S., Byrne, J., van der Grinten, M. G. D., Hartmann, F. J., Heil, W., Konorov, I., Petzoldt, G., Sobolev, Yu., Zimmer, O., Eur. Phys. J. A 23(1), 135 (2005).  DOI 10.1140/epja/i2004-10057-1.  URL https://doi.org/10.1140/epja/i2004-10057-1
  * [15] K. Halbach, Nuclear Instruments and Methods 169(1), 1 (1980).  DOI 10.1016/0029-554X(80)90094-4.  URL http://www.sciencedirect.com/science/article/pii/0029554X80900944
  * [16] J.P. Karch, Design und aufbau des experiments tauspect zur messung der neutronlebensdauer mittels magnetischer wandspeicherung.  Ph.D. thesis, Johannes Gutenberg University Mainz (2017).  URL http://doi.org/10.25358/openscience-1364
  * [17] P. Blümler, Cells 10, 2708 (2021).  DOI 10.3390/cells10102708.  URL https://www.mdpi.com/2073-4409/10/10/2708
  * [18] H.A. Leupold, A.S. Tilak, E.I. Potenziani, IEEE Trans. Magn. 29, 2902 (1993).  DOI 10.1109/20.281092.  URL https://ieeexplore.ieee.org/document/281092
  * [19] A. Abragam, _The Principles of Nuclear Magnetism_.  International series of monographs on physics (Clarendon Press, 1961).  URL https://books.google.de/books?id=9M8U_JK7K54C
  * [20] K.U. Roß, Towards a high precision measurement of the free neutron lifetime with $\tau$spect.  Ph.D. thesis, Johannes Gutenberg University Mainz (2021).  URL http://doi.org/10.25358/openscience-6540
  * [21] J. Kahlenberg, First full-magnetic storage of ultracold neutrons in the $\tau$spect experiment for measuring the neutron lifetime.  Ph.D. thesis, Johannes Gutenberg University Mainz (2020).  URL http://doi.org/10.25358/openscience-5455
  * [22] V.I. Luschikov, Y.V. Taran, Nucl. Instrum. Methods Phys. Res. A 228(1), 159 (1984).  DOI 10.1016/0168-9002(84)90025-1.  URL http://www.sciencedirect.com/science/article/pii/0168900284900251
  * [23] P. Geltenbort, L. Göltl, R. Henneck, K. Kirch, A. Knecht, M. Kuźniak, B. Lauss, T. Lefort, A. Mtchedlishvili, M. Meier, M. Negrazus, G. Petzoldt, P. Ruettimann, V. Vrankovic, G. Zsigmond, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 608(1), 132–138 (2009).  DOI 10.1016/j.nima.2009.06.038.  URL https://www.sciencedirect.com/science/article/pii/S0168900209012418
  * [24] A.T. Holley, L.J. Broussard, J.L. Davis, Rev. Sci. Instrum. 83(7), 073505 (2012).  DOI 10.1063/1.4732822.  URL https://doi.org/10.1063/1.4732822
  * [25] O. Zimmer, Journal of Physics G: Nuclear and Particle Physics 26(1), 67 (2000).  DOI 10.1088/0954-3899/26/1/307.  URL https://dx.doi.org/10.1088/0954-3899/26/1/307
  * [26] E. Tiesinga, P.J. Mohr, D.B. Newell, B.N. Taylor, Journal of Physical and Chemical Reference Data 50(3), 033105 (2021).  DOI 10.1063/5.0064853.  URL https://doi.org/10.1063/5.0064853
  * [27] P. Virtanen, R. Gommers, T.E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S.J. van der Walt, M. Brett, J. Wilson, K.J. Millman, N. Mayorov, A.R.J. Nelson, E. Jones, R. Kern, E. Larson, C.J. Carey, İ. Polat, Y. Feng, E.W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E.A. Quintero, C.R. Harris, A.M. Archibald, A.H. Ribeiro, F. Pedregosa, P. van Mulbregt, SciPy 1.0 Contributors, Nature Methods 17, 261 (2020).  DOI 10.1038/s41592-019-0686-2
  * [28] F. Bloch, Phys. Rev. 70, 460 (1946).  DOI 10.1103/PhysRev.70.460.  URL https://link.aps.org/doi/10.1103/PhysRev.70.460
  * [29] C. Cude-Woods, F.M. Gonzalez, E.M. Fries, T. Bailey, M. Blatnik, N.B. Callahan, J.H. Choi, S.M. Clayton, S.A. Currie, M. Dawid, B.W. Filippone, W. Fox, P. Geltenbort, E. George, L. Hayen, K.P. Hickerson, M.A. Hoffbauer, K. Hoffman, A.T. Holley, T.M. Ito, A. Komives, C.Y. Liu, M. Makela, C.L. Morris, R. Musedinovic, C. O’Shaughnessy, R.W. Pattie, J. Ramsey, D.J. Salvat, A. Saunders, E.I. Sharapov, S. Slutsky, V. Su, X. Sun, C. Swank, Z. Tang, W. Uhrich, J. Vanderwerp, P. Walstrom, Z. Wang, W. Wei, A.R. Young, Phys. Rev. C 106, 065506 (2022).  DOI 10.1103/PhysRevC.106.065506.  URL https://link.aps.org/doi/10.1103/PhysRevC.106.065506
  * [30] R. Workman, et al., Prog. Theor. Exp. Phys. 083C01 (2022)


Note that $Q^{*}$ is independent of the sampling policy of the agent.
Furthermore, $\|\cdot\|_{c}=\|\cdot\|_{\infty}$.

######  Proposition D.1.

Federated $Q$-learning algorithm 3 is equivalent to the FedSAM Algorithm 4
with the following parameters.

  1. 1.

${\bm{\theta}^{i}_{t}}=Q^{i}_{t}-Q^{*}$

  2. 2.

$S_{t}=(S_{t}^{1},\dots,S_{t}^{\rm NewA})$ and
$A_{t}=(A_{t}^{1},\dots,A_{t}^{\rm NewA})$

  3. 3.

${\mathbf{y}^{i}_{t}}=(S_{t}^{i},A_{t}^{i},S_{t+1}^{i},A_{t+1}^{i})$ and
$\mathbf{{\mathbf{Y}_{t}}}=(S_{t},A_{t},S_{t+1},A_{t+1})$

  4. 4.

$\mu^{i}:$ Stationary distribution of the sampling policy of the i’th agent.

  5. 5.

${\mathbf{G}^{i}}({\bm{\theta}^{i}_{t}},{\mathbf{y}^{i}_{t}})_{(s,a)}={\bm{\theta}^{i}_{t}}(s,a)$
$+\mathbbm{1}_{\\{S_{t}^{i}=s,A_{t}^{i}=a\\}}\times\big{[}\gamma\max_{a^{\prime}}\left({\bm{\theta}^{i}_{t}}+Q^{*}(S_{t+1}^{i},a^{\prime})\right)-{\bm{\theta}^{i}_{t}}(S_{t}^{i},A_{t}^{i})-\gamma\max_{a^{\prime}}Q^{*}(S_{t+1}^{i},a^{\prime})\big{]}$

  6. 6.

$\mathbf{b}^{i}({\mathbf{y}^{i}_{t}})_{(s,a)}=\mathbbm{1}_{\\{S_{t}^{i}=s,A_{t}^{i}=a\\}}\left[\mathcal{R}(S_{t}^{i},A_{t}^{i})+\gamma\max_{a^{\prime}}Q^{*}(S_{t+1}^{i},a^{\prime})-Q^{*}(S_{t}^{i},A_{t}^{i})\right]$

where $\mathbbm{1}_{A}$ is the indicator function corresponding to set $A$,
such that $\mathbbm{1}_{A}=1$ is $A$ is true, and $0$ otherwise.

######  Lemma D.1.

Consider the federated $Q$-learning Algorithm 3 as a special case of FedSAM
(as specified in Proposition D.1). Suppose the trajectory
$\\{S_{t}^{i},A_{t}^{i}\\}_{t=0,1,\dots}$ converges geometrically fast to its
stationary distribution as follows
$d_{TV}(P(S_{t}^{i}=\cdot,A_{t}^{i}=\cdot|S_{0}^{i},A_{0}^{i})||\mu^{i}(\cdot,\cdot))\leq\bar{m}\bar{\rho}^{t}$
for all $i=1,2,\dots,{\rm NewA}$. The corresponding
${\bar{\mathbf{G}}}^{i}({\bm{\theta}})$ in Assumption 6.1 for the federated
$Q$-learning is as follows

$\displaystyle{\bar{\mathbf{G}}}^{i}({\bm{\theta}})_{(s,a)}=$
$\displaystyle{\bm{\theta}}(s,a)+\mu^{i}(s,a)\times{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\gamma\max_{a^{\prime}}\left({\bm{\theta}}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}(s,a)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\right].$

Furthermore, we have $m_{1}=2A_{2}\bar{m}$, where $A_{2}$ is specified in
Lemma D.3, $m_{2}=0$, and $\bar{\rho}=\rho$.

######  Lemma D.2.

Consider the federated $Q$-learning as a special case of FedSAM (as specified
in Proposition D.1). The corresponding contraction factor $\gamma_{c}$ in
Assumption 6.2 for this algorithm is
$\gamma_{c}=\left(1-(1-\gamma)\mu_{\min}\right)$, where
$\mu_{\min}=\min_{s,a,i}\mu^{i}(s,a)$

######  Lemma D.3.

Consider the federated $Q$-learning as a special case of FedSAM (as specified
in Proposition D.1). The constants $A_{1}$, $A_{2}$, and $B$ in Assumption 6.2
are as follows: $A_{1}=A_{2}=2$ and $B=\frac{2}{1-\gamma}$.

######  Lemma D.4.

Consider the federated $Q$-learning as a special case of FedSAM (as specified
in Proposition D.1). Assumption 6.4 holds for this algorithm.

###  D.1 Proofs

###### Proof of Proposition D.1.

Items 1-4 are by definition. Furthermore, by the update of the $Q$-learning,
and subtracting $Q^{*}$ from both sides, we have

$\displaystyle\underbrace{{Q}^{i}_{t+1}(s,a)-Q^{*}(s,a)}_{{\bm{\theta}}^{i}_{t+1}(s,a)}=\underbrace{{Q}^{i}_{t}(s,a)-Q^{*}(s,a)}_{{\bm{\theta}^{i}_{t}}(s,a)}$
$\displaystyle+\alpha\mathbbm{1}_{\\{(s,a)=(S_{t}^{i},A_{t}^{i})\\}}\left(\mathcal{R}(S_{t}^{i},A_{t}^{i})+\gamma\max_{a}{Q}^{i}_{t}(S_{t+1}^{i},a)-{Q}^{i}_{t}(S_{t}^{i},A_{t}^{i})\right)$
$\displaystyle=$ $\displaystyle{\bm{\theta}^{i}_{t}}(s,a)$
$\displaystyle+\alpha\mathbbm{1}_{\\{(s,a)=(S_{t}^{i},A_{t}^{i})\\}}\bigg{(}\mathcal{R}(S_{t}^{i},A_{t}^{i})+\gamma\max_{a}\left[\underbrace{{Q}^{i}_{t}(S_{t+1}^{i},a)-Q^{*}(S_{t+1}^{i},a)}_{{\bm{\theta}}^{i}_{t}(S_{t+1}^{i},a)}+Q^{*}(S_{t+1}^{i},a)\right]$
$\displaystyle-\left(\underbrace{{Q}^{i}_{t}(S_{t}^{i},A_{t}^{i})-Q^{*}(S_{t}^{i},A_{t}^{i})}_{{\bm{\theta}}^{i}_{t}(S_{t}^{i},A_{t}^{i})}+Q^{*}(S_{t}^{i},A_{t}^{i})\right)\bigg{)}$
(addition and subtraction) $\displaystyle=$
$\displaystyle{\bm{\theta}^{i}_{t}}(s,a)$
$\displaystyle+\alpha\bigg{(}\underbrace{{\bm{\theta}^{i}_{t}}(s,a)+\mathbbm{1}_{\\{S_{t}^{i}=s,A_{t}^{i}=a\\}}\times\big{[}\gamma\max_{a^{\prime}}\left({\bm{\theta}^{i}_{t}}+Q^{*}(S_{t+1}^{i},a^{\prime})\right)-{\bm{\theta}^{i}_{t}}(S_{t}^{i},A_{t}^{i})-\gamma\max_{a^{\prime}}Q^{*}(S_{t+1}^{i},a^{\prime})\big{]}}_{{\mathbf{G}^{i}}({\bm{\theta}^{i}_{t}},{\mathbf{y}^{i}_{t}})_{(s,a)}}-{\bm{\theta}^{i}_{t}}(s,a)$
$\displaystyle\quad\quad+\underbrace{\mathbbm{1}_{\\{S_{t}^{i}=s,A_{t}^{i}=a\\}}\left(\mathcal{R}(S_{t}^{i},A_{t}^{i})+\gamma\max_{a^{\prime}}Q^{*}(S_{t+1}^{i},a^{\prime})-Q^{*}(S_{t}^{i},A_{t}^{i})\right)}_{\mathbf{b}^{i}({\mathbf{y}^{i}_{t}})_{(s,a)}}\bigg{)},$

which proves items 5 and 6. Furthermore, for the synchronization part of
$Q$-learning, we have

$\displaystyle{Q}^{i}_{t}$ $\displaystyle\leftarrow\frac{1}{{\rm
NewA}}\sum_{j=1}^{\rm NewA}{Q}^{j}_{t}$
$\displaystyle\implies\underbrace{{Q}^{i}_{t}-Q^{*}}_{{\bm{\theta}^{i}_{t}}}$
$\displaystyle\leftarrow\frac{1}{{\rm NewA}}\sum_{j=1}^{\rm
NewA}\underbrace{({Q}^{j}_{t}-Q^{*})}_{{\bm{\theta}^{j}_{t}}},$

which is equivalent to the synchronization step in FedSAM Algorithm 4. Notice
that here we used the fact that all agents have the same fixed point $Q^{*}$.
∎

###### Proof of Lemma D.1.

${\bar{\mathbf{G}}}({\bm{\theta}})_{(s,a)}$ can be found by simply taking
expectation of
${\mathbf{G}^{i}}({\bm{\theta}^{i}_{t}},{\mathbf{y}^{i}_{t}})_{(s,a)}$,
defined in Proposition D.1, with respect to the stationary distribution
$\mu^{i}$. Furthermore, we have

$\displaystyle\|{\bar{\mathbf{G}}}^{i}({\bm{\theta}})-{\mathbb{E}}[{\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}}_{t}^{i})]\|_{c}$
$\displaystyle=$
$\displaystyle\left\|{\mathbb{E}}_{{\mathbf{y}}_{t}^{i}\sim\mu^{i}}[{\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}}_{t}^{i})]-{\mathbb{E}}[{\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}}_{t}^{i})\right\|_{c}$
$\displaystyle=$
$\displaystyle\left\|\sum_{s,a,s^{\prime},a^{\prime}}\left(\mu^{i}(s,a,s^{\prime},a^{\prime})-P(S_{t}^{i}=s,A_{t}^{i}=a,S_{t+1}^{i}=s^{\prime},A_{t+1}^{i}=a^{\prime}|S_{0}^{i},A_{0}^{i})\right){\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}}_{t}^{i})\right\|_{c}$
$\displaystyle=$
$\displaystyle\sum_{s,a,s^{\prime},a^{\prime}}\left|\mu^{i}(s,a,s^{\prime},a^{\prime})-P(S_{t}^{i}=s,A_{t}^{i}=a,S_{t+1}^{i}=s^{\prime},A_{t+1}^{i}=a^{\prime}|S_{0}^{i},A_{0}^{i})\right|.\left\|{\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}}_{t}^{i})\right\|_{c}$
($\|ax\|_{c}=|a|\|x\|_{c}$) $\displaystyle\leq$
$\displaystyle\sum_{s,a,s^{\prime},a^{\prime}}\left|\mu^{i}(s,a,s^{\prime},a^{\prime})-P(S_{t}^{i}=s,A_{t}^{i}=a,S_{t+1}^{i}=s^{\prime},A_{t+1}^{i}=a^{\prime}|S_{0}^{i},A_{0}^{i})\right|.A_{2}\|{\bm{\theta}}\|_{c}$
(Assumption D.3) $\displaystyle=$
$\displaystyle\sum_{s,a,s^{\prime},a^{\prime}}\bigg{|}\mu^{i}(s,a)\mathcal{P}(s^{\prime}|s,a)\pi^{i}(a^{\prime}|s^{\prime})-P(S_{t}^{i}=s,A_{t}^{i}=a|S_{0}^{i},A_{0}^{i})\mathcal{P}(s^{\prime}|s,a)\pi^{i}(a^{\prime}|s^{\prime})\bigg{|}.A_{2}\|{\bm{\theta}}\|_{c}$
(definition of transition probability) $\displaystyle=$
$\displaystyle\sum_{s,a}\bigg{|}\mu^{i}(s,a)-P(S_{t}^{i}=s,A_{t}^{i}=a|S_{0}^{i},A_{0}^{i})\bigg{|}.A_{2}\|{\bm{\theta}}\|_{c}$
$\displaystyle=$ $\displaystyle
2d_{TV}(\mu^{i}(\cdot,\cdot),P(S_{t}^{i}=\cdot,A_{t}^{i}=\cdot|S_{0}^{i},A_{0}^{i})).A_{2}\|{\bm{\theta}}\|_{c}$
$\displaystyle\leq$ $\displaystyle
2A_{2}\|{\bm{\theta}}\|_{c}\bar{m}\bar{\rho}^{t}.$

In addition, we have

$\displaystyle\|{\mathbb{E}}[\mathbf{b}^{i}({\mathbf{y}}_{t}^{i})]\|_{c}=$
$\displaystyle\max_{s,a}\left|P(S_{t}^{i}=s,A_{t}^{i}=a|S_{0}^{i},A_{0}^{i})\left(\mathcal{R}(s,a)+\gamma{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}[\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})]-Q^{*}(s,a)\right)\right|$
$\displaystyle=$ $\displaystyle 0.$ (Bellman optimality equation (127))

∎

###### Proof of Lemma D.2.

$\displaystyle\|{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{1})-{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{2})\|_{c}=\max_{s,a}|{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{1})_{(s,a)}-{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{2})_{(s,a)}|$
$\displaystyle=$
$\displaystyle\max_{s,a}\Bigg{|}{\bm{\theta}}_{1}(s,a)+\mu^{i}(s,a)\times{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\gamma\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}_{1}(s,a)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\right]$
$\displaystyle\quad-\left({\bm{\theta}}_{2}(s,a)+\mu^{i}(s,a)\times{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\gamma\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}_{2}(s,a)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\right]\right)\Bigg{|}$
$\displaystyle=$
$\displaystyle\max_{s,a}\bigg{|}(1-\mu^{i}(s,a))({\bm{\theta}}_{1}(s,a)-{\bm{\theta}}_{2}(s,a))$
$\displaystyle\quad\quad+\gamma\mu^{i}(s,a){\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right]\bigg{|}$
$\displaystyle\leq$
$\displaystyle\max_{s,a}\Bigg{[}\bigg{|}(1-\mu^{i}(s,a))({\bm{\theta}}_{1}(s,a)-{\bm{\theta}}_{2}(s,a))\bigg{|}$
$\displaystyle+\gamma\bigg{|}\mu^{i}(s,a){\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right]\bigg{|}\Bigg{]}$
(triangle inequality) $\displaystyle\leq$
$\displaystyle\max_{s,a}\Bigg{[}(1-\mu^{i}(s,a))\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}$
(definition of $\|\cdot\|_{\infty}$)
$\displaystyle+\gamma\mu^{i}(s,a)\left[\bigg{|}{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\left[\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right]\bigg{|}\right]\Bigg{]}$
($\mu^{i}(s,a)\geq 0$) $\displaystyle\leq$
$\displaystyle\max_{s,a}\Bigg{[}(1-\mu^{i}(s,a))\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}$
$\displaystyle+\gamma\mu^{i}(s,a)\left[{\mathbb{E}}_{S^{\prime}\sim\mathcal{P}(\cdot|s,a)}\bigg{|}\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\bigg{|}\right]\Bigg{]}$
(Jensen’s inequality)

Next, we note that for any functions $f(\cdot)$ and $g(\cdot)$, we have

$\displaystyle\left|(\max_{x}f(x))-(\max_{x}g(x))\right|\leq\max_{x}\left|f(x)-g(x)\right|.$
(128)

The reason is as follows. We have
$\max_{x}f(x)=\max_{x}f(x)-g(x)+g(x)\leq(\max_{x}f(x)-g(x))+(\max_{x}g(x))$.
Hence,
$(\max_{x}f(x))-(\max_{x}g(x))\leq\max_{x}f(x)-g(x)\leq\max_{x}\left|f(x)-g(x)\right|$.
Now suppose $\max_{x}f(x)\geq\max_{x}g(x)$. Then we can apply absolute value
to the left hand side of the inequality, and we get the bound. By a similar
argument for the case $\max_{x}f(x)\leq\max_{x}g(x)$, we get the bound in
(128). Hence, we have
$\left|\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right|\leq\max_{a^{\prime}}|{\bm{\theta}}_{1}(S^{\prime},a^{\prime})-{\bm{\theta}}_{2}(S^{\prime},a^{\prime})|\leq\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\|_{\infty}$.
As a result, we have

$\displaystyle\|{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{1})-{\bar{\mathbf{G}}}^{i}({\bm{\theta}}_{2})\|_{c}\leq$
$\displaystyle\max_{s,a}\left[(1-\mu^{i}(s,a))\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}+\gamma\mu^{i}(s,a)\left[\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\|_{\infty}\right]\right]$
$\displaystyle=$
$\displaystyle\max_{s,a}\left(1-(1-\gamma)\mu^{i}(s,a)\right)\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}$
$\displaystyle\leq$
$\displaystyle\left(1-(1-\gamma)\mu^{i}_{\min}\right)\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}$
$\displaystyle\leq$
$\displaystyle\left(1-(1-\gamma)\mu_{\min}\right)\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}$
$\displaystyle=$
$\displaystyle\left(1-(1-\gamma)\mu_{\min}\right)\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{c}.$

∎

###### Proof of Lemma D.3.

First, for $A_{1}$, we have

$\displaystyle\|{\mathbf{G}^{i}}({\bm{\theta}}_{1},{\mathbf{y}})-{\mathbf{G}^{i}}({\bm{\theta}}_{2},{\mathbf{y}})\|_{c}$
$\displaystyle=$
$\displaystyle\max_{s,a}\bigg{|}{\bm{\theta}}_{1}(s,a)+\mathbbm{1}_{\\{S=s,A=a\\}}\times\big{[}\gamma\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}_{1}(S,A)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\big{]}$
$\displaystyle\quad\quad\quad-\left({\bm{\theta}}_{2}(s,a)+\mathbbm{1}_{\\{S=s,A=a\\}}\times\big{[}\gamma\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}_{2}(S,A)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\big{]}\right)\bigg{|}$
$\displaystyle\leq$
$\displaystyle\max_{s,a}\bigg{[}\left|(1-\mathbbm{1}_{\\{S=s,A=a\\}})({\bm{\theta}}_{1}(s,a)-{\bm{\theta}}_{2}(s,a))\right|$
$\displaystyle\quad\quad+\gamma\mathbbm{1}_{\\{S=s,A=a\\}}\left|\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right|\bigg{]}$
(triangle inequality) $\displaystyle\leq$
$\displaystyle\max_{s,a}\bigg{[}\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}+\gamma\mathbbm{1}_{\\{S=s,A=a\\}}\left|\max_{a^{\prime}}\left({\bm{\theta}}_{1}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}\left({\bm{\theta}}_{2}+Q^{*}(S^{\prime},a^{\prime})\right)\right|\bigg{]}$
(definition of $\|\cdot\|_{\infty}$) $\displaystyle\leq$
$\displaystyle\max_{s,a}\bigg{[}\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\right\|_{\infty}+\gamma\mathbbm{1}_{\\{S=s,A=a\\}}\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\|_{\infty}\bigg{]}$
(By (128)) $\displaystyle\leq$ $\displaystyle
2\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\|_{\infty}$ $\displaystyle=$
$\displaystyle 2\|{\bm{\theta}}_{1}-{\bm{\theta}}_{2}\|_{c}$

Second, for $A_{2}$, we have

$\displaystyle\|{\mathbf{G}^{i}}({\bm{\theta}},{\mathbf{y}})\|_{c}=$
$\displaystyle\max_{s,a}\left|{\bm{\theta}}(s,a)+\mathbbm{1}_{\\{S=s,A=a\\}}\times\big{[}\gamma\max_{a^{\prime}}\left({\bm{\theta}}+Q^{*}(S^{\prime},a^{\prime})\right)-{\bm{\theta}}(S,A)-\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\big{]}\right|$
$\displaystyle\leq$
$\displaystyle\max_{s,a}\left[\left|(1-\mathbbm{1}_{\\{S=s,A=a\\}}){\bm{\theta}}(a,s)\right|+\gamma\mathbbm{1}_{\\{S=s,A=a\\}}\left|\max_{a^{\prime}}\left({\bm{\theta}}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\right|\right]$
(triangle inequality) $\displaystyle\leq$
$\displaystyle\max_{s,a}\left[\left\|{\bm{\theta}}\right\|_{\infty}+\gamma\mathbbm{1}_{\\{S=s,A=a\\}}\left|\max_{a^{\prime}}\left({\bm{\theta}}+Q^{*}(S^{\prime},a^{\prime})\right)-\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})\right|\right]$
(definition of $\|\cdot\|_{\infty}$) $\displaystyle\leq$
$\displaystyle\max_{s,a}\left[\left\|{\bm{\theta}}\right\|_{\infty}+\gamma\max_{a^{\prime}}\left|{\bm{\theta}}(S^{\prime},a^{\prime})\right|\right]$
(By (128)) $\displaystyle\leq$ $\displaystyle
2\left\|{\bm{\theta}}\right\|_{\infty}$ $\displaystyle=$ $\displaystyle
2\left\|{\bm{\theta}}\right\|_{c}.$

Lastly, for $B$, we have

$\displaystyle\|b^{i}({\mathbf{y}}^{i})\|_{c}=$
$\displaystyle\max_{s,a}\left|\mathbbm{1}_{\\{S=s,A=a\\}}\left[\mathcal{R}(S,A)+\gamma\max_{a^{\prime}}Q^{*}(S^{\prime},a^{\prime})-Q^{*}(S,A)\right]\right|$
$\displaystyle\leq$
$\displaystyle\max_{s,a}\mathbbm{1}_{\\{S=s,A=a\\}}\left[\left|\mathcal{R}(S,A)\right|+\gamma\max_{a^{\prime}}\left|Q^{*}(S^{\prime},a^{\prime})\right|+\left|Q^{*}(S,A)\right|\right]$
(triangle inequality) $\displaystyle\leq$
$\displaystyle\max_{s,a}\left[1+\gamma\frac{1}{1-\gamma}+\frac{1}{1-\gamma}\right]$
$\displaystyle=$ $\displaystyle\frac{2}{1-\gamma}.$

∎

###### Proof of Lemma D.4.

The proof follows similar to Lemma C.4. ∎

###### Proof of Theorem 5.2.

By Proposition D.1 and Lemmas D.1, D.2, D.3, and D.4, it is clear that the
federated $Q$ Algorithm 3 satisfies all the Assumptions 6.1, 6.2, 6.3, and 6.4
of the FedSAM Algorithm 4. Furthermore, by the proof of Theorem B.1, we have
$w_{t}=(1-\frac{\alpha{\varphi}_{2}}{2})^{-t}$, and the constant $c$ in the
sampling distribution $q_{T}^{c}$ in Algorithm 1 is
$c=(1-\frac{\alpha{\varphi}_{2}}{2})^{-1}$. In equation (129) we evaluate the
exact value of $w_{t}$.

Furthermore, by choosing step size $\alpha$ small enough, we can satisfy the
requirements in (21), (23), (32), (35). By choosing $K$ large enough, we can
satisfy $K>\tau$, and by choosing $T$ large enough we can satisfy $T>K+\tau$.
Hence, the result of Theorem B.1 holds for this algorithm.

Next, we derive the constants involved in Theorem B.1 step by step. In this
analysis we only consider the terms involving $|\mathcal{S}|$,
$|\mathcal{A}|$, $\frac{1}{1-\gamma}$, $\mathfrak{I}_{\max}$, and
$\mu_{\min}$. Since $\|\cdot\|_{c}=\|\cdot\|_{\infty}$, we choose
$g(\cdot)=\frac{1}{2}\|\cdot\|_{p}^{2}$, i.e. the $p$-norm with
$p=2\log(|\mathcal{S}|)$. It is known that $g(\cdot)$ is $(p-1)$ smooth with
respect to $\|\cdot\|_{p}$ norm [4], and hence
$L=\Theta(\log(|\mathcal{S}|))$. Hence, we have
$l_{cs}=|\mathcal{S}|^{-1/p}=\frac{1}{\sqrt{e}}=\Theta(1)$ and $u_{cs}=1$.
Therefore, we have ${\varphi}_{1}=\frac{1+\psi
u_{cs}^{2}}{1+\psi\ell_{cs}^{2}}=\frac{1+\psi}{1+\frac{\psi}{\sqrt{e}}}\leq
1+\psi$. By choosing
$\psi=(\frac{1+\gamma_{c}}{2\gamma_{c}})^{2}-1=\frac{1+2\gamma_{c}-3\gamma_{c}^{2}}{4\gamma_{c}^{2}}\geq(1-\gamma_{c})=\mu_{\min}(1-\gamma)=\Omega(\mu_{\min}(1-\gamma))$,
which is $\psi=\mathcal{O}(1)$, we have
${\varphi}_{1}=\frac{1+\psi}{1+\frac{\psi}{\sqrt{e}}}=\sqrt{e}\frac{(\frac{1+\gamma_{c}}{2\gamma_{c}})^{2}}{\sqrt{e}+(\frac{1+\gamma_{c}}{2\gamma_{c}})^{2}-1}=\mathcal{O}(1)$,
and

$\displaystyle{\varphi}_{2}=$ $\displaystyle
1-\gamma_{c}\sqrt{\frac{1+\psi}{1+\frac{\psi}{\sqrt{e}}}}=1-\gamma_{c}\sqrt{\frac{(\frac{1+\gamma_{c}}{2\gamma_{c}})^{2}}{1+\frac{(\frac{1+\gamma_{c}}{2\gamma_{c}})^{2}-1}{\sqrt{e}}}}=1-\frac{0.5(1+\gamma_{c})e^{1/4}}{\sqrt{\sqrt{e}-1+\left(\frac{1+\gamma_{c}}{2\gamma_{c}}\right)^{2}}}$
$\displaystyle=$ $\displaystyle
1-\frac{0.5e^{1/4}(2-\mu_{\min}(1-\gamma))}{\sqrt{\sqrt{e}-1+\left(\frac{2-\mu_{\min}(1-\gamma)}{2-2\mu_{\min}(1-\gamma)}\right)^{2}}}>1-\gamma_{c}\sqrt{1+\psi}=1-\gamma_{c}\frac{1+\gamma_{c}}{2\gamma_{c}}=\frac{1-\gamma_{c}}{2}=0.5\mu_{\min}(1-\gamma)$
$\displaystyle=$ $\displaystyle\Omega(\mu_{\min}(1-\gamma))$

${\varphi}_{3}=\frac{L(1+\psi
u_{cs}^{2})}{\psi\ell_{cs}^{2}}=\mathcal{O}\left(\frac{\log(|\mathcal{S}|)(1+\psi)}{\psi}\right)\leq\mathcal{O}\left(\frac{\log(|\mathcal{S}|)}{1-\gamma_{c}}\right)=\mathcal{O}\left(\frac{\log(|\mathcal{S}|)}{\mu_{\min}(1-\gamma)}\right).$

Using ${\varphi}_{2}$, we have

$\displaystyle
w_{t}=\left(1-\frac{\alpha{\varphi}_{2}}{2}\right)^{-t}=\left(1-\alpha/2+\frac{0.25\alpha
e^{1/4}(2-\mu_{\min}(1-\gamma))}{\sqrt{\sqrt{e}-1+\left(\frac{2-\mu_{\min}(1-\gamma)}{2-2\mu_{\min}(1-\gamma)}\right)^{2}}}\right)^{-t}.$
(129)

Further, we have

$l_{cm}=(1+\psi l_{cs}^{2})^{1/2}=\Theta(1)$ $u_{cm}=(1+\psi
u_{cs}^{2})^{1/2}=\Theta(1)$

Since TV-divergence is upper bounded with $1$, we have
$\bar{m}=\mathcal{O}(1)$. By Lemma D.3, we have

$A_{1}=A_{2}=2=\mathcal{O}(1)$

and $A_{1}=A_{2}=\Omega(1)$,

$B=\frac{2}{1-\gamma}=\mathcal{O}\left(\frac{1}{1-\gamma}\right).$

Hence $m_{1}=2A_{2}\bar{m}=\mathcal{O}(1)$. Also, we have $m_{2}=0$.

We choose the $D$-norm in Lemma B.9 as the $2$-norm $\|\cdot\|_{2}$. Hence, by
primary norm equivalence, we have $l_{cD}=\frac{1}{\sqrt{|\mathcal{S}|}}$, and
$u_{cD}=1$, and hence $\frac{u_{cD}}{l_{cD}}=\sqrt{|\mathcal{S}|}$. The rest
of the proof is similar to the proof of Theorem 5.1 where
$\mathfrak{I}_{\max}$ is substituted with $1$. The sample complexity can also
be derived using Corollary B.1.1. ∎