# Efficient Gating of Magnons by Proximity Superconductors

Tao Yu<EMAIL_ADDRESS>School of Physics, Huazhong University of Science
and Technology, Wuhan 430074, China  Gerrit E. W. Bauer
<EMAIL_ADDRESS>WPI-AIMR and Institute for Materials Research and
CSRN, Tohoku University, Sendai 980-8577, Japan  Zernike Institute for
Advanced Materials, University of Groningen, 9747 AG Groningen, Netherlands
Kavli Institute for Theoretical Sciences, University of the Chinese Academy of
Sciences, Beijing 100190, China

###### Abstract

Electrostatic gating confines and controls the transport of electrons in
integrated circuits. Magnons, the quanta of spin waves of the magnetic order,
are promising alternative information carriers, but difficult to gate. Here we
report that superconducting strips on top of thin magnetic films can totally
reflect magnons by its diamagnetic response to the magnon stray fields. The
induced large frequency shifts unidirectionally block the magnons propagating
normal to the magnetization. Two superconducting gates parallel to the
magnetization create a magnonic cavity. The option to gate coherent magnons
adds functionalities to magnonic devices, such as reprogrammable logical
devices and increased couplings to other degrees of freedom.

Introduction.—Electrostatic gating provides a tunable potential to control the
transport channels of electrons in field effect transistors [1]. Potential
barriers size-quantize the electronic wave function, leading to quantum point-
contacts, wires, and dots with application in quantum technology [2, 3, 4].
“Magnonics” employs the magnon quasiparticles, i.e., the bosonic quanta of the
spin wave excitations of the magnetic order, as information carriers in a low-
power alternative to conventional electronics [5, 6, 7, 8, 9, 10, 12, 13, 14,
11]. However, magnons cannot be gated, blocked, or trapped as easily as
electrons, so many mature concepts of electronics cannot directly be applied
in magnonics. Chumak et al. [15] propose an alternatives to gating in a magnon
transistor by inserting a magnonic crystal into a film of yttrium iron garnet
(YIG) that controls the transmission of coherent magnons. Current-biased heavy
metal contacts can modulate the incoherent magnon currents in a YIG channel by
spin injection or heating [16, 17, 18]. However, both mechanisms do not create
potential barriers, the modulation efficiency is low, and power demands are
high. Electrostatic gates on magnetic semiconductors can locally suppress
magnetism [19, 20], but at present the magnetic quality of available materials
remains wanting.

The dispersion of long-wavelength magnetostatic spin waves in thick films or
magnonic crystals can be modulated in a non-reciprocal fashion by the
electromagnetic interaction with metallic gates. The latter can be modelled as
thick perfect [21] or Ohmic conductors [22, 23, 24, 25, 26]. Superconducting
gate can have profound effects as well [27, 28, 29, 30]. Recent research on
ultrathin films accumulates evidence that the physics changes drastically when
thicknesses are reduced down to nanometer scale. In this limit surface and
volume modes merge into perpendicular standing spin waves (PSSW) with spectra
dominated by the exchange interaction. The GHz dynamics is confined to the
lowest PSSW with nearly uniform amplitude normal to the film [31, 32, 33, 34].
In contrast to conventional wisdom for thick films, a metallic cap atop
ultrathin films of YIG only enhances the damping of spin waves [35, 36] with
minor effects on their dispersion. The magnon conductivity in high-quality
nanometer films [37] reaches record values because of the onset of two-
dimensional diffuse magnon transport. Moreover, stray fields emitted by spin
waves in ultrathin films have a relatively short wave length and thereby
cannot penetrate deeply into metallic gates, a limit not accounted for by
previous theories.

In this Letter, we report that floating superconducting films on top of
ultrathin films of a magnetic insulator such as YIG, as illustrated in Fig. 1,
induce chiral frequency shifts of tens of GHz that correspond to magnetic
fields of $O(0.1\,\mathrm{T})$. The chirality is caused by constructive
interference of the Oersted fields of the spin-wave dipolar and induced eddy
currents in the superconductors. The gate generates an effective barrier that
is dynamic, i.e., it depends on the magnon frequency and propagation
direction. A wide superconducting gate totally reflects coherent magnons in
the microwave band propagating normal to the “up” magnetization, but transmits
them when magnetization is flipped. We can thereby control the magnon current
by either modulating the superconductivity in the gates or rotating the
equilibrium magnetization. The on-off ratio is nearly unity over tens of GHz,
in contrast to narrow-band resonant coupling effects [38, 39]. Our set-up is
exceedingly simple and does not require spin-orbit interactions, in contrast
to metal-based designs [40, 41]. Moreover, we predict that two superconducting
strips can nearly perfectly trap spin waves in a sub-micrometer region forming
a magnon waveguide without having to etch the magnetic film. On-chip
implementation of these devices allow to implement magnonic functionalities
such as non-volatility and chirality into quantum technologies.

Figure 1: Unidirectional blocking of magnons by a superconducting metal gate
on a thin magnetic insulator that provides an effective “barrier”. An inserted
thin insulating layer between the conductors and magnetic films (red) can
strongly suppress the interfacial exchange interaction. The stripline on the
left acts as injector and detector of coherent magnons. We address reflection
as a function of angle of an in-plane magnetic field that rotates the
magnetization $\mathbf{M}_{s}$.

Inductive interaction between magnons and superconductors.—We consider a
floating superconducting (SC) or normal metal (NM) gate with thickness $d$ on
a thin YIG film of thickness $s$ as illustrated in Fig. 1. A thin insulating
spacer between them suppresses the proximity effect by an interfacial exchange
interaction. We formulate the inductive coupling of propagating magnons under
the gate that gives rise to “eddy current” [42, 36, 35]. The magnetization is
parallel to and rotates with a weak in-plane applied magnetic field
$\mathbf{H}_{\mathrm{app}}$, which does not affect the superconductivity. We
use a coordinate system in which
$\mathbf{H}_{\mathrm{app}}=H_{\mathrm{app}}\hat{\mathbf{z}}$ and an angle
$\varphi$ relative to the contacts, while the film normal is along
$\hat{\mathbf{x}}$. We assume that the magnetization dynamics is dominated by
isotropic exchange magnons in the thin films and disregard the dipolar
interaction on the dispersion and small mode ellipticity as justified in the
Supplemental Material [43]. With free boundary conditions [31, 32], the
exchange spin waves with amplitudes $m_{y}^{(l)}(\mathbf{k})\approx
im_{x}^{(l)}(\mathbf{k})$ are circularly polarized [43], where $l$ is the band
indices [44, 37].

The dynamic magnetic stray fields emitted by the spin waves induce eddy
currents in conductors that in turn generate Oersted magnetic fields
$\mathbf{H}$ that affect the spin-wave dynamics. We focus on metallic gates
with $d\sim O(10~{}\mathrm{nm})$ because currents are then constant over the
film thickness: London’s penetration depth of SCs
$\lambda_{L}=\sqrt{m_{e}/(\mu_{0}n_{s}e^{2})}\sim O(100~{}\mathrm{nm})$ [46,
45], where $\mu_{0}$ is the vacuum permeability and $m_{e}$ is the effective
mass of electrons with density $n_{s}$. The “skin depth”
$\delta=\sqrt{2/(\mu_{0}\omega\sigma_{c})}$ of NMs with conductivity
$\sigma_{c}$ for ac magnetic fields of frequency $\omega$ [47] is also much
larger than $d$ for good metals at frequencies up to several terahertz (THz).
A pioneering study [21] considers “perfectly conducting” gates with
$\sigma_{c}\rightarrow\infty$ corresponding to $\delta\rightarrow 0$ and
interface boundary condition $\mathbf{B}_{\perp}={0}$, which is very different
from the superconductors discussed below.

The dipolar stray fields $\mathbf{H}^{(l)}_{\mathbf{k}}(\mathbf{r},t)$ of
spin-wave eigenmodes with momentum $\mathbf{k}$ and frequency
$\omega_{\mathbf{k}}$ above the film [43], i.e.,
$\mathbf{r}=(x>0,\boldsymbol{\rho}=y\hat{\mathbf{y}}+z\hat{\mathbf{z}}),$ are
evanescent as $\sim e^{-\left|\mathbf{k}\right|x}$ and nearly uniform across a
gate for wave numbers $\left|\mathbf{k}\right|\ll 1/d$ [32]. When
$\left|\mathbf{k}\right|\ll 1/s$ only the lowest subband contributes. The
components of the magnetic field $\mathbf{H}_{\mathbf{k}}^{(l=0)}$ are locked
with momentum, obeying $k_{z}{H}_{y}^{(0)}=k_{y}{H}_{z}^{(0)}$ and
$\left|\mathbf{k}\right|{H}_{x}^{(0)}=ik_{y}{H}_{y}^{(0)}+ik_{z}{H}_{z}^{(0)}$.
The stray field vanishes when $\mathbf{k}=-|k_{y}|\hat{\mathbf{y}},$ so spin
waves that propagate in the negative $\hat{\mathbf{y}}$-direction cannot
interact with a top gate.

According to Maxwell’s equations $\nabla\cdot\mathbf{E}=0$ and
$\nabla\times\mathbf{E}=i\omega\mu_{0}\mathbf{H}$, the magnetic stray field of
a spin wave with wavevector $\mathbf{k}$ and frequency $\omega$ generates an
electromotive force (emf) that has the form $\mathbf{E}\propto
e^{-\left|\mathbf{k}\right|x+i\mathbf{k}\cdot\boldsymbol{\rho}}$ up to high
frequencies $\sigma_{c}/\varepsilon_{0}\sim 10^{6}$ THz for typical metallic
conductivities $\sigma_{c}\sim 10^{7}~{}\mathrm{\Omega}^{-1}\mathrm{m}^{-1}$.
The normal component is immediately screened, so $E_{x}=0$,
$E_{y}=-\left({i\omega\mu_{0}}/\left|\mathbf{k}\right|\right){H}_{z}^{(0)}$,
and
$E_{z}=\left({i\omega\mu_{0}}/\left|\mathbf{k}\right|\right){H}_{y}^{(0)}$. By
Ohm’s law, the induced electric eddy current in a normal conductor
$J_{x}(x,\boldsymbol{\rho})=0$,
$J_{y}(x,\boldsymbol{\rho})=-\left({i\sigma_{c}\omega\mu_{0}}/\left|\mathbf{k}\right|\right){H}_{z}^{(0)}(x,\boldsymbol{\rho})$,
and
$J_{z}(x,\boldsymbol{\rho})=\left({i\sigma_{c}\omega\mu_{0}}/\left|\mathbf{k}\right|\right){H}_{y}^{(0)}(x,\boldsymbol{\rho})$
is perpendicular to the spin-wave propagation direction since
$\mathbf{k}\cdot\mathbf{J}=0$. Abrikosov vortice lattices of type-II
superconductors are sources of periodic magnetic fields that generate tunable
bandgaps in the magnon spectrum [48]. We will address such effects and that of
different pairing symmetries in the future.

The eddy currents, in turn, generate Oersted magnetic fields that oppose the
original ones (Lenz effect). The effect on the gate itself may be disregarded
since we consider only films much thinner than London’s penetration (SC) and
skin (NM) depths. However, they affect the spin wave dynamics by a field-like
(for SC) or a damping-like (for NM) torque that causes a frequency shift (for
SC) and an additional contribution to the Gilbert damping constant (for NM),
respectively. Here we address the full dynamic response by self-consistently
solving the coupled Maxwell and Landau-Lifshitz-Gilbert (LLG) equations. In
the limit that $\left|\mathbf{k}\right|\ll 1/d,$ the currents are uniform
across the film and we may set $x\rightarrow d/2$. The vector potential
generated by the eddy currents then reads in frequency space [47]
$A_{\xi}(\mathbf{r},\omega)=({\mu_{0}}/{4\pi})\int
d\mathbf{r}^{\prime}{J}_{\xi}\left(x^{\prime}={d}/{2},\boldsymbol{\rho}^{\prime},\omega\right){e^{i\omega|\mathbf{r}-\mathbf{r}^{\prime}|/c}}/{|\mathbf{r}-\mathbf{r}^{\prime}|}$.
Using the Weyl identity [49], we obtain for $x<0$

$A_{\xi}(\mathbf{r},\omega)=({\mu_{0}d}/{2}){J}_{\xi}\left(x^{\prime}={d}/{2},\boldsymbol{\rho},\omega\right){e^{|\mathbf{k}|(x-d/2)}}/{|\mathbf{k}|}.$
(1)

For the LLG equation we require only the transverse ($x$\- and $y$-)
components of the Oersted magnetic field
$\tilde{\mathbf{H}}(\mathbf{r})=\nabla\times\mathbf{A}(\mathbf{r})/\mu_{0}$:
$\tilde{H}_{y}(\mathbf{r},\omega)=(ik_{y}/|\mathbf{k}|)\tilde{H}_{x}(\mathbf{r},\omega)$
with

$\displaystyle\tilde{H}_{x}(\mathbf{r},\omega)$
$\displaystyle=ie^{|\mathbf{k}|(x-d)}{d}\sigma_{c}\omega\mu_{0}({1-e^{-|\mathbf{k}|s}})/(4{|\mathbf{k}|})$
$\displaystyle\times\left(m_{x}^{\mathbf{k}}(\boldsymbol{\rho},\omega)-im_{y}^{\mathbf{k}}(\boldsymbol{\rho},\omega){k_{y}}/{|\mathbf{k}|}\right).$
(2)

The linearized LLG equation in frequency space poses a self-consistency
problem

$\displaystyle-i\omega m_{x}(\mathbf{k})$
$\displaystyle=-i\omega_{\mathbf{k}}(1-i\alpha_{G})m_{x}(\mathbf{k})+\mu_{0}\gamma
M_{s}\tilde{H}_{y}(\mathbf{k},\omega),$ $\displaystyle-i\omega
m_{y}(\mathbf{k})$
$\displaystyle=-i\omega_{\mathbf{k}}(1-i\alpha_{G})m_{y}(\mathbf{k})-\mu_{0}\gamma
M_{s}\tilde{H}_{x}(\mathbf{k},\omega),$

that leads to the modified dispersion relation

$\tilde{\omega}_{\mathbf{k}}=\frac{\omega_{\mathbf{k}}(1-i\alpha_{G})}{1+i\alpha_{m}(|\mathbf{k}|)\left({k_{y}}/{|\mathbf{k}|}+{|k_{y}|}/{|\mathbf{k}|}\right)},$
(3)

where $\alpha_{G}$ is the intrinsic Gilbert damping coefficient and

$\alpha_{m}(|\mathbf{k}|)=({d}/{4})e^{|\mathbf{k}|\left(-\frac{s}{2}-d\right)}(1-e^{-|\mathbf{k}|s})\sigma_{c}\mu_{0}^{2}\gamma
M_{s}/|\mathbf{k}|$ (4)

is dimensionless. In the limit that $\left|\mathbf{k}\right|\ll\\{1/d,1/s\\}$,
$\alpha_{m}(\left|\mathbf{k}\right|)\rightarrow ds\sigma_{c}\mu_{0}^{2}\gamma
M_{s}/4$. In a normal metal, $\sigma_{c}$ is real and $\alpha_{m}>0,$ so only
magnons with positive $k_{y}$ suffer from an additional damping
$\tilde{\alpha}(\mathbf{k})=\alpha_{m}\cos\theta_{\mathbf{k}}\left(1+\mathrm{sgn}(k_{y})\right)$,
where $\cos\theta_{\mathbf{k}}=k_{y}/\left|\mathbf{k}\right|$. For a copper
conductor gate with thickness $d=40$ nm and conductivity $\sigma_{c}\approx
6\times 10^{7}$ $\Omega^{-1}$m-1 on top of a $s=20$ nm thin YIG film with
$\mu_{0}M_{s}=0.18$ T and $\gamma=1.82\times 10^{11}$ s-1T-1, we find
$\alpha_{m}=4.8\times 10^{-4},$ which is of the same order as the intrinsic
damping. Bertelli et al. [35] observed a larger $\alpha_{m}\sim 10^{-2}$ by
nitrogen-vacancy center magnetometry, but for thicker films and only for
positive $\mathbf{k}=|k_{y}|\hat{\mathbf{y}},$ therefore did not yet resolve
the damping chirality. In contrast to thick films [24], the gate hardly
affects the spin wave dispersion Eq. (3).

The chiral damping theory above holds for coherent spin waves with well
defined momentum as excited by narrow striplines, and we may expect similar
effects in the diffuse regime of magnon transport. An asymmetry in the
propagation of carriers into opposite directions has been reported in the
transport of incoherent magnons in YIG films under microwave [50] or spin Hall
effect [51, 52] excitation. References [51, 52] appear to support our results
without having to resort to spin-orbit interactions.

For a superconductor $\sigma_{c}(\omega)$ is complex and can be treated by a
two-fluid model [53, 54, 55, 56]. For simplicity we assume sufficiently low
temperatures to freeze out the quasi-particle excitations, but the effect
sustains when $T\rightarrow 0.85T_{c}$ [43]. The conductivity
$\sigma_{c}(\tilde{\omega}_{\mathbf{k}})=i{n_{s}e^{2}}/(m_{e}\tilde{\omega}_{\mathbf{k}})$
is then purely imaginary, leading to a frequency shift

$\delta\omega_{\mathbf{k}}=\frac{d}{4}e^{-|\mathbf{k}|\left(\frac{s}{2}+d\right)}\frac{1-e^{-\left|\mathbf{k}\right|s}}{\left|\mathbf{k}\right|}\mu_{0}^{2}\gamma
M_{s}\frac{n_{s}e^{2}}{m_{e}}\left(\frac{k_{y}}{\left|\mathbf{k}\right|}+\frac{|k_{y}|}{\left|\mathbf{k}\right|}\right),$
(5)

which is real and positive definite. Its chirality is complete since
$\delta\omega_{\mathbf{k}}$ vanishes for negative $k_{y}$ and arbitrary
$k_{z}$. It is typically quite large: For a $d=40$ nm NbN superconducting film
with electron density $n_{s}=10^{29}/\mathrm{m}^{3}$ [57] on top of a
$s=20~{}\mathrm{nm}$ YIG film the shift amounts to $\delta\omega=45$ GHz when
$k_{y}=\left|\mathbf{k}\right|\ll\\{1/d,1/s\\}.$ This value corresponds to the
Zeeman energy of an applied magnetic field of $255$ mT. Figure 2 summarizes
these features for the chirality of frequency shift [(a)] and spin wave
dispersion with momentum $k_{y}\hat{\mathbf{y}}$ [(b)]. The singularity in
Fig. 2(b) around $k_{y}=0$ has no physical consequences, since the group
velocities are positive (negative) for $k_{y}>0$ ($k_{y}<0$), but vanishes for
Kittel mode at $k_{y}=0$.

Figure 2: Chirality of the frequency shift [(a)] and spin wave dispersion with
momentum $k_{y}\hat{\mathbf{y}}$ [(b)] induced by a superconducting gate. The
frequency shift for positive $k_{y}$ bars spin wave propagation in a large
frequency window of $\sim 20~{}\mathrm{GHz}$ as indicated by the region
between the dashed lines in (b). A field $\mu_{0}H_{\mathrm{app}}=300$ mT is
applied in the $\mathbf{\hat{z}}$-direction.

Unidirectional blocking of magnons.—The above results imply attractive
functionalities created by superconductivity in wave magnonics. A
superconducting top gate on a magnetic film forms a nearly perfect switch. It
is opaque for ballistic spin waves launched, but becomes fully transmitting
simply by heating it above the critical superconducting temperature.

The superconducting gate couples most efficiently to spin waves that propagate
normal to the magnetization (Damon-Eshbach configuration) as in Fig. 2(b). In
order to assess the effect we consider a wide superconducting gate at
$w_{1}\leq y\leq w_{2}$ with width $w_{2}-w_{1}\gg\lambda\gg 1/k_{y},$ where
$\lambda$ is a magnon relaxation length. It acts as a potential barrier that
reflects magnons below an energy threshold just as electric gates in field
effect transistors, but now with a $\mathbf{k}$-dependent barrier height.

We place a stripline with thickness $d_{s}=80$ nm and width $w_{s}=150$ nm
centered at the origin biased by a uniform ac current of frequency
$\nu_{s}=2\pi\times\omega_{s}=55$ GHz and current density
$I_{s}=10^{6}~{}\mathrm{A/cm^{2}}$. When parallel to the external dc field and
magnetization in the underlying YIG, it launches spin waves with positive
$k_{y}^{\ast}=2\pi/0.45$ $\mathrm{\mu m}^{-1}$ into half space as shown in
Fig. 3(a) in the form of a snapshot of the excited magnetization
$\left|\mathbf{m}\right|$ [33, 58, 59, 60, 61]. The Gilbert damping
$\alpha_{G}=10^{-4}$ governs the decay length
$\lambda\sim\sqrt{(\alpha_{\mathrm{ex}}\mu_{0}\gamma
M_{s})(\nu_{s}-\mu_{0}\gamma
H_{\mathrm{app}})}/(\alpha_{G}\nu_{s})=24.2~{}\mathrm{\mu m}$, but it is not a
simple exponential since the stripline of finite width also emits waves around
$k_{y}^{\ast}$ that cause the observed interference pattern. A superconducting
gate made from a NbN film with thickness $d=40$ nm covers the YIG film from
$y=w_{1}=3~{}\mathrm{\mu}$m to $w_{2}=100~{}\mathrm{\mu}$m. Below the gate
$w_{1}\leq y\leq w_{2}$, the diamagnetic field ($k_{y}^{\ast}d\ll
1,k_{y}^{\ast}s\ll 1$)

$\displaystyle\tilde{H}_{x}(y,t)$
$\displaystyle=-\frac{n_{s}e^{2}}{m_{e}}\mu_{0}\frac{ds}{4}\left(m_{x}(y,t)+\frac{1}{\omega_{s}}\frac{dm_{y}(y,t)}{dt}\right),$
$\displaystyle\tilde{H}_{y}(y,t)$
$\displaystyle=-\frac{n_{s}e^{2}}{m_{e}}\mu_{0}\frac{ds}{4}\left(-\frac{1}{\omega_{s}}\frac{dm_{x}(y,t)}{dt}+m_{y}(y,t)\right),$

enters the LLG equation. We solve the time dependent problem in the steady
state and show representative
$\left|\mathbf{m}\right|\equiv\sqrt{m_{x}^{2}+m_{y}^{2}}$ in Fig. 3(b).
Technique details are referred to the Supplemental Material [43]. We clearly
observe the total reflection of spin waves at the gate edge due to the excited
Oersted field from the superconductor that only exists at its edge [43]. The
reflected spin waves penetrate the left half-space in Fig. 3(b), which remains
silent in the absence of the gate [Fig. 3(a)] because the pumping is chiral.
The amplitude of the magnetization between source and gate $0<y<w_{1}$ is
enhanced by a factor 2, i.e., incoming and reflected spin waves interfere
constructively. The reflection at the stripline is very weak and cannot
generate standing waves in the region $0<y<w_{1}$. Nevertheless, a large
number of coherent emitted and reflected waves with different wave lengths
coexist and cause complex interference fringes (refer to Supplemental Material
[43] with other parameters). On the other hand, the transmission of spin waves
that impinge from the right of the superconducting strip are not affected at
all, i.e., the device acts as a spin-wave isolator. Replacing the
superconductor by $d=40$ nm copper strip, we only enhance the damping of the
spin waves without causing reflection [43].

Figure 3: Excited magnetization $|\mathbf{m}|$ by a narrow stripline at $y=0$
without (a) and with (b) a superconducting gate that covers the region from
$3$ to $100~{}\mathrm{\mu m}$. The spin waves are totally reflected at the
gate edge and leak into the left half space.

Spin-wave confinement.—When we rotate the magnetization by 90∘ to become
normal to the gate and stripline, the emitted spin waves
$\mathbf{k}=\left(0,0,k_{z}\right)$ propagate parallel to the magnetization.
The frequency shift of spin waves with positive and negative $k_{z}$ under the
gates on both sides of the stripeline as sketched in Fig. 2(a) are smaller
than Damon-Eshbach configuration, but still very substantial. The device
therefore forms a cavity, trapping the spin waves analogous to, but without
having to nanofabricate, a magnetic film. The standing waves modulate the
density of states that are strongly enhanced at the subband edges, which may
lead to strong coupling to the stripline photons and other degrees of freedom.

We substantiate these expectations by numerical modelling for a YIG film with
two superconducting gates located at $w_{1}\leq\left|z\right|\leq w_{2}$ on
both sides of a stripline at the origin as illustrated in Fig. 4(a). The spin
waves now feel the back-action magnetic field
$\tilde{H}_{x}(z,t)=-({n_{s}e^{2}}/{m_{e}})\mu_{0}{ds}m_{x}(z,t)/4$ and
$\tilde{H}_{y}(z,t)=0$. We self-consistently solve the LLG equation [43] for
cavity widths $\delta w=2w_{1}=1$ and $0.5\,\mathrm{\mu m}$ and a stripline
frequency interval $\nu_{s}\in[53.25,56]$ GHz indicated in Fig. 4(b), while
the other parameters are the same as in Fig. 3. We choose frequencies typical
in propagating magnon spectroscopy and stripline widths that can be deposited
by state-of-the-art fabrication techniques. The stripline now excites spin
waves with equal amplitude to both sides that are reflected by the
superconducting gates and interfere. The steady states in Fig. 4(c) are nearly
perfectly trapped standing spin waves with odd symmetry. The standing wave
resonance for a hard-wall potential are $\nu_{n}=\mu_{0}\gamma
H_{\mathrm{app}}+\alpha_{\mathrm{ex}}\mu_{0}\gamma M_{s}(n\pi/\delta w)^{2}$,
viz. $\nu_{1}\left(\delta w=0.5~{}\mathrm{\mu m}\right)=53.6$ GHz and
$\nu_{3}\left(\delta w=1~{}\mathrm{\mu m}\right)=54$ GHz. The boundary pinning
strongly suppresses the Kittel mode at $\nu_{0}=53.2$ GHz. The excitation
frequency $\nu_{s}=54.5$ GHz is close to the resonances of both cavities.

Figure 4: Spin waves trapped by superconducting gates on both sides of a
microwave stripline atop a magnetic thin film magnetized along
$\hat{\mathbf{z}},$ i.e., normal to the gate. (a) is a top view of the device.
(b) shows the frequency shift of magnons under a uniform gate with wave number
$k_{z}$. The microwave frequencies in (c) and (d) lie in the window indicated
by the vertical dashed lines. In (c), we plot snapshots of the maximum
amplitudes $m_{x}$ for two gate distances $\delta w=\\{1,0.5\\}\,\mathrm{\mu
m}$ at a frequency close to the standing wave frequencies with index
$n=\\{3,1\\}$. In (d) we plot the excitation efficiencies with and without
gating for the $n=1$ resonance of the $0.5\,\mathrm{\mu m}$ cavity.

Figure 4(d) illustrates the enhanced excitation efficiency at a standing wave
resonance. The maximum at $\nu_{s}\sim 54.25$ GHz is not much above the hard-
wall estimate of $53.6$ GHz, indicating efficient confinement. However, the
resonance widths of $\sim 1$ GHz corresponds to a Gilbert damping of
$\alpha_{G}=0.02$ that is much larger than the intrinsic one, indicating
substantial gate leakage that can be suppressed by using a thicker
superconducting film. Nevertheless, comparison with the ungated result in Fig.
4(d) already shows a cavity enhancement of the magnetization dynamics by an
order of magnitude at the resonance.

Microstructuring of YIG films into stripes is usually accompanied by
substantial deterioration of the magnetic quality [62]. The trapping of spin
waves by superconducting gates only requires deposition of two metal films
with a finite gap without introducing additional roughness. Moreover, the
option to modulate the trapping offers an easy reprogramming of magnonic
logical circuits.

Discussion and conclusion.—The predicted total unidirectional reflection is
surprising and seems to violate thermodynamic principles. However, we address
here a coherent scattering process in a finite system and not a diode-like
transport between reservoirs. Furthermore, the physics of chiral reflection of
spin waves under a superconducting gate is very different from conventional
potential scattering. The reflection of spin waves is a complicated process,
in which an approaching wave packet adiabatically generates the diamagnetic
current in the superconductor and the associated magnetic fields that push up
the magnon gap. The non-equilibrium scattering problem of coherent spin waves
injected by an external source that interact with a superconductor gate is not
subject to the constraints from linear response.

In conclusion, we predict a non-reciprocal frequency shift of spin waves by
the diamagnetism of nearby superconductors, which leads to functionalities
such as unidirectional total reflection and confinement of spin waves in
magnetic films by top gates. When the gates turn into normal conductors the
chiral frequency shift turns into a chiral damping. Both effects enrich the
tool box of information communication and processing technology in photonics
[49, 63], plasmonics [64, 65, 66], acoustics [67, 68], electronics [69, 70],
superconductivity [71, 72], and spintronics [73, 74].

###### Acknowledgements.

This work is financially supported by the startup grant of Huazhong University
of Science and Technology (Grants No. 3004012185 and No. 3004012198) as well
as JSPS KAKENHI Grant No. 19H00645.

## References

  * [1] Y. Imry and R. Landauer, Rev. Mod. Phys. 71, S306 (1999).
  * [2] D. Loss and D. P. DiVincenzo, Phys. Rev. A 57, 120 (1998).
  * [3] J. Yoneda, K. Takeda, T. Otsuka, T. Nakajima, M. R. Delbecq, G. Allison, T. Honda, T. Kodera, S. Oda, Y. Hoshi, et al., Nat. Nanotechnol. 13, 102 (2018).
  * [4] W. Huang, C. H. Yang, K. W. Chan, T. Tanttu, B. Hensen, R. C. C. Leon, M. A. Fogarty, J. C. C. Hwang, F. E. Hudson, K. M. Itoh,et al., Nature 569, 532 (2019).
  * [5] B. Lenk, H. Ulrichs, F. Garbs, and M. Muenzenberg, Phys. Rep. 507, 107 (2011).
  * [6] A. V. Chumak, V. I. Vasyuchka, A. A. Serga, and B. Hillebrands, Nat. Phys. 11, 453 (2015).
  * [7] D. Grundler, Nat. Nanotechnol. 11, 407 (2016).
  * [8] V. E. Demidov, S. Urazhdin, G. de Loubens, O. Klein, V. Cros, A. Anane, and S. O. Demokritov, Phys. Rep. 673, 1 (2017).
  * [9] A. Brataas, B. van Wees, O. Klein, G. de Loubens, and M. Viret, Phys. Rep. 885, 1 (2020).
  * [10] Barman et al., J. Phys.: Condens. Matter 33, 413001 (2021).
  * [11] B. Z. Rameshti, S. V. Kusminskiy, J. A. Haigh, K. Usami, D. Lachance-Quirion, Y. Nakamura, C.-M. Hu, H. X. Tang, G. E. W. Bauer, and Y. M. Blanter, Phys. Rep. 979, 1 (2022).
  * [12] L. R. Walker, Phys. Rev. 105, 390 (1957).
  * [13] R. W. Damon and J. R. Eshbach, J. Phys. Chem. Solids 19, 308 (1961).
  * [14] D. D. Stancil and A. Prabhakar, Spin Waves–Theory and Applications (Springer, New York, 2009).
  * [15] A. V. Chumak, A. A. Serga, and B. Hillebrands, Nat. Commun. 5, 4700 (2014).
  * [16] L. J. Cornelissen, J. Liu, B. J. van Wees, and R. A. Duine, Phys. Rev. Lett. 120, 097702 (2018).
  * [17] T. Wimmer, M. Althammer, L. Liensberger, N. Vlietstra, S. Geprägs, M. Weiler, R. Gross, and H. Huebl, Phys. Rev. Lett. 123, 257201 (2019).
  * [18] J. Liu, X-Y. Wei, G. E. W. Bauer, J. Ben Youssef, and B. J. van Wees, Phys. Rev. B 103, 214425 (2021).
  * [19] F. Matsukura, Y. Tokura, and H. Ohno, Nat. Nano. 10, 209 (2015).
  * [20] B. Huang, G. Clark, D. R. Klein, D. MacNeill, E. N.-Moratalla, K. L. Seyler, N. Wilson, M. A. McGuire, D. H. Cobden, D. Xiao, W. Yao, P. J.-Herrero, and X. D. Xu, Nat. Nano. 13, 544 (2018).
  * [21] Seshadri, Proc. IEEE 58, 506 (1970).
  * [22] M. L. Sokolovskyy, J. W. Klos, S. Mamica, and M. Krawczyk, J. Appl. Phys. 111, 07C515 (2012).
  * [23] M. Mruczkiewicz, M. Krawczyk, G. Gubbiotti, S. Tacchi, Y. A. Filimonov, D. V. Kalyabin, I. V. Lisenkov, and S. A. Nikitov, New J. Phys. 15 113023 (2013).
  * [24] M. Mruczkiewicz and M. Krawczyk, J. Appl. Phys. 115, 113909 (2014).
  * [25] M. Mruczkiewicz, E. S. Pavlov, S. L. Vysotsky, M. Krawczyk, Y. A. Filimonov, and S. A. Nikitov, Phys. Rev. B 90, 174416 (2014).
  * [26] M. Mruczkiewicz, P. Graczyk, P. Lupo, A. Adeyeye, G. Gubbiotti, and M. Krawczyk, Phys. Rev. B 96, 104411 (2017).
  * [27] I. A. Golovchanskiy, N. N. Abramov, V. S. Stolyarov, V. V. Bolginov, V. V. Ryazanov, A. A. Golubov, and A. V. Ustinov, Adv. Funct. Mater. 28, 1802375 (2018).
  * [28] I. A. Golovchanskiy, N. N. Abramov, V. S. Stolyarov, V. V. Ryazanov, A. A. Golubov, and A. V. Ustinov, J. Appl. Phys. 124, 233903 (2018).
  * [29] I. A. Golovchanskiy, N. N. Abramov, V. S. Stolyarov, P. S. Dzhumaev, O. V. Emelyanova, A. A. Golubov, V. V. Ryazanov, and A. V. Ustinov, Adv. Sci. 6, 1900435 (2019).
  * [30] I. A. Golovchanskiy, N. N. Abramov, V. S. Stolyarov, A. A. Golubov, V. V. Ryazanov, and A. V. Ustinov, J. Appl. Phys. 127, 093903 (2020).
  * [31] C. Bayer, J. Jorzick, B. Hillebrands, S. O. Demokritov, R. Kouba, R. Bozinoski, A. N. Slavin, K. Y. Guslienko, D. V. Berkov, N. L. Gorn, and M. P. Kostylev, Phys. Rev. B 72, 064427 (2005).
  * [32] T. Yu, C. P. Liu, H. M. Yu, Y. M. Blanter, and G. E. W. Bauer, Phys. Rev. B 99, 134424 (2019).
  * [33] T. Yu, Y. M. Blanter, and G. E. W. Bauer, Phys. Rev. Lett. 123, 247202 (2019).
  * [34] J. L. Chen, T. Yu, C. P. Liu, T. Liu, M. Madami, K. Shen, J. Y. Zhang, S. Tu, M. S. Alam, K. Xia, M. Z. Wu, G. Gubbiotti, Y. M. Blanter, G. E. W. Bauer, and H. M. Yu, Phys. Rev. B 100, 104427 (2019).
  * [35] I. Bertelli, B. G. Simon, T. Yu, J. Aarts, G. E. W. Bauer, Y. M. Blanter, and T. van der Sar, Adv. Quan. Tech. 4, 2100094 (2021).
  * [36] S. A. Bunyaev, R. O. Serha, H. Y. Musiienko-Shmarova, A. J. E. Kreil, P. Frey, D. A. Bozhko, V. I. Vasyuchka, R. V. Verba, M. Kostylev, B. Hillebrands, G. N. Kakazei, and A. A. Serga, Phys. Rev. Applied 14, 024094 (2020).
  * [37] X. Y. Wei, O. A. Santos, C. H. S. Lusero, G. E. W. Bauer, J. B. Youssef, and B. J. van Wees, arXiv:2112.15165.
  * [38] T. Yu, H. C. Wang, M. A. Sentef, H. M. Yu, and G. E. W. Bauer, Phys. Rev. B 102, 054429 (2020).
  * [39] K. G. Fripp, A. V. Shytov, and V. V. Kruglyak, Phys. Rev. B 104, 054437 (2021).
  * [40] J. Lan, W. C. Yu, R. Q. Wu, and J. Xiao, Phys. Rev. X 5, 041049 (2015).
  * [41] K. Szulc, P. Graczyk, M. Mruczkiewicz, G. Gubbiotti, and M. Krawczyk, Phys. Rev. Applied 14, 034063 (2020).
  * [42] M. A. Schoen, J. M. Shaw, H. T. Nembach, M. Weiler, and T. J. Silva, Phys. Rev. B 92, 184417 (2015).
  * [43] See Supplemental Material at […] for the justification of the role of dipolar interaction, derivation of PSSW, and numerical details for the gated magnetization dynamics.
  * [44] T. Yu, C. Wang, M. A. Sentef, and G. E. W. Bauer, Phys. Rev. Lett. 126, 137202 (2021).
  * [45] J. R. Schrieffer, Theory of Superconductivity (W. A. Benjamin, New York, 1964).
  * [46] C. Kittel, Introduction to Solid State Physics, 5th. ed. (Wiley, NewYork, 1976).
  * [47] J. D. Jackson, Classical Electrodynamics (Wiley, New York, 1998).
  * [48] O. V. Dobrovolskiy, R. Sachser, T. Brächer, T. Bóttcher, V. V. Kruglyak, R. V. Vovk, V. A. Shklovskij, M. Huth, B. Hillebrands, and A. V. Chumak, Nat. Phys. 15, 477 (2019).
  * [49] L. Novotny and B. Hecht, Principles of Nano-Optics (Cambridge University Press, Cambridge, England, 2006).
  * [50] J. Liu, F. Feringa, B. Flebus, L. J. Cornelissen, J. C. Leutenantsmeyer, R. A. Duine, and B. J. van Wees, Phys. Rev. B 99, 054420 (2019).
  * [51] R. Schlitz, S. Vélez, A. Kamra, C.-H. Lambert, M. Lammel, S. T. B. Goennenwein, and P. Gambardella, Phys. Rev. Lett. 126, 257201 (2021).
  * [52] J. H. Han, Y. B. Fan, B. C. McGoldrick, J. Finley, J. T. Hou, P. X. Zhang, and L. Q. Liu, Nano Lett. 21, 7037 (2021).
  * [53] R. A. Kaindl, M. A. Carnahan, J. Orenstein, D. S. Chemla, H. M. Christen, H. Y. Zhai, M. Paranthaman, and D. H. Lowndes, Phys. Rev. Lett. 88, 027003 (2001).
  * [54] J. Demsar, R. D. Averitt, A. J. Taylor, V. V. Kabanov, W. N. Kang, H. J. Kim, E. M. Choi, and S. I. Lee, Phys. Rev. Lett. 91, 267002 (2003).
  * [55] R. A. Kaindl, M. A. Carnahan, D. S. Chemla, S. Oh, and J. N. Eckstein, Phys. Rev. B 72, 060510(R) (2005).
  * [56] T. Yu and M. W. Wu, Phys. Rev. B 96, 155311 (2017).
  * [57] S. P. Chockalingam, M. Chand, J. Jesudasan, V. Tripathi, and P. Raychaudhuri, Phys. Rev. B 77, 214503 (2008).
  * [58] I. Bertelli, J. Carmiggelt, T. Yu, B. G. Simon, C. C. Pothoven, G. E. W. Bauer, Y. M. Blanter, J. Aarts, and T. van der Sar, Sci. Adv. 6, eabd3556 (2020).
  * [59] T. Schneider, A. A. Serga, T. Neumann, B. Hillebrands, and M. P. Kostylev, Phys. Rev. B 77, 214411 (2008).
  * [60] V. E. Demidov, M. P. Kostylev, K. Rott, P. Krzysteczko, G. Reiss, and S. O. Demokritov, Appl. Phys. Lett., 95, 112509 (2009).
  * [61] T. Yu, Z. Luo, and G. E. W. Bauer, arXiv:2206.05535.
  * [62] Q. Wang, B. Heinz, R. Verba, M. Kewenig, P. Pirro, M. Schneider, T. Meyer, B. Lägel, C. Dubs, T. Brächer, and A. V. Chumak, Phys. Rev. Lett. 122, 247202 (2019).
  * [63] P. Lodahl, S. Mahmoodian, S. Stobbe, A. Rauschenbeutel, P. Schneeweiss, J. Volz, H. Pichler, and P. Zoller, Nature (London) 541, 473 (2017).
  * [64] F. J. Rodríguez-Fortuño, G. Marino, P. Ginzburg, D. O’Connor, A. Martínez, G. A. Wurtz, and A. V. Zayats, Science 340, 328 (2013).
  * [65] K. Y. Bliokh, D. Smirnova, and F. Nori, Science 348, 1448 (2015).
  * [66] K. Y. Bliokh and F. Nori, Phys. Rep. 592, 1 (2015).
  * [67] H. Nassar, B. Yousefzadeh, R. Fleury, M. Ruzzene, A. Alú, C. Daraio, A. N. Norris, G. Huang, and M. R. Haberman, Nat. Rev. Mat. 5, 667 (2020).
  * [68] C. Rasmussen, L. Quan, and A. Alú, J. Appl. Phys. 129, 210903 (2021).
  * [69] N. Reiskarimian and H. Krishnaswamy, Nat. Commun. 7, 11217 (2016).
  * [70] A. Nagulu, N. Reiskarimian, and H. Krishnaswamy, Nat. Electro 3, 241 (2020).
  * [71] F. Ando, Y. Miyasaka, T. Li, J. Ishizuka, T. Arakawa, Y. Shiota, T. Moriyama, Y. Yanase, and T. Ono, Nature 584, 373 (2020).
  * [72] Y. M. Itahashi, T. Ideue, Y. Saito, S. Shimizu, T. Ouchi, T. Nojima, and Y. Iwasa, Sci. Adv. 6, eaay9120 (2020).
  * [73] S.-H. Yang, Appl. Phys. Lett. 116, 120502 (2020).
  * [74] S.-H. Yang, R. Naaman, Y. Paltiel, and S. S. P. Parkin, Nat. Rev. Phys. 3, 328 (2021).


# Trust-Region-Free Policy Optimization for Stochastic Policies

Mingfei Sun
University of Oxford
<EMAIL_ADDRESS>
&Benjamin Ellis
University of Oxford
<EMAIL_ADDRESS>
&Anuj Mahajan
University of Oxford
<EMAIL_ADDRESS>
Sam Devlin
Microsoft Research
<EMAIL_ADDRESS>
&Katja Hofmann
Microsoft Research
<EMAIL_ADDRESS>
&Shimon Whiteson
University of Oxford
<EMAIL_ADDRESS>

###### Abstract

Trust Region Policy Optimization (TRPO) is an iterative method that
simultaneously maximizes a _surrogate objective_ and enforces a trust region
constraint over consecutive policies in each iteration. The combination of the
surrogate objective maximization and the trust region enforcement has been
shown to be crucial to guarantee a monotonic policy improvement. However,
solving a trust-region-constrained optimization problem can be computationally
intensive as it requires many steps of conjugate gradient and a large number
of on-policy samples. In this paper, we show that the trust region constraint
over policies can be safely substituted by a _trust-region-free_ constraint
without compromising the underlying monotonic improvement guarantee. The key
idea is to generalize the surrogate objective used in TRPO in a way that a
monotonic improvement guarantee still emerges as a result of constraining the
maximum advantage-weighted ratio between policies. This new constraint
outlines a conservative mechanism for iterative policy optimization and sheds
light on practical ways to optimize the generalized surrogate objective. We
show that the new constraint can be effectively enforced by being conservative
when optimizing the generalized objective function in practice. We call the
resulting algorithm Trust-REgion-Free Policy Optimization (TREFree) as it is
free of any explicit trust region constraints. Empirical results show that
TREFree outperforms TRPO and Proximal Policy Optimization (PPO) in terms of
policy performance and sample efficiency.

Keywords:

policy optimization; deep reinforcement learning

#### Acknowledgements

Mingfei Sun is also affiliated with Microsoft Research. He is partially
supported by funding from Microsoft Research. The experiments were made
possible by a generous equipment grant from NVIDIA.

##  1 Introduction

Trust Region Policy Optimization (TRPO) [9] is an iterative method that
optimizes stochastic policies with a trust region constraint. One of the key
ideas in TRPO is to simultaneously optimize a _surrogate objective_ and
enforce a trust region constraint over consecutive policies at each iteration.
The use of surrogate objectives stems from the seminal work of [6], which
modifies the policy gradient (PG) objective [12] by substituting the on-policy
state distribution with a distribution induced by the policy from the
preceding iteration. [9] show that, despite the mismatch between what the
policy update should optimize, i.e. the PG objective, and what is optimized in
practice, i.e. the surrogate objective, a monotonic improvement guarantee for
policy performance can still emerge from constraining the policy update at
each iteration. The resulting TRPO algorithm thus strictly enforces a
Kullback-Leibler (KL) divergence constraint between consecutive policies, and
seeks to solve a KL-constrained surrogate objective optimization at each
iteration. This combination of the surrogate objective maximization and the
trust region enforcement has also been shown to be crucial for policy
improvement in practice [9, 2].

However, solving a KL-constrained optimization problem can be computationally
intensive [10]. In particular, TRPO use a quadratic approximation of the KL
that augments natural policy gradients [7] with a line-search step that
critically ensures KL enforcement [9]. This procedure requires many steps of
conjugate gradient and a large number of on-policy samples making it both
computationally intensive and sample inefficient [13]. Many follow-up studies
attempt to improve TRPO, for example by leveraging Kronecker-factored
approximated curvature to approximate the trust region [13], solving the KL-
regularized optimization analytically via Expectation-Maximization [1, 5],
transforming TRPO into an unconstrained optimization by policy space
projections [3] or integrating the constraint into differentiable layers [8].

In this paper, we propose simplifying policy optimization by completely
removing the trust region constraint, without compromising the underlying
monotonic improvement guarantee. Specifically, instead of the framework of
surrogate objective optimization [6, 9], we generalize the surrogate objective
used in TRPO in a way that a monotonic improvement guarantee still emerges as
a result of constraining the maximum advantage-weighted ratio between
policies. This new constraint is different from the trust region constraint in
TRPO in that it does not seek to impose any divergence constraint over
consecutive policies. Instead, it outlines a conservative mechanism to bound
the maximum advantage-weighted ratios in each iteration, and sheds light on
practical ways to directly optimize the generalized surrogate objective. We
show that the new constraint can be simply enforced by being conservative when
optimizing the generalized objective function in practice. Furthermore, we
present Trust-REgion-Free Policy Optimization (TREFree), a practical policy
optimization method for optimizing stochastic policies. Empirical results show
that TREFree is effective in optimizing policies, outperforming TRPO and PPO
in both performance and sample efficiency.

##  2 Preliminaries

##### Markov decision process (MDP).

Single-agent RL can be modelled as an infinite-horizon discounted Markov
decision process (MDP) $\\{\mathcal{S},\mathcal{A},P,r,d_{0},\gamma\\}$, where
$\mathcal{S}$ is a finite set of states, $\mathcal{A}$ is a finite set of
actions,
$P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the
transition probability distribution,
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function,
$d_{0}:\mathcal{S}\rightarrow\mathbb{R}$ is the initial state distribution and
$\gamma\in[0,1)$ is the discount factor. Let $\pi$ denote a stochastic policy
$\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$, the performance for a
stochastic policy $\pi(a|s)$ is defined as:
$J(\pi)\triangleq\mathbb{E}_{s_{0}\sim
d_{0},a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim
P(\cdot|s_{t},a_{t})}\big{[}\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\big{]}$
The action-value function $Q_{\pi}$ and value function $V_{\pi}$ are defined
as:
$Q_{\pi}(s_{t},a_{t})\triangleq\mathbb{E}_{t}\Big{[}\sum_{l=0}^{\infty}\gamma^{l}r(s_{t+l},a_{t+l})\Big{]}$,
$V_{\pi}(s_{t})\triangleq\mathbb{E}_{a_{t}\sim\pi(\cdot|s_{t})}\Big{[}Q_{\pi}(s_{t},a_{t})\Big{]}$.
Accordingly, the advantage function is defined as $A_{\pi}(s,a)\triangleq
Q_{\pi}(s,a)-V_{\pi}(s)$.

##### TRPO.

Define the discounted state distribution as:
${d_{\pi}(s)\triangleq\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s|\pi,d_{0})}$. The
following equation is useful [6]:

$J(\tilde{\pi})=J(\pi)+\sum_{s}d_{\tilde{\pi}}(s)\sum_{a}\tilde{\pi}(a|s)A_{\pi}(s,a).$
(1)

The complex dependency of $d_{\tilde{\pi}}(s)$ on $\tilde{\pi}$ makes the
right hand side (RHS) difficult to optimize directly. [9] proposed to consider
the following surrogate objective:

$L_{\pi}(\tilde{\pi})\triangleq
J(\pi)+\sum_{s}d_{\pi}(s)\sum_{a}\tilde{\pi}(a|s)A_{\pi}(s,a),$ (2)

where $d_{\tilde{\pi}}$ is replaced with $d_{\pi}$. TRPO introduces the idea
of bounding the distribution change via the policy divergence. Specifically,
define
$D_{\mathrm{TV}}^{\mathrm{max}}(\pi,\tilde{\pi})\triangleq\max_{s}D_{\mathrm{TV}}\big{(}\pi(\cdot|s),\tilde{\pi}(\cdot|s)\big{)}$,
where $D_{\mathrm{TV}}$ is the total variation (TV) divergence.

######  Theorem 2.1.

([9]) Let $\alpha\triangleq D_{\mathrm{TV}}^{\mathrm{max}}(\pi,\tilde{\pi})$,
then the following bound holds: $J(\tilde{\pi})\geq
L_{\pi}(\tilde{\pi})-\frac{4\epsilon\gamma}{(1-\gamma)^{2}}\alpha^{2}$, where
$\epsilon=\max_{s,a}\lvert A_{\pi}(s,a)\rvert$.

Since the TV divergence and the Kullback-Leibler (KL) divergence are related
as follows:
$D_{\mathrm{TV}}^{2}(\pi,\tilde{\pi})\leq\frac{1}{2}D_{\mathrm{KL}}(\pi,\tilde{\pi})$,
we then have the following $J(\tilde{\pi})\geq
L_{\pi}(\tilde{\pi})-\frac{2\epsilon\gamma}{(1-\gamma)^{2}}D_{\mathrm{KL}}^{\mathrm{max}}(\pi,\tilde{\pi})$,
where
$D_{\mathrm{KL}}^{\mathrm{max}}(\pi,\tilde{\pi})\triangleq\max_{s}D_{\mathrm{KL}}(\pi,\tilde{\pi})$.
This forms the foundation of many policy optimization methods, including TRPO
[9] and Proximal Policy Optimization (PPO) [10]. The KL divergence imposed
over the consecutive policies, $\pi$ and $\tilde{\pi}$, is also called the
trust region. In practice, TRPO adopts a robust way to take large update steps
by using a constraint (rather than a penalty) on the KL divergence, and also
considers using the expected KL divergence, instead of the maximum over all
states:

$\max_{\tilde{\pi}}\quad\mathbb{E}_{(s,a)\sim
d_{\pi}}\Big{[}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}A_{\pi}(s,a)\Big{]},\quad\text{s.t.}\quad\mathbb{E}_{s}\big{[}D_{\mathrm{KL}}(\pi(\cdot|s),\tilde{\pi}(\cdot|s))\big{]}\leq\delta,$
(3)

where $\delta$ is a hyperparameter to specify the trust region. PPO with ratio
clipping further simplifies such trust region constraint by leverage ratio
clipping and considers the following optimization problem:

$\max_{\tilde{\pi}}\mathbb{E}_{d_{\pi}}\big{[}\min\big{(}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}A_{\pi},\mathrm{clip}(\frac{\tilde{\pi}(a|s)}{\pi(a|s)},1-\epsilon,1+\epsilon)A_{\pi}\big{)}\big{]},\quad\text{
where }\epsilon\text{ is the clipping hyperparameter. }$ (4)

##  3 Conservative policy optimization

We show in this section that the trust region constraint over policies can be
safely substituted by a _trust-region-free_ constraint when we consider a
generalized form of the surrogate objective function. We also present a
monotonic improvement guarantee for stochastic policies with the generalized
surrogate objective and the trust-region-free constraint.

###  3.1 Optimization of stochastic policies

######  Definition 3.1.

Define state-action function: $A(s,a)\triangleq
r(s,a)+\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}[f(s^{\prime})]-f(s)$, where
$f$ is a function $f:\mathcal{S}\rightarrow\mathbb{R}$.

Consider updating a stochastic policy from $\pi$ to $\tilde{\pi}$ via policy
gradients. The following proposition from [2] is useful.

######  Proposition 3.2.

For any stochastic policies $\tilde{\pi}$, $\pi$, and the state-action
function defined above,

$J(\tilde{\pi})-J(\pi)=\mathbb{E}_{s\sim
d_{\tilde{\pi}}(s),a\sim\tilde{\pi}}[A(s,a)]-\mathbb{E}_{s\sim
d_{\pi}(s),a\sim\pi}[A(s,a)].$ (5)

This proposition generalizes (1) to a broader family of functions $A(s,a)$.
One can easily verify that the advantage function of $\pi$, i.e.,
$A_{\pi}(s,a)$, satisfies the definition with $f$ function as the value
function. In this case, (5) is equivalent to (1). Furthermore, this
proposition implies that the performance difference between any two policies
can be described by their state-action distribution shift, i.e.,
$d_{\tilde{\pi}}(s)\tilde{\pi}(a)-d_{\pi}(s)\pi(a)$, weighted by a function
$A(s,a)$. In practice, it would be very unlikely to have access to
$d_{\tilde{\pi}}(s)$. We thus leverage the same trick used in TRPO to
substitute $d_{\tilde{\pi}}(s)$ with the state distribution induced by policy
$\pi$, i.e., $d_{\pi}(s)$, and consider the following objective:

$G_{\pi}(\tilde{\pi})\triangleq\mathbb{E}_{s\sim
d_{\pi}(s),a\sim\pi(\cdot|s)}\big{[}\big{(}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\big{)}A(s,a)\big{]}.$
(6)

This new objective generalizes the surrogate objective in (3) to any function
defined in 3.1. We have the following bound,

######  Theorem 3.3.

For any two stochastic policies $\tilde{\pi}$ and $\pi$, the following bound
holds:

$J(\tilde{\pi})-J(\pi)\geq
G_{\pi}(\tilde{\pi})-\frac{2\gamma}{1-\gamma}(\delta+\epsilon),\quad\text{
where
}\delta=\max_{s,a}\left|\big{(}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\big{)}A(s,a)\right|\text{
and }\epsilon=\left|\sum_{a}\pi(a|s)A(s,a)\right|.$

To simplify further analysis, we call
$\big{(}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\big{)}$ the _ratio deviation_.
This theorem states that the policy improvement gap can be effectively bounded
by the maximum product of the ratio deviation
$\big{(}\frac{\tilde{\pi}(a|s)}{\pi(a|s)}-1\big{)}$ and the state-action
function $A(s,a)$. Moreover, as this product also appears in the definition of
$G_{\pi}(\tilde{\pi})$ in (6), one can thus consider constraining it when
optimizing $G_{\pi}(\tilde{\pi})$. This is what we call the _conservative
policy optimization_. We discuss how this conservative policy update can be
implemented in practice in the next section.

Theorem 3.3 differs from Theorem 2.1 in two respects. First, it presents a
lower bound for the performance improvement with respect to a state-action
function defined in Definition 3.1. This $A(s,a)$ does not necessarily need to
be the advantage function. Second, instead of imposing the TV constraint over
the policies as in Theorem 2.1, the above theorem considers the maximum
product of the ratio deviation and the state-action function. According to
[11], the TV constraint between any two policies can be equivalently
translated into a constraint over ratio deviations. In this sense, TRPO is
essentially a special case of Theorem 3.3 by leveraging the TV to bound the
ratio deviations under the assumption that the advantage function should be
small. Namely, TRPO optimizes the policy regardless of how the magnitude of
the advantage might change throughout optimization. Consequently, TRPO may
fail to optimize the policy when the advantage function is large in magnitude
at some state-action sample even though the TV divergence is well bounded at
one iteration.

###  3.2 Practical policy optimization methods

Algorithm 1 TREFree algorithm

for iterations $i=1,2,...$ do

for actor $=1,2,...,N$ do

Run policy $\pi$ in environment

Compute advantage estimates $\hat{A}_{\pi}$

end for

for epoch $=1,2,...,K$ do

Sample $M$ samples $\\{(s,a)\\}$ from previous rollouts.

Compute
$\mathcal{L}(\theta)\triangleq\frac{1}{M}\sum\limits_{s,a}\min\Big{(}\big{(}\frac{\tilde{\pi}_{\theta}(a|s)}{\pi(a|s)}-1\big{)}\hat{A}_{\pi}(s,a),\delta\Big{)}$.

Maximize $\mathcal{L}(\theta)$ w.r.t $\theta$ via gradient descent.

end for

$\pi\leftarrow\tilde{\pi}_{\theta}$.

end for

We now present the practical policy optimization methods for stochastic
policies. We first offer intuitions to understand the underlying idea of
conservative policy optimization in the above theorems.

Theorem 3.3 is closely related to some existing policy optimization methods.
For example, optimizing $G_{\pi}(\tilde{\pi})$ without any conservative
constraint is equivalent to the policy gradient method [12]. Also, there are
two ways to impose such conservative constraints: the ratio-conservative and
the objective-conservative, which refer to removing the incentive of
increasing the ratio deviations (corresponding to PPO [10]) or the objective,
respectively, when optimizing the objective function.

##### Non-conservative

With no consideration of the conservative policy update principle, one can
directly optimize $G_{\pi}(\tilde{\pi})$ with the state-action function chosen
as the advantage function of $\pi$. Such policy optimization is performed by
_policy gradient methods_ [12]: $\max_{\theta}\quad\mathbb{E}_{(s,a)\sim
d_{\pi}}\big{[}\frac{\tilde{\pi}_{\theta}(s,a)}{\pi(s,a)}A_{\pi}(s,a)\big{]}$.
Theorem 3.3 implies that optimizing the above objective with the same set of
sampled data for multiple times (i.e., multi-epoch optimization as in [10]),
could incur a significant degradation in policy performance, since the product
between the ratio deviation and the advantage can be large. Thus, applying
policy gradients for multiple epoch optimization does not guarantee policy
improvement [6].

##### Ratio-conservative

One can take into account the conservative policy update rule by constraining
the ratio deviations. Specifically, when optimizing $G_{\pi}(\tilde{\pi})$
with the state-action function as the advantage, one can clip the ratio
deviations to remove the incentive of inducing unexpected large deviations
(i.e., _ratio-conservative_), as follows
$\max_{\theta}\quad\mathbb{E}_{(s,a)\sim
d_{\pi}}\big{[}\mathrm{clip}\big{(}\frac{\tilde{\pi}_{\theta}(s,a)}{\pi(s,a)}-1,-\lambda,\lambda\big{)}A_{\pi}(s,a)\big{]}$,
where $\lambda>0$ is a hyper-parameter for ratio deviation clipping. This new
objective resembles the ratio clipping objective (4) used in _PPO_ [10], which
has been shown to be effective in practice, especially with a normalized
advantage function. However, such ratio clipping scheme ignores the potential
effect of the advantage function on the ratio deviation, and thus can fail to
monotonically improve the policy performance when the advantage is large at
some state-action point and dominates
$\big{(}\frac{\tilde{\pi}_{\theta}(s,a)}{\pi(s,a)}-1\big{)}A_{\pi}(s,a)$ for a
small ratio deviation. Furthermore, solely bounding the divergence between
policies, as used in trust region methods for policy optimization [9, 2], may
not be a good option in practice, as it is not sufficient for policy
improvement when the advantage function fluctuates greatly across state-action
samples.

##### Objective-conservative

We can instead apply the clipping scheme to the objective. Namely, we can clip
the objective directly to achieve this conservative update principle, as
follows:

$\max_{\theta}\quad\mathbb{E}_{(s,a)\sim
d_{\pi}}\Big{[}\min\Big{(}\big{(}\frac{\tilde{\pi}_{\theta}(s,a)}{\pi(s,a)}-1\big{)}A_{\pi}(s,a),\delta\Big{)}\Big{]},$

where $\delta>0$ is a hyper-parameter to control the conservativeness when
optimizing the objective. The $\delta$ operates as a threshold beyond which
the objective quantity have no contribution to the optimization. We call it
_objective conservative_ , and the resulting algorithm _Trust-REgion-Free
Policy Optimization (TREFree)_ as it is free of any explicit trust region
constraint. TREFree is detailed in Algorithm 1.

##  4 Experiments

In this section, we compare TREFree with TRPO and PPO across the Mujoco
continuous control tasks. We adopt the same strategy as in TRPO [9] and PPO
[10] to normalize the observations, rewards, and advantages. Specifically, the
observations, rewards and advantages are normalized to zero mean and unit
variance using a running mean and standard deviation. Both observations and
rewards are normalized using a running mean and standard deviation per-
timestep over the whole training process, while the advantages are normalized
only within a training batch. Moreover, we leverage the actor-critic framework
by parameterizing the actor and critic with 2-layered perceptrons, each of
which has 64 hidden units and is activated with $\tanh$. Also, the actor and
critic share parameters by reusing the first layer of their neural networks,
which has been reported to stabilize the training and improve performance [9,
10, 4]. The policy is modeled as a Gaussian distribution, with mean and
variance parameterized by the actor neural network.

Figure 1:  Contrasting TREFree with TRPO and PPO on Mujoco benchmark tasks;
training with a margin size $0.01$.

We now compare TREFree with TRPO and PPO across the Mujoco continuous control
tasks with different control complexity [4], We used the publicly available
and widely used repository (https://github.com/openai/baselines) as the
baseline implementation. For TRPO and PPO, we use the default hyper-parameters
given in [9, 10]. We also sweep over the clipping range for PPO and use the
best performing value as the baseline. Furthermore, we heuristically decay the
learning rate of TREFree and PPO linearly from $0.0003$ to $0$, as we found
this annealing strategy stabilizes training for both PPO and TREFree. We
heuristically set $\delta$ in TREFree to $0.01$ as it is found to perform well
across all tasks.

Figure 2: Contrasting ratio ranges.

The performance comparison on the Mujoco continuous control tasks is presented
in Figure 1. Overall, TREFree performs better than the baselines on all the
tasks except Hopper. TREFree outperforms TRPO and PPO by a large margin in
terms of the final policy performance on tasks Ant, HalfCheetah, Humanoid,
HumanoidStandup and Walker2d. These five Mujoco environments are more
complicated than Hopper. The training curves in Figure 1 also show how TREFree
often outpaces other baselines in improving policy performance. Though TREFree
is outperformed by PPO and TRPO on Hopper, the performance curves of all these
methods in this specific environment fluctuates greatly over time, and overlap
each other.

We also report the ratio ranges of different methods in Figure 2 to show the
underlying differences between TREFree and the baseline methods. The
probability ratios $\frac{\tilde{\pi}(a|s)}{\pi(a|s)}$ are an important
indicator in TRPO and PPO training as they are closely related to the total
variation divergence [10]. Figure 2 shows that the ratios in both TRPO and
TREFree are better bounded than in PPO, where the ratios grow without bound.
However, TREFree constrains ratios in a dramatically different way from TRPO:
TRPO bounds ratios between $[-2,1]$ (log-scale) in a symmetrical way, while
TREFree bounds ratios between $[0,1]$, which implies that the policy is most
often updated to increase the probability at empirical samples. This contrast
in ratio ranges suggests that TREFree is fundamentally different from trust
region methods.

## References

  * [1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller.  Maximum a posteriori policy optimisation.  arXiv preprint arXiv:1806.06920, 2018.
  * [2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel.  Constrained policy optimization.  In International Conference on Machine Learning, pages 22–31. PMLR, 2017.
  * [3] Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann.  Projections for approximate policy iteration algorithms.  In International Conference on Machine Learning, pages 181–190. PMLR, 2019.
  * [4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  arXiv preprint arXiv:1606.01540, 2016.
  * [5] Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado van Hasselt.  Muesli: Combining improvements in policy optimization.  arXiv preprint arXiv:2104.06159, 2021.
  * [6] Sham Kakade and John Langford.  Approximately optimal approximate reinforcement learning.  In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
  * [7] Sham M Kakade.  A natural policy gradient.  Advances in neural information processing systems, 14, 2001.
  * [8] Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann.  Differentiable trust region layers for deep reinforcement learning.  arXiv preprint arXiv:2101.09207, 2021.
  * [9] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In International conference on machine learning, pages 1889–1897. PMLR, 2015.
  * [10] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [11] Mingfei Sun, Vitaly Kurin, Guoqing Liu, Sam Devlin, Tao Qin, Katja Hofmann, and Shimon Whiteson.  You may not need ratio clipping in ppo.  arXiv preprint arXiv:2202.00079, 2022.
  * [12] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In Advances in neural information processing systems, pages 1057–1063, 2000.
  * [13] Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba.  Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.  Advances in neural information processing systems, 30:5279–5288, 2017.


$\begin{bmatrix}\frac{1-(1+|z|^{2})\,e^{-|z|^{2}}}{1-e^{-|z|^{2}}}&\frac{e^{-|z|^{2}/2}\big{(}1-|z|^{2}-e^{-|z|^{2}}\big{)}}{1-e^{-|z|^{2}}}\\\
\frac{e^{-|z|^{2}/2}\big{(}1-|z|^{2}-e^{-|z|^{2}}\big{)}}{1-e^{-|z|^{2}}}&\frac{1-(1+|z|^{2})\,e^{-|z|^{2}}}{1-e^{-|z|^{2}}}\end{bmatrix}\,.$

Claim 6.6 shows that

$\displaystyle\mathbb{E}\Big{[}\log|D\widehat{g}(0)|^{2}|D\widehat{g}(0)|^{2}\,$
$\displaystyle\log|D\widehat{g}(z)|^{2}|D\widehat{g}(z)|^{2}\mid
g(0)=g(z)=0\Big{]}$
$\displaystyle=\sigma^{4}\,\Bigg{(}\big{(}\beta_{0}+\log\sigma^{2}\big{)}^{2}+\big{(}\beta_{1}-\log\sigma^{2}\big{)}^{2}|\theta|^{2}+\sum_{m=2}^{\infty}\beta_{m}^{2}|\theta|^{2m}\Bigg{)}\,,$

with

$\sigma^{2}=\frac{1-(1+|z|^{2})\,e^{-|z|^{2}}}{1-e^{-|z|^{2}}}\qquad\text{and}\qquad\theta=\frac{e^{-|z|^{2}/2}\big{(}1-|z|^{2}-e^{-|z|^{2}}\big{)}}{1-(1+|z|^{2})e^{-|z|^{2}}}\,.$

Finally, we observe that

$\det\big{[}\operatorname{Cov}\big{(}\widehat{g}(0),\widehat{g}(z)\big{)}\big{]}=1-e^{-|z|^{2}}$

and plug everything into the definition (84) of $c_{2}^{\prime\prime}$ to get

$\displaystyle c_{2}^{\prime\prime}$
$\displaystyle=\frac{1}{4\pi}\int_{\mathbb{C}}\frac{\sigma^{4}}{1-e^{-|z|^{2}}}\bigg{(}\big{(}\beta_{0}+\log\sigma^{2}\big{)}^{2}-\beta_{0}^{2}+\big{(}\beta_{1}-\log\sigma^{2}\big{)}^{2}|\theta|^{2}+\sum_{m=2}^{\infty}\beta_{m}^{2}|\theta|^{2m}\bigg{)}\,{\rm
d}m(z)$ $\displaystyle=\frac{1}{4}\,\bigg{(}I_{1}+I_{2}+I_{3}\bigg{)}\,,$

with $I_{1},I_{2},I_{3}$ given by (82). Plugging this into (84) gives what we
wanted. ∎

###### Proof of Theorem 2.2.

The proof follows immediately by combining Lemma 6.2, Lemma 6.5 and Lemma 6.3.
∎

## Acknowledgments

We would like to thank Alon Nishry, Aditya Potukuchi and Mikhail Sodin for
very helpful discussions. The research of M.M. is supported in part by NSF
grant DMS-2137623. The research of O.Y. is supported by the ERC Advanced Grant
692616 and by ISF Grants 1903/18 and 1288/21.

## References

  * [1] K. Alishahi, M. Zamani, _The spherical ensemble and uniform distribution of points on the sphere_. Electron. J. Probab., 20 (2015).
  * [2] M. Ancona, T. Letendre, _Zeros of smooth stationary Gaussian processes_. Electron. J. Probab. 26 (2021).
  * [3] D. Armentano, C. Beltrán, M. Shub, _Minimizing the discrete logarithmic energy on the sphere: the role of random polynomials_. Trans. Amer. Math. Soc. 363 (2011), 2955–2965.
  * [4] J. Azaïs, M. Wschebor, _Level sets and extrema of random processes and fields_. John Wiley & Sons, Inc., 2009.
  * [5] C. Beltrán.  _The state of the art in Smale’s 7th problem._ In Foundations of computational mathematics, 1–15. Cambridge Univ. Press, 2013.
  * [6] C. Beltrán, J. Marzo, J. Ortega-Cerdà, _Energy and discrepancy of rotationally invariant determinantal point processes in high dimensional spheres_. J. Complexity 37 (2016), 76–109.
  * [7] J. Ben Hough, M. Krishnapur, Y. Peres, B. Virág, _Zeros of Gaussian analytic functions and determinantal point processes_. Amer. Math. Soc., 2009.
  * [8] L. Bétermin, E. Sandier, _Renormalized energy and asymptotic expansion of optimal logarithmic energy on the sphere_. Constr. Approx., 47 (2018), 39–74.
  * [9] B. Błaszczyszyn, D. Yogeshwaran, J. E. Yukich, _Limit theory for geometric statistics of point processes having fast decay of correlations_. Ann. Probab., 47 (2019), 835–895.
  * [10] E. Bogomolny, O. Bohigas, P, Leboeuf, _Distribution of roots of random polynomials_. Phys. Rev. Lett., 68 (1992), 2726–2729.
  * [11] E. Bogomolny, O. Bohigas, P, Leboeuf, _Quantum Chaotic Dynamics and Random Polynomials_. J. Stat. Phys., 85 (1996), 639–679.
  * [12] S. Borodachov, D. Hardin, E. Saff, _Discrete energy on rectifiable sets_.  Springer, 2019.
  * [13] J. Brauchart, D. Hardin, E. Saff, _The next-order term for optimal Riesz and logarithmic energy asymptotics on the sphere_. Contemp. Math., 578 (2012), 31–61.
  * [14] J. Buckley, A. Nishry, _Gaussian complex zeroes are not always normal: limit theorems on the disc_. Probab. Math. Phys., 3 (2022), 675–706.
  * [15] V. de la Torre, J. Marzo, _Expected energy of zeros of elliptic polynomials_. Preprint, 2022. arXiv:2211.07599
  * [16] R. Feng, S. Zelditch, _Random Riesz energies on compact Kähler manifolds_. Trans. Amer. Math. Soc., 365 (2013), 5579–5604.
  * [17] L. Gass, _Cumulants asymptotics for the zeros counting measure of real Gaussian processes_. Preprint, 2021. arXiv:2112.08247
  * [18] J. Hannay, _Chaotic analytic zero points: exact statistics for those of a random spin state_. J. Phys. A, 29 (1996).
  * [19] S. Janson, _Gaussian Hilbert spaces_. Cambridge University Press, 1997.
  * [20] E. Kostlan, _On the distribution of roots of random polynomials_. From Topology to Computation: Proceedings of the Smalefest (1993), 419–431.
  * [21] M. Krishnapur, _From random matrices to random analytic functions_. Ann. Probab. 37 (2009), 314–346.
  * [22] A. B. Lauritsen, _Floating Wigner crystal and periodic jellium configurations_. J. Math. Phys. 62 (2021).
  * [23] M. Michelen, J. Sahasrabudhe, _Random polynomials: the closest roots to the unit circle_. Preprint, 2020. arXiv:2010.10869
  * [24] L. M. Milne-Thomson, _The Calculus of Finite Differences_. Macmillan & Co., 1951.
  * [25] F. Nazarov, M. Sodin, _Correlation Functions for Random Complex Zeroes: Strong Clustering and Local Universality_. Commun. Math. Phys. 310 (2012), 75–98.
  * [26] F. Nazarov, M. Sodin, _Fluctuations in random complex zeroes: asymptotic normality revisited_. Int. Math. Res. Not. IMRN 24 (2011), 5720–5759.
  * [27] F. Nazarov, M. Sodin, _What is $\ldots$a Gaussian entire function?_ Notices Amer. Math. Soc. 57 (2010), 375–377.
  * [28] M. Reitzner, M. Schulte, _Central limit theorems for $U$-statistics of Poisson point processes_. Ann. Probab. 41 (2013), 3879–3909.
  * [29] D. Ruelle, _Statistical mechanics: Rigorous results_. World Scientific, 1999.
  * [30] A. N. Shiryaev, _Probability_. Springer-Verlag, 1996.
  * [31] M. Shub, S. Smale, _Complexity of Bezout’s theorem. II. Volumes and probabilities_. In Computational algebraic geometry 109, 267–285, 1993.
  * [32] S. Smale, _Mathematical problems for the next century_. In Mathematics: Frontiers and Perspectives, 271–294. Amer. Math. Soc., 2000.
  * [33] M. Sodin, _Zeros of Gaussian analytic functions_. Math. Res. Lett., 7 (2000), 371–381.
  * [34] M. Sodin, B. Tsirelson, _Random complex zeroes. I. Asymptotic normality_. Israel J. Math., 144 (2004), 125–149.
  * [35] T. Speed, _Cumulants and partition lattices 1._ Austral. J. Statist. 25 (1983): 378-388.
  * [36] S. Steinerberger, _On the logarithmic energy of points on ${\mathbb{S}}^{2}$_. J. Anal. Math., 148 (2022), 187–211.

##  Appendix A Proof of Proposition 2.5

The starting point is a sufficiently general version for the Kac-Rice formula,
given by the following theorem from [4].

######  Theorem A.1.

[4, Theorem 6.4] Let $Z:U\to\mathbb{R}^{d}$ be a Gaussian random field with
$U\subset\mathbb{R}^{d}$ open. Assume that:

  * •

the function $t\mapsto Z(t)$ is almost surely smooth;

  * •

for each $t\in U$ we have $\det\operatorname{Cov}(Z(t))\neq 0$;

  * •

$\mathbb{P}\big{(}\exists t\in U,Z(t)=0,\det(Z^{\prime}(t))=0\big{)}=0$.

Suppose further that one has another random field
$Y^{t}:U\times\mathbb{R}^{\ell}\to\mathbb{R}^{m}$ satisfying the following
conditions:

  * •

$Y^{t}$ is measurable and almost surely $(t,w)\mapsto Y^{t}(w)$ is continuous;

  * •

for each $t$ the random process $(s,w)\mapsto(Z(s),Y^{t}(w))$ is (real)
Gaussian.

Let $H:U\times\mathcal{C}(\mathbb{R}^{\ell},\mathbb{R}^{m})\to\mathbb{R}$ be a
bounded and continuous function, where the topology on
$\mathcal{C}(\mathbb{R}^{\ell},\mathbb{R}^{m})$ is the topology of uniform
convergence on compact sets. Then, for each compact $I\subset U$, we have

$\mathbb{E}\left(\sum_{t\in
I,Z(t)=0}H(t,Y^{t})\right)=\int_{I}\mathbb{E}\left(|\det(Z^{\prime}(t))|\,H(t,Y^{t})\mid
Z(t)=0\right)p_{Z(t)}(0)\,dt$

where $p_{Z(t)}(0)$ is the density of the random vector $Z(t)$ evaluated at
$0$.

With Theorem A.1 at our disposal, we can prove the following version of it for
Gaussian analytic functions, which will take us most of the way towards
Proposition 2.5.

######  Lemma A.2.

Let $f$ be a Gaussian analytic function so that for any $k$ distinct points
$w_{1},\ldots,w_{k}$ we have
$\det\big{[}\operatorname{Cov}(f(w_{1}),\ldots,f(w_{k}))\big{]}\neq 0$. For a
compact set $K\subset\mathbb{C}$, let $\mathcal{Z}_{k}(K)$ denote the set of
distinct $k$-tuples of zeros of $f$ in $K$. Then for bounded and continuous
functions $h_{j}:\mathbb{C}^{2}\to\mathbb{R}$, $1\leqslant j\leqslant k+\ell$,
we have

$\displaystyle\mathbb{E}\bigg{[}\bigg{(}\int_{K^{\ell}}\prod_{i=1}^{\ell}h_{k+i}(f(z_{i}),z_{i})\,{\rm
d}m(\mathbf{z})\bigg{)}$
$\displaystyle\sum_{(y_{1},\ldots,y_{k})\in\mathcal{Z}_{k}(K)}\prod_{j=1}^{k}h_{j}(f^{\prime}(y_{j}),y_{j})\bigg{]}$
$\displaystyle=\int_{K^{\ell}}\int_{K^{k}}\frac{{\Lambda}(\mathbf{u},\mathbf{z};h,f)}{\pi^{k}\det\big{[}\operatorname{Cov}\big{(}(f(u_{j}))_{j=1}^{k}\big{)}\big{]}}\,{\rm
d}m(\mathbf{u})\,{\rm d}m(\mathbf{z})\,,$

where

${\Lambda}(\mathbf{u},\mathbf{z};h,f)\stackrel{{\scriptstyle{\rm
def}}}{{=}}\mathbb{E}\bigg{[}\Big{(}\prod_{j=1}^{k}|f^{\prime}(u_{j})|^{2}h_{j}(f^{\prime}(u_{j}),u_{j})\Big{)}\Big{(}\prod_{i=1}^{\ell}h_{k+i}(f(z_{i}),z_{i})\Big{)}\,\Big{|}\,f(u_{j})=0,\
\forall j\in[k]\bigg{]}\,.$

###### Proof.

We can assume without loss of generality that the $h_{j}$’s are non-negative.
Let $\delta>0$ and consider the set

$U_{\delta}\stackrel{{\scriptstyle{\rm
def}}}{{=}}\big{\\{}(y_{1},\ldots,y_{k})\in
K^{k}\,:\,|y_{i}-y_{j}|\geqslant\delta,\ \forall
i\not=j\big{\\}}\subset\mathbb{C}^{k}\,.$

We define $Z:U_{\delta}\to\mathbb{C}^{k}$ by

$Z(\mathbf{u})\stackrel{{\scriptstyle{\rm
def}}}{{=}}\big{(}f(u_{1}),\ldots,f(u_{k})\big{)}$

and note that $Z$ is a smooth Gaussian process. It is straightforward to check
that

$|\det\big{(}Z^{\prime}(\mathbf{u})\big{)}|=\prod_{j=1}^{k}|f^{\prime}(u_{k})|\,,$

and so, if $t\in U_{\delta}$ is such that $Z(t)=\det Z^{\prime}(t)=0$ we must
have a $z\in K$ with $f(z)=f^{\prime}(z)=0$. By [7, Lemma 2.4.2], this event
has probability zero.

Define
$Y^{\mathbf{u}}:U_{\delta}\times\mathbb{C}^{\ell}\to\mathbb{C}^{\ell+k}$ by

$Y^{\mathbf{u}}(\mathbf{z})\stackrel{{\scriptstyle{\rm
def}}}{{=}}\big{(}f^{\prime}(u_{1}),\ldots
f^{\prime}(u_{k}),f(z_{1}),\ldots,f(z_{\ell})\big{)}$

and note that $Y^{\mathbf{u}}$ is measurable, continuous and Gaussian.
Finally, we set

$H(\mathbf{u},Y^{\mathbf{u}})\stackrel{{\scriptstyle{\rm
def}}}{{=}}\bigg{(}\prod_{j=1}^{k}h_{j}(f^{\prime}(u_{j}),u_{j})\bigg{)}\bigg{(}\int_{K^{\ell}}\prod_{i=1}^{\ell}h_{k+i}(f(z_{i}),z_{i})\,{\rm
d}m(\mathbf{z})\bigg{)}$

and note that $H$ is bounded and continuous. By the above, we may apply
Theorem A.1 and see that

$\displaystyle\mathbb{E}$
$\displaystyle\left[\left(\int_{K^{\ell}}\prod_{i=1}^{\ell}h_{k+i}(f(z_{i}),z_{i})\,{\rm
d}m(\mathbf{z})\right)\sum_{(y_{1},\ldots,y_{k})\in\mathcal{Z}_{k}(K)\cap
U_{\delta}}\prod_{j=1}^{k}h_{j}(f^{\prime}(y_{j}),y_{j})\right]$
$\displaystyle=\int_{U_{\delta}}\mathbb{E}\Big{[}\,|\det[Z^{\prime}(\mathbf{u})]|\,H(\mathbf{u},Y^{\mathbf{u}})\mid
Z(\mathbf{z})=0\Big{]}\,p_{Z(\mathbf{z})}(0)\,{\rm d}m(\mathbf{u})\,.$

where $p_{Z(\mathbf{z})}$ is the density function of the Gaussian vector
$\big{(}f(u_{1}),\ldots,f(u_{k})\big{)}$, and hence

$p_{Z(\mathbf{z})}(0)^{-1}=\pi^{k}\det\big{[}\operatorname{Cov}\big{(}(f(u_{j}))_{j=1}^{k}\big{)}\big{]}\,.$

Since we assumed that the $h_{j}$’s are non-negative, we can send
$\delta\downarrow 0$ and apply the monotone convergence theorem, which implies
the lemma. ∎

###### Proof of Proposition 2.5.

Let $\\{\psi_{m}\\}$ be a sequence of continuous functions so that
$|\psi_{m}|\leqslant|\psi_{m+1}|$ and $\psi_{m}(s)\xrightarrow{m\to\infty}\log
s$ for all $s>0$. For a compact set $K\subset\mathbb{C}$ we apply Lemma A.2
and observe that

$\displaystyle
n^{-a}\mathbb{E}\left[\left(\int_{K^{b}}\prod_{i=1}^{\ell}\psi_{m}(|\widehat{f}(z_{i})|)\,{\rm
d}\mu(\mathbf{z})\right)\sum_{(y_{1},\ldots,y_{a})\in\mathcal{Z}_{a}(K)}\prod_{j=1}^{a}\psi_{m}(|D\widehat{f}(y_{j})|)^{p_{j}}\right]$
(85)
$\displaystyle=n^{-a}\int_{K^{b}}\int_{K^{a}}\frac{\mathbb{E}\left[\left(\prod_{j=1}^{a}|f^{\prime}(u_{j})|^{2}\psi_{m}(|D\widehat{f}(u_{j})|)^{p_{j}}\right)\left(\prod_{i=1}^{b}\psi_{m}(|f(z_{i})|)\right)\,|\,f(u_{j})=0\,\forall
j\right]}{\pi^{a}\det\operatorname{Cov}\left((f(u_{j}))_{j=1}^{a}\right)}\,{\rm
d}m(\mathbf{u})\,{\rm d}\mu^{\otimes b}(\mathbf{z})$
$\displaystyle=\int_{K^{b}}\int_{K^{a}}\frac{\mathbb{E}\left[\left(\prod_{j=1}^{a}|D\widehat{f}(u_{j})|^{2}\psi_{m}(|D\widehat{f}(u_{j})|)^{p_{j}}\right)\left(\prod_{i=1}^{b}\psi_{m}(|f(z_{i})|)\right)\,|\,f(u_{j})=0\,\forall
j\right]}{\det\operatorname{Cov}\left((\widehat{f}(u_{j}))_{j=1}^{a}\right)}\,{\rm
d}\mu^{\otimes a}(\mathbf{u})\,{\rm d}\mu^{\otimes b}(\mathbf{z})$

where $f=f_{n}$ is our random polynomial (1). We claim that

$\mathbb{E}\left[\left(\int_{\mathbb{C}^{b}}\prod_{i=1}^{b}|\log|\widehat{f}(z_{i})||\,{\rm
d}\mu^{\otimes
b}(\mathbf{z})\right)\sum_{(y_{1},\ldots,y_{a})\in\mathcal{Z}_{a}(\mathbb{C})}\prod_{j=1}^{a}|\log|D\widehat{f}(y_{j})||^{p_{j}}\right]<\infty\,.$
(86)

Assuming that (86) holds, we can finish the proof of the proposition. Simply
take $K\uparrow\mathbb{C}$ and $m\to\infty$ in (85) and apply the dominated
convergence theorem, which is justified by (86). To prove (86), simply apply
Lemma A.2 as in (85), but with $|\psi_{m}|$ instead of $\psi_{m}$. As both
sides are non-deceasing when $m$ increase and as $K$ exhaust $\mathbb{C}$, we
can apply the monotone convergence theorem to obtain equality of the limits.
Finally, by the assumption of Proposition 2.5, the right-hand-side is finite,
and hence (86) holds and we are done. ∎

##  Appendix B Positivity of the variance

In order to show that the limiting variance is non-trivial, it is sufficient
to prove that $c_{1}\neq c_{2}$. In particular, we will prove

######  Fact B.1.

We have $c_{2}-c_{1}>0.025$.

We will tackle $I_{1}$ and $I_{2}$ separately, beginning with $I_{1}$.

######  Fact B.2.

$I_{2}\geqslant 1.348$.

###### Proof.

Note that for $s\geqslant 1$ we have
$|\beta_{1}|>\left|\log(1-\frac{s}{e^{s}-1})\right|$ and have

$\left(\beta_{1}-\log\left(1-\frac{s}{e^{s}-1}\right)\right)^{2}\geqslant\left(\beta_{1}+\frac{3}{2}\frac{s}{e^{s}-1}\right)^{2}\,.$
(87)

In particular, we have

$\displaystyle
I_{2}\geqslant\int_{1}^{\infty}\big{(}1-e^{-s}\big{)}^{-1}\Big{(}1-\frac{s}{e^{s}-1}\Big{)}^{2}\left(\beta_{1}+\frac{3}{2}\frac{s}{e^{s}-1}\right)^{2}\bigg{(}\frac{e^{-s/2}(1-s-e^{-s})}{1-(s+1)e^{-s}}\bigg{)}^{2}\,{\rm
d}s\,.$

Letting $h_{2}$ denote the integrand appearing on the right-hand side, we note
that $h_{2}$ is monotone increasing for $s\in[0,1]$ and so we may bound

$I_{2}\geqslant\int_{0}^{\infty}h_{2}(s)\,{\rm d}s-h_{2}(1)\,.$ (88)

Exact computation gives

$\displaystyle\int_{0}^{\infty}h_{2}(s)\,{\rm d}s$
$\displaystyle=\frac{\pi^{2}}{2}\left(\frac{1}{12}+\frac{\pi^{2}}{10}-\frac{\gamma}{3}(1-\gamma)\right)-3\zeta(3)\left(\frac{1}{4}+\gamma\right)-\frac{\gamma}{2}(3-\gamma)-\frac{19}{16}>1.408$
$\displaystyle h_{2}(1)$ $\displaystyle=\frac{(2\gamma
e-4d-2\gamma+7)^{2}}{4(e-1)^{5}}\leqslant.06\,.$

∎

To handle $I_{1}$ we first simplify slightly.

######  Fact B.3.

$\displaystyle I_{1}$
$\displaystyle=-\frac{(1-\gamma)^{2}}{2}\left(1+\frac{\pi^{2}}{3}\right)+I_{1}^{\prime}>-.384+I_{1}^{\prime}$

where

$I_{1}^{\prime}=\int_{0}^{\infty}\big{(}1-e^{-s}\big{)}^{-1}\Big{(}1-\frac{s}{e^{s}-1}\Big{)}^{2}\Big{[}2\beta_{0}\log\Big{(}1-\frac{s}{e^{s}-1}\Big{)}+\log^{2}\Big{(}1-\frac{s}{e^{s}-1}\Big{)}\Big{]}\,{\rm
d}s$

###### Proof.

This follows from computing

$\int_{0}^{\infty}\left[1-\big{(}1-e^{-s}\big{)}^{-1}\Big{(}1-\frac{s}{e^{s}-1}\Big{)}^{2}\right]\,{\rm
d}s=\frac{1}{2}+\frac{\pi^{2}}{6}$

along with recalling that $\beta_{0}=1-\gamma$. ∎

We now similarly lower bound $I_{1}^{\prime}$ in a similar fashion to $I_{2}$.

######  Fact B.4.

$I_{1}^{\prime}\geqslant-.472$

###### Proof.

Note that the integrand of $I_{1}^{\prime}$ is positive for $s\in[0,1]$. For
$s\geqslant 1$ we may bound

$I_{1}^{\prime}\geqslant\int_{1}^{\infty}(1-e^{-s})^{-1}\left(1-\frac{s}{e^{s}-1}\right)^{2}\left(-2\beta_{0}\cdot\left(\frac{s}{e^{s}-1}+\left(\frac{s}{e^{s}-1}\right)^{2}\right)+\left(\frac{s}{e^{s}-1}\right)^{2}\right)\,{\rm
d}s\,.$

Letting $h_{1}$ denote the integrand on the right-hand side we see that
$h_{1}\leqslant 0$ and so we may bound

$I_{1}^{\prime}\geqslant\int_{0}^{\infty}h_{1}(s)\,{\rm d}s-|h_{1}(1)|\,.$

Exact computation again gives

$\displaystyle\int_{0}^{\infty}h_{1}(s)\,{\rm d}s$
$\displaystyle=\pi^{2}\left(\frac{1}{2}-\frac{2\gamma}{3}+\frac{\pi^{2}}{15}(1-2\gamma)\right)+\zeta(3)(18\gamma-11)-\frac{\gamma}{2}+\frac{5}{12}>-.472$

∎

###### Proof of Fact B.1.

Write

$\displaystyle 4(c_{2}-c_{1})$
$\displaystyle=I_{1}+I_{2}+I_{3}+\frac{\pi^{2}}{6}+\gamma(\gamma-2)-\zeta(3)$
$\displaystyle\geqslant-0.384-0.472+1.348+\frac{\pi^{2}}{6}+\gamma(\gamma-2)-\zeta(3)$
$\displaystyle\geqslant 0.1$

where on the first inequality we used $I_{3}\geqslant 0$ and Facts B.2, B.3,
and B.4. ∎


# DGFN: Double Generative Flow Networks

Elaine Lau1,2, Nikhil Vemgal1,2, Doina Precup1,2,3, Emmanuel Bengio4
1Mila, 2McGill University,
3Google DeepMind, 4 Valence Labs
{tsoi.lau<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep learning is emerging as an effective tool in drug discovery, with
potential applications in both predictive and generative models. Generative
Flow Networks (GFlowNets/GFNs) are a recently introduced method recognized for
the ability to generate diverse candidates, in particular in small molecule
generation tasks. In this work, we introduce double GFlowNets (DGFNs). Drawing
inspiration from reinforcement learning and Double Deep Q-Learning, we
introduce a target network used to sample trajectories, while updating the
main network with these sampled trajectories. Empirical results confirm that
DGFNs effectively enhance exploration in sparse reward domains and high-
dimensional state spaces, both challenging aspects of de-novo design in drug
discovery.

##  1 Introduction

One of the greatest challenges in modern medicine currently lies in the
discovery and development of novel therapeutics for disease treatment. This
challenge is most evident in the field of infectious diseases, where the
creation of new antibiotics has been challenging due to substantial research
costs, lengthy timelines, and limited returns. In recent years, a promising
alternative approach has emerged in the form of Generative Flow Networks
(GFlowNets) [2, 3]. GFlowNets tackle the sampling problem by learning to
sample trajectories approximately proportionally to their quality, as captured
by a reward function. This approach encourages the discovery of a diverse set
of high-reward samples, offering the potential to significantly accelerate the
drug discovery and development process.

Despite existing research on credit assignment for GFlowNets, a central
challenge continues to be the improvement of exploration and exploitation
within the GFlowNets framework [8, 7, 11, 4]. In environments like small
molecule generation, where the rewards are sparse [2], GFlowNets may encounter
difficulties in breaking away from the current best mode, thus reducing the
chances for the agent to discover new modes in the environment [10, 3].
Therefore, it is important to find new ways to enhance exploration efficiency
in sparse-reward domains.

In this work, we take inspiration from the double deep Q-learning (DDQN)
algorithm from reinforcement learning [14], and introduce double GFlowNets
(DGFNs). This approach simply involves employing a target network, which acts
as the delayed version of the online network and from which we generate
trajectories. Intuitively, this prevents the data distribution on which the
online model is trained to become too peaked, too “opinionated", too quickly.
We apply DGFNs to two standard GFN tasks: hypergrid (where the complexity and
sparsity can be controlled well) and small molecule generation (which is more
illustrative of real applications) [2]. Our empirical findings demonstrate
that DGFNs finds all modes faster in hypegrid, and uncover a greater number of
high-reward modes in the fragment-based molecular design task. These
observations provide strong evidence that our proposed strategy indeed
encourages diverse exploration, promoting better coverage of the state space
and the discovery of diverse candidate solutions.

##  2 Preliminaries

We begin by introducing GFlowNets, following previous work [2, 8]. Consider a
directed acyclic graph $G=(S,A)$, where each vertex $s\in S$ represents a
state and $s\to s^{\prime}\in A$ a state transition. Notably, $s_{0}$ is the
initial state, with no incoming edges, while $s_{f}$ is the sink state, with
no outgoing edges. A state $s_{n}$ is considered terminal if $s_{n}\to
s_{f}\in A$. Each state is assumed to represent some object
$s_{n}=x\in\mathcal{X}$. We sample such objects by sampling trajectories
starting from $s_{0}$, and following “actions" $a$ drawn from $A$. This yields
a sequence $\tau=(s_{0}\rightarrow s_{1}\rightarrow\ldots s_{n}\rightarrow
s_{f})$, referred to as a complete trajectory. Let $\mathcal{T}$ be the set of
all possible trajectories. Assuming a probability distribution over the edges
from each node, let $F(\tau)$ denote the flow of $\tau$, representing its
unnormalized probability. The edge flow is defined as $F\left(s\rightarrow
s^{\prime}\right)=\sum_{\tau\in\mathcal{T}:(s\rightarrow
s^{\prime})\in\tau}F(\tau)$. The state flow is defined as
$F(s)=\sum_{\tau\in\mathcal{T}:s\in\tau}F(\tau)$. Using these concepts, we can
define forward and backward policies, $P_{F}$ and $P_{B}$, as follows::

$P_{F}(s\to s^{\prime})=\frac{F(s\to
s^{\prime})}{F(s)},\;\;P_{B}(s^{\prime}\to s)=\frac{F(s\to
s^{\prime})}{F(s^{\prime})}$ (1)

Given a non-negative reward function $R:\mathcal{X}\rightarrow\mathbb{R}_{\geq
0}$, let the terminal edge flows be $F(s_{n}\to s_{f})=R(x=s_{n})$. The
primary objective of GFlowNets is to train a generative policy such that the
likelihood of sampling $x\in\mathcal{X}$ is proportional to $R(x)$ , where

$p(x)=\sum_{\tau=\left(s_{0}\rightarrow\cdots\rightarrow
s_{n}=x\right)}P_{F}(\tau)=\sum_{\tau}\prod_{t=1}^{n}P_{F}\left(s_{t}\mid
s_{t-1}\right)$ (2)

This is achieved by _balancing_ flows such that the total quantity of flow is
preserved. [2] In particular, this can be expressed through the trajectory
balance (TB) [8] condition, where for all trajectories $\tau$:

$Z\prod_{t=1}^{n}P_{F}\left(s_{t}\mid
s_{t-1}\right)=R(x)\prod_{t=1}^{n}P_{B}\left(s_{t-1}\mid s_{t}\right)$ (3)

Here, $Z$ represents the total flow, i.e. $Z=\sum_{x\in\mathcal{X}}R(x)$. This
equality can be turned into an objective and used to learn parameterized
$P_{F}$, $P_{B}$, and $Z$ [8, 7]. Satisfying this constraint across all
complete trajectories ensures that $P_{F}(x)\propto R(x)$. Throughout our
experiments, we adopt trajectory balance as the primary training objective.

###  2.1 Related Work

Double Deep Q-Learning (DDQN) In reinforcement learning, it is well-known that
the Q-learning algorithm for learning optimal policies suffers from
overestimation bias [15, 5]. This overestimation leads to collapsing the
exploration too quickly, slowing down the learning process. Double Deep
Q-Networks (DDQN) were introduced to mitigate this issue, by decoupling action
selection and value estimation [15]. DDQN uses two separate Q-networks: a
target network and an online network. The target network is used to compute
the Q-learning target, which is used to update the weights of the online value
network. The latter is employed for action selection. The target network is
updated periodically by copying the weights of the online network. DDQN has
been shown empirically to provide more stable Q-value estimates, thereby
enhancing performance.

Improving GFlowNets A number of works have delved into enhancing the training
process by manipulating the sampling distribution. For example, Thompson
Sampling GFlowNets [11], improve exploration by maintaining uncertainty
through ensembling and using it within Thompson sampling. Replay buffers
coupled with reward-prioritized replay sampling [12, 16] have been shown to
enhance GFlowNet training dynamics. However, existing work only considers
using single networks to generate trajectories.

##  3 Double Generative Flow Networks (DGFN)

In environments characterized by large state spaces or sparse reward signals,
the standard approach to GFlowNet training can induce training instability
and/or lead to mode collapse [12]. To address this limitation and promote
exploration, we draw inspiration from the DDQN idea. We introduce a target
network, whose primary role is to sample trajectories, on which the the online
network’s loss is computed. In our experiments, this approach safeguards
against over-optimization of the GFlowNet, particularly in environments
characterized by sparse rewards, where the agent may be pushed to exploit
known modes if it is able to learn about them quickly, thereby exploring too
little. The target network dampens this problem. The target network is
periodically copied from the online network, but this process is a bit
different from standard DDQN. Because the initial sampled trajectories tend to
be relatively random and low reward, in the initial stages of training, we
update the target network more frequently. Subsequently, we transition to
periodic updates employing a Polyak averaging technique. A full description of
the approach is given in Algorithm 1.

Input: Initial phase length $T^{I}$, update period $T^{U}$

Initialize online flow network $F_{\theta}$, target flow network
$F_{\theta^{\prime}}$, $\alpha<1$

for  _each training step $t=1$ to $T$_ do

Sample a batch of $M$ trajectories
$\tau=\left\\{s_{0}\rightarrow\cdots\rightarrow s_{n}\right\\}$ from
$F_{\theta^{\prime}}$

Compute loss of the online network using sampled trajectories

if  _$t <T^{I}\textup{ or }t\bmod T^{U}\equiv 0$_ then

$\theta^{\prime}\leftarrow\alpha\theta+(1-\alpha)\theta^{\prime}$

end for

Algorithm 1 Double GFlowNets (DGFNs)

##  4 Experiments

We study the performance of the proposed DGFNs in comparison to conventional
GFlowNets with different objective functions on two benchmark tasks: hypergrid
and molecule generation.

###  4.1 Hypergrid Environment: High Dimensional, Sparse Rewards

In the synthetic hypergrid environment introduced by Bengio et al. [2], the
goal is to sample trajectories in a $D$-dimensional grid-world with side
length $H$. The initial state is $(0,...,0)$, and actions increment a
coordinate by 1 within the bounds of $D$ and $H$. The agent can terminate at
any state.

We use a 6-dimensional grid with side lengths $H=8,10,12$. The reward function
is defined as in [2, 8]:
$R(x)=R_{0}+R_{1}\prod_{i}\mathbb{I}\left(0.25<\left|x_{i}/H-0.5\right|\right)+R_{2}\prod_{i}\mathbb{I}\left(0.3<\left|x_{i}/H-0.5\right|<0.4\right)$
with $0<R_{0}\ll R_{1}<R_{2}$, $R_{1}=1/2,R_{2}=2$. We opt for a more
challenging environment by setting $R_{0}=10^{-3}$, making exploration less
rewarding for the agent. We measure the $L_{1}$ error between the true reward
distribution and the empirical distribution over the sampled terminal states.
Additionally, we track the number of modes discovered over the sampled
terminal states.

Figure 1: Left: Percentage of modes discovered over the trajectories sampled.
Right: $L_{1}$ distance between empirical and target distribution over the
trajectories sampled.

The results are shown in Fig. 1 and appendix A.1, with means and standard
errors computed over 5 independent runs on $DGFN_{TB}$, $DGFN_{SubTB}$, and
baseline models: $GFN_{TB}$ and $GFN_{SubTB}$. “TB" refers to the objective
mentioned in equation 3, while “SubTB" is a recent objective proposed by Madan
et al. [7] to learn from partial trajectories rather than complete
trajectories. As hypothesized, the DGFNs find modes across the hypergrid
faster than conventional GFlowNet methods. In particular, the gap becomes
wider in more complex environments.

###  4.2 Small molecule synthesis

To assess the capabilities of DGFNs for drug discovery, we conduct experiments
involving the generation of small molecules. Specifically, this task involves
creating molecules with low binding energy to the soluble epoxide hydrolase
(sEH) protein, employing a docking prediction approach originally introduced
by Trott and Olson [13]. The agent is tasked with a sequential decision-making
process, determining attachment points for molecular building blocks, while
adhering to the constraints of chemical validity. This task is particularly
challenging, primarily due to the vast state space, estimated at around
$10^{16}$ distinct states, and number of available actions at each state,
which can range from $100$ to $2000$. The reward function $R$ relies on a pre-
trained proxy model developed by Bengio et al. [2]. For implementation
details, please refer to Appendix A.1.

We train three models: $DGFN_{TB}$ and two baseline models, $GFN_{TB}$ and
$GFN_{SubTB}$. To ensure the reliability of our findings, we report reward
means and standard error over 5 independent random seeds. We calculate the
number of modes with rewards greater than a $0.9$ threshold. Fig. 2 shows that
$GFN_{TB}$ exhibits more pronounced fluctuations during the training process,
whereas $DGFN_{TB}$ has lower variance. Additionally, $DGFN_{TB}$ surpasses
$GFN_{SubTB}$ in the discovery of modes with rewards exceeding $0.9$.

We also compute the diverse Top-K (i.e. the set of top molecules such that
their pairwise Tanimoto similarity is at most 0.7, c.f. [2]) and Top-K Reward
for the generated molecules, showed in Table 1. Here, $DGFN_{TB}$ matches the
baselines. This shows that $DGFN_{TB}$ is able to find a greater diversity of
solutions without sacrificing reward. Appendix A.2 provides examples of the
top 12 molecules generated by our model.

Table 1: Results on the molecule synthesis task. Mean and standard error over 3 runs. Algorithm | Diverse Top-100 | Diverse Top-1000 | Top-100 Reward | Top-1000 Reward
---|---|---|---|---
DGFN-TB | 1.035$\pm$0.002 | 1.017$\pm$0.002 | 1.036$\pm$0.002 | 1.020$\pm$0.002
GFN-TB | 0.972$\pm$0.028 | 0.836$\pm$0.097 | 0.982$\pm$0.024 | 0.931$\pm$0.044
GFN-SubTB | 1.017$\pm$0.001 | 0.992$\pm$0.002 | 1.017$\pm$0.001 | 0.996$\pm$0.002
Figure 2: Left: Average reward as a function of trajectories sampled. Right:
Number of modes with rewards $R>0.90$ as a function of trajectories sampled.

##  5 Discussion and Conclusion

In this work, we introduced the concept of Double Generative Flow Networks
(DGFNs), in order to improve the training stability and the ability of
GFlowNets to explore well in large state spaces with sparse rewards. This
issue is especially important in drug discovery, where sampling molecules
demands a more robust exploration strategy. Our empirical results in hypergrid
and molecule synthesis tasks demonstrate the effectiveness of DGFN in
promoting diversity in sample generation and enhancing stability. More work
remains: more extensive testing of DGFNs across different tasks, the
development of a more comprehensive theoretical framework for this approach,
and ultimately, the exploration of more techniques inspired by RL and
generative modeling to improve the stability of GFlowNets.

## Acknowledgements

We genuinely appreciate the funding support from Fonds Recherche Quebec
through the FACS-Acquity grant and the National Research Council of Canada.
Mila has been instrumental in providing the computational resources for this
project. We also want to acknowledge Jarrid Rector Brooks and Moksh Jain for
their valuable discussions on related work. This work is partially done at
Valence Labs.

## References

  * [1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A Next-generation Hyperparameter Optimization Framework.  arXiv e-prints, page arXiv:1907.10902, July 2019.
  * [2] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio.  Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation.  arXiv e-prints, page arXiv:2106.04399, June 2021.
  * [3] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio.  GFlowNet Foundations.  arXiv e-prints, page arXiv:2111.09266, November 2021.
  * [4] Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio.  Bayesian Structure Learning with Generative Flow Networks.  arXiv e-prints, page arXiv:2202.13903, February 2022.
  * [5] Hado Hasselt.  Double q-learning.  In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
  * [6] Greg Landrum.  Rdkit: Open-source cheminformatics. 2006.  Google Scholar, 2006.
  * [7] Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin.  Learning GFlowNets from partial episodes for improved convergence and stability.  arXiv e-prints, page arXiv:2209.12782, September 2022.
  * [8] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio.  Trajectory balance: Improved credit assignment in GFlowNets.  arXiv e-prints, page arXiv:2201.13259, January 2022.
  * [9] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A Distributed Framework for Emerging AI Applications.  arXiv e-prints, page arXiv:1712.05889, December 2017.
  * [10] Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio.  Generative Augmented Flow Networks.  arXiv e-prints, page arXiv:2210.03308, October 2022.
  * [11] Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio.  Thompson sampling for improved exploration in GFlowNets.  arXiv e-prints, page arXiv:2306.17693, June 2023.
  * [12] Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani.  Towards Understanding and Improving GFlowNet Training.  arXiv e-prints, page arXiv:2305.07170, May 2023.
  * [13] Oleg Trott and Olson Arthur.  AutoDock Vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization and multithreading.  Inc. J Comput Chem 2010, January 2010.
  * [14] Hado van Hasselt, Arthur Guez, and David Silver.  Deep Reinforcement Learning with Double Q-learning.  arXiv e-prints, page arXiv:1509.06461, September 2015.
  * [15] Hado van Hasselt, Arthur Guez, and David Silver.  Deep Reinforcement Learning with Double Q-learning.  arXiv e-prints, page arXiv:1509.06461, September 2015.
  * [16] Nikhil Vemgal, Elaine Lau, and Doina Precup.  An Empirical Study of the Effectiveness of Using a Replay Buffer on Mode Discovery in GFlowNets.  arXiv e-prints, page arXiv:2307.07674, July 2023.

##  Appendix A Appendix

###  A.1 Experiment details: Hypergrid

The model architecture for both the forward and backward policies remains
consistent with the original GFlowNets models [8, 7], using Adam as the
optimizer. All models were trained using a batch size of 64 for a total of
640,000 trajectories. Hyperparameter tuning was conducted via Optuna [1],
which automates hyperparameter optimization with Ray Tune [9]. For dimension
6, side length 10, the optimal hyperparameters for $DGFN_{TB}$ were found to
be an initial phase length of $T^{I}=698$ and an update period of $T^{U}=137$.
For $DGFN_{SubTB}$, the optimal settings were an initial phase length of
$T^{I}=794$ and an update period of $T^{U}=149$.

Figure 3: Left: Average reward as a function of trajectories sampled. Right:
Number of modes with rewards R > 0.90 as a function of trajectories sampled.

###  A.2 Experiment details: Small Molecule Synthesis

In our experiments, we used the dataset and proxy model provided by Bengio et
al. [2]. The model architecture has the same implementation as the GFlowNet’s
trajectory balance [8]. Additionally, we incorporated the AutoDock Vina
library [13] for binding energy estimation and relied on the RDKit library [6]
for chemistry routines.

For the experimental setup, we trained the models for a total of 10,000
iterations using a batch size of 64. The temperature coefficient for the
reward function, denoted as $\beta$ [2], was set to 96. To determine the
optimal initial phase length $T^{I}$ and update period $T^{U}$, we conducted a
grid search over the following values:
$T^{I}\in\\{500,1000,1500,2000,2500,3000\\}$ and
$T^{U}\in\\{50,100,150,200,250,300,350\\}$. Our experiments revealed that
$T^{I}$ of 2000 and $T^{U}$ of 200 produced the best results.

Figure 4: Top-12 molecules generated by $DGFN_{TB}$