# Proposal for a long-lived quantum memory using matter-wave optics with Bose-
Einstein condensates in microgravity

Elisa Da Ros§ Institut für Physik and IRIS, Humboldt-Universität zu Berlin,
Newtonstr. 15, Berlin 12489, Germany  Simon Kanthak§ Institut für Physik and
IRIS, Humboldt-Universität zu Berlin, Newtonstr. 15, Berlin 12489, Germany
Erhan Sağlamyürek  Currently at Lawrence Berkeley National Lab, Berkeley,
California, 94720, USA and Department of Physics, University of California,
Berkeley, California, 94720, USA Department of Physics and Astronomy,
University of Calgary, Calgary, Alberta T2N 1N4, Canada  Department of
Physics, University of Alberta, Edmonton, Alberta T6G 2E1, Canada  Mustafa
Gündoğan<EMAIL_ADDRESS>Institut für Physik and IRIS,
Humboldt-Universität zu Berlin, Newtonstr. 15, Berlin 12489, Germany  Markus
Krutzik<EMAIL_ADDRESS>Institut für Physik and IRIS,
Humboldt-Universität zu Berlin, Newtonstr. 15, Berlin 12489, Germany
Ferdinand-Braun-Institut (FBH), Gustav-Kirchoff-Str.4, 12489 Berlin

###### Abstract

Bose-Einstein condensates are a promising platform for optical quantum
memories, but suffer from several decoherence mechanisms, leading to short
memory lifetimes. While some of these decoherence effects can be mitigated by
conventional methods, density dependent atom-atom collisions ultimately set
the upper limit of quantum memory lifetime to s-timescales in trapped Bose-
Einstein condensates. We propose a new quantum memory technique that utilizes
microgravity as a resource to minimize such density-dependent effects. We show
that by using optical atom lenses to collimate and refocus the freely
expanding atomic ensembles, in an ideal environment, the expected memory
lifetime is only limited by the quality of the background vacuum. We
anticipate that this method can be experimentally demonstrated in Earth-bound
microgravity platforms or space missions, eventually leading to storage times
of minutes and unprecedented time-bandwidth products of $10^{10}$.

BEC, quantum memory, quantum information, atomic physics

Optical quantum memories (QMs) are devices that can faithfully and reversibly
store and recall the quantum states of light. They are required in many
applications in quantum information science such as long-distance quantum
communications Sangouard  _et al._ (2011); Heshami  _et al._ (2016),
deterministic generation of multiphoton states Nunn  _et al._ (2013) and
quantum computation Gouzien and Sangouard (2021). A recent idea is to deploy
QMs in space in order to enable globe-spanning quantum networks Gündoğan  _et
al._ (2021); Liorni  _et al._ (2021); Sidhu  _et al._ (2021); Wallnöfer  _et
al._ (2022), ultra-long baseline Bell experiments Cao  _et al._ (2018);
Mohageg  _et al._ (2021); Gündoğan  _et al._ (2021); Lu  _et al._ (2022) and
probing the interplay between gravity and quantum physics Barzel  _et al._
(2022) for which a storage time, $\tau_{\mathrm{mem}}$, of around
$\sim$$1\text{\,}\mathrm{s}$ is needed. Several atomic systems have been
proven useful for such reversible mapping between light and matter qubits.
These include single defects in diamond Fuchs  _et al._ (2011); Sukachev  _et
al._ (2017); Bradley  _et al._ (2019), rare-earth ion doped crystals Ledingham
_et al._ (2012); Jobez  _et al._ (2015); Gündoğan  _et al._ (2015), trapped
ions Langer  _et al._ (2005); Harty  _et al._ (2014); Wang  _et al._ (2021),
single trapped atoms Specht  _et al._ (2011); Langenfeld  _et al._ (2021), and
warm Cho  _et al._ (2016); Wolters  _et al._ (2017); Katz and Firstenberg
(2018); Kaczmarek  _et al._ (2018) and cold Riedl  _et al._ (2012); Wang  _et
al._ (2019); Bao  _et al._ (2012); Heller  _et al._ (2020); Saglamyurek  _et
al._ (2021) atomic gases. Among these platforms, cold-atomic gases have
recently been deployed in space for a number of experiments: optical atomic
clocks Liu  _et al._ (2018); the first Bose-Einstein condensate (BEC) on board
a sounding rocket Becker  _et al._ (2018) and the International Space Station
(ISS) Aveline  _et al._ (2020). In addition to these, missions using cold
atoms in space are being envisioned Alonso  _et al._ (2022) for gravity and
dark matter exploration Tino  _et al._ (2019); El-Neaj  _et al._ (2020), and
currently in development for ultra-cold atom research including atom
interferometry Frye  _et al._ (2021) and advanced atomic clocks on board the
ISS Cacciapuoti  _et al._ (2020). Cold atom based QMs would share the same
technical infrastructure with these experiments.

Figure 1: Protocol for a long-lived quantum memory utilizing interaction-
driven expansion and delta-kick collimation (DKC) of a Bose-Einstein
condensate (BEC) in microgravity. a) $\Lambda$-type three-level structure
together with the employed light fields during b) different stages of the size
evolution of the BEC. The quantum state of single photon pulses is imprinted
into an internal excitation of a BEC shortly after its release from an optical
dipole trap (ODT). Brief exposure of the BEC by two consecutive optical
lensing potentials allows to stop and subsequently revert the interaction-
driven expansion via DKC. This protocol allows for transition between the
complementary density regimes needed for an efficient write-in and read-out of
the memory at high optical depths (ODs) and large coherence times for long-
time storage at low atomic densities, respectively.

A BEC platform has unique advantages over cold atoms (obeying a thermal
distribution) for optical QMs due to the inhibition of thermal motion
(allowing long memory lifetime) and its high atomic density (leading to
efficient operation). However, condensates are still affected by several
decoherence mechanisms. Among these, decoherence due to magnetic field
inhomogeneities Riedl  _et al._ (2012); Saglamyurek  _et al._ (2021) can be
mitigated by employing rephasing protocols based on dynamical decoupling Dudin
_et al._ (2013) and those caused by AC Stark shifts that are due to
inhomogeneous optical trapping beams can be prevented by employing magic
wavelength techniques Lundblad  _et al._ (2010); Dudin  _et al._ (2010). On
the other hand, losses due to atom-atom collisions are usually not reversible,
and cannot be mitigated by such measures. The collisions of cold atoms with
the background gas (i.e. 1-body collisions) can be controlled only with the
vacuum quality, while the collision rates between 2 or 3 atoms within the cold
ensemble (i.e. 2-body and 3-body collision) increase with increasing atom
density. These processes become relevant beyond storage times of
$\sim$1\text{\,}\mathrm{m}\mathrm{s}$$. However, a maximum storage time of
around $\sim$1\text{\,}\mathrm{s}$$ has been observed with bright pulses in a
Sodium BEC by tuning the atom-atom collision cross sections via external
magnetic fields Zhang  _et al._ (2009).

In this work, we propose a novel quantum storage scheme that exploits matter-
wave optics to tune the density of the atomic ensemble to minimize the effects
of density-dependent collisions. This is achieved by letting the condensate
expand after writing the quantum state of incoming photons into an internal
state of the atoms in the condensate, which is followed by employing the delta
kick collimation (DKC) technique Ammann and Christensen (1997); Myrskog  _et
al._ (2000); Kovachy  _et al._ (2015); Deppner  _et al._ (2021), first to
collimate and then to refocus the BEC for efficient read-out of the stored
excitation. This protocol is carried out in a microgravity environment, which
prevents the fall of the centre of mass without the need for any types of
inhomogeneous field to levitate the atoms. We show that this technique would
allow storage times that are orders of magnitude beyond what is possible in
ground-based experiments and, in fact, only limited by the quality of the
background vacuum. We expect our protocol to reach a few minutes of storage
time with the state-of-the-art background vacuum values Hogan  _et al._
(2011); Nirrengarten  _et al._ (2006).

We assume a pure BEC initially trapped in an optical dipole trap (ODT), as
illustrated in Fig. 1. To circumvent decoherence due to AC-Stark shifts, the
quantum state of single photon pulses is imprinted within the BEC only shortly
after its release from the trap. Timing of the write pulse is set to mode-
match the light intensity and atomic density distributions with negligible
reduction in optical depth (OD). During free expansion, the internal energy is
converted into kinetic energy, yielding a reduction in the density and
therefore in the 2-body collisions. After a set time $T_{0}$, the BEC is
exposed to a tailored, optical potential for a short duration of
$\tau_{\text{DKC}}$. This way, the BEC experiences a delta-kick, which acts as
an optical atom lens Kovachy  _et al._ (2015); Luan  _et al._ (2018); Kanthak
_et al._ (2021); Gochnauer  _et al._ (2021), resulting in a narrow momentum
distribution. After a chosen collimation time $T_{\text{C}}$, a second DKC
pulse is applied to refocus the ensemble. At this point
($T_{\text{C}}+2\,T_{0}$) it is possible to faithfully recall the stored
quantum information at the original higher OD. Our protocol thus allows to
transition between the complementary density regimes needed for an efficient
write-in and read-out at high ODs and coherent storage in a dilute quantum gas
by exploiting the mean-field driven expansion of a self-interacting BEC. Given
by the point-like source characteristics and single-mode properties of the
BEC, the dispersion of the ensemble can be shaped after release from the
trapping potential by DKC to nearly stop and finally revert the expansion.

The quantum memory itself is based on a $\Lambda$-type three-level system as
represented in Fig. 1. The states $\ket{g}$ and $\ket{s}$ represent the ground
states of the hyperfine structure of the 87Rb D1 line,
$\ket{5^{2}S_{1/2},F=1}$ and $\ket{5^{2}S_{1/2},F=2}$, respectively, while
$\ket{e}$ is the excited state $\ket{5^{2}P_{1/2},F=1}$. Collinear probe and
control beams address the $\ket{g}\longleftrightarrow\ket{e}$ and
$\ket{s}\longleftrightarrow\ket{e}$ transitions, respectively. The use of
collinear beams Namazi  _et al._ (2017) ensures both the optimal spin storage
and phase-matching condition which in turn eliminates decoherence due to
recoil collisions Saglamyurek  _et al._ (2021).

Although ensemble-based memories generally follow similar considerations
Gorshkov  _et al._ (2007), we choose to incorporate the Autler–Townes
splitting (ATS) method Saglamyurek  _et al._ (2018, 2021) into our approach as
it requires lower OD and control power for efficient storage of broadband
pulses compared to other memory protocols implemented in cold-atom systems,
such as electromagnetically induced transparency Geng  _et al._ (2014), which
makes it more attractive for applications in quantum information science.
Furthermore, lower requisites on these properties make the ATS protocol more
robust against four-wave mixing noise Saglamyurek  _et al._ (2021), which is
another important feature for practical applications.

We predict the dynamics of the BEC through a variational ansatz to numerically
solve the time-dependent Gross–Pitaevskii equation

$\text{i}\hbar\frac{\partial}{\partial t}\psi(\vec{r},t)=\\\
\left[-\frac{\hbar^{2}}{2m}\nabla^{2}+V_{\text{DKC}}(\vec{r},t)+U_{0}\rho(\vec{r},t)\right]\psi(\vec{r},t)$
(1)

following a scaling approach Pérez-García  _et al._ (1997, 1996), where
$V_{\text{DKC}}(\vec{r},t)$ represents the lensing potential,
$U_{0}=4\pi\hbar^{2}\text{Re}(a_{\text{sc}})N_{0}/m$ characterizes the
interaction and is defined by the real part of the s-wave scattering length
$a_{\text{sc}}$ Harber  _et al._ (2002) for a ground state BEC with $N_{0}$
atoms of mass $m$.

As trial functions, we simply utilize a Gaussian ansatz for the atomic
wavefunction $\psi(\vec{r},t)$ with corresponding spatial density distribution
$\rho(t)$ expressed as

$\rho(t)=\left|\psi(t)\right|^{2}=\underbrace{\frac{N_{0}}{(2\pi)^{3}}\prod_{\xi\in\\{x,y,z\\}}\sigma_{\xi}^{-1}(t)}_{\text{\normalsize$=\rho_{0}(t)$}}\exp\left(-\frac{\xi^{2}}{2\sigma_{\xi}^{2}(t)}\right),$
(2)

where $\rho_{0}(t)$ represents the peak density. The standard deviation
$\sigma_{\xi}(t)$ of the atomic density can then be related to the Thomas-
Fermi radius $\sigma_{\xi}=R_{\xi}/\sqrt{7}$, knowing the parabolic shape of a
BEC Corgier  _et al._ (2020).

The lensing potential is well described by the harmonic approximation
$V_{\text{DKC}}(\vec{r},t)=1/2m\sum_{\xi}\omega_{\xi}^{2}\xi^{2}$ if the BEC
is located close to the center of the optical trap and if the characteristic
size of the generating beams is $w_{0}\gg\sigma_{\xi}$. Any anharmonicity of
the lensing potential would cause lens aberrations and will ultimately limit
the achievable storage times and efficiency of the information read-out due to
the attainable collimation times Deppner  _et al._ (2021) and minimum sizes
during refocusing Kovachy  _et al._ (2015), respectively.

For our case study, we initialize an isotropic BEC of size
$\sigma=$3\text{\,}\mathrm{\SIUnitSymbolMicro m}$$ with $N_{0}=1\times 10^{5}$
87Rb atoms. After $T_{0}=$1\text{\,}\mathrm{s}$$ of free expansion, we apply a
DKC potential with trap frequency $\omega=2\pi\cdot$2.25\text{\,}\mathrm{Hz}$$
for a symmetrically centered box pulse of duration
$\tau_{\text{DKC}}=$5\text{\,}\mathrm{ms}$$. After further evolution for
$T_{\text{C}}$, the BEC is exposed again to the same DKC potential. The trap
frequency is chosen to mode-match the final and initial wavefunction. The
total storage time of the memory in this protocol is then approximately
$\tau_{\text{mem}}\approx T_{\text{C}}+2\,T_{0}$.

In general, illumination of the ensemble with an inhomogeneous light field, as
needed for optical DKC, can affect the system coherence. Assuming a crossed
beam ODT at $1064\text{\,}\mathrm{nm}$ as origin of the lensing potential, we
calculate the differental AC-Stark shift of the two ground states of 87Rb to
be $\delta\nu_{\text{AC}}\ll\tau_{\text{DKC}}^{-1}$, granting a negligible
decoherence during the DKC pulses.

Figure 2: BEC dynamics and intrinsic memory efficiency. Time evolution of a)
the BEC size $\sigma$, b) corresponding decay rates $\gamma_{\text{2-body}}$
and its effect on c) the intrinsic memory efficiency (i.e., not including
write-in and read-out efficiencies) due to only 2-body (dashed lines) and
combination of 2- and 1-body collisions (solid lines). We compare our memory
protocol with storage times $\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$
(blue) and $\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$ (orange)
together with the trapped case (yellow). The inlets show a zoom onto the
respective changes during the focus of the BEC for the case
$\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$, in linear scaling. The black
vertical lines indicate the timing of the DKC pulses to collimate and refocus
the BEC for the case with $\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$.

Assuming that the untrapped ensemble is shielded from any magnetic field
inhomogenities, the overall efficiency of photon retrieval can then be
expressed as a function of the atomic density distribution and peak density as

$\eta_{\text{tot}}(t)=\eta_{\text{DKC}}^{2}\cdot\eta_{\text{1}}(t)\cdot\eta_{\text{2}}(\rho_{0}(t))\cdot\eta_{\text{ATS}}(\rho(t)),$
(3)

where $\eta_{\text{DKC}}$ indicates the efficiency of the DKC procedure,
$\eta_{\text{1}}(t)$ and $\eta_{\text{2}}(\rho_{0}(t))$ are associated with
the 1-body and 2-body collisions, respectively, and finally the factor
$\eta_{\text{ATS}}(\rho(t))$ represents the combined efficiencies of the ATS
write-in and read-out steps.

For the purpose of this work, $\eta_{\text{DKC}}$ is set to 1. This
corresponds to the assumption of ideal harmonic potentials for the
implementation of the DKC procedures, and thus neglects possible lens
aberrations that would affect the atom distribution Kovachy  _et al._ (2015).
This ideal case might be accomplished in experiments utilizing time-averaged
optical potentials Albers  _et al._ (2022).

The efficiency associated with the atom losses due to the collisions with the
background gas within the vacuum chamber, $\eta_{\text{1}}(t)$, follows an
exponential decay. In Nirrengarten  _et al._ (2006), Nirrengarten et al.
achieve a lifetime of about $115\text{\,}\mathrm{s}$ associated with a
background pressure of $3\text{\times}{10}^{-11}\text{\,}\mathrm{m}\bar{}$. We
set this value as the 1-body collision lifetime $\tau_{1}$ used within this
work. The term $\eta_{\text{2}}(\rho_{0}(t))$ expresses the exponential decay
of the memory efficiency due to two-body collisions between cold Rb atoms. For
the states and densities taken into account in this work, the contribution of
the 3-body collisions is negligible compared to the other collisional losses
Burt  _et al._ (1997).

Figure 3: Intrinsic memory efficiency for different initial BEC sizes
$\sigma_{0}$ and common initial peak density
$\rho_{\mathrm{0}}(\text{0})=$2.3\text{\times}{10}^{14}\text{\,}{\mathrm{cm}}^{-3}$$.
The black lines indicate the timing of the DKC pulses to collimate and refocus
the BEC. The inlet shows the evolution of the 2-body decay rate shortly after
release and during refocusing, in linear scaling.

Figure 2 (a) shows the comparison of the time evolution of the condensate size
for two different collimation times (blue and orange lines) with the scenario
in which the BEC is not released from the trap at all (yellow line). The color
code is the same for the other panels in the figure. In both expanding cases,
the condensate peak density drops from its initial value
$\rho_{\mathrm{0}}(\text{0})=$2.3\text{\times}{10}^{14}\text{\,}{\mathrm{cm}}^{-3}$$
to
$\rho_{\mathrm{0}}(T_{\text{0}}=$1\text{\,}\mathrm{s}$)=$7\text{\times}{10}^{6}\text{\,}{\mathrm{cm}}^{-3}$$
at the time of the first DKC pulse. After a duration of
$T_{\text{C}}=$3\text{\,}\mathrm{s}$$ (blue) or
$T_{\text{C}}=$98\text{\,}\mathrm{s}$$ (orange), the second DKC pulse is
applied, which causes the BEC to refocus and reach again its minimum size at
$\tau_{\text{mem}}\approx T_{\text{C}}+2\,T_{\text{0}}$, when the memory is
read out. The corresponding storage times are
$\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$ (blue) or
$\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$ (orange). Upon refocusing,
the initial density of the ensemble is fully recovered and, subsequently, the
BEC keeps expanding again (as shown by the blue line after
$5\text{\,}\mathrm{s}$). The inset shows the detailed evolution of the BEC
size around the focus for the case with $T_{\text{C}}=$3\text{\,}\mathrm{s}$$.

The time evolutions of peak density corresponding to the simulated sizes of
Fig. 2 (a) allow to calculate the time-dependent 2-body collisions decay rate
$\gamma_{\text{2-body}}(t)$ as Egorov (2012); Saglamyurek  _et al._ (2021)

$\gamma_{\text{2-body}}(t)=\frac{4~{}h\text{Im}(a_{\text{sc}})\rho_{0}(t)}{m},$
(4)

where $\text{Im}(a_{\text{sc}})$ is the imaginary part of the s-wave
scattering length. Figure 2 (b) shows a reduction of more than 7 orders of
magnitude in the 2-body decay rate during $T_{0}$, compared to the trapped
case with constant peak density. The inset highlights again the behaviour
around the focus for the case with $T_{\text{C}}=$3\text{\,}\mathrm{s}$$.

With this time-dependent decay rate, the intrinsic efficiency due to 2-body
collisions is found upon integration over time as

$\eta_{\text{2}}(\rho_{0}(t))=\exp\left(-\int_{0}^{t}\frac{\rho_{0}(t^{\prime})}{\kappa}\mathrm{d}t^{\prime}\right)$
(5)

where $\kappa=m/\left(4h\text{Im}(a_{\text{sc}})\right)$.

The intrinsic memory efficiency,
$\eta_{\text{1}}(t)\cdot~{}\eta_{\text{2}}(\rho_{0}(t))$, is plotted in Fig. 2
(c). The solid lines include the effects of 1- and 2-body collisions whereas
the dashed lines show only the effects of 2-body collisions. It is evident
from the figure, that the constant high density of the trapped case yields a
lifetime limit to around $100\text{\,}\mathrm{ms}$ by 2-body collisions. On
the contrary, the lifetime can be extended for the expanding and refocused
cases, proving that the memory decay rate can be tuned by varying the BEC
density via DKC pulses. The spike that occurs for $\gamma_{\text{2-body}}$
after the second DKC pulse (see Fig. 2 (b)), due to the increased density
during refocusing, results in a sudden drop in intrinsic efficiency, as seen
in the inset of Fig. 2 (c). Nonetheless, this drop is not particularly
detrimental: from 0.95 to 0.86. The inset focuses on the comparison between
the effects of 1- and 2-body collisions,
$\eta_{\text{1}}\cdot~{}\eta_{\text{2}}$, and the single 2-body contribution,
$\eta_{\text{2}}$, around the focus, for the case
$\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$. It highlights how the main
loss contribution on the short timescales is given by 2-body collisions (e.g.
the contribution of $\gamma_{\text{1-body}}$ is limited to about $4\%$). The
1-body collisions become, instead, the dominant loss mechanism at long storage
times. For instance for $\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$,
the intrinsic efficiency due to $\gamma_{\text{2-body}}(t)$ is still around
0.90, whereas the collisions with the background gas brings it down to 0.38.

The initial atom density distribution of the BEC determines the expansion rate
of the ensemble, which consequently affects the achievable intrinsic memory
efficiency. Figure 3 shows the trends of the intrinsic memory efficiency for a
set of three isotropic BECs with different initial sizes $\sigma_{0}$. All
cases have, though, a common initial peak density
$\rho_{\mathrm{0}}(\text{0})=$2.3\text{\times}{10}^{14}\text{\,}{\mathrm{cm}}^{-3}$$
to grant a common initial $\gamma_{\text{2-body}}(0)$. The ensembles expand
for $1\text{\,}\mathrm{s}$, until collimation, and are then refocused after
$T_{\text{C}}=$3\text{\,}\mathrm{s}$$. For smaller initial size, the ensemble
expands faster, yielding lower accumulated losses from the 2-body collisions
and therefore granting higher intrinsic efficiencies. The inset displays the
trends of the 2-body decay rate for the three cases during the initial
expansion phase and around the focus.

Figure 4: Calculated overall efficiency as a function of the storage time in
BECs of initial size $\sigma_{0}=$3\text{\,}\mathrm{\SIUnitSymbolMicro m}$$.
For each data point the readout is performed after refocusing, i.e. at high
density. The inset shows the comparison between the overall retrieval
efficiency $\eta_{\text{tot}}$(blue), the efficiency factor
$\eta_{\text{ATS}}$ and the intrinsic memory efficiency
$\eta_{1}\cdot\eta_{2}$ as a function of time for
$T_{\text{C}}=$3\text{\,}\mathrm{s}$$ (i.e.,
$\tau_{\text{mem}}\approx$5\text{\,}\mathrm{s}$$) during the focus of the BEC.

In order to calculate the overall efficiency, $\eta_{\text{tot}}$, we estimate
the optimal write-in and read-out efficiency associated with an ATS protocol
(in this case in backward retrieval) as Rastogi  _et al._ (2019)

$\eta_{\text{ATS}}(\rho(t))\approx\left(1-e^{-d(t)/(2F)}\right)^{2}e^{-d(t)/(2F)},$
(6)

where $F\approx 2\pi B/\Gamma$ is the ‘ATS factor’ depending on the bandwidth
$B$ and the optical transition linewidth $\Gamma$. The effective OD, $d(t)$,
results from integration along the probe beam propagation of the spatial
overlap between the Thomas-Fermi density distribution of the ensemble and
intensity profile of the probe beam Saglamyurek  _et al._ (2021). In this
work, we consider a probe beam with a waist of
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$, which is smaller than the radius of
the BEC $\sigma(t)$, and a Gaussian temporal profile with full width at half
maximum $\tau_{p}=$2.4\text{\,}\mathrm{ns}$$. The associated bandwidth,
$B=$180\text{\,}\mathrm{MHz}$$, satisfies the condition for optimal ATS
efficiency associated with the condensate OD and the optical transition
linewidth $\Gamma$ Rastogi  _et al._ (2019).

The total efficiency for a series of on-demand read-out cases with different
storage times and initial size
$\sigma_{0}=$3\text{\,}\mathrm{\SIUnitSymbolMicro m}$$ is calculated through
Eq. 3 and presented in Fig. 4. The inset highlights separately the
contributions of the ATS efficiency factor $\eta_{\text{ATS}}$ (yellow) and of
the intrinsic efficiency $\eta_{\text{1}}\cdot\eta_{\text{2}}$ (orange) to the
total efficiency $\eta_{\text{tot}}$ (blue) for the case with
$5\text{\,}\mathrm{s}$ storage time. It is worth noting how the increased
2-body decay rate that derives from the enhanced density in the focus leads to
a shift in the efficiency peaks: the peak in the total efficiency is
registered consistently $\delta\tau\approx$2\text{\,}\mathrm{ms}$$ before the
peak density is achieved. The magnitude of this effect depends on the
expansion rate of the BEC: a slower expansion of the ensemble generally
corresponds to a larger value of $\delta\tau$. In the case with the slowest
expansion considered in Fig. 3, corresponding to the initial size
$\sigma_{0}=$6.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$$ (yellow line), for
instance, the shift would rise to $\delta\tau\approx$7\text{\,}\mathrm{ms}$$.
Each point presented in the main figure is, thus, associated with the peak
total write-in and readout efficiency after the BEC refocusing. Assuming ideal
lensing potentials, overall efficiencies $>$35% can be reached for storage
times up to $100\text{\,}\mathrm{s}$, neglecting external sources of noise.
With a storage time of $\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$, the
proposed memory will thus exhibit an unprecedented time-bandwidth product of
$\sim$1\text{\times}{10}^{10}$$.

In conclusion, by focusing on density-dependent effects on the memory
efficiency, we proposed a new experimental protocol that extends the storage
time of BEC-based quantum memories. This method relies on matter-wave lensing
to tune the density of an expanding pure BEC in microgravity. We show storage
times up to $\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$, which are
ultimately limited by the vacuum quality and the consequent collisions with
the background gas. Note that aberrations due to anharmonicity of the lensing
potentials or possible technical field inhomogeneities are not accounted for
in our simulations. Nevertheless, this method can open up the way to push the
limit of cold atom based quantum memories to
$\tau_{\text{mem}}\approx$100\text{\,}\mathrm{s}$$ storage time, which would
be of interest for growing demands of long storage time memories for space
operations Cao  _et al._ (2018); Mohageg  _et al._ (2021); Gündoğan  _et al._
(2021); Lu  _et al._ (2022).

Furthermore, this protocol that relies on interaction-driven expansion to
decrease the atomic density, and consequently the collisional losses, followed
by matter-wave optics techniques to collimate and refocus the ensemble has
advantages that are not restricted to the quantum memory field, but can have
applications for a broader audience. Proof of principle tests of this
technique should be readily possible within long baseline facilities Hardman
_et al._ (2016); Asenbaum  _et al._ (2017); Wodey  _et al._ (2020),
microgravity platforms on ground, e.g. drop tower facilities Deppner  _et al._
(2021); Müntinga  _et al._ (2013) and ultimately in space Becker  _et al._
(2018); Aveline  _et al._ (2020); Frye  _et al._ (2021).

##  I Acknowledgements

This work is supported by the German Space Agency (DLR) with funds provided by
the BMWK under grant number No. 50WM2055 (OPTIMO-II) and No. 50WM2250B
(QUANTUS+). M. G. further acknowledges funding from the European Union’s
Horizon 2020 research and innovation programme under the Marie Skłodowska-
Curie grant agreement No. 894590.

§ E. D. R. and S. K. contributed equally.

## References

  * Sangouard  _et al._ (2011) N. Sangouard, C. Simon, H. Riedmatten, and N. Gisin, Rev Mod Phys 83, 33 (2011).
  * Heshami  _et al._ (2016) K. Heshami, D. G. England, P. C. Humphreys, P. J. Bustard, V. M. Acosta, J. Nunn, and B. J. Sussman, Journal of Modern Optics 63, 2005 (2016), pMID: 27695198, https://doi.org/10.1080/09500340.2016.1148212 .
  * Nunn  _et al._ (2013) J. Nunn, N. K. Langford, W. S. Kolthammer, T. F. M. Champion, M. R. Sprague, P. S. Michelberger, X.-M. Jin, D. G. England, and I. A. Walmsley, Phys. Rev. Lett. 110, 133601 (2013).
  * Gouzien and Sangouard (2021) E. Gouzien and N. Sangouard, Phys. Rev. Lett. 127, 140503 (2021).
  * Gündoğan  _et al._ (2021) M. Gündoğan, J. S. Sidhu, V. Henderson, L. Mazzarella, J. Wolters, D. K. L. Oi, and M. Krutzik, npj Quant. Inf. 7, 128 (2021).
  * Liorni  _et al._ (2021) C. Liorni, H. Kampermann, and D. Bruß, New J. Phys. (2021).
  * Sidhu  _et al._ (2021) J. S. Sidhu, S. K. Joshi, M. Gündoğan, T. Brougham, D. Lowndes, L. Mazzarella, M. Krutzik, S. Mohapatra, D. Dequal, G. Vallone, P. Villoresi, A. Ling, T. Jennewein, M. Mohageg, J. G. Rarity, I. Fuentes, S. Pirandola, and D. K. L. Oi, IET Quantum Communication 2, 182 (2021).
  * Wallnöfer  _et al._ (2022) J. Wallnöfer, F. Hahn, M. Gündoğan, J. S. Sidhu, F. Wiesner, N. Walk, J. Eisert, and J. Wolters, Communications Physics 5, 169 (2022).
  * Cao  _et al._ (2018) Y. Cao, Y.-H. Li, W.-J. Zou, Z.-P. Li, Q. Shen, S.-K. Liao, J.-G. Ren, J. Yin, Y.-A. Chen, C.-Z. Peng, and J.-W. Pan, Phys. Rev. Lett. 120, 140405 (2018).
  * Mohageg  _et al._ (2021) M. Mohageg, L. Mazzarella, D. V. Strekalov, N. Yu, A. Zhai, S. Johnson, C. Anastopoulos, J. Gallicchio, B. L. Hu, T. Jennewein, S.-Y. Lin, A. Ling, C. Marquardt, M. Meister, A. Roura, L. Wörner, _et al._ , “The deep space quantum link: Prospective fundamental physics experiments using long-baseline quantum optics,” (2021), arXiv:2111.15591 .
  * Gündoğan  _et al._ (2021) M. Gündoğan, T. Jennewein, F. K. Asadi, E. D. Ros, E. Sağlamyürek, D. Oblak, T. Vogl, D. Rieländer, J. Sidhu, S. Grandi, L. Mazzarella, J. Wallnöfer, P. Ledingham, L. LeBlanc, M. Mazzera, _et al._ , “Topical white paper: A case for quantum memories in space,” (2021), arXiv:2111.09595 .
  * Lu  _et al._ (2022) C.-Y. Lu, Y. Cao, C.-Z. Peng, and J.-W. Pan, Rev. Mod. Phys. 94, 035001 (2022).
  * Barzel  _et al._ (2022) R. Barzel, M. Gündoğan, M. Krutzik, D. Rätzel, and C. Lämmerzahl, “Gravitationally induced entanglement dynamics of photon pairs and quantum memories,”  (2022), arXiv:2209.02099 .
  * Fuchs  _et al._ (2011) G. D. Fuchs, G. Burkard, P. V. Klimov, and D. D. Awschalom, Nature Physics 7, 789 (2011).
  * Sukachev  _et al._ (2017) D. D. Sukachev, A. Sipahigil, C. T. Nguyen, M. K. Bhaskar, R. E. Evans, F. Jelezko, and M. D. Lukin, Phys. Rev. Lett. 119, 223602 (2017).
  * Bradley  _et al._ (2019) C. E. Bradley, J. Randall, M. H. Abobeih, R. C. Berrevoets, M. J. Degen, M. A. Bakker, M. Markham, D. J. Twitchen, and T. H. Taminiau, Phys. Rev. X 9, 031045 (2019).
  * Ledingham  _et al._ (2012) P. M. Ledingham, W. R. Naylor, and J. J. Longdell, Phys. Rev. Lett. 109, 093602 (2012).
  * Jobez  _et al._ (2015) P. Jobez, C. Laplane, N. Timoney, N. Gisin, A. Ferrier, P. Goldner, and M. Afzelius, Phys. Rev. Lett. 114, 230502 (2015).
  * Gündoğan  _et al._ (2015) M. Gündoğan, P. M. Ledingham, K. Kutluer, M. Mazzera, and H. de Riedmatten, Phys. Rev. Lett. 114, 230501 (2015).
  * Langer  _et al._ (2005) C. Langer, R. Ozeri, J. D. Jost, J. Chiaverini, B. DeMarco, A. Ben-Kish, R. B. Blakestad, J. Britton, D. B. Hume, W. M. Itano, D. Leibfried, R. Reichle, T. Rosenband, T. Schaetz, P. O. Schmidt, and D. J. Wineland, Phys. Rev. Lett. 95, 060502 (2005).
  * Harty  _et al._ (2014) T. P. Harty, D. T. C. Allcock, C. J. Ballance, L. Guidoni, H. A. Janacek, N. M. Linke, D. N. Stacey, and D. M. Lucas, Phys. Rev. Lett. 113, 220501 (2014).
  * Wang  _et al._ (2021) P. Wang, C.-Y. Luan, M. Qiao, M. Um, J. Zhang, Y. Wang, X. Yuan, M. Gu, J. Zhang, and K. Kim, Nature Communications 12, 233 (2021).
  * Specht  _et al._ (2011) H. P. Specht, C. Nölleke, A. Reiserer, M. Uphoff, E. Figueroa, S. Ritter, and G. Rempe, Nature 473, 190 (2011).
  * Langenfeld  _et al._ (2021) S. Langenfeld, P. Thomas, O. Morin, and G. Rempe, Phys. Rev. Lett. 126, 230506 (2021).
  * Cho  _et al._ (2016) Y.-W. Cho, G. T. Campbell, J. L. Everett, J. Bernu, D. B. Higginbottom, M. T. Cao, J. Geng, N. P. Robins, P. K. Lam, and B. C. Buchler, Optica 3, 100 (2016).
  * Wolters  _et al._ (2017) J. Wolters, G. Buser, A. Horsley, L. Béguin, A. Jöckel, J.-P. Jahn, R. J. Warburton, and P. Treutlein, Phys. Rev. Lett. 119, 060502 (2017).
  * Katz and Firstenberg (2018) O. Katz and O. Firstenberg, Nature Communications 9, 2074 (2018).
  * Kaczmarek  _et al._ (2018) K. T. Kaczmarek, P. M. Ledingham, B. Brecht, S. E. Thomas, G. S. Thekkadath, O. Lazo-Arjona, J. H. D. Munns, E. Poem, A. Feizpour, D. J. Saunders, J. Nunn, and I. A. Walmsley, Phys. Rev. A 97, 042316 (2018).
  * Riedl  _et al._ (2012) S. Riedl, M. Lettner, C. Vo, S. Baur, G. Rempe, and S. Dürr, Phys. Rev. A 85, 022318 (2012).
  * Wang  _et al._ (2019) Y. Wang, J. Li, S. Zhang, K. Su, Y. Zhou, K. Liao, S. Du, H. Yan, and S.-L. Zhu, Nature Photonics 13, 346 (2019).
  * Bao  _et al._ (2012) X.-H. Bao, A. Reingruber, P. Dietrich, J. Rui, A. Dück, T. Strassel, L. Li, N.-L. Liu, B. Zhao, and J.-W. Pan, Nature Physics 8, 517 (2012).
  * Heller  _et al._ (2020) L. Heller, P. Farrera, G. Heinze, and H. de Riedmatten, Phys. Rev. Lett. 124, 210504 (2020).
  * Saglamyurek  _et al._ (2021) E. Saglamyurek, T. Hrushevskyi, A. Rastogi, L. W. Cooke, B. D. Smith, and L. J. LeBlanc, New Journal of Physics 23, 043028 (2021).
  * Liu  _et al._ (2018) L. Liu, D.-S. Lü, W.-B. Chen, T. Li, Q.-Z. Qu, B. Wang, L. Li, W. Ren, Z.-R. Dong, J.-B. Zhao, W.-B. Xia, X. Zhao, J.-W. Ji, M.-F. Ye, Y.-G. Sun, _et al._ , Nat. Commun. 9, 2760 (2018).
  * Becker  _et al._ (2018) D. Becker, M. D. Lachmann, S. T. Seidel, H. Ahlers, A. N. Dinkelaker, J. Grosse, O. Hellmig, H. Müntinga, V. Schkolnik, T. Wendrich, A. Wenzlawski, B. Weps, R. Corgier, T. Franz, N. Gaaloul, _et al._ , Nature 562, 391 (2018).
  * Aveline  _et al._ (2020) D. C. Aveline, J. R. Williams, E. R. Elliott, C. Dutenhoffer, J. R. Kellogg, J. M. Kohel, N. E. Lay, K. Oudrhiri, R. F. Shotwell, N. Yu, and R. J. Thompson, Nature 582, 193 (2020).
  * Alonso  _et al._ (2022) I. Alonso, C. Alpigiani, B. Altschul, H. Araujo, G. Arduini, J. Arlt, L. Badurina, A. Balaz, S. Bandarupally, B. C. B. M. Barone, M. Barsanti, S. Bass, A. Bassi, B. Battelier, C. F. A. Baynham, _et al._ , “Cold atoms in space: Community workshop summary and proposed road-map,” (2022), arXiv:2201.07789 .
  * Tino  _et al._ (2019) G. M. Tino, A. Bassi, G. Bianco, K. Bongs, P. Bouyer, L. Cacciapuoti, S. Capozziello, X. Chen, M. L. Chiofalo, A. Derevianko, W. Ertmer, N. Gaaloul, P. Gill, P. W. Graham, J. M. Hogan, _et al._ , The European Physical Journal D 73, 228 (2019).
  * El-Neaj  _et al._ (2020) Y. A. El-Neaj, C. Alpigiani, S. Amairi-Pyka, H. Araújo, A. Balaž, A. Bassi, L. Bathe-Peters, B. Battelier, A. Belić, E. Bentine, J. Bernabeu, A. Bertoldi, R. Bingham, D. Blas, V. Bolpasi, _et al._ , EPJ Quantum Technology 7, 6 (2020).
  * Frye  _et al._ (2021) K. Frye, S. Abend, W. Bartosch, A. Bawamia, D. Becker, H. Blume, C. Braxmaier, S.-W. Chiow, M. A. Efremov, W. Ertmer, P. Fierlinger, T. Franz, N. Gaaloul, J. Grosse, C. Grzeschik, _et al._ , EPJ Quantum Technology 8, 1 (2021).
  * Cacciapuoti  _et al._ (2020) L. Cacciapuoti, M. Armano, R. Much, O. Sy, A. Helm, M. P. Hess, J. Kehrer, S. Koller, T. Niedermaier, F. X. Esnault, D. Massonnet, D. Goujon, J. Pittet, P. Rochat, S. Liu, _et al._ , The European Physical Journal D 74, 164 (2020).
  * Dudin  _et al._ (2013) Y. O. Dudin, L. Li, and A. Kuzmich, Phys. Rev. A 87, 031801 (2013).
  * Lundblad  _et al._ (2010) N. Lundblad, M. Schlosser, and J. V. Porto, Phys. Rev. A 81, 031611 (2010).
  * Dudin  _et al._ (2010) Y. O. Dudin, R. Zhao, T. A. B. Kennedy, and A. Kuzmich, Phys. Rev. A 81, 041805 (2010).
  * Zhang  _et al._ (2009) R. Zhang, S. R. Garner, and L. V. Hau, Phys. Rev. Lett. 103, 233602 (2009).
  * Ammann and Christensen (1997) H. Ammann and N. Christensen, Phys. Rev. Lett. 78, 2088 (1997).
  * Myrskog  _et al._ (2000) S. H. Myrskog, J. K. Fox, H. S. Moon, J. B. Kim, and A. M. Steinberg, Phys. Rev. A 61, 053412 (2000).
  * Kovachy  _et al._ (2015) T. Kovachy, J. M. Hogan, A. Sugarbaker, S. M. Dickerson, C. A. Donnelly, C. Overstreet, and M. A. Kasevich, Phys. Rev. Lett. 114, 143004 (2015).
  * Deppner  _et al._ (2021) C. Deppner, W. Herr, M. Cornelius, P. Stromberger, T. Sternke, C. Grzeschik, A. Grote, J. Rudolph, S. Herrmann, M. Krutzik, A. Wenzlawski, R. Corgier, E. Charron, D. Guéry-Odelin, N. Gaaloul, _et al._ , Phys. Rev. Lett. 127, 100401 (2021).
  * Hogan  _et al._ (2011) J. M. Hogan, D. M. S. Johnson, S. Dickerson, T. Kovachy, A. Sugarbaker, S.-w. Chiow, P. W. Graham, M. A. Kasevich, B. Saif, S. Rajendran, P. Bouyer, B. D. Seery, L. Feinberg, and R. Keski-Kuha, General Relativity and Gravitation 43, 1953 (2011).
  * Nirrengarten  _et al._ (2006) T. Nirrengarten, A. Qarry, C. Roux, A. Emmert, G. Nogues, M. Brune, J.-M. Raimond, and S. Haroche, Phys. Rev. Lett. 97, 200405 (2006).
  * Luan  _et al._ (2018) T. Luan, Y. Li, X. Zhang, and X. Chen, Review of Scientific Instruments 89, 123110 (2018), https://doi.org/10.1063/1.5046815 .
  * Kanthak  _et al._ (2021) S. Kanthak, M. Gebbe, M. Gersemann, S. Abend, E. M. Rasel, and M. Krutzik, New Journal of Physics 23, 093002 (2021).
  * Gochnauer  _et al._ (2021) D. Gochnauer, T. Rahman, A. Wirth-Singh, and S. Gupta, Atoms 9 (2021), 10.3390/atoms9030058.
  * Namazi  _et al._ (2017) M. Namazi, G. Vallone, B. Jordaan, C. Goham, R. Shahrokhshahi, P. Villoresi, and E. Figueroa, Phys. Rev. Applied 8, 064013 (2017).
  * Gorshkov  _et al._ (2007) A. V. Gorshkov, A. André, M. Fleischhauer, A. S. Sørensen, and M. D. Lukin, Phys. Rev. Lett. 98, 123601 (2007).
  * Saglamyurek  _et al._ (2018) E. Saglamyurek, T. Hrushevskyi, A. Rastogi, K. Heshami, and L. J. LeBlanc, Nature Photonics 12, 774 (2018).
  * Geng  _et al._ (2014) J. Geng, G. T. Campbell, J. Bernu, D. B. Higginbottom, B. M. Sparkes, S. M. Assad, W. P. Zhang, N. P. Robins, P. K. Lam, and B. C. Buchler, New Journal of Physics 16, 113053 (2014).
  * Pérez-García  _et al._ (1997) V. M. Pérez-García, H. Michinel, J. I. Cirac, M. Lewenstein, and P. Zoller, Phys. Rev. A 56, 1424 (1997).
  * Pérez-García  _et al._ (1996) V. M. Pérez-García, H. Michinel, J. I. Cirac, M. Lewenstein, and P. Zoller, Phys. Rev. Lett. 77, 5320 (1996).
  * Harber  _et al._ (2002) D. M. Harber, H. J. Lewandowski, J. M. McGuirk, and E. A. Cornell, Phys. Rev. A 66, 053616 (2002).
  * Corgier  _et al._ (2020) R. Corgier, S. Loriani, H. Ahlers, K. Posso-Trujillo, C. Schubert, E. M. Rasel, E. Charron, and N. Gaaloul, New Journal of Physics 22, 123008 (2020).
  * Albers  _et al._ (2022) H. Albers, R. Corgier, A. Herbst, A. Rajagopalan, C. Schubert, C. Vogt, M. Woltmann, C. Lämmerzahl, S. Herrmann, E. Charron, W. Ertmer, E. M. Rasel, N. Gaaloul, and D. Schlippert, Communications Physics 5, 60 (2022).
  * Burt  _et al._ (1997) E. A. Burt, R. W. Ghrist, C. J. Myatt, M. J. Holland, E. A. Cornell, and C. E. Wieman, Phys. Rev. Lett. 79, 337 (1997).
  * Egorov (2012) M. Egorov,  _Coherence and collective oscillations of a two-component Bose-Einstein condensate_ , Ph.D. thesis (2012).
  * Rastogi  _et al._ (2019) A. Rastogi, E. Saglamyurek, T. Hrushevskyi, S. Hubele, and L. J. LeBlanc, Phys. Rev. A 100, 012314 (2019).
  * Hardman  _et al._ (2016) K. S. Hardman, P. J. Everitt, G. D. McDonald, P. Manju, P. B. Wigley, M. A. Sooriyabandara, C. C. N. Kuhn, J. E. Debs, J. D. Close, and N. P. Robins, Phys. Rev. Lett. 117, 138501 (2016).
  * Asenbaum  _et al._ (2017) P. Asenbaum, C. Overstreet, T. Kovachy, D. D. Brown, J. M. Hogan, and M. A. Kasevich, Phys. Rev. Lett. 118, 183602 (2017).
  * Wodey  _et al._ (2020) E. Wodey, D. Tell, E. M. Rasel, D. Schlippert, R. Baur, U. Kissling, B. Kölliker, M. Lorenz, M. Marrer, U. Schläpfer, M. Widmer, C. Ufrecht, S. Stuiber, and P. Fierlinger, Review of Scientific Instruments 91, 035117 (2020), https://doi.org/10.1063/1.5141340 .
  * Müntinga  _et al._ (2013) H. Müntinga, H. Ahlers, M. Krutzik, A. Wenzlawski, S. Arnold, D. Becker, K. Bongs, H. Dittus, H. Duncker, N. Gaaloul, C. Gherasim, E. Giese, C. Grzeschik, T. W. Hänsch, O. Hellmig, _et al._ , Phys. Rev. Lett. 110, 093602 (2013).


# Deep Policy Iteration for High-Dimensional Mean Field Games

Mouhcine Assouli  Badr Missaoui

###### Abstract

This paper introduces Deep Policy Iteration (DPI), a novel approach that
combines the MFDGM [1] method and the Policy Iteration method [2, 3] to
address high-dimensional stochastic Mean Field Games. The Deep Policy
Iteration employs three neural networks to approximate the solutions of
equations. These networks are trained to satisfy each equation and its
corresponding forward-backward conditions. Unlike existing approaches that are
limited to separable Hamiltonians and lower dimensions, DPI extends its
capabilities to effectively solve high-dimensional MFG systems, encompassing
both separable and non-separable Hamiltonians. To evaluate the reliability and
efficacy of DPI, a series of numerical experiments is conducted. The results
obtained using DPI are compared with those obtained using the MFDGM method and
the Policy Iteration Method. This comparative analysis provides insights into
the performance of DPI and its advantages over existing methods.

###### keywords:

Mean Field Games , Deep Learning , Policy Iteration , Non-Separable
Hamiltonian

latexText page 7 contains only floats

[inst1]organization=College of Computing, UM6P,addressline=Lot 660,
city=BenGuerir, postcode=43150, country=Morocco

[inst2]organization=College of Computing, UM6P,addressline=Lot 660,
city=BenGuerir, postcode=43150, country=Morocco

##  1 Introduction

Mean Field Games (MFG) theory, introduced by Lasry and Lions [4], provides a
framework for analyzing Nash equilibria in differential games involving a
large number of agents. This theory is characterized by a mathematical
formulation consisting of a system of coupled partial differential equations
(PDEs). Specifically, the system comprises a forward-time Fokker-Planck
equation (FP) and a backward-time Hamilton-Jacobi-Bellman equation (HJB),
which govern the evolution of the population density and the value function,
respectively. MFGs have garnered significant attention and have been
extensively studied in various fields, such as autonomous vehicles [5, 6],
finance [7, 8], economics [9, 10, 11]. In the general case, the MFG system is
described as,

$\left\\{\begin{array}[]{rrrrr}-\partial_{t}\phi-\nu\Delta\phi+H(x,\rho,\nabla\phi)=0,\
in&E,\\\
\partial_{t}\rho-\nu\Delta\rho-\operatorname{div}\left(\rho\nabla_{p}H(x,\rho,\nabla\phi)\right)=0,\
in&E,\\\ \rho(0,x)=\rho_{0}(x),\ \ \phi(T,x)=g(x,\rho(T,x)),\
in&\Omega,\end{array}\right.$ (1)

where, $E=[0,T]\times\Omega,$ $\Omega$ bounded subset of $\mathbb{R}^{d}$ and
$g$ denotes the terminal cost. The Hamiltonian $H$ with separable structure is
defined as

$H(x,\rho,p)=inf_{v}\\{-p.v+L_{0}(x,v)\\}-f_{0}(x,\rho)=H_{0}(x,p)-f_{0}(x,\rho),$
(2)

The solution of (1) exists and is unique under the standard assumptions of
convexity of $H$ in the second variable and monotonicity of $f$ and $g$ [12],
see also refs [13, 4] for more details. For non-separable Hamiltonians, where
the Hamiltonian of the MFG depends jointly on $\rho$ and $p$, the existence
and uniqueness of the solution for MFGs of congestion type has been
investigated by Achdou and Porretta in [14] and Gomes et al. in [15].

The numerical solution of (1) holds significant importance in the practical
application of MFG theory. However, due to the strong coupling and forward-
backward structure of the two equations in (1), they cannot be solved
independently or jointly using simple forward-in-time methods. Extensive
research has been conducted in this area, leading to the proposal of various
methods with successful applications, as seen in [16, 4] and references
therein. Nevertheless, these methods often face challenges in terms of
computational complexity, particularly when dealing with high-dimensional
problems [17, 18].

To address this challenge, deep learning methods, such as Generative
Adversarial Networks (GANs) [12, 19], have been utilized to reformulate MFGs
as primal-dual problems. However, these existing methods require the
Hamiltonian $H$ to be separable in terms of $\rho$ and $p$. Unfortunately,
none of these methods adequately cover the case of a non-separable Hamiltonian
with a generic structure.

To the best of our knowledge, two promising numerical approaches have recently
emerged to overcome this limitation. The first approach is the Policy
Iteration (PI) method (in the context of MFG) [3], which is a numerical method
based on finite difference techniques. PI can be seen as a modification of the
fixed-point procedure, where, at each iteration, the HJB equation is solved
for a fixed control. The control is then updated separately after the value
function is updated. The second approach is the MFDGM method [1], which is a
deep learning method based on the Deep Galerkin Method. This method offers a
promising solution for MFGs with non-separable Hamiltonians.

Following the work in [2, 3], we introduce the policy iteration algorithms for
(1) with periodic boundary conditions.

We first define the Lagrangian as the Legendre transform of $H$ :

$L(m,q)=\sup_{p\in\mathbb{R}^{d}}\\{p\cdot q-H(\rho,p)\\}$

Policy Iteration Algorithm: Given $R>0$ and given a bounded, measurable vector
field $q^{(0)}:\mathbb{T}^{d}\times[0,T]\rightarrow\mathbb{R}^{d}$ with
$\left|q^{(0)}\right|\leq R$ and
$\left\|\operatorname{div}q^{(0)}\right\|_{L^{r}(E)}\leq R$, iterate:

(i) Solve

$\begin{cases}\partial_{t}\rho^{(n)}-\epsilon\Delta\rho^{(n)}-\operatorname{div}\left(\rho^{(n)}q^{(n)}\right)=0,&\text{
in }E,\\\ \rho^{(n)}(x,0)=\rho_{0}(x)&\text{ in }\Omega.\end{cases}$ (3)

(ii) Solve

$\begin{cases}-\partial_{t}\phi^{(n)}-\epsilon\Delta\phi^{(n)}+q^{(n)}D\phi^{(n)}-\mathcal{L}\left(m^{(n)},D\phi^{(n-1)},q^{(n)}\right)=0&\text{
in }E,\\\ \phi^{(n)}(x,T)=\phi_{T}(x)&\text{ in }\Omega,\end{cases}$ (4)

where for
$(\rho,p,q)\in\mathbb{R}^{+}\times\mathbb{R}^{d}\times\mathbb{R}^{d}$,

$\mathcal{L}(m,p,q)=p\cdot q-H(m,p).$

(iii) Update the policy

$q^{(n+1)}(x,t)=\arg\max_{|q|\leq R}\left\\{q\cdot
D\phi^{(n)}(x,t)-L\left(\rho^{(n)},q\right)\right\\}\quad\text{ in }E.$ (5)

_Contributions_ This paper contributes by introducing a novel approach, called
Deep Policy Iteration (DPI), which combines elements of the MFDGM method and
the PI method to solve high-dimensional stochastic MFG. Inspired by [20, 21],
we employ three neural networks to approximate the unknown solutions of
equations (3)-(5), trained to satisfy each equation and the associated
forward-backward conditions. While most existing methods are restricted to
problems with separable Hamiltonians and lower dimensions, DPI overcomes these
limitations and can effectively solve high-dimensional MFG systems,
encompassing both separable and non-separable Hamiltonians. To assess the
reliability and effectiveness of DPI, we conducted a series of numerical
experiments. In these experiments, we compare the results obtained using DPI
with those obtained using the MFDGM method and the PI method.

_Contents_ The subsequent sections of this paper are organized as follows:
Section 2 provides an introduction to our approach, outlining its key aspects.
In Section 3, we delve into a comprehensive examination of PI and MFDGM
methods. Moving on to Section 4, we explore the numerical performance of our
proposed algorithms. To evaluate the efficacy of our method, we employ a
straightforward analytical solution in Section 4.1. Furthermore, we employ a
well-established one-dimensional traffic flow problem [5], renowned for its
non-separable Hamiltonian, to compare DPI and PI in Section 4.2.

##  2 Methodology

The proposed methodology consists of two main steps: In the first step, the
algorithm computes the density and cost of a given policy. This is achieved by
solving a set of coupled partial differential equations (PDEs) that describe
the dynamics of the system.

In the second step, the algorithm updates the policy established in the first
step and computes a new policy that minimizes the expected cost given the
density and value function. This involves solving a separate optimization
problem for each agent, which can be done using deep learning techniques.
Here, we proceed by using a distinct neural network for each of the three
variables: the density function, the value function, and the policy.

To assess the accuracy of these approximations, we use a loss function based
on the residual of each equation to update the parameters of the neural
networks. These neural networks are trained based on the Deep Galerkin Method
[20]. Within each iteration, the DPI method repeats the two steps until
convergence. At each iteration, the algorithm computes a better policy by
taking into account the expected behavior of the other agents in the
population. The resulting policy is a solution to the MFG that describes the
optimal behavior of the agents in the population, given the interactions
between them.

We initialize the neural networks as a solution to our system and define:

$q_{\tau}(t,x)=N_{\tau}(t,x),\ \ \ \phi_{\omega}(t,x)=N_{\omega}(t,x),\ \ \
\rho_{\theta}(t,x)=N_{\theta}(t,x).$ (6)

Our training strategy starts by solving (3). We compute the loss (7) at
randomly sampled points $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$, and
$\\{x_{s}\\}_{s=1}^{S}$ from $\Omega$.

$\textsc{Loss}_{total}^{(3)}=\textsc{Loss}^{(3)}+\textsc{Loss}_{cond}^{(3)},$
(7)

where

$\displaystyle\textsc{Loss}^{(3)}$
$\displaystyle=\frac{1}{B}\sum_{b=1}^{B}\Big{|}\partial_{t}\rho_{\theta}(t_{b},x_{b})-\nu\Delta\rho_{\theta}(t_{b},x_{b})$
$\displaystyle\quad\quad\quad-\operatorname{div}\left(\rho_{\theta}(t_{b},x_{b})q_{\tau}(t_{b},x_{b}))\right)\Big{|}^{2},$

and

$\textsc{Loss}_{cond}^{(3)}=\frac{1}{S}\sum_{s=1}^{S}\Big{|}\rho_{\theta}(0,x_{s})-\rho_{0}(x_{s})\Big{|}^{2}.$

We then update the weights of $\rho_{\theta}$ by back-propagating the loss
(7). We do the same to (4) with the given $\rho_{\theta}$. We compute (8) at
randomly sampled points $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$, and
$\\{x_{s}\\}_{s=1}^{S}$ from $\Omega$,

$\textsc{Loss}_{total}^{(4)}=\textsc{Loss}^{(4)}+\textsc{Loss}_{cond}^{(4)},$
(8)

where

$\displaystyle\textsc{Loss}^{(4)}$
$\displaystyle=\frac{1}{B}\sum_{b=1}^{B}\Big{|}\partial_{t}\phi_{\omega}(t_{b},x_{b})+\nu\Delta\phi_{\omega}(t_{b},x_{b})$
$\displaystyle\quad\quad+q_{\tau}(t_{b},x_{b}))\nabla\phi_{\omega}(t_{b},x_{b})-L(\rho_{\theta}(t_{b},x_{b}),q_{\tau}(t_{b},x_{b}))\Big{|}^{2},$

and

$\textsc{Loss}_{cond}^{(4)}=\frac{1}{S}\sum_{s=1}^{S}\Big{|}\phi_{\omega}(T,x_{s})-g(x_{s},\rho_{\theta}(T,x_{s}))\Big{|}^{2}.$

We then update the weights of $\phi_{\omega}$ by backpropagating the loss (8).
Finally, we update $q_{\tau}$ by computing the loss (9) at randomly sampled
points $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$.

$\textsc{Loss}_{total}^{(5)}=\textsc{Loss}^{(5)}+\textsc{Loss}_{cond}^{(5)},$
(9)

where

$\displaystyle\textsc{Loss}^{(5)}$
$\displaystyle=\frac{1}{B}\sum_{b=1}^{B}q_{\tau}(t_{b},x_{b}))\nabla\phi_{\omega}(t_{b},x_{b})-L(\rho_{\theta}(t_{b},x_{b}),q_{\tau}(t_{b},x_{b})),$

and

$\textsc{Loss}_{cond}^{(5)}=\frac{1}{B}\sum_{b=1}^{B}\Big{|}q_{\tau}(t_{b},x_{b})-\nabla_{p}H(\rho_{\theta}(t_{b},x_{b}),\nabla\phi_{\omega}(t_{b},x_{b}))\Big{|}^{2}.$

To help readers better understand our methodology, we have summarized it in
the form of an algorithm 1. This algorithm serves as a visual representation
of our steps, allowing others to replicate our methodology and validate our
results. The convergence of the neural network approximation was previously
analyzed in [20, 1]. Furthermore, the convergence of policy iteration was
examined in [2, 3] with the Banach fixed point method. It is important to note
that by experimenting with diverse network structures and training approaches,
one can enhance the performance and robustness of the neural networks utilized
in the model. Hence, selecting the best possible combination of architecture
and hyperparameters for the neural networks is crucial for achieving the
desired outcomes.

$L$, $\nu$ diffusion parameter, $g$ terminal cost.

Initialize neural networks $N_{\tau_{0}}$, $N_{\omega_{0}}$ and
$N_{\theta_{0}}$

Train

for n=0,1,2…,K-1 do

(i) solve (3)

Sample batch $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$, and
$\\{x_{s}\\}_{s=1}^{S}$ from $\Omega$

$\textsc{L}^{(3)}\leftarrow\frac{1}{B}\sum_{b=1}^{B}\Big{|}\partial_{t}\rho_{\theta_{n}}(t_{b},x_{b})-\nu\Delta\rho_{\theta_{n}}(t_{b},x_{b})$

$\quad\quad\quad\quad\quad\quad\quad-\operatorname{div}\left(\rho_{\theta_{n}}(t_{b},x_{b})q_{\tau_{n}}(t_{b},x_{b}))\right)\Big{|}^{2}$

$\textsc{L}_{cond}^{(3)}\leftarrow\frac{1}{S_{1}}\sum_{s=1}^{S}\Big{|}\rho_{\theta_{n}}(0,x_{s})-\rho_{0}(x_{s})\Big{|}^{2}.$

Backpropagate $\textsc{Loss}_{total}^{(3)}$ to $\theta_{n+1}$ weights.

(ii) solve (4)

Sample batch $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$, and
$\\{x_{s}\\}_{s=1}^{S}$ from $\Omega$.

$\textsc{L}^{(4)}\leftarrow\frac{1}{B}\sum_{b=1}^{B}\Big{|}\partial_{t}\phi_{\omega_{n}}(t_{b},x_{b})+\nu\Delta\phi_{\omega_{n}}(t_{b},x_{b})$

$\quad\quad\quad\quad+q_{\tau_{n}}(t_{b},x_{b}))\nabla\phi_{\omega_{n}}(t_{b},x_{b})-L(\rho_{\theta_{n+1}}(t_{b},x_{b}),q_{\tau_{n}}(t_{b},x_{b}))\Big{|}^{2}$

$\textsc{L}_{cond}^{(4)}\leftarrow\frac{1}{S}\sum_{s=1}^{S}\Big{|}\phi_{\omega_{n}}(T,x_{s})-g(x_{s},\rho_{\theta_{n+1}}(T,x_{s}))\Big{|}^{2}.$

Backpropagate $\textsc{Loss}_{total}^{(4)}$ to $\omega_{n+1}$ weight.

(i) solve (5)

Sample batch $\\{(t_{b},x_{b})\\}_{b=1}^{B}$ from $E$.

$\textsc{L}^{(5)}\leftarrow\frac{1}{B}\sum_{b=1}^{B}q_{\tau_{n}}(t_{b},x_{b}))\nabla\phi_{\omega_{n+1}}(t_{b},x_{b})$

$\quad\quad\quad\quad\quad-L(\rho_{\theta_{n+1}}(t_{b},x_{b}),q_{\tau_{n}}(t_{b},x_{b})),$

$\textsc{L}_{cond}^{(5)}\leftarrow\frac{1}{B}\sum_{b=1}^{B}\Big{|}q_{\tau_{n}}(t_{b},x_{b})-\nabla_{p}H(\rho_{\theta_{n+1}}(t_{b},x_{b}),\nabla\phi_{\omega_{n+1}}(t_{b},x_{b}))\Big{|}^{2}.$

Backpropagate $\textsc{Loss}_{total}^{(5)}$ to $\tau_{n+1}$ weight.

return $\theta_{K}$, $\omega_{K}$

Algorithm 1 Deep Policy Iteration Method

##  3 Related Works

In this section, we present a literature review that pertains to our study.
Specifically, we concentrate on two pertinent sources that provide insight
into our research. It is important to note that previous literature has
primarily focused on the separable case already presented on [19, 12].
However, our study seeks to broaden this by exploring the general MFG
problems. To accomplish this objective, we conduct a thorough analysis of the
two identified sources and draw inspiration from them. This allows us to gain
a more comprehensive understanding of how decision-making criteria interact in
intricate situations. Additionally, this review illuminates potential
difficulties that may arise when examining the non-separable case and
demonstrates how our approach can contribute to addressing these issues.

Policy Iteration method: In [3], the authors introduced the Policy iteration
method, which proved to be the first successful approach for solving systems
of mean-field game partial differential equations with non-separable
Hamiltonians. The method involved proposing two algorithms based on policy
iteration, which iteratively updated the population distribution, value
function, and control. Since the control was fixed, these algorithms only
required the solution of two decoupled, linear PDEs at each iteration, which
reduced the complexity of the equations; refer to [2].

The authors presented a revised version of the policy iteration algorithm,
which involves updating the control during each update of the population
distribution and value function. Their novel approach strives to make
significant strides in the field and deepen our understanding of the subject
matter. However, due to the computational complexity of the method, it was
limited to low-dimensional problems.

One major limitation of this method was that it may not work in the separable
case. To address these issues, our study proposes a new approach using neural
networks. We will use the same policy iteration method but instead of using
the finite difference method to solve the two decoupled, linear PDEs, we will
use neural networks inspired by DGM. This will allow us to overcome the
computational challenges of the original method and extend its applicability
to the separable case. With this approach, we expect to significantly improve
the computational complexity of the method, which will allow us to apply it to
a wider range of high-dimensional problems.

Overall, our study seeks to extend the applicability of the PI method to more
complex and higher-dimensional problems by incorporating neural networks into
its framework. We believe that this approach has the potential to make a
significant contribution to the field of mean-field game theory and could have
practical implications in fields such as finance, economics, and engineering.

MFDGM: The methodology proposed by [1] involves the use of two neural networks
to approximate the population distribution and the value function. The
accuracy of these approximations is optimized by using a loss function based
on the residual of the first equation (HJB) to update the parameters of the
neural networks. The process is then repeated using the second equation (FP)
and new parameters to further improve the accuracy of the approximations. In
contrast, methods based on Generative Adversarial Networks (GANs) [19, 12]
cannot solve MFGs with non-separable Hamiltonians.

This novel methodology significantly improves the computational complexity of
the method by utilizing neural networks trained simultaneously with a single
hidden layer and optimizing the approximations through a loss function. This
makes the method more efficient and scalable, enabling its application to a
wider range of high-dimensional problems in general mean-field games,
including separable or non-separable Hamiltonians and deterministic or
stochastic models. Additionally, comparisons to previous methods demonstrate
the efficiency of this approach, even with multilayer neural networks.

Our research builds upon prior investigations and offers substantial
advancements in methodologies for enhancing the efficiency and precision of
solving general mean-field games. This is achieved through the integration of
PI, which reduces the complexity of the equations, and neural networks,
resulting in improved computational performance and accuracy.

##  4 Numerical Experiments

To assess the efficacy of the proposed algorithm (1), we conducted experiments
on two distinct problems. Firstly, we utilized the example dataset provided in
[12, 1], which has an explicitly defined solution structure that facilitates
straightforward numerical comparisons. Secondly, we evaluated the algorithm’s
performance on a well-studied one-dimensional traffic flow problem [5], which
is known for its non-separable Hamiltonian.
To ascertain the reliability of our approach, we compared the performance of
three different algorithms, namely PI, DPI, and MFDGM, on the same dataset.
Through this evaluation, we aimed to determine the effectiveness of our
proposed algorithm in comparison to existing state-of-the-art methods.

###  4.1 Analytic Comparison

We test the effectiveness of the DPI method, we compare its performance with a
simple example of the analytic solution characterized by a separable
Hamiltonian. In order to simplify the comparison process, we select the
spatial domain as $\Omega=[-2,2]^{d}$ and set the final time as $T=1.$ This
allows us to easily analyze and compare the results obtained from both
methods. For

$\begin{array}[]{cc}H_{0}(x,p)=\frac{||p||^{2}}{2}-\beta\frac{||x||^{2}}{2},\
\ \ f_{0}(x,\rho)=\gamma\ln(\rho),\\\ g(x)=\alpha\frac{||x||^{2}}{2}-(\nu
d\alpha+\gamma\frac{d}{2}\ln\frac{\alpha}{2\pi\nu}),\end{array}$ (10)

and $\nu=\beta=1$, where

$\alpha=\frac{-\gamma+\sqrt{\gamma^{2}+4\nu^{2}\beta}}{2\nu}=\frac{-\gamma+\sqrt{\gamma^{2}+4}}{2}.$

The corresponding MFG system is:

$\left\\{\begin{array}[]{rrrrr}-\partial_{t}\phi-\Delta\phi+\frac{||\nabla\phi||^{2}}{2}-\frac{||x||^{2}}{2}=\gamma\ln(\rho),\\\
\partial_{t}\rho-\Delta\rho-\operatorname{div}\left(\rho\nabla\phi\right)=0,\\\
\rho(0,x)=(\frac{1}{2\pi})^{\frac{d}{2}}e^{-\frac{||x||^{2}}{2}},\\\
\phi(T,x)=\alpha\frac{||x||^{2}}{2}-(\alpha
d+\gamma\frac{d}{2}\ln\frac{\alpha}{2\pi}),\end{array}\right.\vspace{-0.25cm}$
(11)

and the explicit formula is given by

$\begin{array}[]{cc}\phi(t,x)=\alpha\frac{||x||^{2}}{2}-(\alpha
d+\gamma\frac{d}{2}\ln\frac{\alpha}{2\pi})t,\\\
\rho(t,x)=(\frac{1}{2\pi})^{\frac{d}{2}}e^{-\frac{||x||^{2}}{2}}.\end{array}$
(12)

Test 1: Assuming a congestion-free scenario ($\gamma=0$) in a one-dimensional
system of partial differential equations 11, we utilize Algorithm 1 to obtain
results by using a minibatch size of 50 samples at each iteration. Our
approach employs neural networks with one hidden layer consisting of 100
neurons each, with Softplus activation function for $N_{\omega}$ and Tanh
activation function for $N_{\theta}$ and $N_{\tau}$. To train the neural
networks, we use ADAM with a learning rate of $10^{-4}.$ and weight decay of
$10^{-3}.$ We adopt ResNet as the architecture of the neural networks, with a
skip connection weight of 0.5. We kept the same parameters for the second
method MFDGM. The numerical results are presented in Figure 1, where we
compare the approximate solutions obtained by two methods to the exact
solutions at different times. To assess the performance of the methods, we
compute the relative error between the model predictions and the exact
solutions on a $100\times 100$ grid within the domain $[0,1]\times[-2,2]$, see
Figure 2. Furthermore, we monitored the convergence of our approach by
plotting the two residual losses, as defined in Algorithm 1 and the MFDGM
algorithm, in Figure 3. Optimal selection of architecture and hyperparameters
for the neural networks plays a crucial role in attaining the desired outcomes
for both methods see experiments in [1].

Figure 1: Comparison of Exact Solution and Prediction using DPI and MFDGM in
one-Dimension at t=(0.25, 0.5, 0.75) with $\gamma=0$

Figure 2: Comparison of Relative Error for $\phi$ and $\rho$ using Two Methods
with $\gamma=0$.

Figure 3: Comparison of losses for $\phi$ and $\rho$ using Two Methods with
$\gamma=0$.

Test 2: In order to investigate the impact of congestion on the previous case,
we repeat the same experiment with the same parameters, but this time we
consider a non-zero congestion parameter $(\gamma=0.1)$. We present the
numerical results in Figures 4, 5, 6.

Figure 4: Comparison of Exact Solution and Prediction using DPI and MFDGM in
One Dimension at t=(0.25, 0.5, 0.75) with $\gamma=0.1$

Figure 5: Comparison of Relative Error for $\phi$ and $\rho$ using Two Methods
with $\gamma=0.1$.

Figure 6: Comparison of losses for $\phi$ and $\rho$ using Two Methods with
$\gamma=0.1$.

In comparison to the MFDGM algorithm, DPI demonstrates a higher level of
effectiveness, particularly when applied to congestion scenarios. This
superiority can be attributed to DPI’s ability to reduce the complexity of
Partial Differential Equations (PDEs) through the incorporation of a policy
neural network. Furthermore, to mitigate errors such as MFDGM, we enforce
training DPI within the boundaries. This additional measure ensures improved
accuracy in the training process. By leveraging this neural network, DPI
offers a more suitable approach for handling congestion-related tests,
surpassing the capabilities of the MFDGM algorithm.
Test 3: In this test, we aim to solve (11) using PI. To enable the application
of this method, we incorporate periodic boundary conditions into (6). In their
research, the authors utilized Policy Iteration methods to solve a Partial
Differential Equation (PDE) system through finite-difference approximation.
They adopted a uniform grid $\mathcal{G}$ and centered second-order finite
differences for the discrete Laplacian while computing the Hamiltonian and
divergence term in the FP equation using the Engquist-Osher numerical flux for
conservation laws. The symbol $\sharp$ represented the linear differential
operators at the grid nodes. Specifically, in the context of one dimension,
they implemented a uniform discretization of $E$ with $I$ nodes $x_{i}=ih$,
where $h=1/I$ is the space step. To discretize time, they employed an implicit
Euler method for both the time-forward FP equation and the time-backward HJB
equation, with a uniform grid on the interval $[0,T]$ with $N+1$ nodes
$t_{n}=nt$, where $t=T/N$ was the time step. To prevent any confusion, we
adhere to the previously established notation. Therefore, we use $\phi$,
$\rho$, and the policy $q$ to represent the vectors that approximate the
solution on $\mathcal{G}$, and by $\phi_{n},\rho_{n},$ and $q_{n}$ the vectors
on $\mathcal{G}$ approximating the solution and policy at time $t_{n}$. The
algorithm we will use for the fully discretized System 11 is as follows: we
start with an initial guess $q^{(0)}_{n}:\mathcal{G}\to\mathbb{R}^{2d}$ for
$n=0,...,N,$ and initial and final data
$\rho_{0},\phi_{N}:\mathcal{G}\to\mathbb{R}$. We then iterate on $k\geq 0$
until convergence,
(i) Solve for $n=0,\ldots,N-1$ on $\mathcal{G}$

$\left\\{\begin{array}[]{l}\rho_{n+1}^{(k)}-dt\left(\Delta_{\sharp}\rho_{n+1}^{(k)}+\operatorname{div}_{\sharp}\left(\rho_{n+1}^{(k)}q_{n+1}^{(k)}\right)\right)=\rho_{n}^{(k)}\\\
\rho_{0}^{(k)}=\rho_{0}\end{array}\right.$

(ii) Solve for $n=N-1,\ldots,0$ on $\mathcal{G}$

$\left\\{\begin{aligned}
\phi_{n}^{(k)}&-dt\left(\Delta_{\sharp}\phi_{n}^{(k)}-q_{n,\pm}^{(k)}\cdot
D_{\sharp}\phi_{n}^{(k)}\right)\\\
&=\phi_{n+1}+dt\left(\frac{1}{2}\left|q_{n+1,\pm}^{(k)}\right|^{2}+x_{i}^{2}+\gamma
Ln\left(\rho_{n+1}^{(k)}\right)\right)\\\
\phi_{N}^{(k)}&=\phi_{N}\end{aligned}\right.$

(iii) Update the policy $q_{n}^{(k+1)}=D_{\sharp}\phi_{n}^{(k)}$ on
$\mathcal{G}$ for $n=0,\ldots,N$, and set $k\leftarrow$ $k+1$. In the
following test, we choose $\gamma=0$, T = 1 for the final time, and $K=50$.
The grid consisted of $I=200$ nodes in space and $N=200$ nodes in time. The
initial policy was initialized as $q^{(0)}_{n}\equiv(0,0)$ on $\mathcal{G}$
for all $n$.

Figure 7: Comparison of Exact Solution and Prediction using PI in One
Dimension at t=(0.25, 0.5, 0.75) with $\gamma=0$

The results of the study Figure 7 showed that policy iteration is not
effective in solving MFG problems in separable cases. However, the use of deep
learning instead of finite difference methods has exhibited promise in
extending the applicability of this approach to MFG in the separable case. In
future examinations, we will conduct a thorough assessment to explore the
potential of DPI in addressing more intricate MFG problems.

Test 4: The effectiveness of our approach in high-dimensional cases is tested
in this experiment. We solve the MFG system (11) for dimensions 100 and
present the results in Figure 8. To train the neural networks, we use a
minibatch of 256 samples. The neural networks have a single hidden layer
consisting of 256 neurons each and utilize the Softplus activation function
for $N_{\omega}$ and the Tanh activation function for $N_{\tau}$ and
$N_{\theta}$. We use ADAM with a learning rate of $10^{-4}$, weight decay of
$10^{-4}$, and employ ResNet as the architecture with a skip connection weight
of 0.5. To enhance the result, we incorporated the ideas presented in MFDGM,
employing the approach of simultaneously training the neural networks for each
equation.

Figure 8: The relative Error for $\phi$ and $\rho$ DPI Method for d=100

###  4.2 Trafic flow

To assess the DPI method, we conducted experiments on a traffic flow problem
for an autonomous vehicle used to test the effectiveness of MFDGM. We
specifically chose this problem because it is characterized by a non-separable
Hamiltonian, making it a more complex and challenging problem. In our
experiments, we applied the PI and DPI Methods to the problem and evaluated
its performance in stochastic cases $\nu=0.1$. We consider the traffic flow
problem on the spatial domain $\Omega=[0,1]$ with dimension $d=1$ and final
time $T=1$. The terminal cost $g$ is set to zero and the initial density
$\rho_{0}$ is given by a Gaussian distribution,
$\rho_{0}(x)=0.05-0.9\exp\left(\frac{-1}{2}\left(\frac{x-0.5}{0.1}\right)^{2}\right)$.
The corresponding MFG system is,

$\left\\{\begin{array}[]{rrrrr}\phi_{t}+\nu\Delta\phi-\frac{1}{2}||\phi_{x}||^{2}+(1-\rho)\phi_{x}=0\\\
\rho_{t}-\nu\Delta\rho-\operatorname{div}((\phi_{x}-(1-\rho))\rho)=0\\\
\rho(x,0)=0.05-0.9\exp(\frac{-1}{2}(\frac{x-0.5}{0.1})^{2}),\\\
\phi(x,T)=0.\end{array}\right.\vspace{-0.25cm}$ (13)

Test 1: In this test, we aim to solve (13) with periodic boundary conditions,
we utilized Algorithm 1 with a minibatch size of 50 samples at each iteration
to obtain results. Our approach involved the use of neural networks with
different hidden layers, each consisting of 100 neurons. Specifically, for
$N_{\theta}$, we employed three hidden layers with the Gelu activation
function, while for $N_{\omega}$ and $N_{\tau}$, we used a single hidden layer
with the Sin activation function. During the training process, we employed the
ADAM optimizer with a learning rate of $10^{-4}$ and weight decay of
$10^{-3}$. To construct the neural networks, we adopted the ResNet
architecture, incorporating a skip connection weight of 0.5.
To assess the performance of the methods, we conducted an analysis of the
numerical results presented in Figure 9. This figure provides visual
representations of the density solution as well as the $L^{\infty}$ Distance
between $\rho^{(k)}$, $\phi^{(k)}$ and $q^{(k)}$ from DPI and the final
solution $\rho^{*}$, $\phi^{*}$ and $q^{*}$ from fixed point algorithm.

Figure 9: the density solution obtained with DPI and $L^{\infty}$ Distance
between $\rho^{(k)}$, $\phi^{(k)}$ and $q^{(k)}$ from DPI and the final
solution $\rho^{*}$, $\phi^{*}$ and $q^{*}$ from fixed point algorithm

Test 2: The purpose of this test is to utilize the PI method in order to solve
(13) with periodic boundary conditions. Similar to Test 3, we employ PI
algorithm to solve the fully discretized (13) as follows: we start with an
initial guess $q^{(0)}_{n}:\mathcal{G}\to\mathbb{R}^{2d}$ for $n=0,...,N,$ and
initial and final data $\rho_{0},\phi_{N}:\mathcal{G}\to\mathbb{R}$. We then
iterate on $k\geq 0$ until convergence:
(i) Solve for $n=0,\ldots,N-1$ on $\mathcal{G}$

$\left\\{\begin{array}[]{l}\rho_{n+1}^{(k)}-dt\left(\Delta_{\sharp}\rho_{n+1}^{(k)}+\operatorname{div}_{\sharp}\left(\rho_{n+1}^{(k)}q_{n+1}^{(k)}\right)\right)=\rho_{n}^{(k)}\\\
\rho_{0}^{(k)}=\rho_{0}\end{array}\right.$

(ii) Solve for $n=N-1,\ldots,0$ on $\mathcal{G}$

$\left\\{\begin{aligned}
\phi_{n}^{(k)}&-dt\left(\Delta_{\sharp}\phi_{n}^{(k)}-q_{n,\pm}^{(k)}\cdot
D_{\sharp}\phi_{n}^{(k)}\right)\\\
&=\phi_{n+1}+\frac{dt}{2}\left(\left|q_{n+1,\pm}^{(k)}\right|^{2}+\left|1-\rho_{n+1}^{(k)}\right|^{2}+\left(1-\rho_{n+1}^{(k)}\right)q_{n+1,\pm}^{(k)}\right)\\\
\phi_{N}^{(k)}&=\phi_{N}\end{aligned}\right.$

(iii) Update the policy $q_{n}^{(k+1)}=D_{\sharp}\phi_{n}^{(k)}$ on
$\mathcal{G}$ for $n=0,\ldots,N$, and set $k\leftarrow$ $k+1$. In the
following test, we choose $\gamma=0$, T = 1 for the final time, and $K=50$.
The grid consisted of $I=200$ nodes in space and $N=200$ nodes in time. The
initial policy was initialized as $q^{(0)}_{n}\equiv(0,0)$ on $\mathcal{G}$
for all $n$. Figure 10 presents the numerical results, offering visual
depictions of both the density solution and the $L^{\infty}$ Distance between
$\rho^{(k)}$, $\phi^{(k)}$ and $q^{(k)}$ from policy iteration and the final
solution $\rho^{*}$, $\phi^{*}$ and $q^{*}$ from fixed point algorithm.

Figure 10: the density solution obtained with policy iteration and
$L^{\infty}$ Distance between $\rho^{(k)}$, $\phi^{(k)}$ and $q^{(k)}$ from
policy iteration and the final solution $\rho^{*}$, $\phi^{*}$ and $q^{*}$
from fixed point algorithm

The findings of this study revealed that policy iteration outperforms deep
policy iteration in terms of effectiveness and speed. Consequently, we can
deduce that traditional methods are commonly employed to address such
problems. However, it is important to note that traditional methods have
limitations, such as the curse of dimensionality. Hence, there is a growing
interest in utilizing deep learning approaches as an alternative solution.

##  5 Conclusion

We have presented a novel approach that combines the MFDGM method and the PI
method to tackle high-dimensional stochastic MFG. Our approach exhibits a
higher level of effectiveness, particularly in congestion scenarios, compared
to the MFDGM algorithm. Additionally, we have observed that while our DPI
approach demonstrates effectiveness for general MFG, the traditional PI method
outperforms DPI in terms of effectiveness and speed when dealing with non-
separable Hamiltonians. It can be deduced that traditional methods are
commonly employed to address such problems in low-dimensional settings, but
DPI extends its capabilities to high-dimensional scenarios.

## References

  * [1] M. Assouli, B. Missaoui, Deep learning for mean field games with non-separable hamiltonians, Chaos, Solitons & Fractals 174 (2023) 113802.  doi:https://doi.org/10.1016/j.chaos.2023.113802.
  * [2] S. Cacace, F. Camilli, A. Goffi, A policy iteration method for mean field games, ESAIM: Control, Optimisation and Calculus of Variations 27 (2021) 85.
  * [3] M. Lauriére, J. Song, Q. Tang, Policy iteration method for time-dependent mean field games systems with non-separable hamiltonians, arXiv preprint arXiv:2110.02552 (2021).
  * [4] J.-M. Lasry, P.-L. Lions, Mean field games, Japanese journal of mathematics 2 (1) (2007) 229–260.
  * [5] K. Huang, X. Di, Q. Du, X. Chen, A game-theoretic framework for autonomous vehicles velocity control: Bridging microscopic differential games and macroscopic mean field games, arXiv preprint arXiv:1903.06053 (2019).
  * [6] H. Shiri, J. Park, M. Bennis, Massive autonomous uav path planning: A neural network based mean-field game theoretic approach, in: 2019 IEEE Global Communications Conference (GLOBECOM), IEEE, 2019, pp. 1–6.
  * [7] P. Cardaliaguet, C.-A. Lehalle, Mean field game of controls and an application to trade crowding, Mathematics and Financial Economics 12 (3) (2018) 335–363.
  * [8] P. Casgrain, S. Jaimungal, Algorithmic trading in competitive markets with mean field games, SIAM News 52 (2) (2019) 1–2.
  * [9] Y. Achdou, J. Han, J.-M. Lasry, P.-L. Lions, B. Moll, Income and wealth distribution in macroeconomics: A continuous-time approach, Tech. rep., National Bureau of Economic Research (2017).
  * [10] Y. Achdou, F. J. Buera, J.-M. Lasry, P.-L. Lions, B. Moll, Partial differential equation models in macroeconomics, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372 (2028) (2014) 20130397\.
  * [11] D. A. Gomes, L. Nurbekyan, E. Pimentel, Economic models and mean-field games theory, Publicaoes Matematicas, IMPA, Rio, Brazil (2015).
  * [12] A. T. Lin, S. W. Fung, W. Li, L. Nurbekyan, S. J. Osher, Apac-net: Alternating the population and agent control via two neural networks to solve high-dimensional stochastic mean field games, arXiv preprint arXiv:2002.10113 (2020).
  * [13] J.-M. Lasry, P.-L. Lions, Jeux à champ moyen. ii–horizon fini et contrôle optimal, Comptes Rendus Mathématique 343 (10) (2006) 679–684.
  * [14] Y. Achdou, A. Porretta, Mean field games with congestion, Annales de l’Institut Henri Poincare (C) Non Linear Analysis 35 (06 2017).  doi:10.1016/j.anihpc.2017.06.001.
  * [15] D. A. Gomes, V. K. Voskanyan, Short-time existence of solutions for mean-field games with congestion, Journal of the London Mathematical Society 92 (3) (2015) 778–799.
  * [16] Y. Achdou, P. Cardaliaguet, F. Delarue, A. Porretta, F. Santambrogio, Y. Achdou, M. Laurière, Mean field games and applications: Numerical aspects, Mean Field Games: Cetraro, Italy 2019 (2020) 249–307.
  * [17] P. Hammer, Adaptive control processes: a guided tour (r. bellman) (1962).
  * [18] R. Bellman, Dynamic programming, Science 153 (3731) (1966) 34–37.  arXiv:https://www.science.org/doi/pdf/10.1126/science.153.3731.34, doi:10.1126/science.153.3731.34.
URL https://www.science.org/doi/abs/10.1126/science.153.3731.34

  * [19] H. Cao, X. Guo, M. Laurière, Connecting gans, mfgs, and ot, arXiv preprint arXiv:2002.04112 (2020).
  * [20] J. Sirignano, K. Spiliopoulos, Dgm: A deep learning algorithm for solving partial differential equations, Journal of computational physics 375 (2018) 1339–1364.
  * [21] M. Raissi, Deep hidden physics models: Deep learning of nonlinear partial differential equations, The Journal of Machine Learning Research 19 (1) (2018) 932–955.