# Multisource Holography

Grace Kuo  Reality Labs Research, MetaUSA ,  Florian Schiffers  Reality Labs
Research, MetaUSA ,  Douglas Lanman  Reality Labs Research, MetaUSA ,  Oliver
Cossairt  Reality Labs Research, MetaUSA and  Nathan Matsuda  Reality Labs
Research, MetaUSA

###### Abstract.

Holographic displays promise several benefits including high quality 3D
imagery, accurate accommodation cues, and compact form-factors. However,
holography relies on coherent illumination which can create undesirable
speckle noise in the final image. Although smooth phase holograms can be
speckle-free, their non-uniform eyebox makes them impractical, and speckle
mitigation with partially coherent sources also reduces resolution. Averaging
sequential frames for speckle reduction requires high speed modulators and
consumes temporal bandwidth that may be needed elsewhere in the system.

In this work, we propose multisource holography, a novel architecture that
uses an array of sources to suppress speckle in a single frame without
sacrificing resolution. By using two spatial light modulators, arranged
sequentially, each source in the array can be controlled almost independently
to create a version of the target content with different speckle. Speckle is
then suppressed when the contributions from the multiple sources are averaged
at the image plane. We introduce an algorithm to calculate multisource
holograms, analyze the design space, and demonstrate up to a 10 dB increase in
peak signal-to-noise ratio compared to an equivalent single source system.
Finally, we validate the concept with a benchtop experimental prototype by
producing both 2D images and focal stacks with natural defocus cues.

Figure 1.  We propose a new architecture for holographic displays specifically
designed for speckle reduction. Instead of a single coherent source of
illumination, our design uses a grid of multiple sources, which sum
incoherently at the image plane. By using two spatial light modulators (SLMs)
with an air gap in between, we break correlations between the multiple sources
enabling high resolution holograms with significantly suppressed speckle. We
experimentally demonstrate speckle reduction on both 2D images (left) and
focal stacks with natural defocus blur (right).

##  1\. Introduction

Computer generated holography uses a spatial light modulator (SLM) to mimic
the wavefront coming from a three-dimensional (3D) object. This enables high
resolution displays with accurate per-pixel focal cues, and recent user
studies demonstrated that holographic displays have the potential to drive the
human accommodation response (Kim et al., 2022b), offering a solution to the
vergence-accommodation conflict of stereoscopic displays (Hoffman et al.,
2008). Holography is a particularly promising technology for head-mounted
displays (HMDs) since it also enables compact form-factors, can compensate for
optical aberrations, and can correct for eyeglass prescriptions entirely in
software (Maimone et al., 2017).

However, holographic displays rely on spatially coherent illumination to
achieve 3D cues (Lee et al., 2020a), which can create speckle in the displayed
content. Speckle is a phenomenon that occurs with coherent light when random
path length differences interfere at the image plane, creating a noisy pattern
of dark and bright spots due to random constructive and destructive
interference (Goodman, 2007). This effect is undesirable since it hides
details in the hologram and creates noisy, visually unappealing images.
Although reducing the illumination coherence can suppress speckle, it also
reduces resolution and depth of field (Deng and Chu, 2017).

Smooth phase holograms offer a different option to control speckle: by
removing randomness in the image plane phase, all interference is constructive
and speckle is eliminated (Maimone et al., 2017). However, these holograms
have highly non-uniform energy distribution in the eyebox, greatly reducing
practicality (Yoo et al., 2021). In addition, the focal cues generated are
limited and often exhibit unnatural ringing. In user studies, these focal cues
were not effective at driving accommodation (Kim et al., 2022b).

Another option for speckle reduction is temporal multiplexing, when several
frames with unique speckle patterns are shown in rapid sequence such that they
are visually averaged by the eye. However, this requires high speed SLMs with
frame rates in the kilohertz range, and the amount of despeckling increases
sub-linearly with the number of averaged frames. Reducing the number of frames
needed for speckle control could increase the amount of speckle reduction with
the same number of frames, could allow for more flexibility choosing
modulators, or could free temporal bandwidth to address other challenges, for
example, increasing eyebox size or field of view (FoV) through scanning.

We propose a novel architecture for speckle reduction in holographic displays
that can create natural defocus cues with a uniform eyebox at the full SLM
resolution, all in a single frame. To do this, we modify the illumination
setup of a traditional holographic display, which typically consists of a
single light source that generates a coherent plane wave at the SLM. In our
architecture, we replace this single source with a grid of multiple sources,
which each generate a plane wave at a different angle of incidence. By using
sources that are incoherent with each other, the speckle patterns from each
source average at the image plane, reducing speckle contrast.

However, with a single SLM, each source creates a shifted copy of the same
hologram, creating haze and doubling in the displayed content. To address
this, we propose using two SLMs spaced a few millimeters apart axially. This
arrangement creates a modulation response that varies with the angle of
incidence, similar to how volume holograms use their thickness to create
angular selectivity (Heanue et al., 1994). With the two SLMs, we can
independently control the output image from each source, removing the doubling
artifacts while continuing to get the speckle-reduction benefits of the
multiple sources. We refer to this architecture, including both the array of
sources and the two SLMs, as multisource holography.

In summary, we make the following contributions:

  * •

We introduce the multisource holography architecture and a corresponding
hologram generation algorithm. We demonstrate full resolution holograms with
natural defocus cues and uniform eyebox in a single frame.

  * •

In simulation, we demonstrate improvements of 10 dB in peak signal-to-noise
ratio (PSNR) compared to an equivalent single source system with the same
degrees of freedom. We further show that multisource holography with no
temporal averaging outperforms temporal multiplexing with 6 jointly optimized
frames.

  * •

We analyze how the source spacing and number of sources impact hologram
quality and provide guidance on the minimum spacing needed to achieve full
resolution.

  * •

We validate the multisource holography concept with a full-color benchtop
prototype. We introduce a customized calibration procedure and experimentally
demonstrate low speckle holograms for both planar images and focal stacks with
natural blur.

##  2\. Related Work

#### Smooth Phase Holograms

As described above, smooth phase holograms eliminate speckle by enforcing
near-constant phase at the image plane, which removes randomness and ensures
interference between neighboring points is always constructive. Enforcing a
specific phase at the image plane requires complex modulation at the SLM, so a
practical option is the double phase amplitude coding (DPAC) method, which can
almost entirely remove speckle (Maimone et al., 2017; Shi et al., 2021). Even
without complex modulation, one can achieve low phase variation at the image
plane through gradient descent with uniform phase initialization at the SLM
(Chakravarthula et al., 2019) or by explicitly enforcing a piece-wise constant
phase in the loss function (Choi et al., 2021a). Although these smooth phase
approaches can create high quality and speckle-free two dimensional (2D)
images, defocus blur is limited and contains unnatural ringing. To address
this issue, holograms can be optimized to target natural-appearing blur while
encouraging image phase to remain smooth (Kavaklı et al., 2023; Yang et al.,
2022). However, the amount of blur is still limited, and all smooth phase
holograms concentrate energy into a small region of the eyebox, making these
systems very sensitive to eye movement and imperfections in the user’s eye.
Furthermore, in a recent user study, smooth phase holograms were not effective
at driving accommodation (Kim et al., 2022b).

#### Random Phase Holograms

Random phase at the image plane, which generates scattering similar to a
diffuse object, enables natural defocus blur and uniform energy distribution
in the eyebox (Yoo et al., 2021). However, this same randomness reintroduces
speckle due to interference from different random path lengths. For 2D images,
by letting the phase at the image plane be a free variable, one can use
iterative approaches to shape the phase such that speckle is minimized from a
particular viewing angle (Saxton and Gerchberg, 1972; Fienup, 1982). Adding
spatial “don’t care” regions can further enhance image quality in the regions
of interest (Georgiou et al., 2008). However, for 3D content, the number of
degrees of freedom on the SLM is insufficient to suppress speckle everywhere
at once. One option is to let out-of-focus content be unconstrained, which
enables better in-focus imagery but creates additional speckle in defocused
regions (Choi et al., 2021a; Kuo et al., 2020). Generating natural defocus
blur with low speckle over the whole volume requires additional despeckling
approaches that cannot be achieved through the algorithm alone.

#### Partial Coherence

Decreasing the coherence of the illumination can reduce speckle by imposing an
incoherent blur on the output image through wavelength diversity (temporal
partial coherence) or angular diversity (spatial partial coherence) (Deng and
Chu, 2017; Zhao et al., 2022). Spatial partial coherence in holographic
displays has been demonstrated using an echelon stair (Lee et al., 2019), and
temporal partial coherence has been demonstrated with different light sources,
such as light emitting diodes (LEDs) (Moon et al., 2014; Kozacki et al., 2022)
and superluminescent LEDs (SLEDs) (Primerov et al., 2019; Peng et al., 2021).
However, partially coherent sources result in a direct trade-off between
resolution and speckle reduction, which is incompatible with a high
resolution, low speckle display. Lee et al. (2020a) designed a partially
coherent light source specifically to balance resolution, depth of field, and
speckle, but the trade-off still exists so despeckling is limited without
further resolution reduction. Similar coherence properties have been explored
in the context of interferometric 3D sensing (Kotwal et al., 2023) and
transmission matrix characterization (Gkioulekas et al., 2015; Kotwal et al.,
2020), and we refer interested readers to these sources for an in-depth
analysis.

#### Temporal Multiplexing

To achieve despeckling without sacrificing resolution, one can display many
holograms in sequence, each with a unique speckle pattern. Due to human
persistence of vision, the user sees an average of the displayed images,
effectively suppressing speckle. Systems with 8 to 24 frames of temporal
multiplexing per color have been demonstrated with high speed modulators such
as digital micromirror devices (DMDs) (Lee et al., 2020b; Curtis et al.,
2021), ferro-electronic liquid crystal on silicon (FLCoS) (Lee et al., 2022),
and micro-electromechanical systems (MEMS) (Choi et al., 2022). These prior
works achieve state-of-the-art image quality for temporal multiplexing by
jointly optimizing all frames and accounting for the limited bit depth of
these high speed SLMs. However, to create a fully life-like HMD, one needs to
refresh content at least 1.8 kHz (Cuervo et al., 2018); achieving these
refresh speeds with temporal multiplexing requires updating content between
sub-frames, which current algorithms do not support.

In addition, reducing the number of frames needed for speckle control could
free the temporal bandwidth for other uses. For example, Lee et al. (2020b)
demonstrated increased viewing angle (in other words, increased étendue) by
scanning illumination angle over time. Approaches like these could help
overcome the fundamental étendue limits (Park and Lee, 2022) of holographic
displays, but they reduce the amount of temporal bandwidth available for
despeckling.

#### Multiple Modulators

Our system is capable of reducing speckle in a single frame through the use of
two cascaded SLMs, taking advantage of the compression that layered displays
can provide. In conventional optics, layered modulators can be used to break
the trade-off between spatial and angular resolution in light field displays
(Wetzstein et al., 2012). Similarly, in diffractive optics, Ye et al. (2014)
showed that static layered diffractive elements can control the 4D
bidirectional reflectance distribution function (BRDF) under incoherent
illumination, and Peng et al. (2017) used pairs of static diffractive optical
elements (DOEs) to create different holograms based on the relative
translations between the DOEs. Although these prior publications have
different application spaces, they demonstrate that two layered modulators can
create several (more than two) different images, highlighting the compressive
nature of the layered displays.

In our system, we take advantage of compression in layered displays to achieve
more despeckling than in non-compressive systems (such as temporal
multiplexing) with the same number of degrees of freedom. We note that
interferometer-inspired setups (Choi et al., 2021b; Wang et al., 2019), which
also use multiple SLMs for image quality enhancement, are not designed to take
advantage of potential compression, and therefore have limited speckle
reduction based on the degrees of freedom in the two modulators.

#### Multiple Incoherent Sources

Despeckling in our system is achieved through multiple discrete sources of
illumination that are incoherent with each other. To our knowledge, the only
prior work with similar illumination is that of Jo et al. (2022) in which
multiple sources are used for étendue expansion while simultaneously providing
some despeckling. Like our work, they show that multiple sources with a single
modulation plane create uncontrollable replicas in the final image. However,
they use a binary amplitude mask in the Fourier plane to break correlations
between replicas, as where we use a second SLM with a small air gap, which is
more amenable to a compact system and provides additional degrees of freedom
for better image quality. Finally, since Jo et al. (2022) target étendue
expansion as their application, they fix the number and locations of the
sources such that, at any position in the image, a maximum of 9 different
sources are averaged for speckle reduction. We demonstrate that speckle
reduction can be dramatically increased with more sources, and we analyze the
effect of number of sources and source spacing on image quality.

#### Camera-Based Calibration

Even if speckle is theoretically reduced, any non-idealities in the optical
system can cause additional speckle in practice due to mismatch between the
model used in optimization and the true system. To account for imperfections,
one can design a model of the optical system with learnable parameters, then
fit the unknowns in an offline calibration process using experimentally
captured data (Peng et al., 2020; Choi et al., 2021a; Chakravarthula et al.,
2020; Kavaklı et al., 2022). A special case of camera-based calibration is the
“active” approach proposed by Peng et al. (2020), in which the SLM pattern is
fine-tuned online to a particular image based on camera feedback. Although
these holograms do not generalize to new content, they highlight what is
feasible with a given experimental system. To best demonstrate the potential
of multisource holography, we use both offline calibration with a physically-
based model and online active camera-in-the-loop.

Figure 2.  System Architecture: Multisource holography uses an array of
mutually incoherent sources that each generate a plane wave at a different
angle. (a) With a single SLM, all sources are modulated with the same pattern
but propagate in different directions, creating replicas of the content.
Generating an image with this configuration is a poorly posed problem. (b) We
propose adding a second SLM a small distance $\Delta z$ in front of the first.
This enables different modulation function for different angles of incidence,
enabling unique content for each source. By jointly optimizing the two SLM
patterns, the holograms from each source line up correctly, removing replica
artifacts. Since the sources are incoherent with each other, their intensities
add at the image plane which suppresses speckle through averaging.

##  3\. System Overview

A traditional holographic display uses an SLM to shape an incoming coherent
beam to form an image. Denoting the complex modulation function of the SLM as
$\mathbf{s}(\vec{x})$, we can write the image formation model as

(1) $\displaystyle\mathbf{g}_{z}(\vec{x})$
$\displaystyle=\mathcal{P}_{z}\big{\\{}\mathbf{p}(\vec{x})\odot\mathbf{s}(\vec{x})\big{\\}},$
$\displaystyle\mathbf{I}_{z}(\vec{x})$
$\displaystyle=|\mathbf{g}(\vec{x})|^{2},$

where $\vec{x}$ is the 2D spatial coordinate at the SLM,
$\mathbf{g}_{z}(\cdot)$ and $\mathbf{I}_{z}(\cdot)$ are, respectively, the
electric field and intensity a distance $z$ from the SLM, and
$\mathbf{p}(\cdot)$ is the complex field illuminating the SLM, which is most
commonly a plane wave of unit energy, $\mathbf{p}(\vec{x})=1$. Finally,
$\odot$ denotes pointwise multiplication, and $\mathcal{P}_{z}\\{\cdot\\}$ is
the angular spectrum method (ASM) propagation operator defined as

(2) $\displaystyle\mathcal{P}_{z}\left\\{\mathbf{s}(\vec{x})\right\\}$
$\displaystyle=\mathcal{F}^{-1}\bigl{\\{}\mathcal{F}\\{\mathbf{s}(\vec{x})\\}\odot\mathcal{H}_{z}(\vec{u})\bigr{\\}},$
(3) $\displaystyle\mathcal{H}_{z}(\vec{u})$
$\displaystyle=\begin{cases}\exp\left(\frac{2\pi
jz}{\lambda}\sqrt{1-\|\lambda\vec{u}\|^{2}}\right),&\text{ if
}\sqrt{\|\vec{u}\|^{2}}<\frac{1}{\lambda},\\\ 0,&\text{ otherwise,
}\end{cases}$

where $\mathcal{F}\\{\cdot\\}$ is the 2D Fourier transform operator and
$\vec{u}$ is the 2D coordinate in frequency space (Matsushima and Shimobaba,
2009). Here, we assume monochromatic illumination with wavelength $\lambda$;
see Supplement for an extension to broadband sources.

To generate a hologram, one can use first order methods like gradient descent
to optimize for an SLM pattern that creates a given target image:

(4)
$\displaystyle\mathbf{s}=\operatorname*{argmin}_{\mathbf{s}}\sum_{z}\|\mathbf{I}_{z}(\vec{x})-\mathbf{\hat{I}}_{z}(\vec{x})\|_{2}^{2},$

where $\mathbf{\hat{I}}_{z}(\vec{x})$ is the target intensity at a given
plane, and optimization is performed over a dense range of propagation
distances in the volume of interest.

To encourage natural defocus cues, we render the target images with realistic
blur based on incoherent illumination where the blur kernel size is determined
by the maximum diffraction angle of the SLM (see Supplement for details).
However, the holographic display aims to control a 3D volume of light using
only a single 2D SLM pattern, making the optimization problem overdetermined.
As a result, the 3D volume cannot be matched exactly and uncontrollable
speckle noise is visible in the image, particularly as the image volume grows.

###  3.1. Despeckling with Multiple Illumination Sources

Our goal is to reduce speckle in holographic displays. Our basic strategy is
common in the despeckling literature: produce several versions of the image
that each have a unique speckle pattern, and when these copies of the image
are superimposed, the speckle is reduced through averaging (Goodman, 2007). To
create different versions of the image, we propose using multiple sources of
illumination. When sources are placed at different locations behind a
collimating lens, as shown in Fig. 2a, each source illuminates the SLM from a
different angle:

(5)
$\displaystyle\mathbf{p}(\vec{x};\vec{m_{i}})=e^{j(\vec{x}\cdot\vec{m_{i}})},$

where $\cdot$ denotes inner product and $\vec{m_{i}}$ is the phase slope (in
radians per meter) of the $i$-th source at the SLM plane, which is related to
illumination angle of incidence by

(6) $\displaystyle\vec{\theta}=\frac{\lambda\vec{m}_{i}}{2\pi}.$

Note that, unlike the work of Jo et al. (2022), we are not using the sources
to expand étendue. Therefore, we choose small slopes for $\vec{m}_{i}$, within
the range of angles that the SLM is able to create natively.

If the different sources are all incoherent with each other, they will not
exhibit interference effects at the image plane when they are combined.
Instead, the multisource image is the sum of the individual source intensities
as follows:

(7) $\displaystyle\mathbf{g}_{z,m_{i}}(\vec{x})$
$\displaystyle=\mathcal{P}_{z}\big{\\{}\mathbf{p}(\vec{x},\vec{m}_{i})\odot\mathbf{s}(\vec{x})\big{\\}}$
(8) $\displaystyle\mathbf{I}_{z}(\vec{x})$
$\displaystyle=\sum_{i}\big{|}\mathbf{g}_{z,m_{i}}(\vec{x})\big{|}^{2}$

This achieves part of our goal: each source creates a unique speckle pattern,
so speckle contrast is reduced when the individual source intensities are
combined. However, for a useful display, the final output intensity
$\mathbf{I}_{z}$ should have the potential to be shaped into arbitrary target
images, which is not the case in this configuration. To demonstrate the
problem, we make the small angle approximation to the ASM kernel, and derive
the following relationship (see Supplement for derivation):

(9)
$\displaystyle\mathbf{g}_{z,m_{i}}(\vec{x})=\mathbf{g}_{z,0}(\vec{x}+\tfrac{\lambda
z}{2\pi}\vec{m}_{i})\odot e^{j(\vec{x}\cdot\vec{m}_{i})}$

where $\mathbf{g}_{z,0}(\cdot)$ is the electric field from an on-axis plane
wave ($\vec{m}=\vec{0}$). This means that the output electric field from the
$i$-th source is a translated copy of electric field with on-axis
illumination, up to a carrier wave. In other words, a single ideal SLM has
infinite memory effect (Freund et al., 1988).

Based on Eqs. 8 and 9, the total intensity with all the sources can be written
as

(10)
$\displaystyle\mathbf{I}_{z}(\vec{x})=\big{|}\mathbf{g}_{z,0}(\vec{x})\big{|}^{2}*\sum_{i}\delta(\vec{x}+\tfrac{\lambda
z}{2\pi}\vec{m}_{i}),$

where $*$ denotes a 2D convolution. Therefore, producing a given output image
$\mathbf{I}_{z}$ requires deconvolving the source locations. This is a very
poorly posed problem, regardless of the $\vec{m}_{i}$ used, since the result
of the deconvolution, $\big{|}\mathbf{g}_{z,0}(\vec{x})\big{|}^{2}$, is a
physical quantity that must be nonnegative. As a result, multiple sources with
a single SLM is not a viable solution for holographic displays.

Figure 3.  Single Source vs. Multisource (Simulation): In simulation, we
compare four methods for generating a target focal stack (a) with natural
defocus cues. (b) A traditional single source hologram optimized with smooth
phase has no speckle but there are ringing artifacts in the defocused regions.
More importantly, the energy distribution in the eyebox (bottom row) is
extremely non-uniform (note that plots are logarithmic) with a large peak in
the center, which makes the display sensitive to eye imperfections and
requires precise, low latency eye tracking and 3D pupil steering for a usable
display. (c) A single source hologram with random phase achieves an
approximately uniform eyebox distribution, but the image is corrupted by
severe speckle. (d) Multiple sources reduce speckle, but with a single SLM,
correlations between the outputs of each source create haze and doubling in
the displayed hologram. (e) Our multisource holography approach uses two SLMs
(here, one phase SLM and one amplitude SLM) to break correlations between the
individual source outputs. This removes the low frequency artifacts in (d)
while preserving the speckle reduction. Although (e) uses two SLMs, all
simulations have the same degrees of freedom since (b)-(d) are simulated with
a single complex SLM. Out of these approaches, only multisource holography is
capable of creating high quality focal stacks with a practical energy
distribution in the eyebox.

###  3.2. Multisource Holography with Two Modulators

In order to display arbitrary content with multiple incoherent sources, there
cannot be strong correlations between the different source outputs. We want
each angle of illumination to generate a unique pattern, and this requires
that the modulator have an angularly selective response. We achieve this
requirement by adding a second SLM a distance $\Delta z$ from the first, as
shown in Fig. 2b, which yields the following image formation model:

(11)
$\displaystyle\mathbf{I}_{z}(\vec{x})=\sum_{i}\bigg{|}\mathcal{P}_{z}\Big{\\{}\mathcal{P}_{\Delta
z}\big{\\{}\mathbf{p}(\vec{x};\vec{m}_{i})\odot\mathbf{s}_{1}(\vec{x})\big{\\}}\odot\mathbf{s}_{2}(\vec{x})\Big{\\}}\bigg{|}^{2},$

where $\mathbf{s}_{1}(\cdot)$ and $\mathbf{s}_{2}(\cdot)$ are the modulation
functions of the two SLMs. To see how the second SLM breaks the correlation
between sources, let $\mathbf{g}_{\Delta z,0}(\vec{x})$ be the electric field
just before the second SLM given on-axis illumination. Applying Eq. 9, we can
describe the electric field after the second SLM as

(12) $\displaystyle\mathbf{g}_{\Delta z,0}(\vec{x}+\tfrac{\lambda\Delta
z}{2\pi}\vec{m}_{i})\odot\mathbf{s}_{2}(\vec{x})\odot
e^{j(\vec{x}\cdot\vec{m}_{i})}.$

Here, the electric field is translated based on the source angle, then
pointwise multiplied by the modulation function of the second SLM. As long as
the relative translation between any two sources is at least one SLM pixel,
then the output fields (and therefore the final intensities) will be
substantially decorrelated, breaking the memory effect (Freund et al., 1988).
This gives the following condition on the source spacing:

(13) $\displaystyle\Delta m\geq\frac{2\pi p}{\lambda\Delta z},$

where $\Delta m$ is the spacing between sources ($\Delta
m=\|\vec{m}_{i}-\vec{m}_{j}\|^{2}$), and $p$ is the SLM pixel pitch, assumed
be the same for both SLMs.

As long as Eq. 13 is met, our multisource holography setup can create
different content for each source. Conceptually, each source “sees” a
different relative translation between the two SLM patterns. Therefore, we
would like to design the SLMs so each of these translations creates the
desired target image for each source. This is similar to the work of Peng et
al. (2017), where pairs of static DOEs are combined with different
translations to create unique images. Based on their results, where several
unique holograms were created from a single DOE pair, we expect our system can
also create the desired output for more than two sources simultaneously, even
though there are only two SLMs. In other words, we expect the system to be
compressive, which allows our system to generate more incoherent copies of the
image, resulting in more despeckling, than other systems (for example,
temporal multiplexing) with the same number of degrees of freedom.

In practice, we jointly solve for both SLM patterns using the using the model
in Eq. 11 and solving the optimization problem,

(14)
$\displaystyle\mathbf{s}_{1},\mathbf{s}_{2}=\operatorname*{argmin}_{\mathbf{s}_{1},\mathbf{s}_{2}}\sum_{z}\|\mathbf{I}_{z}(\vec{x})-\mathbf{\hat{I}}_{z}(\vec{x})\|_{2}^{2},$

using ADAM (Kingma and Ba, 2014).

Figure 4. Source Configuration Analysis: We assess the impact of source
spacing (a) and number of sources (b) on peak signal-to-noise-ratio (PSNR) for
a natural scene (top) and contrast at 1 arcmin (bottom). Here, we assume the
system is scaled so 1 arcmin corresponds to the maximum SLM resolution. (a)
When sources are close together, all sources are within the memory effect
region (i.e. do not meet Eq. 13), so each source generates a similar output,
creating blur in the final image. Although a small blur increases PSNR, it
decreases resolution creating a dip in the contrast metric at small spacings.
As the source spacing increases, more sources leave the memory effect region
and contrast at 1 arcmin increases, demonstrating that full resolution is
possible when the sources are spaced sufficiently far apart. Example images at
two different spacings (indicated by the orange dots) are shown on the right.
(b) As the number of sources grows, PSNR increases due to better speckle
suppression. However, for large numbers of sources the SLMs cannot fully
control the outputs of all sources, creating haze in the final image (see 100
source example). This effect is captured by the contrast metric, which
decreases after about 36 sources.

##  4\. Simulation

To demonstrate the improvements that multisource holography provides, we
optimize holograms in simulation to generate a focal stack with rendered
incoherent blur, similar to what one would see in a natural scene (Fig. 3a).
Our focal stack, shown in Fig. 3a, covers a $10\text{\,}\mathrm{mm}$ range in
SLM space (from $z=$15\text{\,}\mathrm{mm}$$ to $z=$25\text{\,}\mathrm{mm}$$)
with a blur radius of 4 pixels per millimeter of defocus, matched to the
maximum diffraction angle of an SLM with $8\text{\,}\mathrm{\SIUnitSymbolMicro
m}$ pixels (see Supplement for an explanation of these parameters).
Simulations are conducted in red-green-blue
($\lambda=$640\text{\,}\mathrm{nm}$,$520\text{\,}\mathrm{nm}$,$450\text{\,}\mathrm{nm}$$)
assuming monochromatic illumination, and we supervise the loss at 15 evenly
spaced planes. Optimization is implemented in PyTorch on an Nvidia A6000 GPU
at $2\times$ the SLM resolution in each direction to avoid aliasing.

###  4.1. Single Source Holograms

Using the traditional configuration with a single source and single SLM, it’s
difficult to create practical, high quality holograms with natural defocus. To
demonstrate the challenges, we solve Eq. 4 using the model in Eq. 1, where we
assume a complex SLM. Although most off-the-shelf modulators control either
phase or amplitude, but not both, we choose a complex SLM for this simulation
as it has the same number of degrees of freedom as our 2-SLM multisource
approach.

For single source holograms, the SLM initialization has a big impact on the
result. We consider two different initializations: constant (in both phase and
amplitude) versus uniform random. In both cases, after initialization we
iteratively optimize the SLM pattern using ADAM in Pytorch based on the loss
function in Sec. 3; however, even after optimization the final SLM pattern is
influenced by the starting point. For example, with constant initialization,
the phase of $\mathbf{g}_{z}(\vec{x})$ tends to be low variance (Yoo et al.,
2021), resulting in a smooth phase hologram (Fig. 3b). Similarly, with random
initialization, $\mathbf{g}_{z}(\vec{x})$ tends to be high variance, resulting
in a random phase hologram (Fig. 3c). As shown in Fig. 3b, the smooth phase
simulation has low speckle noise, but exhibits unnatural ringing in defocused
regions. In contrast, the random phase hologram has more natural defocus
effects but contains substantial speckle noise.

Although ringing may seem like an acceptable trade-off for speckle removal,
smooth phase holograms are impractical for near-eye displays due to their
eyebox energy distribution, shown in the bottom row for the green channel. The
eyebox, which is created by an eyepiece of focal length $f$, is the area where
the user’s pupil is located (see Fig. 2). The electric field at the eyebox,
$\mathbf{e}(\cdot)$, is described by

(15)
$\displaystyle\mathbf{e}(\vec{u})=\mathcal{F}\\{\mathbf{g}_{z_{0}}(\vec{x})\\},$

where $z_{0}$ is the propagation distance from the SLM to the focal plane of
the eyepiece, and $\vec{u}$ is the spatial coordinate at the eyebox
($\vec{u}=\vec{x}/\lambda f$) (Goodman, 2005).

The eyebox of the smooth phase hologram (Fig. 3b, bottom) has a very strong
peak in the center, with almost 5 orders of magnitude more energy in the peak
than in the eyebox periphery. This presents several challenges for a practical
display since the eyebox energy is mostly concentrated into an area only a
handful of microns across. First, this means that small eye movements, even
those contained within the theoretical eyebox (Kuo et al., 2020), cause the
eye to miss the peak, and then the user will not see the image. See Fig. 9 for
an example of this effect. Second, since the light is concentrated into a
small point on the user’s pupil, “floaters” (debris in the vitreous humor) or
other imperfections in the eye can cause substantial artifacts in the
hologram. These imperfections are barely noticeable in daily life since the
image on the retina is typically an integral over the full pupil; however, in
a smooth phase hologram, only a small part of the pupil is sampled.
Computationally removing the effects of floating debris is unrealistic as it
would require detailed, real-time mapping of every user’s eyes. Even if eye
imperfections could be overcome, user studies suggest that the small eyebox of
smooth phase holograms cannot effectively drive accommodation: Kim et al.
(2022b) found much lower accommodative gain for smooth phase holograms
compared to random phase holograms. As a result of all of these restrictions,
we believe smooth phase holograms cannot achieve compelling 3D content with
good image quality for all users.

Random phase holograms, on the other hand, simulate a diffuse surface at the
object, which scatters light to cover the full theoretical eyebox (Fig. 3c,
bottom), but this comes at the cost of speckle. Although random phase
holograms can be low speckle for a 2D scene, for a 3D focal stack the degrees
of freedom on the SLM are insufficient to control speckle at all planes, even
with a complex modulator. Not only does this speckle hide detail and make
images visually unappealing, the high frequency speckle can also interfere
with the human accommodation response which expects low spatial frequencies in
defocused regions (Kim et al., 2022b). As a result, with a single source,
neither smooth nor random phase holograms can produce high quality images that
drive accommodation without additional speckle reduction.

###  4.2. Multisource Holograms

Our multisource holography approach achieves the benefits of random phase
holograms while adding substantial despeckling to reduce noise and produce
more natural defocus cues. However, as described in Sec. 3.1, adding more
sources with a single SLM results in a poorly posed optimization problem that
is not able to display arbitrary content. Figure 3d shows an example with a
$4\times 4$ grid of sources and a single complex SLM. Although there is
substantial noise reduction compared to random phase with a single source, the
resulting image contains low frequency artifacts, as expected, due to the
strong correlations between the outputs of each source. However, even with a
single SLM, the multisource hologram is able to create an approximately
uniform eyebox when initialized with a random pattern, albeit with some
periodic structure due to the sources.

Our final design uses an array of sources with two SLMs, as described in Sec.
3.2, where the gap between the SLMs creates an angularly selective response
that breaks the correlations between sources. Figure 3e shows a simulation of
this configuration with a $4\times 4$ grid of sources such that all sources
are outside the memory effect region (Eq. 13) for all wavelengths of interest.
Of our two SLMs, spaced $\Delta z=$2\text{\,}\mathrm{mm}$$ apart, the first
SLM modulates phase only, and the second SLM modulates amplitude only,
creating the exact same degrees of freedom as in the prior simulations. We
initialize the SLMs with uniform random phase and amplitude, respectively.
Figure 3e demonstrates that the second SLM successfully breaks the
correlations between the sources, removing the low frequency artifacts of Fig.
3d while substantially suppressing speckle compared to Fig. 3c. This
simulation shows that multisource holography can create natural defocus cues
with low speckle, no ringing artifacts, and uniform energy distribution in the
eyebox.

Figure 5. Comparison with Temporal Multiplexing (Simulation): Our multisource
approach with no temporal multiplexing (one frame per color) outperforms a
traditional single source hologram with a phase only SLM and 6 jointly
optimized frames per color when generating a focal stack with natural blur.
Our multisource simulation uses $5\times 5$ sources and $\Delta m=75$ rad/mm.

###  4.3. Source Configuration Analysis

The number of sources and their arrangement are key design choices in
multisource holography, so next we analyze the impact of these parameters.
Figure 4a illustrates the effect of source spacing. Sources were arranged in a
$4\times 4$ grid and the distance between neighboring sources, $\Delta m$, was
varied. We simulate a $\Delta z=$2\text{\,}\mathrm{mm}$$ gap between the SLMs,
and as before, we use a phase SLM as the first modulator and an amplitude SLM
as the second, each with an $8\text{\,}\mathrm{\SIUnitSymbolMicro m}$ pixel
pitch. Simulations were done for a single wavelength of
$520\text{\,}\mathrm{nm}$, and the number of sources within the memory effect
region, defined in Eq. 13, are indicated by the background color in the plots.

Figure 4a (top plot) shows PSNR as a function of $\Delta m$ for a natural
scene. When the sources are within the memory effect region of the two SLMs,
they create correlated patterns. Similar to the scenario with only one SLM
(Sec. 3.1), the resulting output image is described by a convolution (Eq. 10).
In this case, since the sources are close together, this creates a small blur
instead of the dramatic ghost artifacts in Fig. 3d. Since this blur reduces
noise effectively, and PSNR is not a metric that’s sensitive to high
resolution features, the PSNR is highest at small source spacing. However,
this blur is not desirable for a high resolution holographic display.

To quantify the system’s ability to display high frequency features, we
simulate a binary grating with a period of two SLM pixels, the highest spatial
frequency the SLM can produce. We optimize for a focal stack and measure the
Michelson contrast,
$(I_{\text{max}}-I_{\text{min}})/(I_{\text{max}}+I_{\text{min}})$ in focus,
averaged over a $100\times 100$ pixel area. Assuming an
$8\text{\,}\mathrm{\SIUnitSymbolMicro m}$ SLM pixel and an eyepiece with focal
length $f=$27.5\text{\,}\mathrm{mm}$$, this corresponds to the contrast at 30
cycles per degree or 1 arcmin resolution, on par with the human visual system.
We test with the focal plane at three different locations in the volume
($z=$15.7\text{\,}\mathrm{mm}$,$20\text{\,}\mathrm{mm}$,$24.3\text{\,}\mathrm{mm}$$)
and report the average contrast.

Figure 4a (bottom plot) shows this contrast as a function of source spacing.
When $\Delta m=0$, the sources are on top of each other. This is equivalent to
a single source, which, although noisy, can display high resolution features.
Once the sources move slightly apart, they are fully within the memory effect
region of the two SLMs, so the the output is blurred and contrast drops. As
the spacing between the sources increases, progressively more sources leave
the memory effect region and the contrast at 1 arcmin increases, demonstrating
that multisource holography can create high resolution features when sources
are spaced sufficiently far apart. Since the memory effect cutoff (Eq. 13)
also depends on the gap between the SLMs, a similar trend holds when $\Delta
z$ is varied; see Supplement for an analysis of $\Delta z$.

Next, we consider how the number of sources impacts hologram quality. Figure
4b shows PSNR (top) and contrast at 1 arcmin (bottom) as a function of the
number of sources. Sources are arranged in an evenly spaced square grid, with
$\Delta m=50$ radians/mm spacing, which is outside the memory effect region.
As the number of sources increases, there is additional despeckling due to
more incoherent averaging, and this results in an increase in both PSNR and
contrast at 1 arcmin (note that contrast is also negatively affected by
speckle). Although there are only 2 SLMs, the image quality continues to
improve far beyond two sources. This demonstrates the compressive nature of
the system since it implies that each source is still able to create the
correct pattern at full resolution using a limited number of degrees of
freedom.

However, compressive systems still have limits and eventually there are not
sufficient degrees of freedom to uniquely create the correct content for each
source. Looking at the simulated image with 100 sources, one can see haze
caused by some sources creating incorrect content. Once again, PSNR does not
reflect this trend, since the additional haze (which is not well captured by
PSNR) is balanced by further speckle reduction. However, our contrast metric
is a better proxy: around 36 sources, the contrast at 1 arcmin starts to
decrease, reflecting this performance limit. This suggests that the best image
quality is with a $6\times 6$ grid, which achieves 29.4 dB PSNR, over 10 dB
higher than the single source baseline.

We’d like to point out that there is a substantial design space for
multisource holography. Future work includes analyzing sources that are not
confined to a grid, exploring extended sources, and varying the source
intensities. In addition, the source parameters could be optimized
specifically for a dataset of natural images, analogous to the work of Baek et
al. (2021). However, these explorations are out of scope for this paper.

###  4.4. Time Multiplexing Comparison

So far we have restricted our comparisons to single source holograms made with
a single frame, but a common approach to speckle reduction is time
multiplexing. In this approach, several holograms are displayed in rapid
succession, and due to persistence of vision, the user sees an average of the
displayed frames. High speed modulators have made this method increasingly
practical, and prior work (Choi et al., 2022; Lee et al., 2022) has
demonstrated that temporal multiplexing can create natural defocus blur with a
uniform eyebox.

Our method is not meant to be a replacement to temporal multiplexing; the two
approaches are orthogonal and can be combined for even more speckle reduction.
Since noise reduction goes with the square root of the number of uncorrelated
images, temporal multiplexing provides diminishing returns with increasing
frame rate. Additional despeckling may be necessary to reduce noise to an
imperceptible level, even with high speed modulators.

In addition, reducing the necessary temporal bandwidth could help with another
fundamental challenge in holography: limited étendue, which results in a
trade-off between FoV and eyebox size. One practical option to overcome this
limitation is to scan the location of either the FoV or eyebox (Lee et al.,
2020b), enabling expanded étendue without eye tracking. However, scanning also
requires temporal bandwidth, which is no longer available for despeckling. By
providing substantial despeckling in a single frame, multisource holography
could open new paths for increasing étendue.

Figure 5 compares our multisource holography approach to temporal multiplexing
with 6 jointly optimized frames per color. Similar to recent work using
temporal multiplexing (Choi et al., 2021b), our holograms are computed using
iterative optimization where all 6 multiplexed frames are summed together
before computing the loss function. Then, all the frames are simultaneously
updated by the optimizer. As in prior simulations, we target a focal stack
with 15 planes and natural defocus blur. Our multisource simulation uses one
phase and one amplitude SLM, 25 sources with $\Delta m=75$ rad/mm, and only a
single frame per color.

Qualitatively, the two approaches have similar noise levels and image quality,
with multisource visually outperforming temporal multiplexing in white
regions. Quantitatively, multisource holography exceeds 6 frame temporal
multiplexing in PSNR and structural similarity index measure (SSIM) over the
focal stack. In the temporal multiplexing example, we simulated a phase only
SLM, which differs from the simulations in Sec. 4.1 and Sec. 4.3, since this
is the most realistic choice given currently available hardware. If fact, most
high speed SLMs are even more restricted; the SLMs capable of this much
multiplexing are typically binary or have limited bit depth, although in this
simulation we assume no quantization. As the number of temporally multiplexed
frames increases, the quality eventually exceeds that of multisource
holography (see Supplement for an example), but it comes at the cost of
temporal bandwidth.

##  5\. Experimental System Calibration

We have shown in simulation that multisource holography is a promising
technique, but in practice, achieving high quality experimental results
requires accurate knowledge of system parameters such as the source locations
and positions of the two SLMs. To calibrate our experimental system, we adapt
the approaches of Peng et al. (2020) and Chakravarthula et al. (2020) to
multisource holography by designing a physics-inspired forward model where
unknown parameters are learned from a dataset of experimentally captured SLM-
image pairs. Next, we go into the details of this model and the calibration
procedure.

###  5.1. System Model with Learnable Parameters

#### SLM Model

Our model starts with the digital values sent to the SLMs. For each SLM, these
values are passed through a learned lookup table (LUT) which describes the
mapping from digital input to phase. The LUT is parameterized by 256
coefficients (one for each possible input value), and the LUT is made
differentiable using 1D interpolation.

Next, the phase is convolved with a small learnable kernel that represents
cross-talk between pixels due to field fringing (Moser et al., 2019; Apter et
al., 2004; Persson et al., 2012). Field fringing is a phenomenon of liquid-
crystal-on-silicon (LCoS) SLMs in which the output phase is blurred by the
gradual transition at pixel boundaries of the electric field that modulates
the liquid crystal layer. Since this effect is sub-pixel, we upsample the
phase values by $2\times$ in each direction before applying the convolution
kernel ($5$ pixels in the upsampled space).

For each SLM, the phase with field fringing is converted to an electric field
(assuming uniform amplitude), yielding the complex modulation functions
$\mathbf{s}_{1}(\vec{x})$ and $\mathbf{s}_{2}(\vec{x})$.

#### Source Model

Each source is assumed to be a plane wave with learnable angle of incidence
and learnable relative intensity. For each source, we parameterize the angle
of incidence as a 2D location in Fourier space; simulating a delta function at
that location and then taking the Fourier transform and multiplying by the
relative intensity yields the input field for a given source,
$\mathbf{p}(\vec{x};\vec{m}_{i})$.

Figure 6.  Schematic of Experimental Setup: Our benchtop prototype uses two
SLMs with a 4$f$ system in between. A second 4$f$ relays both SLMs to the
correct positions in front of a bare sensor, which is mounted on a linear
motion stage. Irises in the Fourier planes remove higher orders from the SLMs.
To create the multiple sources, we use a superluminescent light emitting diode
(SLED) passed through a fiber splitter. Due to the low coherence of the SLED,
the fiber outputs are mutually incoherent, as required by our method. A
beamsplitter allows for switching between single source and multisource
illumination for comparisons.

#### Propagation Model

We adapt the ideal ASM propagation model (Eq. 2) to include aberrations by
multiplying the ASM kernel (Eq. 3) by a complex learnable pupil function. To
further enable modeling of spatially varying aberrations, different locations
of the input field should have variable pupil functions. Therefore, we learn a
$9\times 16$ grid of pupil functions, and we perform bilinear interpolation to
get the intermediate values.

However, applying a fully-spatially varying model is very computationally
intensive. To avoid computing a different pupil function for each point of the
input field, we instead take a stochastic, patch-based approach: during
optimization, we randomly choose a patch of the input field (about $1200\times
1200$ pixels in the upsampled coordinates) and use the pupil function that
corresponds to the center of that patch. Over the course of optimization, this
approximates the smoothly varying aberrations, with the added advantage of
reducing the memory requirements of the model by only simulating a fraction of
the FoV in each iteration. See Supplement for more details on how aberrations
are parametrized.

#### SLM Alignment

If the two SLMs are not perfectly aligned with sub-pixel accuracy, we need to
account for their relative positions in the model. After propagating the field
from the first SLM, we apply a learned warping function that transforms the
field into the coordinate space of the second SLM. Our warping function, based
on the thin-plate spline model (TPS) of Duchon (1977), can account for non-
radial distortion between the two SLMs, enabling accurate alignment even when
there are non-ideal optics between the modulators. The warping is implemented
in a differentiable manner in Kornia (Riba et al., 2020) using bilinear
interpolation separately on the real and imaginary parts of the complex field.

#### Model Summary

We put together all the components of the model as follows: starting with the
first SLM, we use our SLM model to covert the digital input values into a
complex modulation function. This is multiplied by the source field, then
propagated a distance $\Delta z$ using our modified ASM propagator with
spatially varying pupil functions. The field is then warped to match the
coordinate space of the second SLM and multiplied by the complex modulation
function $\mathbf{s}_{2}(\vec{x})$, which, once again, is computed with the
SLM model described above. Finally, we apply the ASM propagator with spatial
variance a second time to propagate a distance $z$, then take the absolute
value squared to simulate intensity on the sensor. This process is repeated
for each source in the system while summing the contributions.

###  5.2. Calibration Procedure

To fit the learnable parameters of our model, we collect an experimental
dataset of SLM-image pairs and optimize for the unknown parameters using
gradient descent in PyTorch. We use random patterns on the SLM, which have
similar statistics to the random phase holograms we aim to display. To further
facilitate the optimization process, we apply a Gaussian filter on the input
phase with a standard deviation varying from 4 pixels to zero pixels (no
blur). This creates training data with larger features that are especially
helpful when optimizing the TPS and the source position parameters, which do
not converge correctly with high-frequency content alone. We capture datasets
with both single source illumination and multisource illumination.

Since the low frequency SLM inputs are less sensitive to field fringing and
aberrations, we use the single source blurred patterns to optimize the TPS
warping function before fitting the rest of the model. We also optimize a
second similar warping function to align the final intensity to the camera
capture. Once the alignment functions are close to accurate, we use the
remaining single source dataset with all spatial frequencies to fit the other
parameters.

After the single source model is optimized, we use the multisource dataset to
fit the source locations and intensities. Finally we fine-tune the other
parameters using the multisource data to get the complete model. We repeat
this process for each color separately.

Note that unlike many learned models in prior work (Choi et al., 2021a;
Chakravarthula et al., 2020), our model does not contain any black-box neural
networks; all parameters are physically meaningful. This limits the number of
learnable parameters, which in turn means less training data is required, the
model optimizes quickly, and the chance of over-fitting is low. For example,
our training dataset contains only about 300 captures per color per source
configuration, and training takes approximately 10 minutes on an Nvidia GV100.
Although we only capture training data at a single propagation distance $z$,
we find that the model extends well to other planes without retraining.

###  5.3. Active Camera-in-the-Loop

To highlight the potential of multisource holography, we additionally use the
“active” camera-in-the-loop (CiTL) method proposed by Peng et al. (2020),
where feedback from a camera in the system is used to fine-tune the SLM
pattern(s) for a specific image or focal stack. We pre-optimize the SLM
patterns using our learned model, display the patterns on the experimental
system, and continue optimization while replacing the model output with the
captured image before back-propagation. For focal stacks, we capture the
experimental images at a different location in the volume at each iteration,
and we fine-tune the alignment between the capture to the model output using
cross-correlation on a patch-wise basis. Final results are captured after
updates are complete, with one static pair of SLM patterns for all depths.

Figure 7. 2D Results (Experiment): Although a single source random phase
hologram can theoretically control speckle well for a 2D image, the
experimental 2D capture (top) has visible speckle when one zooms in. Our
multisource configuration with $4\times 4$ sources (middle) has noticeably
reduced speckle while maintaining high frequency features. PSNR is shown in
the bottom left.  Figure 8.  Focal Stack Results (Experiment): Focal stacks
created by a single source hologram with random phase (top) suffer from severe
speckle noise since there are insufficient degrees of freedom on the SLM to
control speckle throughout a 3D volume. Our multisource approach with $4\times
4$ sources (middle) greatly reduces speckle, enabling experimental focal
stacks with natural defocus cues. PSNR calculated over the full focal stack is
shown in the bottom left.

##  6\. Experimental Results

We demonstrate multisource holography on a benchtop experimental system,
depicted in Fig. 6. To create the multiple sources, we split the output of a
fiber-coupled light source using cascaded 1:4 fiber splitters (Thorlabs
TWQ560HA) to create 16 different sources, which are arranged in a $4\times 4$
grid. By choosing a superluminescent light emitting diode (SLED, Exalos
EXC250011-00), which has a very short coherence length, we find that the
outputs of the 16 different fibers are mutually incoherent without explicitly
adding path length differences. However, the SLED has a spectral bandwidth of
about $10\text{\,}\mathrm{nm}$, which is not accounted for in our model, and
we discuss this limitation more in Sec. 7. Although the spectral bandwidth of
a laser would match our model better and result in improved resolution (Deng
and Chu, 2017), we found that the longer coherence length of a laser made it
challenging to consistently break the coherence between fiber outputs, even
with added path length differences. This is not a fundamental challenge as one
could use an array of laser diodes instead of splitting a single laser output.

The multiple sources are spaced $4\text{\,}\mathrm{mm}$ apart, held in a 3D
printed housing. Combined with the $f_{c}=$500\text{\,}\mathrm{mm}$$
collimating lens, this yields $\Delta m=79$ rad/mm, $99$ rad/mm, and $110$
rad/mm for red, green, and blue respectively, which are outside the memory
effect region of Eq. 13. The angles of incidence at the SLM are within $\pm
0.69^{\circ}$ (see Eq. 5), which is well within the paraxial approximation as
assumed in Sec. 3. A beamsplitter in front of the sources lets the
illumination be toggled between the multisource configuration and a
traditional single source, and a linear polarizer ensures the beam is
correctly polarized for the SLMs.

The system uses two phase only LCoS SLMs (Holoeye Pluto-2.1-VIS-016), and a
4$f$ system with 1:1 magnification ($f_{1}=f_{2}=$200\text{\,}\mathrm{mm}$$)
relays the first SLM to a distance $\Delta z=$2\text{\,}\mathrm{mm}$$ behind
the second SLM. A second 4$f$ system ($f_{3}=$200\text{\,}\mathrm{mm}$$,
$f_{4}=$150\text{\,}\mathrm{mm}$$) relays the SLMs to the camera sensor.
Irises in the Fourier planes of both 4$f$ systems filter higher orders from
the SLMs.

SLM patterns are optimized using the calibrated model outlined in Sec. 5. SLMs
are initialized with uniform random phase, and we jointly optimize both SLMs,
even for the single source case. Different SLM patterns are optimized for each
color (central wavelengths at $638\text{\,}\mathrm{nm}$,
$510\text{\,}\mathrm{nm}$, and $455\text{\,}\mathrm{nm}$, for red, green, and
blue respectively) and displayed in sequence.

Images are captured on a monochrome camera sensor (XIMEA MC089MG-SY), which is
mounted on a brushless translation stage (Thorlabs DDS050) to enable focal-
stack capture from $z=$15\text{\,}\mathrm{mm}$$ to
$z=$25\text{\,}\mathrm{mm}$$, defined in SLM space. Note that the actual
distances at the camera are slightly less due to the demagnification of the
second 4$f$ system. Since the sensor is monochrome, color results are captured
sequentially and combined in post-processing. After capture, images are
rectified into the SLM coordinate space using bilinear interpolation, un-
modulated areas of the image are cropped out, and the relative intensities of
the color channels are adjusted.

###  6.1. 2D Results

Figure 7 shows a 2D experimental capture on our system at
$z=$20\text{\,}\mathrm{mm}$$ comparing single source and multisource
holograms. Although a traditional single source holographic display can
theoretically create very high quality 2D holograms, in practice there is
still speckle noise visible in a random phase hologram. Even in the 2D
scenario where single source performs quite well, multisource holography still
provides noticeable despeckling, improving PSNR by 4.7 dB in this example.
Both results are optimized with active CiTL as described in Sec. 5.3; see
Supplement for results without this fine tuning.

###  6.2. Focal Stack Results

However, the true benefits of multisource holography become most apparent when
displaying 3D content. Using our calibrated model, we optimize the SLM
patterns while targeting a focal stack with natural blur. We use the same blur
parameters and propagation distances as the simulations (Sec. 4).

Figure 8 shows the experimentally captured results. As expected from our
simulations, the hologram made with a single source is severely corrupted by
speckle. In comparison, multisource holography can generate low speckle images
over the whole volume, complete with natural defocus cues, resulting in a 7.4
dB PSNR increase calculated on the full focal stack. As a reminder, our
multisource holograms are random phase, which creates an approximately uniform
energy distribution in the eyebox (see Supplement for a visualization), and
are produced with only one frame per color. Similar to the 2D images, these
results are all captured with active CiTL; versions without active CiTL are
included in the Supplement.

##  7\. Discussion

We have demonstrated both in simulation and experiment that multisource
holography can provide significant despeckling without resolution loss,
enabling focal stacks with realistic blur. However, there are several
directions for further investigation.

#### Pupil-Aware Holography

Our holograms (like most in the literature) are simulated assuming the entire
eyebox is fully contained within the user’s pupil. This is atypical for
conventional (non-holographic) near-eye displays where the eyebox is usually
larger than the pupil size to give users freedom to move their eyes without
leaving the eyebox. However, in a holographic display, substantial artifacts
can occur when only a fraction of the eyebox enters the user’s eye. Our
initial simulations suggest that multisource holography could improve image
quality given unknown pupil locations.

To demonstrate, we simulate 2D holograms optimized using the pupil-aware loss
proposed by Chakravarthula et al. (2022), in which random pupil locations are
sampled during optimization (Fig. 9). Smooth phase holograms (Fig. 9b) have
excellent image quality when the pupil is centered, but a pupil at the edge of
the eyebox sees a low intensity, completely incorrect version of the image.
Random phase holograms (Fig. 9c) have approximately uniform intensity but are
corrupted by speckle regardless of pupil position, even for a 2D images with
no focal cues. In comparison, multisource holography (Fig. 9d) can produce a
clean image for pupil locations over the whole eyebox. Extending this concept
to light fields (Choi et al., 2022; Padmanaban et al., 2019) is another
direction of future work.

Figure 9. Pupil-Invariance (Simulation): When the user’s pupil does not cover
the full eyebox, holograms can have significant artifacts, even for 2D images.
To demonstrate, we optimize holograms with the pupil aware loss of
Chakravarthula et al. (2022). We show examples at two different pupil
positions in the eyebox, visualized in (a), where the eyebox extent for each
color is depicted with dotted lines. The total intensity of the simulated
image relative to a centered pupil is shown in the top right of each
simulation. Smooth phase holograms (b) can create high quality images when the
pupil is centered, but when the pupil is translated, image content is highly
corrupted and has very low intensity. Random phase holograms (c) have
approximately uniform intensity when the pupil moves but the image is very
noisy due to speckle. Multisource holography (d) can create a low noise image
that’s invariant to the pupil position, which is desirable for a practical
display.

#### Source Design and Étendue Expansion

Although we analyzed several important parameters of the source design in Sec.
4.3, we restricted our analysis to a grid of uniform intensity sources within
the étendue of the native SLM. There may be additional performance gains from
different source configurations such as extended sources, optimized source
locations, or variable source intensities. In addition, by increasing the
spacing between sources, multisource holography may be able to expand system
étendue, similar to the work of Jo et al. (2022), helping with another
fundamental problem in holographic displays.

#### Multisource Holography with 1 SLM

Using two SLMs may not be feasible for all applications. An alternative is to
replace one of the SLMs with a static DOE. This creates an angularly-selective
response similar to the two SLMs, breaking correlations between sources and
enabling many of the benefits of multisource holography. However, the reduced
degrees of freedom mean that fewer sources can simultaneously generate the
correct pattern, so the amount of despeckling will be reduced. To improve
performance with only a single active modulator, co-optimization of the DOE
with the other system parameters could be investigated, similar to the work of
Baek et al. (2021).

#### Compact Architecture

Our experimental system is a large benchtop setup containing multiple 4$f$
systems, but we envision multisource holography could be built into a compact
architecture. Starting with the design of Kim et al. (2022a), which uses a
waveguide to illuminate a reflective phase only SLM, we propose coupling the
multiple sources into the waveguide to generate the multisource illumination.
For the second SLM, we suggest using a transmissive amplitude modulator,
placed just before the eyepiece. However, without a 4$f$ relay system, SLM
higher orders must be taken into account in the model (Gopakumar et al., 2021)
or filtered using compact volume holograms (Bang et al., 2019).

#### SLED Bandwidth

We took advantage of the short coherence length of the SLED to create the
multiple sources used in our experimental setup. However, the SLED has a
bandwidth of about $10\text{\,}\mathrm{nm}$ while our model and analysis in
Sec. 4.3 assumes monochromatic light. We include in the Supplement a complete
model that accounts for the spectral bandwidth of the source and a practical
optimization strategy based on Peng et al. (2021) for this scenario. However,
modeling a larger bandwidth source has higher computational cost, so we chose
to assume monochromatic illumination during optimization. We expect our
results would improve with more accurate modeling of the SLED, but we found
that our monochromatic model was sufficient to show the benefits of
multisource holography. See Supplement for a visualization of the effect of
the SLED.

#### Computation Speed

Computational cost is a limitation of our method, since the image formation
model requires separately simulating the contributions from each source.
Furthermore, all our simulations were conducted at $2\times$ resolution in
each dimension, resulting in computation times of about half an hour to
generate a focal stack. For example, for a 1080 $\times$ 1920 modulator with
16 sources, we run 2000 iterations, each of which takes about 0.8 sec.
Upsampling may not be necessary in all scenarios, and in these cases
computation time drops to about 0.2 sec per iteration, but compute is still a
limitation. To address this, neural networks offer a promising path towards
real-time computation, as they have already been demonstrated for single
source holography (Peng et al., 2020; Shi et al., 2021; Yang et al., 2022),
albeit only for smooth phase so far. Adapting these approaches to multisource
holography will be necessary for a practical display.

##  8\. Conclusion

We introduced a new architecture for holographic displays that uses an array
of mutually incoherent sources and two SLMs to reduce speckle. To our
knowledge, our design is the first single-frame method that can generate low
speckle holograms at full resolution with realistic focal cues and a uniform
eyebox. We analyzed the concept in simulation, explored the design space, and
validated with a benchtop experimental setup capable of producing high quality
focal stacks. In conclusion, we believe multisource holography is a promising
path to address some of the key open problems in holographic displays.

## References

  * (1)
  * Apter et al. (2004) Boris Apter, Uzi Efron, and Eldad Bahat-Treidel. 2004.  On the fringing-field effect in liquid-crystal beam-steering devices.  _Applied Optics_ 43, 1 (2004), 11–19.
  * Baek et al. (2021) Seung-Hwan Baek, Ethan Tseng, Andrew Maimone, Nathan Matsuda, Grace Kuo, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, and Felix Heide. 2021.  Neural Etendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display.  _arXiv_ (2021).
  * Bang et al. (2019) Kiseung Bang, Changwon Jang, and Byoungho Lee. 2019.  Compact noise-filtering volume gratings for holographic displays.  _Optics Letters_ 44, 9 (2019), 2133–2136.
  * Chakravarthula et al. (2022) Praneeth Chakravarthula, Seung-Hwan Baek, Ethan Tseng, Andrew Maimone, Grace Kuo, Florian Schiffers, Nathan Matsuda, Oliver Cossairt, Douglas Lanman, and Felix Heide. 2022.  Pupil-aware holography.  _arXiv_ (2022).
  * Chakravarthula et al. (2019) Praneeth Chakravarthula, Yifan Peng, Joel Kollin, Henry Fuchs, and Felix Heide. 2019.  Wirtinger holography for near-eye displays.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 213.
  * Chakravarthula et al. (2020) Praneeth Chakravarthula, Ethan Tseng, Tarun Srivastava, Henry Fuchs, and Felix Heide. 2020.  Learned hardware-in-the-loop phase retrieval for holographic near-eye displays.  _ACM Transactions on Graphics_ 39, 6 (2020).  https://doi.org/10.1145/3414685.3417846
  * Choi et al. (2022) Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, Matthew O’Toole, and Gordon Wetzstein. 2022.  Time-multiplexed neural holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Choi et al. (2021a) Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021a.  Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays.  _ACM Transactions on Graphics_ 40, 6 (2021).  https://doi.org/10.1145/3478513.3480542
  * Choi et al. (2021b) Suyeon Choi, Jonghyun Kim, Yifan Peng, and Gordon Wetzstein. 2021b.  Optimizing image quality for holographic near-eye displays with Michelson holography.  _Optica_ 8, 2 (2021), 143–146.
  * Cuervo et al. (2018) Eduardo Cuervo, Krishna Chintalapudi, and Manikanta Kotaru. 2018.  Creating the perfect illusion: What will it take to create life-like virtual reality headsets?. In _Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications_. 7–12.
  * Curtis et al. (2021) Vincent R Curtis, Nicholas W Caira, Jiayi Xu, Asha Gowda Sata, and Nicolas C Pégard. 2021.  DCGH: Dynamic computer generated holography for speckle-free, high fidelity 3D displays. In _2021 IEEE Virtual Reality and 3D User Interfaces (VR)_. IEEE, 1–9.
  * Deng and Chu (2017) Yuanbo Deng and Daping Chu. 2017.  Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays.  _Scientific Reports_ 7, 1 (2017), 5893.
  * Duchon (1977) Jean Duchon. 1977.  Splines minimizing rotation-invariant semi-norms in Sobolev spaces. In _Constructive Theory of Functions of Several Variables: Proceedings of a Conference Held at Oberwolfach April 25–May 1, 1976_. Springer, 85–100.
  * Fienup (1982) James R Fienup. 1982.  Phase retrieval algorithms: a comparison.  _Applied Optics_ 21, 15 (1982), 2758–2769.
  * Freund et al. (1988) Isaac Freund, Michael Rosenbluh, and Shechao Feng. 1988.  Memory effects in propagation of optical waves through disordered media.  _Physical Review Letters_ 61, 20 (1988), 2328.
  * Georgiou et al. (2008) A Georgiou, J Christmas, N Collings, J Moore, and WA Crossland. 2008.  Aspects of hologram calculation for video frames.  _Journal of Optics A: Pure and Applied Optics_ 10, 3 (2008), 035302.
  * Gkioulekas et al. (2015) Ioannis Gkioulekas, Anat Levin, Frédo Durand, and Todd Zickler. 2015.  Micron-scale light transport decomposition using interferometry.  _ACM Transactions on Graphics (ToG)_ 34, 4 (2015), 1–14.
  * Goodman (2005) Joseph W Goodman. 2005.  _Introduction to Fourier optics_.  Roberts and Company publishers.
  * Goodman (2007) Joseph W Goodman. 2007.  _Speckle phenomena in optics: theory and applications_.  Roberts and Company Publishers.
  * Gopakumar et al. (2021) Manu Gopakumar, Jonghyun Kim, Suyeon Choi, Yifan Peng, and Gordon Wetzstein. 2021.  Unfiltered holography: Optimizing high diffraction orders without optical filtering for compact holographic displays.  _Optics Letters_ 46, 23 (2021), 5822–5825.
  * Heanue et al. (1994) John F Heanue, Matthew C Bashaw, and Lambertus Hesselink. 1994.  Volume holographic storage and retrieval of digital data.  _Science_ 265, 5173 (1994), 749–752.
  * Hoffman et al. (2008) David M Hoffman, Ahna R Girshick, Kurt Akeley, and Martin S Banks. 2008.  Vergence–accommodation conflicts hinder visual performance and cause visual fatigue.  _Journal of Vision_ 8, 3 (2008), 33–33.
  * Jo et al. (2022) Youngjin Jo, Dongheon Yoo, Dukho Lee, Minkwan Kim, and Byoungho Lee. 2022.  Multi-illumination 3D holographic display using a binary mask.  _Optics Letters_ 47, 10 (2022), 2482–2485.
  * Kavaklı et al. (2023) Koray Kavaklı, Yuta Itoh, Hakan Urey, and Kaan Akşit. 2023.  Realistic defocus blur for multiplane computer-generated holography. In _2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)_. IEEE, 418–426.
  * Kavaklı et al. (2022) Koray Kavaklı, Hakan Urey, and Kaan Akşit. 2022.  Learned holographic light transport.  _Applied Optics_ 61, 5 (2022), B50–B55.
  * Kim et al. (2022b) Dongyeon Kim, Seung-Woo Nam, Byounghyo Lee, Jong-Mo Seo, and Byoungho Lee. 2022b.  Accommodative holography: Improving accommodation response for perceptually realistic holographic displays.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–15.
  * Kim et al. (2022a) Jonghyun Kim, Manu Gopakumar, Suyeon Choi, Yifan Peng, Ward Lopes, and Gordon Wetzstein. 2022a.  Holographic glasses for virtual reality. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv_ (2014).
  * Kotwal et al. (2020) Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas. 2020.  Interferometric transmission probing with coded mutual intensity.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 74–1.
  * Kotwal et al. (2023) Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas. 2023.  Swept-angle synthetic wavelength interferometry. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8233–8243.
  * Kozacki et al. (2022) Tomasz Kozacki, Maksymilian Chlipala, Juan Martinez-Carranza, Rafał Kukołowicz, and Moncy Sajeev Idicula. 2022.  LED near-eye holographic display with a large non-paraxial hologram generation.  _Optics Express_ 30, 24 (2022), 43551–43565.
  * Kuo et al. (2020) Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone. 2020.  High resolution étendue expansion for holographic displays.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 66.
  * Lee et al. (2022) Byounghyo Lee, Dongyeon Kim, Seungjae Lee, Chun Chen, and Byoungho Lee. 2022.  High-contrast, speckle-free, true 3D holography via binary CGH optimization.  _Scientific Reports_ 12, 1 (2022), 2811.
  * Lee et al. (2020b) Byounghyo Lee, Dongheon Yoo, Jinsoo Jeong, Seungjae Lee, Dukho Lee, and Byoungho Lee. 2020b.  Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing.  _Optics Letters_ 45, 8 (2020), 2148–2151.
  * Lee et al. (2019) Dukho Lee, Changwon Jang, Kiseung Bang, Seokil Moon, Gang Li, and Byoungho Lee. 2019.  Speckle reduction for holographic display using optical path difference and random phase generator.  _IEEE Transactions on Industrial Informatics_ 15, 11 (2019), 6170–6178.
  * Lee et al. (2020a) Seungjae Lee, Dongyeon Kim, Seung-Woo Nam, Byounghyo Lee, Jaebum Cho, and Byoungho Lee. 2020a.  Light source optimization for partially coherent holographic displays with consideration of speckle contrast, resolution, and depth of field.  _Scientific Reports_ 10, 1 (2020), 18832.
  * Maimone et al. (2017) Andrew Maimone, Andreas Georgiou, and Joel S Kollin. 2017.  Holographic near-eye displays for virtual and augmented reality.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017), 85.
  * Matsushima and Shimobaba (2009) Kyoji Matsushima and Tomoyoshi Shimobaba. 2009.  Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields.  _Optics Express_ 17, 22 (2009), 19662–19673.
  * Moon et al. (2014) Eunkyong Moon, Myeongjae Kim, Jinyoung Roh, Hwi Kim, and Joonku Hahn. 2014.  Holographic head-mounted display with RGB light emitting diode light source.  _Optics Express_ 22, 6 (2014), 6526–6534.
  * Moser et al. (2019) Simon Moser, Monika Ritsch-Marte, and Gregor Thalhammer. 2019.  Model-based compensation of pixel crosstalk in liquid crystal spatial light modulators.  _Optics Express_ 27, 18 (2019), 25046–25063.
  * Padmanaban et al. (2019) Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019.  Holographic near-eye displays based on overlap-add stereograms.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–13.
  * Park and Lee (2022) Jae-Hyeung Park and Byoungho Lee. 2022.  Holographic techniques for augmented reality and virtual reality near-eye displays.  _Light: Advanced Manufacturing_ 3, 1 (2022), 137–150.
  * Peng et al. (2021) Yifan Peng, Suyeon Choi, Jonghyun Kim, and Gordon Wetzstein. 2021.  Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration.  _Science Advances_ 7, 46 (2021), eabg5040.
  * Peng et al. (2020) Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020.  Neural holography with camera-in-the-loop training.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–14.
  * Peng et al. (2017) Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017.  Mix-and-match holography.  _ACM Trans. Graph._ 36, 6 (2017), 191–1.
  * Persson et al. (2012) Martin Persson, David Engström, and Mattias Goksör. 2012.  Reducing the effect of pixel crosstalk in phase only spatial light modulators.  _Optics Express_ 20, 20 (2012), 22334–22343.
  * Primerov et al. (2019) Nikolay Primerov, Jean Dahdah, Stefan Gloor, Nicolai Matuschek, Tim von Niederhäusern, Antonino Castiglia, Marco Malinverni, Christian Mounir, Marco Rossetti, Marcus Duelk, et al. 2019\.  P-203: Late-News Poster: Integrated Full-Color RGB Superluminescent LED Module for Micro-Displays. In _SID Symposium Digest of Technical Papers_ , Vol. 50. Wiley Online Library, 1731–1734.
  * Riba et al. (2020) Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. 2020.  Kornia: An open source differentiable computer vision library for Pytorch. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 3674–3683.
  * Saxton and Gerchberg (1972) W.O. Saxton and R.W. Gerchberg. 1972.  Gerchberg—Saxton algorithm: Experimental realisation and modification for the problem of formation of multimode laser beams.  _Optik_ 35, 2 (1972), 237–246.  https://doi.org/10.1070/qe2009v039n06abeh013642
  * Shi et al. (2021) Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021.  Towards real-time photorealistic 3D holography with deep neural networks.  _Nature_ 591, 7849 (2021), 234–239.
  * Wang et al. (2019) Di Wang, Nan-Nan Li, Chao Liu, and Qiong-Hua Wang. 2019.  Holographic display method to suppress speckle noise based on effective utilization of two spatial light modulators.  _Optics Express_ 27, 8 (2019), 11617–11625.
  * Wetzstein et al. (2012) Gordon Wetzstein, Douglas R Lanman, Matthew Waggener Hirsch, and Ramesh Raskar. 2012.  Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting.  (2012).
  * Yang et al. (2022) Daeho Yang, Wontaek Seo, Hyeonseung Yu, Sun Il Kim, Bongsu Shin, Chang-Kun Lee, Seokil Moon, Jungkwuen An, Jong-Young Hong, Geeyoung Sung, et al. 2022\.  Diffraction-engineered holography: Beyond the depth representation limit of holographic displays.  _Nature Communications_ 13, 1 (2022), 6012.
  * Ye et al. (2014) Genzhi Ye, Sundeep Jolly, V Michael Bove Jr, Qionghai Dai, Ramesh Raskar, and Gordon Wetzstein. 2014.  Toward BxDF display using multilayer diffraction.  _ACM Transactions on Graphics (TOG)_ 33, 6 (2014), 1–14.
  * Yoo et al. (2021) Dongheon Yoo, Youngjin Jo, Seung-Woo Nam, Chun Chen, and Byoungho Lee. 2021.  Optimization of computer-generated holograms featuring phase randomness control.  _Optics Letters_ 46, 19 (2021), 4769–4772.
  * Zhao et al. (2022) Zijie Zhao, Junyi Duan, and Juan Liu. 2022.  Speckle reduction in holographic display with partially spatial coherent illumination.  _Optics Communications_ 507 (2022), 127604.