# Time-multiplexed Neural Holography:
A Flexible Framework for Holographic Near-eye Displays with Fast Heavily-
quantized Spatial Light Modulators — Supplemental Material

Suyeon Choi<EMAIL_ADDRESS>Stanford UniversityUSA ,  Manu Gopakumar
<EMAIL_ADDRESS>Stanford UniversityUSA ,  Yifan Peng
<EMAIL_ADDRESS>Stanford UniversityUSA ,  Jonghyun Kim
<EMAIL_ADDRESS>NVIDIA and Stanford UniversityUSA ,  Matthew O’Toole
<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA and  Gordon Wetzstein
<EMAIL_ADDRESS>Stanford UniversityUSA

††submissionid: 578††copyright: acmlicensed††journal: TOG††journalyear:
2022††journalvolume: 00††journalnumber: 00††article: 00††publicationmonth:
00††doi: 0000/0000

This supplementary document includes implementation details of our holographic
display prototype, complementary derivations related to wave propagation and
optimization models, and additional experimental results. Refer also to the
supplementary video for better visualization.

Here we list the abbreviations and notations used across this document. These
are consistent with those in the main paper.

SLM: |  a spatial light modulator
---|---
CGH: |  computer-generated holography
STFT: |  the Short-time Fourier transform (STFT)
ASM: |  the angular spectrum method (Goodman, 2005)
GS: |  the Gumber-Softmax operation (Jang et al., 2016; Maddison et al., 2016)
CITL: |  the camera-in-the-loop optimization technique (Peng et al., 2020)
SGD: |  stochastic gradient descent phase retrieval (Peng et al., 2020)

NH: |  a 2D wave propagation model that is trained using an SGD-based camera-in-the-loop training strategy aka neural holography (Peng et al., 2020); once trained, this model is used to generate new holograms using an SGD solver
---|---
NH3D: |  wave propagation model using CNNs operating on the complex-valued field at the SLM plane before ASM propagation and also directly after propagation to the target planes (Choi et al., 2021)

##  S1. Additional Details on Hardware

In this section, we describe the hardware implementation of our benchtop 3D
holographic display prototype. Figure S1 shows the system schematic and
photograph of our implementation, including a display and a capture unit, that
are connected under a closed-loop framework. Specifically, the SLM is TI
DLP6750Q1EVM with a resolution of 1,280 $\times$ 800, a pixel pitch of 10.8
$\mu$m, and a bit depth of 4 bits per pixel. The laser is a FISBA RGBeam
fiber-coupled module with three optically aligned laser diodes with a maximum
output power of 50 mW. The measured wavelengths are 636.4, 517.7, and 440.8
nm. In our implementation, color images are captured as separate exposures for
each wavelength and then cast in post-processing.

Other components including the collimating lenses, the relay imaging lenses,
the filtering iris, and the beam splitter (Thorlabs BS016) are shown in Figure
S1. All images are captured with a FLIR Grasshopper3 12.3 MP color USB3 sensor
through a Canon EF 50mm lens. The Canon lens and sensor are synchronized in
hardware via Arduino (Uno SMD) controller to enable programmable varifocal
display and acquisition. The capture unit is assembled on a motorized
translation stage to enable the acquisition capability from different
horizontal viewpoints. In such a way, we are able to acquire holographic
images to both form the training dataset and showcase diverse 3D cues.

Figure S1. Schematic and prototype photograph of our holographic display.

In the calibration step, we use a similar procedure to that described in the
relevant work (Peng et al., 2020) and apply a planar homography from the field
of computer vision to accurately register the captured images to the ground-
truth images. Our implementation uses a target binary pattern consisting of 18
$\times$ 11 white dots with the interval between the centers of neighboring
two dots set 70 pixels. Accordingly, the region of interest has a resolution
of 1,190 $\times$ 700 pixels.

##  S2. Additional Details on Software

###  S2.1. From a variant of projected gradient descent to surrogate gradient
with unit Jacobian

Here, we derive the relationship between the variant of projected gradient
descent in the manuscript and the gradient descent with the surrogate gradient
of unit matrix Jacobian.

1) Based on the projected gradient descent rule described in the main paper
(Eq. 4), we consider the projection step of the last iteration together with
the phase update step of the current iteration as one iteration:

$\displaystyle\phi^{(k-1)}$
$\displaystyle\leftarrow\Pi_{\mathcal{Q}}\left(\widehat{\phi}^{(k-1)}\right)=q\left(\widehat{\phi}^{(k-1)}\right).$
$\displaystyle\widehat{\phi}^{(k)}$
$\displaystyle\leftarrow\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial\phi}\right)^{T}\\!\\!\mathcal{L}\left(s\big{|}f_{\textrm{\tiny
CNN}}\left(e^{i\phi^{(k-1)}}\right)\big{|},a_{\text{target}}\right).$

2) Then, we substitute the first row into the second row:

$\displaystyle\widehat{\phi}^{(k)}$ $\displaystyle\leftarrow
q\left(\widehat{\phi}^{(k-1)}\right)-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
q}\right)^{T}\\!\\!\mathcal{L}\left(s\big{|}f_{\textrm{\tiny
CNN}}\left(e^{iq\left(\widehat{\phi}^{(k-1)}\right)}\right)\big{|},a_{\text{target}}\right).$

Note that all of these variants (including the one in the manuscript)
generally fail to work well for holographic phase retrieval, because the
projection taken every step vanishes the phase update with the gradient term.

3) Thus, we relax this hard constraint by leaving out the projection in the
first term; this variant of projected gradient descent applying the projection
only in the second term does not suffer from being stuck. This is also a
special case of surrogate gradient where we use the surrogate gradient of unit
Jacobian $\frac{\partial\widehat{q}}{\partial\phi}=I$. This means the gradient
with respect to the quantized phase $q$ is simply passed to the gradient with
respect to the continuous phase $\phi$. We use the following algorithm as the
representative of the projected gradient descent:

(1) $\displaystyle\widehat{\phi}^{(k)}$
$\displaystyle\leftarrow\widehat{\phi}^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
q}\right)^{T}\\!\\!\mathcal{L}\left(s\big{|}f_{\textrm{\tiny
CNN}}\left(e^{iq\left(\widehat{\phi}^{(k-1)}\right)}\right)\big{|},a_{\text{target}}\right)$
(2)
$\displaystyle=\widehat{\phi}^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
q}\cdot\frac{\partial\widehat{q}}{\partial\phi}\right)^{T}\\!\\!\mathcal{L}\left(s\big{|}f_{\textrm{\tiny
CNN}}\left(e^{iq\left(\widehat{\phi}^{(k-1)}\right)}\right)\big{|},a_{\text{target}}\right).$

Note that Eq. 2 is identical to Eq. 5 in the main paper. The recent paper by
Lee et al. (2022) uses hard-sigmoid as a surrogate gradient which has unit
gradient within the valid range, and we note that this falls into the category
of variants of gradients we describe in this section and our Gumbel-Softmax
based approach outperforms it with a large margin as shown in Fig. 3.

Figure S2. Experimentally captured results with the camera-in-the-loop
calibration using the naive (left) and the surrogate gradient (right)
quantization. PSNR metrics are indicated.

###  S2.2. Camera-in-the-loop with highly quantized SLMs

Here we describe a more accurate camera-in-the-loop procedure for highly
quantized SLMs. The camera-in-the-loop procedure proposed by Peng et al.
(2020) approximates the gradient of the physical forward model $f$ to that of
simulated model $\widehat{f}$:

$\displaystyle\phi^{(k)}$
$\displaystyle\leftarrow\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial\phi}\right)^{T}$
(3)
$\displaystyle\simeq\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
f}\cdot\frac{\partial\widehat{f}}{\partial\phi}\right)^{T}.$

However, note that we always have to quantize the phase before displaying it
on the SLM. Thus, technically, $f$ should reads as
$f\left(q\left(\phi\right)\right)$. Again, while we do not have access to the
gradient of the quantization function $q$, we can approximate it with the
surrogate gradient $\frac{\partial\widehat{q}}{\partial\phi}$:

$\displaystyle\phi^{(k)}$
$\displaystyle\leftarrow\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
f\left(q\right)}\cdot\frac{\partial\widehat{f}\left(q\right)}{\partial\phi}\right)^{T}$
$\displaystyle=\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
f}\cdot\frac{\partial\widehat{f}}{\partial q}\cdot\frac{\partial
q}{\partial\phi}\right)^{T}$ (4)
$\displaystyle\simeq\phi^{(k-1)}-\alpha\\!\left(\frac{\partial\mathcal{L}}{\partial
f}\cdot\frac{\partial\widehat{f}}{\partial
q}\cdot\frac{\partial\widehat{q}}{\partial\phi}\right)^{T}.$

In Fig. S2, we compare two update rules Eq. (3) and Eq. (4). We see that with
the approximation with the surrogate gradient, image quality is noticeably
improved.

###  S2.3. Setting parameters for quantized phase optimization

In this subsection we describe parameters used in quantized phase
optimization. We run 2,000 iterations with early stopping with a learning rate
of 0.01 for 1 frame and that of 0.02 for 8 frames. We note that the surrogate
gradients, if used with the Sigmoid or functions with clipping, require a
higher learning rate to avoid getting stuck in local minima which leads to
poor performance. During optimization, gradually annealing the slope helps the
optimization by better approximating the step function gradually while
allowing exploration of a large parameter space with a lower slope at the
beginning. The Sigmoid function can be annealed with a parameter $s$
multiplied with the input $x$, so we refer to the Sigmoid function as
$\sigma\left(s\cdot x\right)$. The Gumbel-Softmax can be annealed with three
parameters including the temperature parameter of Softmax $\tau$, the width
parameter $w$ which corresponds to the interval of discrete levels, and a
scale multiplied to the score function in Eq. 8 of the manuscript. In the
experiments, we tuned $w$ considering the number of phase levels (interval
between neighbour phase displacements) and the scale multiplied to the score
function was increased from 300 to 1,000 during optimization. We used an
annealing schedule of $\tau=\tau_{0}\cdot e^{-c\cdot(t/t_{\textrm{max}})}$ at
iteration $t$ with $c\sim\textrm{ln}2$ and $\tau_{0}\sim 4$.

###  S2.4. Model architecture and training details

The two convolutional neural networks in our model are based on the U-net
architecture as in Choi et al. (2021). We made a slight modification on the
CNN archtectures such that each CNN has 5 layers and 4 input channels of
amplitude, phase, real, and imaginary values of the input field. The output of
$\textsc{cnn}_{\textrm{\tiny SLM}}$ is two channels that are used as real and
imaginary values of the adjusted SLM field. The output of
$\textsc{cnn}_{\textrm{\tiny target}}$ is 1 channel that is used as a
corrected amplitude. As stated in Table 1 of the main paper, the model is
trained over 6 intensity planes, corresponding to 0.0 D, 0.5 D, 1.0 D, 1.5 D,
2.5 D, and 3.0 D in the physical space. The propagation distances from the SLM
are 7.9, 8.1, 8.25, 8.4, 8.6, 8.8, 9.1 cm and the held-out plane is set to 8.6
cm. We use a batch size of 2, and a learning rate of $4e^{-4}$. We note that
the variety of phasemaps are important for model training. For example, we
note that a dataset mainly generated using the SGD algorithm usually consists
of holographic images that have very narrow angular spectrum. Thus, we
generate the dataset with the STFT-based regularizer we present in Eq. S7. In
addition, we generate phasemaps with a set of random parameters, including
learning rates, initial phase distribution, and propagation distances. We
generate 3,000 phases for each channel and capture the intensity at 7 target
planes. Other than the held-out plane, the dataset is divided into training,
validation, and test sets with a ratio of 8:1:1. The training takes around 24
hours to converge. To parameterize and train a look up table for phase
mapping, the phase maps are first one-hot encoded, multiplied with the
parameterized lookup table, and then summed up per pixel before passing
through the full forward model pipeline.

###  S2.5. Natural defocus blur with 2.5D supervision on Quantized SLMs

Figure S3. Simulated evaluation of different RGBD supervised techniques for
generating smooth defocus blur on ideal quantized SLMs. From left to right:
Model ADMM from Choi et al. (Choi et al., 2021), Randomness Prior from Yoo et
al. (Yoo et al., 2021), our Randomness Prior Jointly-optimized implementation,
and our STFT RGBD implementation. Focused regions are highlighted with red
boxes, that from top to bottom, indicate far, center, and near distances. The
ADMM phase smoothness technique from Choi et al. (2021) produces smooth but
very small blur and has strong artifacts because it cannot be adapted to
quantization effectively. The phase randomness technique with individually
optimized frames produces more substantial blur but has poor in-focus image
quality because it uses individually optimized frames. The phase randomness
technique with jointly optimized frames improves on this in-focus image
quality, and both of these phase randomness techniques suffer from out-of-
focus artifacts on the quantized SLMs. Adding the STFT-based loss function
greatly improves the out-of-focus blur and has high image quality.

The 2.5D supervision results in our paper are generated using the multiplane
loss function presented by Choi et al. (2021). For this approach, the depth
map $D$ from an RGBD input is first decomposed into a set of binary masks
$m^{\\{j\\}}$ corresponding to a set of distances $z^{\\{j\\}}$ from the SLM
using closest distance matching,

(5) $m^{(j)}(x,y)=\begin{cases}1,\quad\text{if
}|z^{(j)}-D(x,y)|<|z^{(k)}-D(x,y)|,\forall k\neq j,\\\
0,\quad\text{otherwise.}\end{cases}$

These binary masks are then used to constrain a multiplane loss that pushes
the wavefront to reconstruct the desired RGB amplitude, $a_{\text{target}}$,
at the corresponding in-focus distances from the SLM

$\displaystyle\mathcal{L}_{\tiny\textrm{{2.5D}}}=\frac{1}{J}\sum_{j=1}^{J}\mathcal{L}$
$\displaystyle\Bigg{(}m^{(j)}\circ
s\sqrt{\frac{1}{T}\sum_{t=1}^{T}\Big{|}f_{\textrm{\tiny
model}}\left(e^{iq\left(\phi^{(t)}\right)},z^{\left(j\right)}\right)\Big{|}^{2}},$
(6) $\displaystyle m^{(j)}\circ a_{\text{target}}\Bigg{)},$

where $\circ$ is element-wise multiplication. One challenge with this approach
is that it leaves the out-of-focus parts of the displayed intensities volume
unconstrained, but this can be addressed using additional smooth phase
regularization strategies as discussed further by Choi et al. (2021). However,
the ADMM technique for smooth phase proposed by this work can only produce
very slight blur, and cannot effectively be adapted to quantization.

Alternatively, with time multiplexing, some prior works including (Yoo et al.,
2021) have proposed phase randomness approaches that can produce a much
shallower depth of field. These techniques aim to randomly send light in
differerent directions from each scene point. Over many frames, this results
in scene points that diffusely send light in all directions. Unfortunately,
this technique struggles with producing good image quality in the presence of
quantization because it independently optimizes frames. Quantization also adds
artifacts to the out-of-focus blur with this technique. To overcome these
quantization artifacts and reduce the number of frames needed for smooth out-
of-focus blur, an additional STFT-based loss can be applied to the in-focus
content.

(7)
$\mathcal{L}_{\tiny\textrm{STFT}}=\frac{1}{J}\sum_{j=1}^{J}\overline{\sigma_{\theta}^{2}}\left(m^{(j)}\circ
s\sqrt{\frac{1}{T}\sum_{t=1}^{T}\Big{|}\textrm{STFT}\left(f_{\textrm{\tiny
model}}\left(e^{iq\left(\phi^{(t)}\right)},z\right)\right)\Big{|}^{2}}\right),$

where $\overline{\sigma_{\theta}^{2}}$ is the variance of the STFT over angles
averaged over the spatial locations across the wavefront. This loss pushes the
output of the holographic display to emit light evenly in all directions from
the in-focus points. This mimics the diffuse behavior of most natural coherent
scenes. As demonstrated in Fig. S3, this enables natural blur with 2.5D
supervision on quantized SLMs.

###  S2.6. Time-multiplexing for 3D supervision

Figure S4. Simulated evaluation of focal stack supervision with 1, 2, and 8
frames from left to right on ideal continuous SLMs. Focused regions are
highlighted with red boxes, that from top to bottom, indicate far, center, and
near distances. Even without quantization on an ideal SLM, the supervision
with only 1 or 2 frames is overconstrained by the focal stack and cannot fully
reproduce the desired natural defocus blur.

Our 3D focal stack supervision technique enables very high image quality with
natural defocus effects. This technique relies on the time multiplexing in
order to reproduce the defocus effects as illustrated in Fig. S4. Some prior
works such as Shi et al. (2021) have used similar focal stack supervision with
a single frame but that is only possible with much less blur. This blur is
produced by a low frequency coherent wavefront and cannot match the natural
blur produced by a scene sending light in all directions.

###  S2.7. Time-multiplexing for 4D supervision

Figure S5. Simulated evaluation of different light field to hologram
techniques on ideal continuous SLMs. In the first column we have the full set
of reproduced light field views. From the second column to the fourth column,
we present the comparison of selected views. (Top) OLAS by Padmanaban et al.
(2019) which does not account for the interference of rays has heavily
amplified smooth content in the central view and heavily attenuated rays in
other views. (Middle) Even without quantization on an ideal SLM, our proposed
light field supervision technique with a single frame better covers the light
field views but lacks the degrees of freedom to reproduce all the rays across
all the light field views. (Bottom) Our proposed technique jointly over 8
frames has the degrees of freedom to fully reproduce the light field views.

Our approach can uniquely use a holographic display to reproduce a full set of
light field views. Prior holographic stereogram works did not account for how
interference attenuates and amplifies different rays after converting a light
field into a hologram. With the recently proposed overlap-add stereogram
(OLAS) method (Padmanaban et al., 2019), this interference results in most
rays outside of the central light field view being heavily attenuated by
destructive interference. Additionally, smooth content in the central view
gets amplified by constructive interference. Along with modeling this
interference, time multiplexing is needed to accurately reproduce a set of
light field views that fully cover the diffraction angle of the SLM. Without
time multiplexing, a single coherent wavefront produced with phase modulation
of the SLM’s resolution will not have the degrees of freedom to produce
arbitrary light field views that could naturally occur. The phenomena
discussed here is illustrated in Fig. S5.

##  S3. Additional Experimental Results

In this section, we present extra simulated and experimental results of our 3D
holographic display prototype.

Figure S6. Trade-off between the number of frames and bits for quantized SLMs,
using different optimization algorithms. We optimize SLM phase patterns with a
different number of bits and frames for 14 target images. We simulate the
setup using the ASM model for the green channel using 5 different methods,
including the Naive approach that quantizes only at the end, a variant of the
projected gradient descent approach that we elaborate in Sec. S2.1, the
Surrogate gradients approach with Sigmoid gradient, the Surrogate gradients
with Gumbel-Softmax gradient, and the Gumbel-Softmax approach. We show
averaged PSNR metrics as colormaps. In addition, we mark roughly where it
reaches 50 dB PSNR as a black line, which is achieved by the 8 bit–1 frame
type SLM. Note that the trend of black lines shifts.

#### Exploring the trade-off between number of bits and frames

We explore the trade-off space between the number of frames and the number of
bits an SLM supports. We optimize phase patterns for 14 target images using
the ASM model for the green channel using 5 different methods, including the
Naive approach that quantizes only at the end, a variant of the projected
gradient descent approach that we elaborate in Sec. S2.1, the Surrogate
gradients approach with Sigmoid gradient, the Surrogate gradients with Gumbel-
Softmax gradient, and the Gumbel-Softmax approach. We show averaged PSNR
metrics in colormaps in Fig. S6.

Overall, the projected gradient descent approach improves upon the naive
method, and notably, the surrogate gradient method with the Gumbel-Softmax
gradient outperforms other methods by a large margin. We also denote a line
for each method that roughly matches 50 dB performance that the 8 bit–1 frame
type SLM achieves. Accordingly, we observe the trend that advanced algorithms
shift the line towards the bottom left, which means it can effectively save
the number of bits and frames without sacrificing performance. Note that the
last approach does not quantize during the optimization but only replaces the
forward model with our forward model described in Eq. 6 in the manuscript.
This is analogous to the Naive approach for the Surrogate gradients approach
with Gumbel-Softmax. We note that this continuous relaxation is beneficial
especially in more constrained cases.

#### Full model visualization

Figure S7 visualizes our calibrated model, that includes learned intensity on
SLM plane, learned phase on SLM plane, learned amplitude of the optical filter
on Fourier plane, learned phase of the optical filter on Fourier plane, and
learned lookup table for phase mapping. Note that this visualization includes
many interesting aspects that present in the setup. First, the amplitude at
SLM plane $a_{\textsc{SLM}}$ reveals the envelope of the incident beam as well
as ripples and rings that occur in the physical display system. The phase at
SLM plane $\phi_{\textsc{SLM}}$ shows the phase distortion. The terms at
Fourier plane learn the shape of the physical filter we use in the setup and
especially phase term $\phi_{\mathcal{F}}$ learns a radial phase ramp, and we
note that potential propagation distance error can be learned through this
parameter.

Figure S7. Parameters visualization of the calibrated model of our holographic
display prototype (refer to Section 3 in main text). From left to right: red,
green, and blue channels. From top to bottom: learned intensity on SLM plane,
learned phase on SLM plane, learned amplitude of the optical filter on Fourier
plane, learned phase of the optical filter on Fourier plane, and learned look
up table for phase mapping.

#### Additional 2D results

Figure S8 and Figure S9 present full resolution holographic images and their
close-ups reconstructed with different CGH algorithms, including the ASM with
naive quantization, the ASM with Gumbel-Softmax quantization, our Model with
naive quantization, and our Model with Gumbel-Softmax quantization. For each
algorithm, we assess with displaying one single frame on the SLM as well as
multiplexing 8 frames (which is jointly optimized). Note that the benefits of
Gumbel-Softmax are also less prominent in the ASM case when the image quality
degradation is dominated by the model mismatch, but even then the quantitative
evaluations indicate improved performance. Thus, the benefits of our
quantization techniques are most significant when the model mismatch is
mitigated using the learned model (See columns 5-8). Corresponding
quantitative evaluation is presented in Table 1 and Table 2, indicating
results with 8 multiplexed frames and 1 frame, respectively. PSNR and SSIM
metrics are listed.

Figure S8. Experimental holographic images reconstructed with different CGH
algorithms when displaying one single frame and multiplexing 8 frames.

Figure S9. Experimental holographic images reconstructed with different CGH
algorithms when displaying one single frame and multiplexing 8 frames.

Table 1. PSNR and SSIM metrics of captured 2D results with 8 multiplexed
frames. Among all the methods, the proposed model, in tandem with the Gumbel-
Softmax (GS) quantization, achieves the highest PSNR and SSIM. Images assessed
are shown in Figure S8 (index 1 to 5) and Figure S9 (index 6 to 10) . Methods
(algorithm-propagation operator)
---
| SGD-ASM | SGD-ASM-GS | SGD-ours | SGD-ours-GS
# 1 | 20.29 / 0.821 | 20.61 / 0.829 | 27.41 / 0.947 | 28.22 / 0.954
# 2 | 17.24 / 0.796 | 17.52 / 0.807 | 22.39 / 0.909 | 23.00 / 0.916
# 3 | 20.33 / 0.655 | 20.68 / 0.663 | 25.67 / 0.803 | 26.14 / 0.811
# 4 | 18.43 / 0.632 | 18.63 / 0.643 | 22.35 / 0.815 | 22.70 / 0.811
# 5 | 16.57 / 0.508 | 16.52 / 0.486 | 19.78 / 0.731 | 19.90 / 0.718
# 6 | 18.26 / 0.789 | 18.48 / 0.799 | 23.33 / 0.911 | 23.82 / 0.915
# 7 | 15.16 / 0.066 | 15.08 / 0.063 | 16.82 / 0.088 | 16.74/ 0.088
# 8 | 18.09 / 0.654 | 18.15 / 0.643 | 21.23 / 0.764 | 21.24 / 0.758
# 9 | 18.54 / 0.748 | 18.86 / 0.751 | 22.97 / 0.934 | 23.31 / 0.832
# 10 | 18.62 / 0.820 | 18.83 / 0.827 | 23.09 / 0.889 | 23.50 / 0.898
Avg. | 18.16 / 0.649 | 18.33 / 0.652 | 22.51/ 0.769 | 22.85 / 0.770
Table 2. PSNR and SSIM metrics of captured 2D results with 1 single frame.
Among all the methods, the proposed model, in tandem with the Gumbel-Softmax
(GS) quantization, achieves the highest PSNR and SSIM. Images assessed are
shown in Figure S8 (index 1 to 5) and Figure S9 (index 6 to 10) . Methods
(algorithm-propagation operator)
---
| SGD-ASM | SGD-ASM-GS | SGD-ours | SGD-ours-GS
# 1 | 18.63 / 0.686 | 18.85 / 0.703 | 25.14 / 0.882 | 26.42 / 0.910
# 2 | 16.31 / 0.722 | 16.30 / 0.721 | 20.89 / 0.864 | 22.30 / 0.888
# 3 | 18.80 / 0.511 | 18.93 / 0.512 | 24.09 / 0.720 | 24.72 / 0.744
# 4 | 17.31 / 0.480 | 17.38 / 0.481 | 21.26 / 0.709 | 21.75 / 0.715
# 5 | 15.46 / 0.391 | 15.54 / 0.394 | 19.03 / 0.673 | 19.40 / 0.676
# 6 | 17.48 / 0.719 | 17.53 / 0.724 | 22.33 / 0.875 | 22.96 / 0.889
# 7 | 14.87 / 0.057 | 14.53 / 0.052 | 16.68 / 0.082 | 16.53 / 0.080
# 8 | 16.88 / 0.555 | 17.01 / 0.563 | 20.32 / 0.709 | 20.57 / 0.713
# 9 | 17.55 / 0.632 | 17.57 / 0.642 | 21.75 / 0.771 | 22.27 / 0.783
# 10 | 17.73 / 0.758 | 17.71 / 0.762 | 21.90 / 0.858 | 22.57 / 0.871
Avg. | 17.10 / 0.551 | 17.13 / 0.556 | 21.34 / 0.714 | 21.95 / 0.727

#### Additional 3D results

Figure S10 and Figure S11 further present comprehensive simulation results of
focal stacks and their close-ups reconstructed with different CGH algorithms,
including the AADPM from Shi et al. (Shi et al., 2021), the Model ADMM from
Choi et al. (Choi et al., 2021), the SGD-RGBD from Choi et al. (Choi et al.,
2021), the randomness control from Yoo et al. (Yoo et al., 2021), our STFT
RGBD implementation, and our focal stack implementation. All of these
holograms used are with quantization operations. For each algorithm,
reconstructed images with the camera focus at three different distances (far,
center, near) are shown. We observe that ours outperform the others in
preserving sharp content for in-focus regions while providing more natural
blur for out-of-focus regions, with the focal stack implementation on the
right being the best. Accordingly, we experimentally captured results of the
scene in Figure S10 optimized with native quantization and Gumbel-Softmax (GS)
quantization, as shown in Figure S12 and Figure S13.

Figure S14 and Figure S15 show additional experimental results of 3D
holographic display assessing different CGH algorithms (complimentary to
Figure 6 in the main paper). In this experiment, we compare algorithms of the
SGD-NH3D using RGBD input (Choi et al., 2021) with 1 frame and 8 multiplexed
frames, respectively, SGD-ours using RGBD input without and with Gumbel-
Softmax (GS), and SGD-ours using Focal Stack with GS. Quantitative assessments
are provided as PSNR metrics in the caption, as well as summarized in Table 3.
We also show the behaviour of the interpolation between supervised planes in
Fig. S16.

#### Additional 4D results

Figure S17 presents experimental results of light field reconstructed with
different CGH algorithms, including the ASM-Naive with 1 single frame, our
ASM-GS with 1 single frame, the ASM-Naive with 8 multiplexed frames, and our
ASM-GS with 8 multiplexed frames. For each example scene, we show close-ups of
content at three distances (far, intermediate, near). Our framework leads to
overall higher image fidelity for both the in-focus and out-of-focus regions.

#### Robustness to possible viewpoint Shifts

Figure S18 presents a set of captured results of a holographic scene that
validates the robustness of our image synthesis to possible viewpoint shifts.
The camera is manually translated in horizontal from left to right, for a few
millimeters. We observe no noticeable degradation in image quality over the
viewpoint shifts.

Figure S10. Simulated focal stacks reconstructed with different CGH algorithms
on ideal quantized SLMs. From left to right: AADPM from Shi et al. (Shi et
al., 2021), Model ADMM from Choi et al. (Choi et al., 2021), SGD-RGBD from
Choi et al. (Choi et al., 2021), Randomness Control from Yoo et al. (Yoo et
al., 2021), our STFT RGBD implementation, and our focal stack implementation.
Focused regions are highlighted with red squares, that from top to bottom,
indicate far, center, and near distances.

Figure S11. Simulated focal stacks reconstructed with different CGH algorithms
on ideal quantized SLMs. From left to right: AADPM from Shi et al. (Shi et
al., 2021), Model ADMM from Choi et al. (Choi et al., 2021), SGD-RGBD from
Choi et al. (Choi et al., 2021), Randomness Control from Yoo et al. (Yoo et
al., 2021), our STFT RGBD implementation, and our focal stack implementation.
Focused regions are highlighted with red squares, that from top to bottom,
indicate far, center, and near distances.

Figure S12. Experimentally captured focal stacks reconstructed with different
CGH algorithms on ideal quantized SLMs. From left to right: AADPM from Shi et
al. (Shi et al., 2021), Model ADMM from Choi et al. (Choi et al., 2021), SGD-
RGBD from Choi et al. (Choi et al., 2021), Randomness Control from Yoo et al.
(Yoo et al., 2021), our STFT RGBD implementation, and our focal stack
implementation. Focused regions are highlighted with red squares, that from
top to bottom, indicate far, center, and near distances.

Figure S13. Experimentally captured focal stacks reconstructed with different
CGH algorithms on the Gumbel-Softmax (GS) quantized SLMs. From left to right:
AADPM from Shi et al. (Shi et al., 2021), Model ADMM from Choi et al. (Choi et
al., 2021), SGD-RGBD from Choi et al. (Choi et al., 2021), Randomness Control
from Yoo et al. (Yoo et al., 2021), our STFT RGBD implementation, and our
focal stack implementation. Note that ADMM with GS doesn’t work in this case.
Focused regions are highlighted with red squares, that from top to bottom,
indicate far, center, and near distances.

Figure S14. Comparison of 3D CGH algorithms using experimentally captured
data. In this experiment, we compare algorithms of SGD-NH3D using RGBD input
(Choi et al., 2021) with 1 frame and 8 multiplexed frames, respectively, SGD-
ours using RGBD input with Gumbel-Softmax (GS), and SGD-ours using Focal Stack
with GS. For the top scene from left to right, the corresponding PSNR metrics
are 23.5 dB, 25.6 dB, 28.7 dB, 27.7 dB with respect to the RGBD all-in-focus
targets, and 19.7 dB, 21.6 dB, 23.6 dB, 26.1 dB with respect to the focal
stack. Same metrics for the bottom scene are 28.3 dB, 30.0 dB, 31.0 dB, 30.3
dB and 24.3 dB, 26.5 dB, 27.0 dB, 30.1 dB. For close-ups, red squares indicate
where the camera is focused at three distances (from top to bottom: far,
intermediate, and near).  Figure S15. Comparison of 3D CGH algorithms using
experimentally captured data. In this experiment, we compare algorithms of
SGD-NH3D using RGBD input (Choi et al., 2021) with 1 frame and 8 multiplexed
frames, respectively, our SGD-RGBD with Gumbel-Softmax (GS), and our SGD-Focal
Stack with GS. From left to right, the corresponding PSNR metrics are 21.2 dB,
22.5 dB, 24.7 dB, 24.2 dB with respect to the RGBD all-in-focus targets, and
18.5 dB, 20.3 dB, 22.3 dB, 23.7 dB with respect to the focal stack. For close-
ups, red squares indicate where the camera is focused at three distances (from
top to bottom: far, intermediate, and near).  Figure S16. Interpolation
behaviour of 3D focal stack supervised holograms. We experimentally capture 13
planes and show them at each row of closeups. Note that only odd rows are
supervised while unsupervised planes (even rows) interpolate it smoothly.
Table 3. PSNR metrics of captured 3D results using different CGH algorithms,
including the SGD-NH3D using RGBD input (Choi et al., 2021) with 1 frame and 8
multiplexed frames, respectively, SGD-our model using RGBD input, SGD-our
model using RGBD input with Gumbel-Softmax (GS), and SGD-our model using Focal
Stack (FS) supervision with GS. Images assessed are shown in Figure S14,
Figure S15, and Figure 6 in the main paper. For each cell, the first PSNR is
evaluated with respect to the RGBD all-in-focus targets, while the second one
with respect to the focal stack. Note that the first four columns are
supervised on RGBD input, where ours achieves the best all-in-focus PSNR, and
the fifth column is supervised on a focal stack, and achieves the best
performance on the PSNR metric on the target focal stack. Methods (SGD-
propagation operator)
---
| NH3D, 1 frame | NH3D, 8 frames | ours, 8 frames | ours, w/ GS, 8 frames | ours, w/ GS (FS), 8 frames
Robot | 23.5 / 19.7 | 25.6 / 21.6 | 27.9 / 23.2 |  28.7 / 23.6 | 27.7 / 26.1
Sintel Bamboo | 28.3 / 24.3 | 30.0 / 26.5 | 30.4 / 26.9 |  31.0 / 27.0 | 30.3 / 30.1
Hyperism Room | 21.2 / 18.5 | 22.5 / 20.3 | 24.0 / 21.7 |  24.7 / 22.3 | 24.2 / 23.7
Big Buck Bunny | 24.3 / 21.3 | 25.8 / 23.2 | 26.1 / 24.0 |  26.7 / 24.5 | 25.9 / 26.9

Figure S17. Comparison of 4D CGH algorithms using experimentally captured
data. In this experiment, we compare the OLAS algorithm (Padmanaban et al.,
2019), and the SGD algorithms using our ASM-Naive with 1 single frame, our
ASM-GS with 1 single frame, our ASM-Naive with 8 multiplexed frames, our ASM-
GS with 8 multiplexed frames, and our Model-GS with 8 multiplexed frames. For
close-ups, red squares indicate where the camera is focused at three distances
(from top to bottom: far, intermediate, and near).

Figure S18. Frames extracted from the camera with different spatial shifts.

## References

  * (1)
  * Choi et al. (2021) Suyeon Choi, Manu Gopakumar, Yifan Peng, Jonghyun Kim, and Gordon Wetzstein. 2021.  Neural 3D Holography: Learning Accurate Wave Propagation Models for 3D Holographic Virtual and Augmented Reality Displays.  _ACM Trans. Graph. (SIGGRAPH Asia)_ (2021).
  * Goodman (2005) Joseph W Goodman. 2005\.  _Introduction to Fourier optics_.  Roberts and Company.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ (2016).
  * Lee et al. (2022) Byounghyo Lee, Dongyeon Kim, Seungjae Lee, Chun Chen, and Byoungho Lee. 2022.  High-contrast, speckle-free, true 3D holography via binary CGH optimization.  _arXiv preprint arXiv:2201.02619_ (2022).
  * Maddison et al. (2016) Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016\.  The concrete distribution: A continuous relaxation of discrete random variables.  _arXiv preprint arXiv:1611.00712_ (2016).
  * Padmanaban et al. (2019) Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019\.  Holographic Near-eye Displays Based on Overlap-add Stereograms.  _ACM Trans. Graph._ 38, 6 (2019).
  * Peng et al. (2020) Yifan Peng, Suyeon Choi, Nitish Padmanaban, and Gordon Wetzstein. 2020.  Neural holography with camera-in-the-loop training.  _ACM Trans. Graph._ 39, 6 (2020), 1–14.
  * Shi et al. (2021) Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, and Wojciech Matusik. 2021.  Towards real-time photorealistic 3D holography with deep neural networks.  _Nature_ 591, 7849 (2021), 234–239.
  * Yoo et al. (2021) Dongheon Yoo, Youngjin Jo, Seung-Woo Nam, Chun Chen, and Byoungho Lee. 2021.  Optimization of computer-generated holograms featuring phase randomness control.  _Opt. Lett._ 46, 19 (2021), 4769–4772.