# Neural Shadow Mapping

Sayantan Datta<EMAIL_ADDRESS>McGill
UniversityMontrealQCCanada Meta Reality LabsRedmondWAUSA ,  Derek
Nowrouzezahrai<EMAIL_ADDRESS>McGill UniversityMontrealQCCanada ,
Christoph Schied<EMAIL_ADDRESS>and  Zhao Dong<EMAIL_ADDRESS>Meta
Reality LabsRedmondWAUSA

(2022)

###### Abstract.

We present a neural extension of basic shadow mapping for fast, high quality
hard and soft shadows. We compare favorably to fast pre-filtering shadow
mapping, all while producing visual results on par with ray traced hard and
soft shadows. We show that combining memory bandwidth-aware architecture
specialization and careful temporal-window training leads to a fast, compact
and easy-to-train neural shadowing method. Our technique is memory bandwidth
conscious, eliminates the need for post-process temporal anti-aliasing or
denoising, and supports scenes with dynamic view, emitters and geometry while
remaining robust to unseen objects.

Shadow mapping, neural networks

††journalyear: 2022††copyright: acmlicensed††conference: Special Interest
Group on Computer Graphics and Interactive Techniques Conference Proceedings;
August 7–11, 2022; Vancouver, BC, Canada††booktitle: Special Interest Group on
Computer Graphics and Interactive Techniques Conference Proceedings (SIGGRAPH
’22 Conference Proceedings), August 7–11, 2022, Vancouver, BC, Canada††price:
15.00††doi: 10.1145/3528233.3530700††isbn:
978-1-4503-9337-9/22/08††conference: SIGGRAPH ’22 Conference Proceedings;
August 07–11, 2022; Vancouver, BC, Canada††submissionid: papers_195††ccs:
Computing methodologies Visibility††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Rasterization MSM-3 Ours Reference
MSM-9 Ours Reference Hard shadow Soft shadow MSE: 0.216 MSE: 0.171 MSE: 0.117
MSE: 0.093 MSE: 0.101 MSE: 0.165 MSE: 0.025 MSE: 0.062 MSE: 0.170, 4ms MSE:
0.114, 8ms MSE: 0.161, 4.5ms MSE: 0.083, 8ms Figure 1. Our hard and soft
shadowing method approaches the quality of offline ray tracing whilst striking
a favorable position on the performance-accuracy spectrum. On the high-
performance end, we produce higher quality results than $n\times n$ Moment
Shadow Maps (MSM-$n$). We require only vanilla shadow mapping inputs to
generate visual (and temporal) results that approach ray-traced reference,
surpassing more costly denoised interactive ray-traced methods.

##  1\. Introduction

Shadows provide important geometric, depth and shading cues. Real-time hard
and soft shadow rendering remains a challenge, especially on resource-limited
systems. Pre-filtering based methods (Peters and Klein, 2015; Donnelly and
Lauritzen, 2006; Annen et al., 2007) are fast but approximate. They are prone
to light leaking artifacts, reduced shadow contrast, and limited contact
hardening. Interactive ray-tracing (Keller and Waechter, 2009) coupled with
post-process denoising (Schied et al., 2017; Chaitanya et al., 2017) and
upscaling (Xiao et al., 2020) can deliver high quality dynamic shadows, but
even the fastest GPU ray-tracers fall short of the performance demands of
interactive graphics. Low ray-tracing hardware adoption and the added
engineering complexity of integrating GPU ray tracers into rasterization-based
pipelines is another limitation. Pre-computation based methods (Sloan et al.,
2002; Mildenhall et al., 2020) do not generally support dynamic objects or
near-field light transport, and require significant memory.

We propose a machine learning-based method that generates high quality hard
and soft shadows for dynamic objects in real-time. Our approach does not
require ray-tracing hardware, has high performance (¡ 6ms), requires little
memory (¡ 1.5MBs), and is easy to deploy on commodity low-end GPU hardware. We
use the output of “vanilla” rasterization-based shadow mapping (i.e., no
cascades, etc.) to hallucinate temporally-stable hard and soft shadows. We
design a compact neural architecture based on the statistics of penumbra sizes
in a diversity of scenes. The network admits rapid training and generalizes to
unseen dynamic objects. We demonstrate improved quality over state of the art
in high-performance pre-filtering based methods while retaining support for
dynamic scenes and approaching reference-quality results.

We show that careful feature engineering, application and memory aware
architecture design, combined with a novel temporal stability loss results in
a system with many favorable properties: apart from compactness and high-
performance, our output precludes the need for post-process temporal anti-
aliasing (TAA), further reducing the renderer’s bandwidth requirements. We
parameterize our network by emitter size, allowing us to encode both hard and
soft shadow variation into a single trained net. We demonstrate its
effectiveness on several scenes with dynamic geometry, camera, and emitters.
Our results are consistently better than workhorse interactive methods, and
they also rival much slower (and more demanding, system- and hardware-wise)
interactive ray-tracing and denoising-based pipelines. Finally, we discuss
scene dependent optimizations that further reduce our network size and
runtime.

##  2\. Related work

Shadow mapping (Williams, 1978) and its variants are efficient shadowing
methods for point and directional lights in dynamic scenes. Shadow map
resolution and projection leads to shadow aliasing artifacts, with solutions
(e.g., depth biasing (King, 2004; Dou et al., 2014)) leading to secondary
issues and trade-offs. Modern shadow mapping relies on delicately engineered
systems that combine many cascaded maps (Zhang et al., 2006; Engel, 2006).
Here, we refer readers to a comprehensive survey (Eisemann et al., 2011).

Filtering-based methods prefilter (in emitter-space) depth-based visibility to
reduce aliasing. One simple such method weights nearby depth samples (Reeves
et al., 1987); this percentage closer filtering remains a commonly used
technique in interactive applications, with a recent variant that modulates
the filter size based on the relative blocker and receiver positions is used
to approximate soft shadows (Fernando, 2005). More recently, a new class of
filtering methods replace binary depth samples with statistical proxies,
allowing for more sophisticated pre-filtering (Annen et al., 2007; Donnelly
and Lauritzen, 2006; Annen et al., 2008) and coarse approximations of soft
shadows. Moment shadow maps (Peters and Klein, 2015) are the state of the art
of these methods, but it can suffer from banding, aliasing, light leaking in
scenes with high depth complexity.

Screen-space methods treat G-buffers, including screen-projected shadow map
data, leveraging image-space locality and GPU parallelization for efficient
filtering in a deferred shading pipeline. Here, accurate filtering here
requires the determination of an potentially-anisotropic filter kernel (due to
perspective distortion), and so depends non-linearly on the viewing angle
(Zheng and Saito, 2011) and pixel depths (MohammadBagher et al., 2010). Our
method similarly treats image-space G-buffer data, but we instead learn
compositional filters from data. High-fidelity soft shadows also benefit from
occluder depth estimates from both the emitter and shade point, of which only
the former is readily available from the shadow map and the latter can be
approximated using min- (2018) or average-filtering (2010) of the projected
shadow map. Again, we rely on learning compositions of convolution and pooling
layers to model (the effects of) these depth estimates.

Ray tracing hardware opens up an exciting new avenue for dynamic hard and soft
shadows. These methods, however, remain power-inefficient and typically
require post-process denoising (traditional (Schied et al., 2017) or machine
learning-based (Munkberg and Hasselgren, 2020; Chaitanya et al., 2017)) and
TAA (Xiao et al., 2020; Edelsten et al., 2019) to attain modest interactivity.

##  3\. Overview

Rasterized features Ray-traced targets $c_{e}+r_{e}$ $z/z_{f}$ $z-z_{f}$
$c_{c}/d$ Hard shadow Soft shadow Figure 2. Visualizing supervised learning
pairs. The network inputs are the rasterization buffers modulated by the size
of emitter ($r_{e}$). The targets are generated using ray-tracing according to
the corresponding emitter size.

Overall, our approach is straightforward; we generate a set of screen-space
buffers using a G-buffer and a shadow mapping pass before passing them as
inputs to our network. The output of the network is compared against ray-
traced shadows as target during training. Although straightforward, simply
using a UNet (Ronneberger et al., 2015; Nalbach et al., 2016) without our
proposed training and optimization methodology yields a network that is
temporally unstable, bandwidth limited, heavy (¿25MBs) and too slow (¿100ms)
for real-time use. As such, our methodology is focused on making conscious
choices to preserve memory bandwidth while having minimal impact on quality.

We train our network using screen-space buffers as features and corresponding
ray-traced shadows as targets. The approach allows for easy integration into
the rendering pipeline while providing room for integration (possible future
work) into supersampling (Xiao et al., 2020) and neural-shading (Nalbach et
al., 2016). Our approach is also suitable for tiled rendering - popular among
mobile devices. We develop a methodology to select a compact set of features
that preserve necessary information without increasing memory bandwidth. We
then show a simple technique to encode shadows with variable emitter size in a
single network. We design a loss function to enhance the temporal stability of
network without using historical buffers, thus further reducing bandwidth
requirements. We provide several network architecture optimizations aimed at
reducing memory and compute requirements. While our general architecture
supports flexible emitter sizes, we show a recipe to further optimize our
network for a fixed emitter size, enabling further trimming of network layers.

Bistro Conf. Sponza Living Room Avg. $z-z_{f}$ $z/z_{f}$ $c_{e}$ $c_{c}/d$
Figure 3.  Relative sensitivity of the selected features for various scenes.

###  3.1. Supervised training pairs

We use supervised learning to train our neural network. The training examples
are generated using rasterization and ray-tracing for features and targets
respectively. The rasterization pipeline includes a G-buffer pass followed by
a shadow mapping pass. Together they generate the following screen-space
buffers:

  * •

view-space depth $d$ and normal $\mathbf{n}$,

  * •

emitter-to-occluder depth $z$ and emitter-space normal $\mathbf{n_{e}}$,

  * •

pixel-to-emitter distance $z_{f}$, the emitter radius (size) $r_{e}$ for
spherical sources, and

  * •

dot products $\\{c_{e},c_{c}\\}$ of $\mathbf{n}$ with the emitter direction
and $\mathbf{n}$ with the viewing direction.

The ray-tracing pass generates converged images of hard and soft shadows using
a brute force Monte-Carlo sampling and a mild Gaussian filter. An 8x multi-
sample anti-alising (MSAA) is also applied to the ray-traced targets. We do
not however use MSAA in the rasterization pipeline as we expect the network to
implicitly learn anti-aliasing from the target images.

####  3.1.1. Softness control

We train a single network to predict a range of shadows with varying softness.
Note that the same input from the rasterization is used to generate both hard
and soft shadows. The softness is controlled (on a continuous scale) using a
scalar parameter indicating the size of the emitter. For training, the emitter
sizes are encoded as integer textures between 0 to 4, where 0 indicates a
point light and 4 indicates the largest emitter size (diameter 50 cm). Rather
than passing the scalar values to the network as an additional constant
screen-space buffer, a more bandwidth efficient approach is to add the scalar
as a dc-offset to an already existing buffer. We choose the cosine texture
($c_{e}$) to add the emitter-size ($r_{e}$) dc-offset to. The network targets
are also changed corresponding to the selected emitter size. See figure 2.
During inference, the network accepts a scalar between 0 and 4 and
intrinsically interpolates across discrete emitter values the network is
trained with.

###  3.2. Feature selection

While dumping the content of the rasterization pass through a network works,
it is bandwidth inefficient and adds a 2.5ms penalty to the cost of evaluating
the network. The inefficient technique involves adding a feature extraction
network, cascaded before the main network and training the two network end to
end. The feature extraction network consisting of several layers of 1x1
convolutions, compresses all 15 channels of rasterizer output down to 4. A 1x1
convolution layer acts as fully connected layer across channels without
performing any convolution across pixels. We tested a 2-layer deep 1x1
convolution network, recorded an overall error of $6.64\times 10^{-3}$ across
a suite of scenes involving hard and soft shadows.

Our approach eliminates the need for a feature extraction network by
systematic evaluation and selection of the rasterization output buffers. We
first introduce the notion of sensitivity, a metric we use to quantify the
importance of a feature. Sensitivity measures a change in the network output
due to a small perturbation in the input. Intuitively, sensitivity is lower if
a channel’s contribution in explaining the output variation is lower. Absolute
sensitivity $S_{i}$ for the $i^{th}$ input channel $f_{i}$ is given by

(1)
$S_{i}=\mathbb{E}\left[\frac{\left(\phi(f_{i}+\epsilon_{i})-\phi(f_{i})\right)}{0.1\sigma_{i}}\right],\;\;\epsilon_{i}\sim\mathcal{N}(0,0.1\sigma_{i})$

where $\phi$ is the network, the random perturbation texture $\epsilon_{i}$ is
obtained by sampling a normal distribution. The standard deviation
$\sigma_{i}$ corresponding to the $i^{th}$ channel is empirically estimated by
aggregating all pixels in the dataset for that channel. The formula is
repeated several times to reduce sampling noise. To compare the sensitivities
across different training instances, we compute relative sensitivity as
$s_{i}=S_{i}/\sum_{i}S_{i}$.

Vgg-Off Vgg-On Vgg-Off Vgg-On MSE: 0.038 MSE: 0.035 MSE: 0.020 MSE: 0.057 MSE:
0.007 MSE: 0.046 Figure 4. Effect of VGG loss on the final output. VGG loss
produces sharper edges for geometry and shadow silhouettes.

Armed with relative sensitivity as our yardstick, our problem is thus
selecting a subset of features from a set of features
$U=\left\\{d,\mathbf{n},z,\mathbf{n_{e}},z_{f},c_{e},c_{c}\right\\}+\left\\{z-z_{f},z/z_{f},c_{c}/d,\mathbf{n}\cdot\mathbf{n_{e}}\right\\}$.
The first set of buffers are obtained directly from the rasterization while
the second set is a composition from the first set. We take a tiered approach
for selecting the best features. In the first pass, we train our network with
all buffers in set $U$ and reject buffers with low relative sensitivity. We
repeat the process until all buffers have sensitivity higher than 1.5%. Refer
to supplemental111Attached herewith at the end material, section 1.0.2 for
more details. Our final set of buffers is as shown in figure 3. We obtain
nearly the same error ($6.67\times 10^{-3}$) as having a feature extraction
while saving an extra 2.5ms.

###  3.3. Loss function and temporal stability

The loss function plays two main role in defining the characteristics of our
network. It shapes the network to better fit the hard edges for shadow
silhouette and geometry, essentially performing post-process anti-aliasing.
Second, our loss function improves temporal stability without using any
historical buffers for training and inference. Our approach not only saves
memory bandwidth but also enables easier integration into tiled renderers.

We achieve the first objective using a weighted combination of per-pixel
difference and VGG-19 (2015) perceptual loss. The effect of VGG loss on the
final output is shown in figure 4 and clearly shows the anti-aliasing effect
of VGG-19 on hard edges.

Network output (Perturb-On) Perturb-Off Perturb-On Temporal -21% Spatial +19%
Temporal -29% Spatial +14% Figure 5. Comparing the temporal and spatial effect
of perturbation loss. The application of perturbation loss reduces temporal
instability while causing an increase in spatial blur as shown in the cutouts.
We measure temporal instability by comparing the network output between
consecutive frames while we measure spatial error by comparing the network
output with reference. Temporal instability and spatial errors are represented
using false colors purple/gold and red/blue respectively.

Existing methods typically improves temporal stability using historical
buffers to better support the network during inference (Xiao et al., 2020;
Edelsten et al., 2019) and also to reshape the loss function (Kaplanyan et
al., 2019; Holden et al., 2019) during training. In our case, we do not use
historical buffers but use random perturbations of the input buffers for
reshaping the loss landscape during training. Temporal instabilities arise due
to shadow-map aliasing, where shadow map texels do not align one-to-one with
screen pixels. As such small, movement in camera or emitter can cause large
changes in depth comparisons, especially around shadow silhouettes. Inspired
from noise-to-noise training (Lehtinen et al., 2018), we train our network to
learn from pairs of noisy inputs, in addition to the traditional supervised
learning pair. Our network intrinsically learns to damp sharp changes due to
small perturbations with minimal impact on overall quality as shown in figure
5. At each training iteration, we perturb the camera and emitter position by a
small value proportional to the distance from the scene and size of emitter.
For each perturbation, we collect the input buffers for training. The target
is evaluated for only one of the perturbations. We evaluate the network on
each perturbations and minimize the differences in the perturbed outputs as an
additional loss function. All network instances evaluating the perturbed
inputs share the same weights while backpropagation is only enabled through
one instance, as

(2)
$\mathcal{L}=L({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}x_{0}},\tilde{x})+\textstyle\sum_{i=1}^{p}L({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}x_{0}},x_{i}),$

where $L(y,\tilde{y})=\alpha\cdot|y-\tilde{y}|+(1-\alpha)\cdot
VGG19(y,\tilde{y})$, and $x_{i}$ and $\tilde{x}$ are the network outputs and
target. Only one network
output-${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}x_{0}}$
has backpropagation enabled through it. We set $\alpha=0.9$ and the number of
perturbations $p=3$.

###  3.4. Temporal stability measurement

Reduction in temporal instability Hard Soft Perturb. loss TAA (+1.3ms) Bistro
Emitter Bistro Objects Bistro Camera Bistro Mixed Conf. Sponza Living Room
Figure 6. Comparing reduction in temporal instability between Perturbation
Loss and TAA. Reference is trained without perturb. loss or TAA. TAA is
implemented as an additional pass after network (trained without perturbation
loss) evaluation requiring extra 1.3ms.

Several techniques exist for measuring perceptual similarity with respect to a
single reference frame (Wang et al., 2003; Andersson et al., 2020) or
reference video (Mantiuk et al., 2021). These techniques measure the spatio-
temporal difference which indicates the overall reconstruction quality across
screen-space and time. Since we sacrifice spatial quality for temporal
stability, using these metric may not indicate a reduction in temporal
instability due perturbation loss, even when there is a clear visual
improvement in temporal stability. Thus we formulate our own metric to measure
only the temporal changes without considering spatial similarity with
reference. To measure flickering, we find the motion-vector adjusted per-pixel
temporal difference (Yang et al., 2020). Since flickering can be quantified as
an abrupt change in the pixel intensities between frames, we penalize the
large differences more by passing the temporal pixel difference through an
exponential. We aggregate the result across all pixels and frames, with

(3) $E=\frac{1}{P}\sum_{p,t}\left\\{\exp(\alpha D_{t}(p))-1\right\\}~{},$

where $D_{t}(p)=|I_{t}(p)-I_{t-1}(m(p))|$ is a per-pixel difference between
two consecutive frames at time $t$ and $m(p)$ abstracts away the motion-vector
adjusted lookup at pixel $p$ in the previous frame. We set $\alpha=3$, which
controls the penalty for large changes in intensity through time, and the
normalizing factor $P$ is the total number of pixels. We reject pixels that
fail depth and normal comparison with its reprojection.

Figure 6 contrasts the effect on temporal stability between our loss and TAA
(1 last frame): the improvement in temporal stability with our perturbation
loss is strongest in scenes with dynamic emitters and non-negligible, albeit
smaller with dynamic view.

Time (ms) 4.99 0.00 (Optimized) Layer 0 2.01 1.72 0.91 0.93 0.39 0.40 0.57
0.59 Layer 4 0.72 0.69 1.67 1.32 5.98 0.24 Layer 0 $(1K,2K,4)$ $(512,1K,16)$
$(256,512,32)$ $(128,256,64)$ $(64,128,256)$ $(1K,2K,1)$ Figure 7. Layer-wise
performance optimization for a 1024$\times$2048 input.

###  3.5. Network architecture and optimizations

The original UNet (Ronneberger et al., 2015) architecture is too slow (¿100ms)
to fit into a real-time graphics pipeline. As such, we start with trimming the
network down. Our generic network has 5 layers, however, each layer composed
of one 3x3 convolution and one 1x1 convolution layers as opposed to the
standard double 3x3 convolution. A major departure from the original UNet is
using bi-linear interpolation instead of expensive transpose convolutions for
upscaling. We also use algebraic sum instead of a concatenation layer for
merging the skip connections on the decoder side. A positive side effect of
using a sum layer is the reduction in the number of hidden units on the
decoder side. With these modifications we reduce the network size from 25MB to
just 2.5MB, while the runtime is minimized to 28ms. Quantizing the network to
half-precision further reduces the size to 1.5MB and 17ms runtime.

Other modifications for improving temporal stability without affecting
performance includes using Average-pool instead of Max-pool and removing the
skip connection in the first layer. Replacing max-pool with average-pool
reduces extremities during processing and smooths out the output. As raw
shadow map depth values are prone to aliasing noise, removing the first skip
connection ensures the noisy input does not affect the output directly.

Hard-shadow Soft-shadow Before, 17.3ms After, 5.84ms MSE: 0.077  MSE: 0.085
Before, 17.3ms After, 5.84ms MSE: 0.118  MSE: 0.128 Figure 8. Figure showing
the effect of all optimizations in section 3.5. Ours, 8ms, MSE: 0.095 Hard-
shadow MSM-3, 4ms, MSE: 0.110 MSE: 0.071 MSE: 0.121 Ours Reference MSM-3 MSE:
0.077 MSE: 0.092 Ours, 8ms, MSE: 0.097 Soft-shadow  MSM-9, 4.5ms, MSE: 0.156
PCSS, 5.5ms, MSE: 0.112 5SPP + SVGF, 11.5ms, MSE: 0.126 Ours MSM-9 Reference
PCSS 5 SPP+SVGF MSE: 0.051 MSE: 0.081 MSE: 0.148 MSE: 0.059 MSE: 0.047 MSE:
0.071 MSE: 0.063 MSE: 0.085 Left/Right: Superimposed error Middle (zoom-ins):
Network output Red: Excess shadow Blue: Missing shadow Figure 9.  Comparing
our network’s ability to generalize to unseen objects (buddha, bunny, dragon)
with other competing techniques (MSM, PCSS, Raytracing & Denoising) for hard
and soft shadows. MSM-3 and MSM-9 are Moment Shadow Map variants using
3$\times$3 and 9$\times$9 prefiltering kernels.

At this stage, we analyze the performance and error of our network before
optimizing it further. Our validation error is $6.67\times 10^{-3}$ over an
ensemble test scenes. From figure 7, we see that the first layer (combined
encoder-decoder) requires more time compared to the rest of the layers
combined. Moving from inner (#4) to outer layers (#0), the resolution is
quadrupled while the number of channels is halved. Consequently, the effective
cached memory bandwidth doubles as we move from inner to outer layers;
however, with increasing resolution, memory operations are also more prone to
cache misses. In practice, we see more than doubling of runtime as we move
towards the outer layers. Refer supplemental section 3.

We further optimize by changing the first layer which consumes
disproportionately more time. A naive approach is to replace the first layer
with a downsampler on the encoder side and upsampler on the decoder side of
UNet . However, simple downsampling and upsampling loses information contained
in the input and also produces less sharp output. Instead, we flatten a 2x2
square of pixels into 4 separate channels and use the restructured buffer as
the input for second layer. Thus we rearrange the input and change the buffer
dimensions from ($h\times w\times ch$) to ($h/2\times w/2\times 4ch$). More
concretely, instead of feeding the first layer with full resolution
($1024\times 2048$) input with 4 channels, we feed the second layer directly
with quarter resolution ($512\times 1024$) input with 16 channels. We do the
inverse on the decoder side; rearrange 4 output channels into a 2x2 pixel
square. Note that the rearrangement of buffers does not add any extra
temporary storage for the second layer while removing the first layer (Conv2D
operations) completely. On the decoder side, to improve training convergence,
we upscale the first output channel to full resolution using a bi-linear
interpolation. We then add rest of the three channels to the interpolated
output, filling in rest of the details. The performance of our optimized
network is 5.8ms.

###  3.6. Network depth optimizations

a. Model estimating penumbra$r_{e}$EmitterCenter
line$z_{min}$$z_{max}$$z_{f}$$x_{a}<0$$x_{a}>0$Penumbra$\approx
x_{a}+x_{b}$$x_{a}$$x_{b}$b. Estimating
parametersPA$z_{m}$$\theta$$\theta_{\delta}$MDBQ$x_{a}$C$x_{b}$ Figure 10.  A
simplified model to estimate the penumbra size at a given pixel. We assume our
occluder is spherical in shape, forming a convex bounding-sphere around the
occlusion geometry as shown in figure (a) on the left. Figure (b) shows a
simplified diagram to estimate the parameters $\theta,\theta_{\delta}$.

Our optimizations so far are generic and apply across scene and emitter
configurations. Below, we explore scene specific optimizations and tune our
network architecture for compactness. Shallower networks have many pragmatic
benefits: it has exponentially (power of 2) fewer parameters, is faster to
train, and admits faster runtime inference. Instead of relying on adhoc
architecture tuning, we will choose architectures based on their ability to
capture the shadowing effects we target. Specifically, we will build a simple
model to estimate the maximum penumbra size for a scene configuration, and
then relate this size exactly to the depth of the network suited to
reproducing them. We empirically validate our model.

To compute the world space penumbra size, our simplified model assumes a
spherical occluder (or, conservatively, a bounding sphere around occluding
geometry). When generating training samples, we additionally measure the
minimum and maximum occluder distances $z_{max}$, $z_{min}$ (figure 2, a). We
then estimate the penumbra width at a pixel as the sum of inner ($x_{a}$) and
outer ($x_{b}$) penumbra (figure 2, b) as
$\\{x_{a},x_{b}\\}=z_{f}\,\tan(\theta\pm\theta_{\delta})-r_{e}$. We derive the
parameters $\theta,\theta_{\delta}$ in the supplemental, section 4.

Table 1.  MSE for variable penumbra sizes and with 3/5/7-layer nets. Predicted Penumbra Width (in pixels) |  | Network layers $\rightarrow$
---
Receptive field size (in pixels)
| 3 $\rightarrow$ 24
---
| 5 $\rightarrow$ 96
---
| 7 $\rightarrow$ 384
---
21 | 0.005 | 0.005 | 0.012
90 | 0.083 | 0.009 | 0.018
180 | 0.161 | 0.042 | 0.019

After computing a histogram of penumbra sizes in screen-space for each pixel
across all the training frames, we select the highest 95th percentile penumbra
size as a conservative bound on the receptive field size requirements for our
neural architecture.

We can modulate the per-layer convolutional layer parameters (kernel size,
stride) and pooling operation parameters in order to meet the target receptive
field requirements. If we set each convolutional layers to halve the spatial
resolution, the effective receptive field of the network grows with $\times
2^{l}$ for an $l$-layer network. Exclusively using $3\times 3$ kernels, we can
solve for $l=\log_{2}(p_{w}/3)$, where $p_{w}$ is the conservative screen-
space penumbra width.

MSE: 0.095 MSE: 0.096 MSE: 0.096 MSE: 0.097 Figure 11.  Generalization to
untrained objects (red, first row) in a trained scene. Second row visualizes
false color error w.r.t. reference. From left to right, emitter size increases
linearly from 0 (hard shadow) to 50 cm diameter.

Table 1 provides an empirical validation of our technique. We train three
networks with 3,5, and 7 layers using the same dataset. The dataset consist of
scenes with mixture of penumbra sizes. The penumbra size estimates are
computed using our model. During inference, networks with receptive field
lower than the predicted penumbra size perform poorly as marked in red color.
A more detailed analysis of the empirical verification is provided in the
supplemental section 4.1.

##  4\. Results and comparisons

We demonstrate our method on a diversity of scenarios. We augment static
environments (e.g., rooms) included in our training set to include untrained
objects at runtime, illustrating an important use case for interactive
settings like games (figure 6). We train a single network on the Bistro
interior scene with varying emitter sizes, emitter positions and camera
trajectories and introduce the (untrained objects) Buddha, bunny, and dragon
for validation.

##### Comparison.

In the soft-shadowing regime, our comparisons focus on two classes of baseline
methods: first, high-performance rasterization-based approximations such as
Moment Shadow Maps (MSM) (Peters and Klein, 2015) and Percentage Closer Soft
Shadows (PCSS) (Fernando, 2005) that align with our engineering (i.e., fully
rasterization-based; no ray-tracing) and performance targets as our primary
baseline; second, we use interactive GPU ray-tracing with post-process
denoising as a more accurate “interactive” baseline, i.e., 5-SPP raytracing
with SVGF (Schied et al., 2017). Note that, unlike our method, neither MSM nor
PCSS allow explicit control of penumbra style using emitter size; as such, we
adjust the pre-filtering kernel size for MSM and PCSS to achieve a penumbra
size that most closely matches reference renderings. We use kernel sizes of
3$\times$3 for MSM and 9$\times$9 for PCSS. Our PCSS baseline also includes a
depth-aware post-filtering.

Our method consistently improves shadow quality at competitive performances
(figure 9). Refer to our video to observe the temporal stability of our
results. For hard shadows, we compare to $3\times 3$ MSM, obtaining alias-free
shadows without any light leaking. Please refer to the supplemental section 7
for more results, comparisons.

Runtime comparisons are measured at a resolution of 2k$\times$1k on an AMD
5600X CPU and Nvidia 2080Ti GPU. The timings in all figures, both main paper
and supplemental, exclude G-Buffer generation which consistently requires an
additional 2-3 ms (depending on the scene) across all techniques. Each scene
is trained on $\leq 400$ images of resolution 2k$\times$1k on a cluster for
roughly 16 hours (75 training epochs).

##  5\. Limitations

Our technique shares similar limitations to other screen-space methods. The
unavailability of layered depth information, both in camera- (Ritschel et al.,
2009) and emitter-space (Jesus Gumbau, 2018) leads to an ill-posedness of the
problem that results in approximation error. In camera space, the lack of
peeled-depth data complicates the determination of mutual visibility between
pixels. Similarly, computing the blur kernel size to soften shadow silhouettes
relies on the distance between shading points and occluders, which is also
unavailable in our setting. Our network and training methodology are
effectively designed to compensate for this ill-posedness, bridging the visual
gap in a diversity of object/scene arrangements by leveraging complex patterns
inherent in the data. Figure 12 highlights a standard failure case and our
supplemental includes additional discussion (section 8).

Foliage-hard Overlapping Mesh Foliage-soft MSE :0.047 MSE: 0.003 MSE: 0.022
Figure 12.  Limitations of our method with high-frequency/high-depth
complexity. Orange, blue, and green boxes highlight visual artifacts due to
limited emitter and camera depth information, and due to undersampled training
scenarios (i.e., loss of fine details).

##  6\. Conclusions

We presented a compact, fast neural method and training loss suited to
temporally-coherent hard and soft shadows synthesis using only basic shadow
map rasterized inputs. We showed that – with a careful, problem-specific
architecture design and a new, simple temporal loss – a single small network
can learn to hallucinate hard and soft shadows from varying emitter sizes and
for a diversity of scenes. It is robust to the insertion of unseen objects,
requires only a modest training budget, and precludes the need for any post-
process denoising and/or TAA. Our approach yields stable hard and soft shadows
with performance similar to workhorse interactive approximations and higher
quality than (more expensive) GPU-raytracing and denoising (and TAA)
alternatives.

Rasterization-based approaches for soft and hard shadows rely on heuristics
and brittle manual tuning to achieve consistent, visually-desirable results.
Our data-driven approach precludes such tuning, improving shadow quality at a
modest cost, producing plausible and temporally-coherent soft shadows without
any ray-tracing. Ours is a compact neural shading-based framework (Nalbach et
al., 2016) suitable for low-power tiled-rendering systems, striking an
interesting trade-off in a complex design space. We demonstrate benefits that
largely offset the added training and integration complexity.

In the future, pursuing more aggressive neural architecture optimizations,
including quantization and procedural architecture search, could likely
further improve inference performance. When coupled with sparsification using,
e.g., lottery ticket-based method (Frankle and Carbin, 2018), we suspect that
significant additional performance gains are possible, all without sacrificing
quality.

###### Acknowledgements.

We thank the reviewers for their constructive feedback, the ORCA for the
Amazon Lumberyard Bistro model (Lumberyard, 2017), the Stanford CG Lab for the
Bunny, Buddha, and Dragon models, Marko Dabrovic for the Sponza model and
Morgan McGuire for the Bistro, Conference and Living Room models (McGuire,
2017). This work was done when Sayantan was an intern at Meta Reality Labs
Research. While at McGill University, he was also supported by a Ph.D.
scholarship from the Fonds de recherche du Québec – nature et technologies.

## References

  * (1)
  * Andersson et al. (2020) Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020.  FLIP: A Difference Evaluator for Alternating Images.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ 3, 2 (2020), 15:1–15:23.
  * Annen et al. (2007) Thomas Annen, Tom Mertens, Philippe Bekaert, Hans-Peter Seidel, and Jan Kautz. 2007.  Convolution Shadow Maps. In _Proceedings of the 18th Eurographics Conference on Rendering Techniques_ (Grenoble, France) _(EGSR’07)_. Eurographics Association, Goslar, DEU, 51–60.
  * Annen et al. (2008) Thomas Annen, Tom Mertens, Hans P. Seidel, Eddy Flerackers, and Jan Kautz. 2008\.  Exponential Shadow Maps. In _Proceedings of Graphics Interface 2008_ (Windsor, Ont., Canada) _(GI ’08)_. Canadian Information Processing Society, CAN, 155–161.
  * Chaitanya et al. (2017) Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017\.  Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder.  _ACM Trans. Graph._ 36, 4, Article 98 (July 2017), 12 pages.
  * Community (2018) Blender Online Community. 2018\.  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam.
  * Donnelly and Lauritzen (2006) William Donnelly and Andrew Lauritzen. 2006.  Variance Shadow Maps. In _Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games_ (Redwood City, California) _(I3D ’06)_. Association for Computing Machinery, New York, NY, USA, 161–165.
  * Dou et al. (2014) Hang Dou, Yajie Yan, Ethan Kerzner, Zeng Dai, and Chris Wyman. 2014.  Adaptive Depth Bias for Shadow Maps. In _Proceedings of the 18th Meeting of the ACM SIGGRAPH Symposium on I3D Graphics and Games_ (San Francisco, California) _(I3D ’14)_. Association for Computing Machinery, New York, NY, USA, 97–102.
  * Edelsten et al. (2019) Andrew Edelsten, Paula Jukarainen, and Anjul Patney. 2019\.  Truly Next-Gen: Adding Deep Learning to Games and Graphics.  NVIDIA Sponsored Sessions (Game Developers Conference.
  * Eisemann et al. (2011) Elmar Eisemann, Michael Schwarz, Ulf Assarsson, and Michael Wimmer. 2011.  _Real-Time Shadows_ (1st ed.).  A. K. Peters, Ltd., USA.
  * Engel (2006) Wolfgang F Engel. 2006\.  _ShaderX5: Advanced Rendering Techniques_.  C.R. Media.
  * Fernando (2005) Randima Fernando. 2005\.  Percentage-Closer Soft Shadows. In _ACM SIGGRAPH 2005 Sketches_ (Los Angeles, California) _(SIGGRAPH ’05)_. Association for Computing Machinery, New York, NY, USA, 35–es.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The Lottery Ticket Hypothesis: Training Pruned Neural Networks.  _CoRR_ abs/1803.03635 (2018).  arXiv:1803.03635
  * Holden et al. (2019) Daniel Holden, Bang C. Duong, Sayantan Datta, and Derek Nowrouzezahrai. 2019.  Subspace Neural Physics: Fast Data-Driven Interactive Simulation. In _Proceedings of the 18th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (Los Angeles, California) _(SCA ’19)_. Association for Computing Machinery, New York, NY, USA, Article 6, 12 pages.
  * Jesus Gumbau (2018) Mateu Sbert Jesus Gumbau, Miguel Chover. 2018\.  _Screen Space Soft Shadow, GPU Pro 360 Guide to Rendering_ (1st ed.).  A. K. Peters, Ltd., USA.
  * Kallweit et al. (2021) Simon Kallweit, Petrik Clarberg, Craig Kolb, Kai-Hwa Yao, Theresa Foley, Lifan Wu, Lucy Chen, Tomas Akenine-Moller, Chris Wyman, Cyril Crassin, and Nir Benty. 2021.  The Falcor Rendering Framework.  https://github.com/NVIDIAGameWorks/Falcor https://github.com/NVIDIAGameWorks/Falcor.
  * Kaplanyan et al. (2019) Anton S. Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019\.  DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression Using Learned Statistics of Natural Videos.  _ACM Trans. Graph._ 38, 6, Article 212 (nov 2019), 13 pages.
  * Keller and Waechter (2009) Alexander Keller and Carsten Waechter. 2009.  Real-time precision ray tracing.  US patent US20070024615A1.
  * King (2004) Gary King. 2004\.  _Shadow mapping algorithms_.  Nvidia. 354–355 pages.
  * Lehtinen et al. (2018) Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018.  Noise2Noise: Learning Image Restoration without Clean Data.  _CoRR_ abs/1803.04189 (2018).  arXiv:1803.04189
  * Lumberyard (2017) Amazon Lumberyard. 2017\.  Amazon Lumberyard Bistro, Open Research Content Archive (ORCA).  http://developer.nvidia.com/orca/amazon-lumberyard-bistro http://developer.nvidia.com/orca/amazon-lumberyard-bistro.
  * Mantiuk et al. (2021) Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021.  FovVideoVDP: A Visible Difference Predictor for Wide Field-of-View Video.  _ACM Trans. Graph._ 40, 4, Article 49 (July 2021), 19 pages.
  * Marton Tamas (2018) Viktor Heisenberg Marton Tamas. 2018\.  _Practical Screen Space Soft Shadow, GPU Pro 360 Guide to Shadows_ (1st ed.).  A. K. Peters, Ltd., USA.
  * McGuire (2017) Morgan McGuire. 2017\.  Computer Graphics Archive.  https://casual-effects.com/data https://casual-effects.com/data.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  _CoRR_ abs/2003.08934 (2020).  arXiv:2003.08934
  * MohammadBagher et al. (2010) Mahdi MohammadBagher, Jan Kautz, Nicolas Holzschuch, and Cyril Soler. 2010.  Screen-Space Percentage-Closer Soft Shadows. In _ACM SIGGRAPH 2010 Posters_ (Los Angeles, California) _(SIGGRAPH ’10)_. Association for Computing Machinery, New York, NY, USA, Article 133, 1 pages.  https://doi.org/10.1145/1836845.1836987
  * Munkberg and Hasselgren (2020) Jacob Munkberg and Jon Hasselgren. 2020.  Neural Denoising with Layer Embeddings.  _Computer Graphics Forum_ (2020).
  * Nalbach et al. (2016) Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2016.  Deep Shading: Convolutional Neural Networks for Screen-Space Shading.  _CoRR_ abs/1603.06078 (2016).  arXiv:1603.06078
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in PyTorch.  (2017).
  * Peters and Klein (2015) Christoph Peters and Reinhard Klein. 2015.  Moment Shadow Mapping. In _Proceedings of the 19th I3D Symposium_ (San Francisco, CA) _(i3D ’15)_. Association for Computing Machinery, New York, NY, USA, 7–14.
  * Reeves et al. (1987) William T. Reeves, David H. Salesin, and Robert L. Cook. 1987\.  Rendering Antialiased Shadows with Depth Maps.  _SIGGRAPH Comput. Graph._ 21, 4 (Aug. 1987), 283–291.
  * Ritschel et al. (2009) Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009\.  Approximating Dynamic Global Illumination in Image Space. In _Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games_ (Boston, Massachusetts) _(I3D ’09)_. Association for Computing Machinery, New York, NY, USA, 75–82.  https://doi.org/10.1145/1507149.1507161
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  _CoRR_ abs/1505.04597 (2015).  arXiv:1505.04597
  * Schied et al. (2017) Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017.  Spatiotemporal Variance-Guided Filtering: Real-Time Reconstruction for Path-Traced Global Illumination. In _Proceedings of High Performance Graphics_ (Los Angeles, California) _(HPG ’17)_. Association for Computing Machinery, New York, NY, USA, Article 2, 12 pages.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _International Conference on Learning Representations_.
  * Sloan et al. (2002) Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002\.  Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments.  _ACM Trans. Graph._ 21, 3 (July 2002), 527–536.
  * Wang et al. (2003) Z. Wang, Eero Simoncelli, and Alan Bovik. 2003.  Multiscale structural similarity for image quality assessment.  _Conference Record of the Asilomar Conference on Signals, Systems and Computers_ 2, 1398–1402Vol.2.
  * Williams (1978) Lance Williams. 1978\.  Casting Curved Shadows on Curved Surfaces. In _Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’78)_. Association for Computing Machinery, New York, NY, USA, 270–274.
  * Xiao et al. (2020) Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020\.  Neural Supersampling for Real-Time Rendering.  _ACM Trans. Graph._ 39, 4, Article 142 (July 2020), 12 pages.
  * Yang et al. (2020) Lei Yang, Shiqiu Liu, and Marco Salvi. 2020.  A Survey of Temporal Antialiasing Techniques.  _Computer Graphics Forum_ 39, 2 (2020), 607–621.  arXiv:https://onlinelibrary.wiley.com/olddoi/pdf/10.1111/cgf.14018
  * Zhang et al. (2006) Fan Zhang, Hanqiu Sun, Leilei Xu, and Lee Kit Lun. 2006\.  Parallel-Split Shadow Maps for Large-Scale Virtual Environments. In _Proceedings of the 2006 ACM International Conference on Virtual Reality Continuum and Its Applications_ (Hong Kong, China) _(VRCIA ’06)_. Association for Computing Machinery, New York, NY, USA, 311–318.
  * Zheng and Saito (2011) Zhongxiang Zheng and Suguru Saito. 2011.  Screen Space Anisotropic Blurred Soft Shadows. In _ACM SIGGRAPH 2011 Posters_ (Vancouver, British Columbia, Canada) _(SIGGRAPH ’11)_. Association for Computing Machinery, New York, NY, USA, Article 75, 1 pages.  https://doi.org/10.1145/2037715.2037799

Neural Shadow Mapping - Supplemental

##  1\. Feature engineering

####  1.0.1. Overfitting and underfitting

Neural networks are known for their ability to fit a dataset, even when there
is no underlying relationship between the features and the target - a
phenomenon known as overfitting. An extreme example is mapping our ray-traced
targets to white noise textures as features. In this case, the neural network
acts as a hash-map, mapping input to the output. This might work for a very
small dataset but will obviously fail for any larger dataset. In practice,
some features provide crucial information and some do not. Thus, to isolate
noise from information, we need a diverse dataset. In our case, we can create
diversity by collecting data across various emitter and camera positions, and
scenes. In fact, as we generate more training data, we run into the regime of
underfitting, where the network lacks enough capacity to learn the details in
the dataset.

Table 1.  Validation error for various combination of buffers. At each row, we add 2 new features and remove features with sensitivity below 1.5%. $+/-$ indicates set addition and subtraction. Before pruning | After pruning
---|---
| Fea.
---
Count
| Feature
---
Description
| Err
---
$10^{-3}$
| Fea.
---
Count
| Feature
---
Description
| Err
---
$10^{-3}$
2 | $\left\\{z/z_{f},c_{e}\right\\}$ | $6.86$ | No change
4 | $+\left\\{z-z_{f},c_{c}/d\right\\}$ | $6.65$ | No change
6 | $+\left\\{\mathbf{n}\cdot\mathbf{n_{e}},d\right\\}$ | $6.61$ | 4 | $-\left\\{\mathbf{n}\cdot\mathbf{n_{e}},c_{c}/d\right\\}$ | $6.69$

####  1.0.2. Feature selection

As described in the main paper, we use sensitivity as our metric to select and
prune our features from the set
$U=\left\\{d,\mathbf{n},z,\mathbf{n_{e}},z_{f},c_{e},c_{c}\right\\}+\left\\{z-z_{f},z/z_{f},c_{c}/d,\mathbf{n}\cdot\mathbf{n_{e}}\right\\}$.
However, pruning based on sensitivity alone does not lead to a unique
combination of buffers. There are sever combinations with similar sensitivity.
We use validation error as tie-breaking rule in such scenario. For example as
shown in table 1, buffer combinations $\\{z/z_{f},z-z_{f},c_{e},c_{c}/d\\}$
and $\\{z/z_{f},z-z_{f},\mathbf{n}\cdot\mathbf{n},d\\}$ have similar
sensitivity but one has lower validation error.

##  2\. Temporal loss

Frame $t$ Frame $t-1$ Frame $t$ Frame $t-1$ Shadow map (SM) A. Small camera
movement B. Large camera movement Pixel reprojection SM lookup Perturbation 0
Perturbation 1 C. Small SM perturbation D. Large SM perturbation Figure 1.
Comparing the effect of motion-vector and perturbation loss on shadow map
texture lookup. A and C indicate that a small camera movement in time is
equivalent to a small perturbation of SM in space. Similarly, B and D indicate
that a large camera movement is same as large perturbation of SM in space.

In this section, we compare the similarities an dissimilarities between our
perturbation loss with motion vector (Xiao et al., 2020) based loss function.
A motion-vector finds a pixel’s location in the previous frame by projecting
the pixel’s world-space position on the previous frame. A motion-vector based
loss thus computes the motion-vector compensated temporal pixel difference in
the network output as error which is then backpropagated through the network
for learning. Note that a motion-vector based loss uses historical data only
during training.

We argue that our perturbation loss has similar end effect as motion vector
based loss while offering more control over temporal stability and simplicity
in data collection. In case of motion vector loss, due to discrete nature of
pixels, the reprojected pixel in the previous frame may correspond to a
slightly different world-space location. Assuming the emitter is fixed, a
small difference in world space value of a pixel across time may result in
different shadow-map texture lookup, as shown in figure 1 (b). This is
equivalent to perturbing the emitter (our approach) while keeping the camera
position fixed as shown in (d). However, for motion-vector loss, if the
framerate is too high, the world-space difference in pixel may be small. The
pixel and its reprojection may share the same shadow map texel, as shown in
(a). Same is true if the shadow map texels are large. In both cases, the
temporal error is zero and the network does not learn. Conversely, if the
framerate is too low, we may not find a valid reprojection and the error must
be forced to zero. As such, a careful balance of the framerate is required for
motion-vector losses. However, in our case, we can directly adjust the emitter
perturbation such that the differences are large enough for non-zero
backpropagation as shown in figure (d). This is achieved by by setting the
perturbation magnitude proportional to the distance of emitter from the scene
and size of the emitter. Additionally, motion-vector losses rely on long
sequence of key-framed images which adds additional complexity to data
collection pipeline. Our approach do not rely on temporal information, as such
we can sample the scene with arbitrary camera, emitter and object trajectories
with any desired framerate.

##  3\. Performance optimization

We discuss an interesting aspect of UNet architecture that we did not discuss
in the main document.

Table 2.  Layer-wise compute, learnable parameters, and temporary storage read/write access for a 5-layer network processing 1024x2048 resolution inputs. # |  | runtime(ms)/
---
resolution
| Compute
---
(GFlops)
| # parameters
---
($\times 10^{3}$)
| Temp. storage IO
---
(MPixels)
0 |  | 11.0ms
---
(1Kx2K)
7.15 | 3.4 | 278.9
1 |  | 3.67ms
---
(512x1K)
8.05 | 15.4 | 117.4
2 |  | 1.63ms
---
(256x512)
8.05 | 61.4 | 58.7
3 |  | 0.79ms
---
(128x256)
8.05 | 245.8 | 29.3
4 |  | 0.17ms
---
(64x128)
2.68 | 327.68 | 6.29

From table 2, we see that the runtime performance of our network is
proportional to the temporary storage, not the number of compute operations.
Notice how the compute flops are nearly constant for the first 4 layers, yet
the runtime drops as we go down the layers. This indicates the performance is
bounded by memory bandwidth. Required memory bandwidth is proportional to the
size of temporary buffers and how well the buffers are cached. Caching is most
effective when the size of the temporary buffers are small. In fact due to
higher cache misses in the layer 0, we see a more than linear growth in
runtime compared to layer 1. We verified this using a profiler. A detailed
layerwise breakdow of the network runtime (without optimization) is provided
in table 2.

Table 3.  Measured layer-wise compute time for a 5-layer network processing 1024x2048 resolution input. | layer/
---
Resolution
| Encoder
---
Op/Time(ms)
| Decoder
---
Op/Time(ms)
| Total
---
(ms)
| 0
---
(1Kx2K)
| Conv2d + DnSamp
---
(4.27 + 0.72)
| UpSamp + Skip + Conv2d
---
(0.24 + 0.37 + 5.37)
11.0
| 1
---
(512x1K)
| Conv2d + DnSamp
---
(1.38 + 0.63)
| UpSamp + Skip + Conv2d
---
(0.12 + 0.19 + 1.36)
3.67
| 2
---
(256x512)
| Conv2d + DnSamp
---
(0.58 + 0.33)
| UpSamp + Skip + Conv2d
---
(0.07 + 0.10 + 0.55)
1.63
| 3
---
(128x256)
| Conv2d + DnSamp
---
(0.29 + 0.10)
| UpSamp + Skip + Conv2d
---
(0.04 + 0.05 + 0.31)
0.79
| 4
---
(64x128)
| Conv2d
---
(0.17)
0.17

Further performance improvements may be possible through pruning (Frankle and
Carbin, 2018) of the network weights. Performance of our network largely
depends on the size of temporary buffers; not how the buffers are connected.
As such, simply pruning the weights may not improve performance unless we also
reduce the size of temporary storage. Another avenue for exploration is
lowering the precision of temporary storage to 8 bits as the network output is
always between 0 and 1. We leave these optimizations for further exploration
as future work.

##  4\. Network depth optimization

In this section, we first derive the mathematical formula for penumbra width
using our simplified model as described in the main paper. We estimate the
parameters $\theta$, and $\theta_{\delta}$ as follows:

a. Model estimating penumbra$r_{e}$EmitterCenter
line$z_{min}$$z_{max}$$z_{f}$$x_{a}<0$$x_{a}>0$Penumbra$\approx
x_{a}+x_{b}$$x_{a}$$x_{b}$b. Estimating
parametersPA$z_{m}$$\theta$$\theta_{\delta}$MDBQ$x_{a}$C$x_{b}$ Figure 2.  A
simplified model to estimate the penumbra size at a given pixel. We assume our
occluder is spherical in shape, forming a convex bounding-sphere around the
occlusion geometry as shown in figure (a) on the left. Figure (b) shows a
simplified diagram to estimate the parameters $\theta,\theta_{\delta}$.
Predictedpenumbra width distribution 3-layers,Receptive field: 24 pixels
5-layers,Receptive field: 96 pixels 7-layers,Receptive field: 384 pixels
Density $P_{95}$ width: 21 pxls $P_{95}$ width: 90 pxls $P_{95}$ width: 180
pxls Density Density Penumbra width (in pixels) Penumbra width (in pixels)
Penumbra width (in pixels) MSE: 0.005 MSE: 0.005 MSE: 0.012 MSE: 0.083 MSE:
0.009 MSE: 0.018 MSE: 0.161 MSE: 0.042 MSE: 0.019 $\mathbf{P_{95}}$ :
$95^{th}$ percentile Left: Network o/p Right: Superimp. error Red: Excess
shadow Blue:Missing shadow Figure 3. Empirical validation of our penumbra
width prediction model. We predict the penumbra width on the vertical axis and
vary the number of layers on the horizontal axis. The diagonal represents the
optimal number of layers and elements below the diagonal show high error as
the receptive field is not large enough to accommodate the penumbra width.

(1) $z_{m}=PM=\frac{z_{max}+z_{min}}{2}$ (2)
$r_{s}=MD=\frac{z_{max}-z_{min}}{2}$

From $\triangle AMP$,

(3) $AM=\sqrt{z_{m}^{2}+r_{e}^{2}}$

From $\triangle AMD$,

(4)
$\theta_{\delta}=sin^{-1}\frac{MD}{AM}=sin^{-1}\frac{r_{s}}{\sqrt{z_{m}^{2}+r_{e}^{2}}}$

From $\triangle AMP$ and $\triangle BMQ$,

(5) $BQ=\frac{AP\cdot MQ}{MP}=\frac{r_{e}(z_{f}-z_{m})}{z_{m}}$

From $\triangle ABC$,

(6) $\theta=tan^{-1}\frac{BQ+QC}{AC}=tan^{-1}\frac{BQ+r_{e}}{z_{f}}$

Therefore,

(7) $x_{a}=z_{f}\,tan(\theta-\theta_{\delta})-r_{e}$ (8)
$x_{b}=z_{f}\,tan(\theta+\theta_{\delta})-r_{e}.$

Predictedpenumbra width distribution 3-layers,Receptive field: 24 pixels
5-layers ,Receptive field: 96 pixels 7-layers,Receptive field: 384 pixels
Density $P_{95}$ width: 21 pxls $P_{95}$ width: 90 pxls $P_{95}$ width: 180
pxls Density Density Penumbra width (in pixels) Penumbra width (in pixels)
Penumbra width (in pixels) MSE: 0.005 MSE: 0.005 MSE: 0.019 MSE: 0.083 MSE:
0.010 MSE: 0.017 MSE: 0.161 MSE: 0.042 MSE: 0.030 $\mathbf{P_{95}}$ :
$95^{th}$ percentile Left: Network o/p Right: Superimp. error Red: Excess
shadow Blue:Missing shadow Figure 4. Empirical validation of our penumbra
width prediction model. We predict the penumbra width on the vertical axis and
vary the number of layers on the horizontal axis while keeping the compute
flops constant across networks. The flops constrained networks show similar
behavior as the unconstrained networks in figure 3. Predicted penumbra width
(in screen space) Ray-traced reference 3-layersRf: 24 pixels 5-layersRf: 96
pixels 7-layersRf: 384 pixels Predicted penumbra width130 pixels23 pixelsMSE:
0.232 MSE: 0.051 MSE: 0.015 MSE: 0.009 MSE: 0.008 MSE: 0.010 Left: Network o/p
Right: Superimp. error Rf: Receptive field Red: Excess shadow Blue:Missing
shadow Penumbra width (in pixels) Figure 5. Figure showing the accuracy of our
penumbra width prediction model on Sponza. Region with high penumbra width
(blue cutout), as predicted by our model is reproduced accurately only by the
7-layer network.

###  4.1. Empirical verification

Table 4.  Table showing the compute cost and the number of learnable parameters for 3,5,and 7 layer network in figure 3. Network | Flops per pixel | # learnable parameters
---|---|---
3-layer | 8528 | 39.25K
5-layer | 16208 | 653.7K
7-layer | 21968 | 9.501M

In figure 3, we move a mesh object between a ground plane and an emitter
producing penumbra of varying size. For each row in the figure, we predict the
distribution of penumbra size and find the maximum (95th percentile) width
from the distribution. Across the columns, we vary the receptive field of the
network by adjusting the number of layers. Collecting the errors across all
combinations, we notice that the elements below the diagonal have high error
as the receptive field of the network is not large enough to accommodate the
penumbra size. Note that in figure 3, networks do not have the same compute
flops per pixel as shown in table 4.

Table 5.  Table showing the compute cost and the number of learnable parameters for 3,5,and 7 layer network shown in figure 4. Network | Flops per pixel | # learnable parameters
---|---|---
3-layer | 14928 | 110.9K
5-layer | 16208 | 653.7K
7-layer | 15328 | 1.626M

Simply increasing the number of layers in a network also increases the compute
cost. As such, any reduction in error might be attributed to the increased
compute requirement. We verify that the reduced error is indeed due to
increased receptive field of the network and not necessarily due to the
increased computation. As such, we constrain the network flops across the 3,
5, and 7 layer network as shown in table 5. From figure 4, we see that the
error reduces along the diagonal despite using networks that are
computationally equal (almost).

In figure 5, we take a single frame from the Sponza scene and analyze the
validity of our penumbra width prediction across different regions in the
frame. The increasing accuracy of our network output with the number of layers
in a region with large penumbra (yellow cutout) is a strong validation in
support of our model.

0 1 2 3 Network o/p (Untrained objects) Superimposed error MSE: 0.095 MSE:
0.096 MSE: 0.096 MSE: 0.097 Red: Excess shadow Blue: Missing shadow Figure 6.
Demonstration of the effectiveness of our technique on untrained objects
marked in red in the first row. Second row shows the comparison w.r.t
reference with error superimposed over raw network output. Going from left to
right, hard-shadow is indexed with 0 while 3 indicates an emitter of diameter
50cm. Shading is applied post-process in the first row.

##  5\. Data generation, training and inference

Ours, 8ms, MSE: 0.035 Hard-shadow MSM-3, 4ms, MSE: 0.105 MSE: 0.055 MSE: 0.219
Ours Reference MSM-3 MSE: 0.030 MSE: 0.133 Ours, 8ms, MSE: 0.083 Soft-shadow
MSM-9, 4.5ms, MSE: 0.161 PCSS, 5.5ms, MSE: 0.111 5SPP + SVGF, 11.5ms, MSE:
0.094 Ours MSM-9 Reference PCSS 5 SPP+SVGF MSE: 0.085 MSE: 0.187 MSE: 0.113
MSE: 0.093 MSE: 0.065 MSE: 0.159 MSE: 0.217 MSE: 0.075 Left/Right:
Superimposed error Middle (zoom-ins) : Network output Red: Excess shadow Blue:
Missing shadow Figure 7.  Our network’s ability to generalize to unseen
trajectories (within the same scene) compared to output from competing
techniques (MSM, PCSS, Raytracing+Denoising) for hard and soft shadows. MSM-3
and MSM-9 are Moment Shadow Map variants using 3$\times$3 and 9$\times$9
prefiltering kernels. Conference model ©Anat Grynberg and Greg Ward.

Orchestrating scene authoring, data collection, training, inference, and
generating the final results is one of the key challenges to this project. We
author the scenes using Blender (Community, 2018), ensuring all keyframed
camera trajectories do not go outside the emitter frustum. We keyframe several
camera and emitter trajectories covering different scenarios and bake them
into the scenes. We then upload the scenes to a Linux cluster to generate our
training data. A modified version Falcor (Kallweit et al., 2021) based on
Vulkan (originally DX12) is used along with Python scripts to collect, process
and organize the data for training. We use Pytorch (Paszke et al., 2017)
scripts for training which scans through the data in a random order every
epoch. While training, we save N($=3$) best models based on a test error. The
test consist of a small number (10-15) of handpicked images. We run the test
with a probability of 0.01 at each training iteration. Thus the test is run
roughly every 100 training iterations. Once the training is complete, we
select the best model out of N by comparing the error on full training
dataset. We also collect several statistics during the training for analysis.
During inference, we run our model through new trajectories which may have
some overlap with training dataset but are not same. We test the performance
of our network on a local machine (AMD 5600X, Nvidia 2080Ti) using Falcor and
a Cudnn based solution.

The training data is generated using concurrent shadow mapping and ray-tracing
passes. The shadow mapping pass outputs three perturbations of the buffers,
obtained by jittering the emitter and camera positions. The ray-tracing pass
uses 1500 rays per pixel distributed across 8 or more accumulation passes
enabling 8x or more MSAA. The ray-tracing pass simultaneously outputs shadows
with 4 different levels of softness. To summarize, each frame consist of 3
perturbations of our input buffers and 4 antialiased ray-traced images with
increasing softness.

A visual inspection of the generated data is crucial. We compare the
unprocessed rasterized shadows with the ray-traced shadows. A match between
rasterized and ray-traced output is desired and the two should have minimal
and consistent bias, if any. For example, there may be a difference in the
position of shadows between rasterization and ray-tracing due to the offset
used for preventing self-intersection. We find rescaling the scenes to the
same dimension useful for minimizing bias between rasterization and ray-
tracing and generating consistent data. Bias also depend on how we generate
the rays - emitter to scene or vice versa. The two may be different depending
on how backface culling is configured. We prefer emitter to scene ray-tracing
as the setup is closer to shadow-mapping.

##  6\. Network architecture ablation study

We discuss the limitations of various other network architecture we
implemented. The first variation is inspired by Exponential Shadow Maps (Annen
et al., 2008), where we learn the parameter $\alpha$ in the depth test
approximation $e^{\alpha(z-z_{f})}$. To remove non-linearity in the final
network layer, we perform the Adam-SGD optimization in log-space and
exponentiate the output during inference. However, injecting unprocessed input
$z$ directly to the output causes severe shadow-aliasing and our loss function
(VGG-19) is ineffective in correcting the aliasing in log-space. In another
variation, we tried denoising the rasterized (from emitter) depth - $z$ using
a separate network with ray-traced depths as targets. We split the network in
two halves - first half for denosing the depth and second half for processing
the output of the first half into final shadows. We trained the two network
end to end. However, the architecture failed to generate good quality final
output compared to our vanilla network. We also tested other variants of the
same idea but were equally ineffective.

##  7\. Results and comparisons

Figure 7 compares our technique with other competing techniques on
trajectories that were not present in the training. Figure 8 shows the
application of our technique across untrained camera and emitter trajectories.
For each scene, we train a separate network using a variety of emitter and
camera configurations across a range of softness. As such, a single network
can generate both hard and soft shadows where the softness is controlled using
a scalar input. Figure 6 shows that our network generalizes across variety of
shapes that were not present in the training set.

4 0 1 2 3 MSE: 0.078 MSE: 0.072 MSE: 0.069 MSE: 0.069 MSE: 0.070 Network o/p
Superimposed error Bistro MSE: 0.037 MSE: 0.035 MSE: 0.032 MSE: 0.032 MSE:
0.034 Network o/p Superimposed error Sponza MSE: 0.020 MSE: 0.024 MSE: 0.018
MSE: 0.019 MSE: 0.019 Network o/p Superimposed error Living room Red: Excess
shadow Blue: Missing shadow Figure 8. Figure shows the output of our network
(shaded post-process) with varying penumbra sizes in the cutouts (unshaded).
Each scene is trained independently and tested on a validation set. The second
row in each scene shows the error (superimposed on network oputput) w.r.t.
reference. Hard-shadow is indexed 0 while 4 indicates an emitter of diameter
50cm.

##  8\. Limitations and future work

One of the current limitations of our technique is that it does not naturally
extend beyond a single light source. Resolving this is an exciting avenue for
future work. Another limitation is it does not generalize well across a
mixture of scenes with widely different emitter depth distributions. This is
primarily due to the (purposefully) compact size of our networks and their
limited capacity to fit such multi-modal data. To avoid this, one could
potentially train multiple networks across different depth variations and swap
between them networks (i.e., based on emitter distance) at runtime. Artifacts
also arise when the penumbra sizes exceed the receptive field of the neural
network and training tile size; our conservative penumbra size estimate
affords us the opportunity, however, to tailor our architecture’s receptive
field accordingly. Finally, our perturbation loss can sometimes overblur fine
geometric details, such as from thin features like wires, however the gains in
temporal stability are significant; fine tuning here can help to mitigate such
overblurring.

An exciting avenue for future work involves extending our technique to a
finite number of light sources. We can draw inspiration from graph-coloring
approaches (Marton Tamas, 2018) here, where non-overlapping emitter frustums
are grouped together in layers and each layer is filtered independently. A
neural approach could use a network that takes as input buffers from $n$
sources and outputs the result. During training, we can disable (i.e., zero
input) a fraction of the light sources p ($<n$) to permit the network to more
effectively learn the response from each source, i.e., without confounding the
effects between sources.

Finally, runtime optimization of our network is another avenue for
exploration. The performance of a UNet depends primarily on memory bandwidth,
as opposed to compute operations. A customized pruning technique focused on
improving the compute density by reducing the number of temporary buffers is
an interesting direction of research. Also the network output is non-HDR,
which can be exploited to reduce the size of buffers.