# Towards Neural Path Tracing in SRAM

Mark Pupilli ID

###### Abstract

We present an experimental neural path tracer designed to exploit the large
on-chip memory of Graphcore intelligence-processing-units (IPUs). This open
source renderer demonstrates how to map path tracing to the novel software and
hardware architecture and is a useful tool for analysing in-cache neural-
rendering scenarios. Such scenarios will be increasingly important if
rasterisation is replaced by combinations of ray/path tracing, neural-radiance
caching, and AI denoising/up-scaling, for which small neural networks are
already routinely employed. A detailed description of the implementation also
serves as a self-contained resource for more general software design on IPU.

(a) Box

(b) Box + HDR-NIF

(c) Spheres + HDR-NIF

(d) Small BVH + HDR-NIF

(e) Large BVH

Figure 1: Images path traced on a Graphcore Bow-Pod-16. The HDR environment
light is compressed into 97 KiB of neural network weights. The neural-network
weights, activations, and scene BVH reside entirely in on-chip SRAM.

##  1 Introduction

Path tracing Kajiya [1986] has replaced rasterisation for high-quality offline
rendering Fascione et al. [2018], Burley et al. [2018], Christensen et al.
[2018] and real-time rendering seems to be at the start of a similar
transition. For example, the ability of GPUs to train and run small neural-
networks (NNs) at high throughput and low latency Müller [2021] allows the
bulk of the path tracing workload to be approximated by a neural-radiance-
cache Müller et al. [2021]. Whilst expensive, NN compute is accelerated by
dedicated hardware, amenable to reduced precision arithmetic, and
deterministic: all of this is in contrast to the full path tracing algorithm.
Neural networks are also utilised for real-time denoising and spatio/temporal
super-resolution Yang et al. [2023] and on state of the art GPUs up to 88% of
displayed pixels are AI generated as a result NVIDIA [2022]. The increased
visual fidelity from path tracing and the effectiveness of neural-radiance
caches make it a distinct possibility that real-time rendering pipelines
become predominantly neural. The key to realising real-time performance for
networks in use today is keeping network weights in GPU on-chip memory
(SRAM/L1-cache and registers) for as long as possible Müller [2021], Müller et
al. [2022]. This trend raises the question: do current GPU architectures have
the right balance of on-chip memory capacity to off-chip memory bandwidth to
adapt to a fundamental shift in the nature of rendering computations?

IPUs were nominally designed for artificial intelligence (AI) and are in no
way intended for rendering. That said, each chip is massively parallel like a
GPU, but in contrast contains 897MiB SRAM close to the processing cores. Some
key architectural differences between IPU and GPU are summarised in Table 1.
We are interested in using the large on-chip memory to explore neural
rendering configurations that are not currently possible on other hardware. To
this end we present an IPU implementation of path tracing combined with a high
dynamic range (HDR) neural environment lighting. The network we employ has a
similar architecture and size to models used in other neural rendering tasks
(see Section 2.2), and those algorithms often additionally involve some form
of ray-casting/tracing. For this reason our simple application is a useful
tool for reasoning about the performance and viability of in-SRAM neural
rendering, regardless of how future hardware adapts.

Property | IPU Bow GC200 | GPU A100
---|---|---
Independent instruction streams (IIS) | 1472 (tiles) x 6 (threads) = 8832 | 108 (SMs) x 64 (warps) = 6912
SRAM per core | 624 KiB per tile | 192 KiB (up to 164 KiB shared) per SM
Contention free SRAM/registers per IIS |  624/6 = 104 KiB |  (192 + 256)/64 = 7 KiB
SIMD vector width per IIS | 2 float | 32 float
| 4 half | 64 half
Integer compute | Non-vectorized but dual issue | Vectorized but no dual issue
Peak TFLOP/sec (half, non-zero) | 349 | 312
Table 1: Comparison of IPU and GPU (NVIDIA [2020]) variants with similar
silicon area from the same process node (7nm).

##  2 Related Work

###  2.1 Neural Networks in Path Tracing

Monte Carlo (MC) path tracing Kajiya [1986] is a rendering algorithm which
accumulates the contributions of all possible light paths in a scene using
Monte Carlo integration. It exactly integrates the rendering equation in
expectation and therefore simulates many physical effects producing high
fidelity images. Because MC convergence follows an inverse square law it is
inevitable that sampling must stop at some point and give way to a denoising
algorithm, often implemented using sophisticated AI based denoisers Intel®
[2021], Zhang et al. [2021].

Neural networks have also become key to realising real-time path tracing. For
example, in Müller et al. [2020] two neural networks reduce variance of the MC
integration: a normalizing-flow network is employed to learn a proposal
distribution to enable efficient importance sampling of BSDFs and a second
network is used to correct errors introduced by the proposal. In Müller et al.
[2021] a simpler neural-network is trained online using extended path traces
in order that the majority of paths can be stopped after a few bounces, and
then fed through a small MLP (en masse) that returns an estimate of the path’s
remaining radiance. For efficiency, weights are streamed into L1 cache or
registers once and then reused repeatedly on the entire batch of rays Müller
[2021]. Depending on batch-size, NN inference throughput for 128 hidden
neurons and 2 layers (H128, L2) is reported to be between 280M and 390M
samples/sec on a 3090 RTX.

###  2.2 Other Neural Rendering Techniques

Neural radiance fields (NeRFs) Mildenhall et al. [2021] are neural networks
trained to approximate continuous functions from
$\mathbb{R}^{5}\mapsto\mathbb{R}^{4}$:

$(r,g,b,w)=f(x,y,z,\theta,\phi)$ (1)

They map rays in 3D space (point and normalised direction vector in spherical
coordinates) into an RGB colour and a weight. This allows a NeRF to encode
volumetric data: output colour nominally represents the radiance transmitted
along the ray (but in usual practice it is a low dynamic range quantity non-
linearly related to luminance). The weight allows the network to represent
free space and opacity, but also makes the rendering differentiable allowing
it to be part of the training loop. These neural-volume representations have
found utility in a number of domains including semantic perception Zhi et al.
[2021], Blomqvist et al. [2023] and even form the basis for text-to-3D
generative AI models Poole et al. [2022].

####  2.2.1 Prevalence of the Multi-Layer-Perceptron

The function NeRFs encode is low dimensional, so small MLP/relu networks
($10^{5}$ or $10^{6}$ parameters) give acceptable approximations. In
Mildenhall et al. [2021], for example, the architecture is an MLP with 9
layers and 256 hidden neurons. Deep neural networks exhibit a spectral bias
Rahaman et al. [2018] which prevents them from learning high frequency
functions (like natural images) without adding engineered features or
inductive bias to overcome this. In NeRF the low dimensional input coordinates
are embedded into a higher dimensional space using Fourier features Tancik et
al. [2020]. SIREN networks Sitzmann et al. [2020] on the other hand, encourage
the network to learn higher frequencies by using sinusoidal activation
functions and these networks have the added advantage that they can learn to
approximate partial derivatives much more effectively than a similar sized
network that uses Fourier features. In CoConet Bricman and Ionescu [2018] they
train an MLP network to approximate images but use a hand engineered embedding
of pixel coordinates into a higher dimensional input space. A detailed
theoretical analysis on alternative input embeddings is given in Tancik et al.
[2020].

Whilst models discussed so far use MLPs, Minnen et al. [2018] employs a
convolutional network for image compression. They report the compression
artifacts induced by this method are more palatable than JPEG to human
observers (smoothed but not blocky) for equivalent bit-rates. Another non-MLP
example is the neural texturing sytem of Thies et al. [2019] where a
convolutional U-net is used. The network has an encoder/decoder structure
hence the rendering system is entirely NN based (16M parameters).

State of the art neural material/texturing systems also utilise MLPs. In
Kuznetsov et al. [2021] the problem is again tackled using an MLP that
approximates the bi-directional-texture function from
$\mathbb{R}^{7}\mapsto\mathbb{R}^{3}$. Most recently Zeltner et al. [2023]
implements a fully neural material system, combines it with path tracing, and
demonstrates improved performance over the traditional shading pipeline: the
transition away from rasterisation seems imminent.

Since many techniques in this field use the MLP as a core building block,
metrics from our on-chip system should be useful as a baseline for reasoning
about the benefits of large on-chip SRAM for a range of neurally based
rendering algorithms.

##  3 Implementation

We will briefly discuss aspects of the IPU hardware and software architecture
relevant to the implementation, more detail is given in Appendix A. The GC200
processor used in this work contains 1472 tiles (cores) and each tile has six,
multiple-instruction-multiple-data (MIMD), barrel scheduled, hardware threads
called workers. Each worker issues one instruction packet in round-robin
fashion and can dual issue floating point and integer/memory operations. All
threads can work in unison to drive each tile’s accumulating matrix product
(AMP) unit, achieving the peak FLOP rate (Table 1) for matrix multiplies and
convolutions. Tiles can synchronise and communicate using a bulk synchronous
parallel (BSP) execution scheme that alternates between internal or external
exchange of data and local tile computations. The latest variant of the GC200
uses wafer-on-wafer technology and has the designation Bow. Bow IPUs have
identical micro-architecture but a 40% higher clock speed.

###  3.1 IPU Path Tracer Design

The path tracer itself is very simple. Paths are traced through the scene with
no light sampling: contributions are only accumulated if paths hit an emissive
object by chance or if the path escapes the scene and receives a contribution
from the environment light. The environment light field is encoded in a neural
network as described in Section 3.2 so the sampling loop alternates between
path tracing operations and NN inference. Paths are terminated early by
roulette with probability proportional to their radiometric throughput. While
the lack of light sampling makes the implementation sample inefficient, it
simplifies reasoning about performance.

An entirely on-chip solution needs to store ray payload and results, the scene
description, and neural network weights all in SRAM. While 897MiB might seem
plenty for this task we need to consider some of the finer details of the
architecture. First note the memory is split into 1472 tiles, 624 KiB per
core. Cores can not directly access memory of other tiles nor can the cores
issue direct accesses to off-chip memory and there is no unified address
space. This means that the processor is most efficient when each core is
repeatedly accessing data from its local SRAM in parallel and that DRAM access
must be batched into a BSP compatible schedule (Appendix A.2.3).

####  3.1.1 Scene Data

The first design decision we make is to replicate the scene description on
every tile, but spread the neural network weights across the entire chip. The
primary reason for this is that routing ray data between tiles containing, for
example, different treelets, requires fine grained dynamic exchange patterns.
While this is possible in principle using just-in-time (JIT) compiled exchange
sequences, this is not exposed in the currently available SDK which only
allows for exchanges that are fixed at compile time. Batching ray data for
neural network inference has no ordering requirement, so inter-tile exchanges
of inputs and results can easily be pre-compiled and in fact the graph
compiler can make sophisticated data movement decisions for us (see Section
3.3.1).

BVH node, triangle, index and vertex buffers are all serialised into a single
binary chunk. The IPU programming model is built around variables of a single
data type (i.e. tensors) so the idiomatic way of passing the scene data would
be using structures of arrays. However, we found this makes the graph
description overly verbose and increases the code size needed for inter-tile
exchange, so we prefer to serialise all the data into one buffer and pay the
cost of de-serialising the data on each tile. De-serialisation takes around
522 worker cycles (3132 total cycles as we only use a single worker). The
additional benefit of this scheme is possible experimentation with exchange of
treelets Aila and Karras [2010] between tiles and/or DRAM in future.

####  3.1.2 Ray Data and external DRAM

We could distribute rays and the frame-buffer across tiles but early
experiments with this approach proved limiting in terms of both the resolution
of images that can be kept in SRAM and also in terms of ability to store
auxiliary payload data with the rays. From an API design point of view it is
desirable to associate extra data with each ray, for example normals, hit-
point, or other arbitrary output variables. For this reason we choose to
stream ray data from external DRAM allowing use of standard ray data
structures (84 bytes per ray) instead of a compact representation. DRAM
bandwidth in current IPU systems is limited (25.6 GB/sec) but the API allows
us to reserve some compute tiles to act as "smart I/O controllers": 32 tiles
perform ray loads and stores in parallel with 1440 tiles rendering each ray
batch.

The number of rays that each worker thread processes is specified as a graph-
compile time constant. This allows us to balance the workload so that DRAM
fetches are effectively hidden and large enough ray batches are generated to
achieve efficient neural-network inference. (The batch size for each neural
network query is rays-per-worker x 6 x 1440.)

Using DRAM this way allows rendering extremely high resolutions, hiding of
DRAM latency, and has another advantage: many standard resolutions are
divisible by both 1440 and 6 which ensures all worker threads are utilised
(e.g. the 8K format 7680 x 4320 = 3840 x 6 x 1440).

####  3.1.3 IPU Ray Tracing Kernels

The ray tracing implementation is plain C++. The only IPU specific code is: 1)
use of the vector float2 data type (2 x float32 elements) in the ray-slab
intersection test to improve vectorisation; and 2) intrinsics to access the
IPU’s hardware random number generator inline in the path trace kernel (for
example __builtin_ipu_urand_f32()). We do not yet employ ray-packetisation,
which would assist the compiler further with auto-vectorisation. The
simplicity is deliberate, partly because we want to share non-obfuscated
reference code for IPU graphics programming, and partly because the MIMD
nature of the worker threads means there are no execution stream coherence
issues we need to manage. As an indication of optimisation potential and
generated code quality, the triangle intersection routine (derived from
PBRT-v3 Pharr et al. [2016]) compiles to code with a FLOP arithmetic intensity
of 0.23 (67/289) FLOPs per issue with 20/67 of the FLOPs in vectorised
instructions. The compiler emits many 32-bit load/stores with only 20/71 using
64-bit. This suggests there is potential to increase FLOP intensity with an
IPU specific ray packetisation strategy and optimisations that ensure use of
64/128-bit load/stores. For comparison, the same routine compiled with g++-9.4
targeting x86-64+AVX2 generates 0.30 (67/226) FLOPs/issue, 15 of which are
fused multiplies, with no vectorised instructions emitted.

####  3.1.4 Bounding Volume Hierarchy

We found memory optimisations important in building a usable system: next to
neural network weights, the bounding volume hierarchy (BVH) is the most
significant consumer of tile memory. Our BVH nodes do not differ much from
common CPU or GPU implementations with an initial BVH-2 tree built using
Embree Wald et al. [2014] on the host CPU. This is then compacted into an
array of carefully packed, pointer-less nodes, similar to those used in Pharr
et al. [2016], where the first child is implicitly stored next in the array.

We further reduce memory consumption by storing the extent of the bounding
volume in float16 and casting to float32 on-demand at intersection time (the
box origins are stored at float32). This reduces node size from 32 bytes to
24. At BVH construction time we use a special software implemented rounding,
round-to-nearest-not-lower, to cast from float32 to float16 which guarantees
no intersection can be missed as the bounding box is never smaller than it
would have been if stored at float32. Run-time overhead is a negligible cast
back from float16 to float32. There are prior examples of compressed BVH nodes
Benthin et al. [2018], with the smallest typically intended for hardware
implementation (the feasibility of 1-bit hierarchical encoding is demonstrated
in Keely [2014]). Experimenting with sub 16-bit precision seems appealing,
however, on IPU casting between integer and floating-point register files can
have significant overhead, integer bit manipulation instructions are not
vectorised, and the smallest load/stores are 32-bit. Very low precision
encoding also increases the total number of intersection tests due to looser
bounding boxes. Hence, we prefer the compromise of half precision extents,
keeping everything in floating-point. This gives a negligible performance loss
due to additional intersections (< 1% ray throughput on test scenes) yet still
gains a 25% BVH memory saving.

###  3.2 Neural Environment Lighting

Our environment lighting model is conceptually like any other neural image
field (NIF) approximator. It approximates the low dimensional function:
$(r,g,b)=f(u,v)$.

Figure 2: NIF model schematic showing activation sizes with operations between
them.

The function takes input coordinates $u,v$, applies positional embeddings as
in Tancik et al. [2020] and predicts a colour value using a sequence of dense
layers with ReLU activation functions. A schematic for a specific size of the
HDR-NIF model is shown in Figure 2 where hidden size H (128) and number of
dense-relu layers L (4) can be varied. The initial Fourier encoded $u,v$ co-
ordinates are always concatenated with the last odd layer before the middle of
the network, which has a reduced dense layer output size to compensate. The
Fourier feature dimension can be varied but we keep it fixed at 40 in all
experiments in this paper. The final trainable layer has no activation
function allowing the network to regress high dynamic range functions. The
network’s final layer is a static colour-conversion matrix (defaulting to YUV
to RGB). The dynamic range of the RGB training samples is logarithmically
compressed as in LeGendre et al. [2019] (during inference we apply the inverse
exponential tone-mapping to the final output of the network).

####  3.2.1 Training

Training samples are drawn uniformly on normalised $u,v$ coordinates and sub-
pixel samples are taken from the image using a bi-linear filter. During
training we intermittently evaluate the network by feeding $u,v$ coordinates
on a regular grid to reconstruct an image and then compute the PSNR versus the
original input image. The model is implemented in Keras and the network is
trained on a single GC200 IPU.

We train the network with master weights stored in float16 and enable
stochastic rounding Gupta et al. [2015] which the IPU supports in hardware.
This requires a loss scale which we fix at 16384. We use an ADAM optimiser
with learning-rate := 0.001 and internal variables (mean, uncentered-variance)
stored at float32. The loss is a Huber loss with delta := 0.001.

####  3.2.2 Approximating HDR Images

While not the focus of this work some investigation into the behaviour of HDR-
NIF convergence was necessary in order to get accurately lit renders and some
anecdotal results are included in Section 4. We discovered that existing
training regimes such as Minnen et al. [2018], Bricman and Ionescu [2018] do
not behave well for high resolution HDR images, so we make two notable
adjustments. First we employ a Huber loss, which we found to outperform MSE in
terms of accuracy: we hypothesise that encouraging the network to learn high
luminance samples more slowly leads to more stable gradients early in training
but did not investigate further. We also add a final, fixed, linear colour-
space conversion transform to the end of the network: this forces the network
to predict samples in a colour-space of our choosing which we found affects
accuracy (see Section 4.3).

####  3.2.3 Precision

The IPU does not have hardware sin/cos instructions. Fourier feature training
samples are pre-computed with sin/cos at float32 but during inference, where
the Fourier features must be computed on the fly, we use optimised software
float16 sin/cos implementations. This does not noticeably affect
reconstruction quality but improves performance significantly. In path
tracing, where we need float32 $sincos()$, we use an optimised, but generic,
software implementation Moshier [1992]. This is important because the C++
standard library equivalents use double precision which will be emulated on
IPU. Not only is this slow, but the emulation code itself consumes around 10
KiB per tile.

###  3.3 Combining Neural-HDRIs and Path Tracing

Integrating HDR-NIF lighting into the path tracer is conceptually straight
forward: to determine the lighting contribution from the environment to a
given ray, we just convert its direction in the world coordinate frame to
equirectangular $[u,v]$ coordinates and submit these to the NIF which returns
the $[r,g,b]$ light contribution to be multiplied with the accumulated ray
throughput. For this to be efficient we need to run inference on large batches
of ray queries in parallel. The only similar use of NIFs in the literature is
Poole et al. [2022] where the 3D models are represented using NeRF, but the
background is separated, using an neural environment field representation
similar to ours. In their case however, the field approximation is low dynamic
range.

####  3.3.1 Compute Graph Compilation

The Poplar/Poplibs graph compilation framework (Appendix A.2) plans how
matrix-multiplies are distributed across cores/tiles automatically. For many
model configurations considered here, it is possible that a single layer’s
weights fit in one tile’s SRAM. In this case Poplibs can decide to replicate
weights of individual layers across every tile on-demand, maximising
utilisation of the AMP units. It is able to do this because inter-tile
bandwidth is high enough that there is a net performance improvement despite
the overhead of broadcasting weights before executing each layer. It is
interesting to note that Poplibs’ matrix-multiply planner is sophisticated
enough to make this decision completely automatically: on other platforms this
kind of optimisation must be done by hand (as in Müller [2021]).

When the path trace kernel finishes, the queries it produces are not laid out
efficiently for the first matrix multiply in the MLP network. Exchanging
information between IPU cores is fast so this is not a problem. However,
transferring data between IPUs over IPU link is slower than on-chip exchanges.
For this reason when we use multiple IPUs the NIF model is replicated across
all the chips which then run data parallel with no communication between them.

####  3.3.2 Batch-Serialisation

The matrix-multiply planner allows us to trade performance versus memory use.
If we want to increase performance we can allocate the planner a larger
proportion of SRAM for weights and activations. However, when memory use is
close to the limit, the optimiser often resorts to serialisation of the output
channels of each layer to reduce memory needed for temporary activations. We
prefer to manually manage activation memory by batch-serialising the entire
network, which we found leads to better inference throughput and memory use in
this case.

##  4 Results

All results use the same path tracing parameters: a maximum path length of 10
and roulette termination beginning at depth 3. If the reader wishes to
extrapolate performance figures to real-time rendering scenarios they should
bare in mind that state of the art real-time path tracing algorithms typically
sample one path per-every-other pixel, for a few bounces, then execute one or
more neural-networks to complete the task. Here we fully trace paths for every
pixel, then execute a neural network, and repeat taking many samples.

###  4.1 Test Scenes

The test scenes evaluated are shown in Figure 1. These scenes are designed to
give an indicative spread of results that we can expect for the range of scene
sizes and NIF configurations that fit in IPU SRAM. Table 2 contains some
baseline metrics for these scenes. These were measured on an IPU Bow system.
In results where we do not specify Bow the performance was measured on classic
GC200 chips.

Scene | Box | Box | Spheres | Small BVH | Large
---|---|---|---|---|---
|  | \+ NIF | \+ NIF | \+ NIF | BVH
Giga-paths/sec | 0.826 | 0.418 | 1.333 | 0.350 | 0.487
BVH KiB (per tile) | 189 | 189 | 0.258 | 265 | 397
Table 2: Metrics for test scenes from Figure 1 traced on a Bow-Pod-16. NIF
model size is 97KiB (H128, L5).

###  4.2 Precision of Ray Tracing Operations

In Section 3.1.3 we mention that some CPU code lowers to AVX fused multiply
add/subtract (FMA/S) instructions. The IPU has no infinite intermediate
precision FMA/S and we avoided any code that results in fallback to double
emulation, so it is prudent to check the accuracy of basic ray tracing
operations on IPU. In Table 3 we compare normals and primary hit-points
against two CPU implementations: one that uses Embree for ray tracing
operations, and one that runs the IPU kernels on CPU. This allows us to
distinguish disparities due to algorithmic differences from those due to
floating point arithmetic.

We note that differences between the same kernels on IPU/CPU are tiny and
consistent, with no difference at all in the primary hits. Against Embree
errors are still small but there are notable outliers. The first is the hit-
point delta for the box scene, this is because the box scene uses the original
Cornell box data with no scaling applied (so dimension range is of the order
100$\times$ larger than the other scenes). The second notable difference is in
normals for the small/large BVH scenes compared to box/spheres. In this case
the BVH scenes contain many triangles and triangle intersection has the
largest algorithmic difference compared to Embree. In effect the larger error
is due to outliers where rays hit adjacent (but still valid within machine
precision) primitives. In those cases the normal will change dramatically (but
the hit-point much less so). Overall, we don’t believe these differences
should cause any difficulties beyond those normally encountered when ray
tracing with single precision.

Scene | MSE (vs Embree) |  | MSE (vs CPU) |
---|---|---|---|---
| Normal | Hit | Normal | Hit
Box | $1.1\times 10^{-13}$ | $7.6\times 10^{-9}$ | $2.6\times 10^{-16}$ | $0$
Spheres | $2.1\times 10^{-14}$ | $2.5\times 10^{-14}$ | $1.6\times 10^{-16}$ | $0$
Small BVH | $4.5\times 10^{-7}$ | $2.3\times 10^{-14}$ | $1.8\times 10^{-16}$ | $0$
Large BVH | $1.2\times 10^{-7}$ | $7.1\times 10^{-14}$ | $2.2\times 10^{-16}$ | $0$
Table 3: Precision test results showing the worst (across the $x,y,z$
components) mean squared error (MSE) for normals and primary hit-points.

###  4.3 Colourspaces

We found that images with significant dynamic range require careful selection
of the colour-space in which the NN learns its representation. This problem is
exacerbated with smaller batch-sizes ($O(10^{3})$). In extreme examples, such
as where midday sun is the only light source, training can be very unstable if
the network learns in the RGB colour-space. In this case all the energy in the
scene comes from $<<$ 1% of the HDRI’s pixels: in effect the most important
training samples are outliers.

Results of forcing the network to learn in different colour-spaces with a sun-
lit HDRI are shown in Figure 3. Table 4 shows the peak-signal-to-noise-ratio
(PSNR) of the reconstructed NIF versus the orignal HDRI. Note the background
where the NIF is directly hit by primary rays: it contains few perceptible
differences between the three images because the errors in the few sun samples
are only noticeable when used for path tracing. Only YUV gives the correct
result in this case, despite YCoCg having higher PSNR in luminance channel and
RGB space. The YCoCg colour matrix completely de-correlates gradients flowing
back from the loss layer whilst in contrast, the RGB colour-space is known to
have highly correlated components. We hypothesise that having some correlation
between gradients is beneficial, but the network is not able to separate
luminance and chrominance information from RGB samples alone. We believe these
aspects warrant further investigation.

(a) Network learned RGB

(b) Network learned YCoCg

(c) Network learned YUV

Figure 3: Images path traced using HDR-NIFs trained in different colour-spaces. (Network that learns a YUV representation gives light with the correct hue). Color-space | PSNR | PSNR | PSNR
---|---|---|---
| RGB | Luminance | Chrominance
RGB | 32.4 | 37.8 | 28.1
YCoCg | 57.8 | 61.5 | 43.9
YUV | 57.7 | 61.4 | 45.7
Table 4: PSNRs for Borghese Gardens HDR-NIF approximation using different
colour spaces. Highest PSNRs in bold.

###  4.4 NIF Accuracy

In Figure 4 we can see a comparison of path tracing results for the spheres
scene with HDR-NIFs of increasing model size and a reference image rendered in
Blender Community [2018]. This simple scene is used so that we can vary the
NIF model size through a large range without exhausting SRAM. In Figure 4(a)
even though the background is a very poor approximation of the original HDRI
the shadows and caustics look comparable to the higher quality NIFs and
original reference in 4(d). This explains why tiny neural radiance caches can
be so effective: because in that use case the neural-field approximation is
only used to complete paths and never directly visualised.

(a) H64 L2, 311.8M paths/sec.

(b) H256 L4, 149.1M paths/sec.

(c) H1024 L8, 9.3M paths/sec.

(d) Reference (original 4K HDRI).

Figure 4: Comparison of neural path tracing results with different HDR-NIF
model sizes for the urban alley HDRI.

We list the PSNR values for the urban alley HDRI in Table 5 because it is an
outlier in terms of other PSNR results. Figure 7 shows model-sizes versus
PSNRs and path rate (paths/second) for a sweep over 7 different HDRI images:
urban alley can be seen as the left most vertical grouping of scatter points
with significantly lower PSNR than the rest. This may be because it has high
texture and colour variation as well as large dynamic range. In this case we
note that the chromatic PSNR is a poor predictor of reconstruction quality
which is at odds with Table 4.

Model Config | PSNR | PSNR | PSNR
---|---|---|---
| RGB | Luminance | Chrominance
H64 L2 | 29.0 | 31.1 | 14.5
H256 L4 | 29.3 | 35.6 | 14.1
H1024 L8 | 29.4 | 37.3 | 14.0
Table 5: PSNRs for Urban Alley HDR-NIF approximations. Highest PSNRs in bold.

We are aware of perceptual HDR metrics, such as Mantiuk et al. [2011], which
should be preferred over PSNR in some circumstances. However, here we are
jointly interested in the perceptual reconstruction of the directly visible
background, and indirect effects that affect validity of the lighting
simulation. It is not immediately clear how perceptual metrics help us in this
case. For the remainder of the paper we continue to report three variations of
PSNR but recognise their limited utility.

Figure 5: Effect of sample count on path rate (paths/sec), with and without an
HDR-NIF.

###  4.5 Performance

Figure 5 shows a roofline plot of sample count versus path rate when rendering
720 $\times$ 720 images of the box scene on a single IPU GC200 chip from a
Pod-4 (classic) system. Even with relatively low bandwidth the transfers can
be completely overlapped with compute when taking more than 256 samples per
pixel. The DRAM ray transfer bandwidth is ultimately limited by transfer size,
which in turn is limited by the amount of ray-data that fits on 32 I/O tiles
(see Table 6). Note that ray batch data is double buffered on the I/O tiles to
allow saving one batch while reading the next efficiently in a pipeline.

Image | Save+Load Time | Bytes per | Bandwidth)
---|---|---|---
Resolution | (milli-seconds) | Ray-Batch | (GiB/sec)
360$\times$360 | 3.9 | 5443200 | 1.29
720$\times$720 | 4.0 | 5443200 | 1.27
1440$\times$1440 | 3.3 | 5806080 | 1.64
2880$\times$2880 | 3.3 | 5806080 | 1.64
5760$\times$5760 | 3.3 | 7257600 | 1.65
Table 6: Achieved bi-directional DRAM ray transfer bandwidth when using 8
rays-per-worker thread.

Bytes per ray batch in Table 6 does not exhaust I/O tile memory so in
principle, we could increase the DRAM bandwidth slightly by increasing the
batch size, which amounts to increasing rays-per-worker (Section 3.1.2).
However, in Figure 6 we can see that overall path trace rate peaks at 8 rays-
per-worker (and this seems consistent across many NIF configurations). So
unless very small numbers of samples are required, there is no performance
benefit from increasing rays-per-worker beyond 8, despite this limiting the
DRAM bandwidth utilisation. Furthermore, increasing rays-per-worker to 10
exhausts SRAM for larger NIF models (Figure 6) so there is limited room for
manoeuvre and we generally stick to 6 or 8 rays-per-worker.

Figure 6: 4D scatter plots showing the effect of NIF model configuration
(layers and hidden-size) and rays-per-worker thread on path throughput and on-
chip memory consumption.

####  4.5.1 Scaling with Clock Speed

Bow IPU systems have identical micro-architecture to IPU-Classic systems but a
40% higher clock speed. If we take enough samples to maintain overlapped DRAM
transfers then we would expect all on-chip operations to scale perfectly with
clock speed. We measure a single test case (Figure 1(d)) on a Bow system to
check this. Results are given in Table 7 where samples per clock are almost
identical between the two systems, demonstrating the expected scaling.

IPU | Sample-rate | Clock | Sample-rate
---|---|---|---
System | (paths/sec) | (GHz) | per GHz
Pod-4 Classic | 156.5 | 1.33 | 117.7
Bow Pod-4 | 216.4 | 1.85 | 117.0
Table 7: Scaling of neural path trace rate with clock-speed.

###  4.6 Model-size/PSNR versus Path Rate

Figure 7 shows how NIF model configuration affects path sampling rate
(paths/sec) and PSNR (dB). Sample rates are again for a single GC200. As well
as sweeping across NIF size we also sweep across 7 HDR-NIFs, trained on the
HDRIs listed in Appendix B.

Unfortunately, chromatic PSNR (Figure 7) appears insensitive to model size,
despite it having a large effect on correctness of the lighting, but we have
seen that lighting can be affected by outlying samples that will not be
captured by PSNR metrics (Figure 3). Luminance (Figure 7) is more sensitive:
only models with either hidden size above 128, or layer counts above 4, (or
both), reliably result in PSNRs above 60. These are interesting thresholds
because they are precisely the points at which current GPUs can not operate in
the regime that loads weights into SRAM/registers exactly once Müller [2021].
Of course, things are not so clear cut, as we have seen in Figure 4(a), path
tracing effects can have high fidelity even when the background reconstruction
quality is unacceptable.

##  5 Conclusions and Future Work

We have presented results that demonstrate the potential and utility of
Graphcore processors for exploring on-chip neural rendering techniques. The
current system is effective when all data, other than ray streams, fits in
SRAM. The capability to render, even small scenes, entirely from SRAM is a
capability unique to IPUs and we believe it is worth exploring further.

Functionally, the requirement for all BVH data to fit on one tile is currently
the most significant limitation. Algorithms for distributed rendering between
thousands of CPUs, such as Fouladi et al. [2022], may well be applicable at a
smaller scale within a single IPU processor, and we would like to explore
similar algorithms to exchange both treelets and rays between tiles, taking
advantage of the high on-chip communication bandwidth to render larger scenes.
Using the currently idle compute capability of the I/O tiles is also an
interesting possibility, for example to decompress treelets, or to sort and
schedule rays as they are loaded in parallel with the heavier compute.

In terms of performance there is scope for optimisation. For example, tracing
ray packets of size two and storing more than one triangle primitive per BVH
leaf-node would drastically increase the scope for vectorisation and reduce
BVH memory consumption. Combined with more judicious use of larger load/store
instructions it should be possible to increase FLOP intensity. Newer IPU
hardware supports float8 and storing the HDR-NIF weights in this format would
halve the memory required for weights and increase performance of the system’s
neural component. The BVH representation could also be compressed further
using quarter-precision.

Finally, a more sophisticated material and shading system including textures
and importance sampling would be a desirable addition. In current systems the
lack of fine grained, random external DRAM reads for shade on hit texturing
suggests a shade before hit approach Fascione et al. [2018], or alternatively,
fully neural material and geometry systems Park et al. [2019], Kuznetsov et
al. [2021], Zeltner et al. [2023] may be more appropriate given that IPUs are
predominantly AI processors.

Hopefully the work presented here encourages others to experiment with IPU
hardware in this domain with confidence in the basic building blocks we have
provided: all source code is made available here Graphcore [2023]. It is
probably not yet clear which types and sizes of neural networks rendering
hardware should be optimised for, but we believe the IPU is a useful tool in
evaluating the range of possibilities.

##  6 Acknowledgements

We would like to thank Deniz Beker, Yani Donchev, Andrew Fitzgibbon, and Carlo
Luschi for their input and feedback.

Figure 7: 4D scatter plots showing how NIF model size influences path rate
(paths/sec) and PSNR (in luminance 7, chrominance 7, and RGB 7).

## References

  * Kajiya [1986] James T. Kajiya.  The rendering equation.  _SIGGRAPH Comput. Graph._ , 20(4):143–150, August 1986.  ISSN 0097-8930.  doi: 10.1145/15886.15902.  URL https://doi.org/10.1145/15886.15902.
  * Fascione et al. [2018] Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich, and Johannes Meng.  Manuka: A batch-shading architecture for spectral path tracing in movie production.  _ACM Trans. Graph._ , 37(3), aug 2018.  ISSN 0730-0301.  doi: 10.1145/3182161.  URL https://doi.org/10.1145/3182161.
  * Burley et al. [2018] Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece.  The design and evolution of disney’s hyperion renderer.  _ACM Transactions on Graphics (TOG)_ , 37(3):1–22, 2018.
  * Christensen et al. [2018] Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Marc Bannister, Brenton Rayner, Jonathan Brouillat, and Max Liani.  Renderman: An advanced path-tracing architecture for movie rendering.  _ACM Trans. Graph._ , 37(3), August 2018.  ISSN 0730-0301.  doi: 10.1145/3182162.  URL https://doi.org/10.1145/3182162.
  * Müller [2021] Thomas Müller.  tiny-cuda-nn, 4 2021.  URL https://github.com/NVlabs/tiny-cuda-nn.
  * Müller et al. [2021] Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller.  Real-time neural radiance caching for path tracing.  _ACM Trans. Graph._ , 40(4), jul 2021.  ISSN 0730-0301.  doi: 10.1145/3450626.3459812.  URL https://doi.org/10.1145/3450626.3459812.
  * Yang et al. [2023] Sipeng Yang, Yunlu Zhao, Yuzhe Luo, He Wang, Hongyu Sun, Chen Li, Binghuang Cai, and Xiaogang Jin.  Mnss: Neural supersampling framework for real-time rendering on mobile devices.  _IEEE Transactions on Visualization and Computer Graphics_ , pages 1–14, 2023.  doi: 10.1109/TVCG.2023.3259141.
  * NVIDIA [2022] NVIDIA.  NVIDIA Ada GPU science, 2022.  URL https://images.nvidia.com/aem-dam/Solutions/geforce/ada/ada-lovelace-architecture/nvidia-ada-gpu-science.pdf.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4), jul 2022.  ISSN 0730-0301.  doi: 10.1145/3528223.3530127.  URL https://doi.org/10.1145/3528223.3530127.
  * NVIDIA [2020] NVIDIA.  NVIDIA A100 tensor core GPU architecture, 2020.  URL https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper.pdf.
  * Intel® [2021] Intel®.  Intel® open image denoise, 2021.  URL https://www.openimagedenoise.org/.
  * Zhang et al. [2021] Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas.  Deep compositional denoising for high-quality monte carlo rendering.  _Computer Graphics Forum_ , 40(4):1–13, 2021\.  doi: https://doi.org/10.1111/cgf.14337.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14337.
  * Müller et al. [2020] Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller.  Neural control variates, 2020.  URL https://arxiv.org/abs/2006.01524.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ , 65(1):99–106, dec 2021.  ISSN 0001-0782.  doi: 10.1145/3503250.  URL https://doi.org/10.1145/3503250.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison.  In-place scene labelling and understanding with implicit scene representation, 2021.
  * Blomqvist et al. [2023] Kenneth Blomqvist, Francesco Milano, Jen Jen Chung, Lionel Ott, and Roland Siegwart.  Neural implicit vision-language feature fields, 2023.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion, 2022.
  * Rahaman et al. [2018] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville.  On the spectral bias of neural networks.  2018\.  doi: 10.48550/ARXIV.1806.08734.  URL https://arxiv.org/abs/1806.08734.
  * Tancik et al. [2020] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  _CoRR_ , abs/2006.10739, 2020.  URL https://arxiv.org/abs/2006.10739.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  _CoRR_ , abs/2006.09661, 2020.  URL https://arxiv.org/abs/2006.09661.
  * Bricman and Ionescu [2018] Paul Andrei Bricman and Radu Tudor Ionescu.  Coconet: A deep neural network for mapping pixel coordinates to color values, 2018.  URL https://arxiv.org/abs/1805.11357.
  * Minnen et al. [2018] David Minnen, George Toderici, Michele Covell, Troy Chinen, Nick Johnston, Joel Shor, Sung Jin Hwang, Damien Vincent, and Saurabh Singh.  Spatially adaptive image compression using a tiled deep network.  2018\.  doi: 10.48550/ARXIV.1802.02629.  URL https://arxiv.org/abs/1802.02629.
  * Thies et al. [2019] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures, 2019\.  URL https://arxiv.org/abs/1904.12356.
  * Kuznetsov et al. [2021] Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi.  Neumip: Multi-resolution neural materials, 2021.  URL https://arxiv.org/abs/2104.02789.
  * Zeltner et al. [2023] Tizian Zeltner, Fabrice Rousselle, Andrea Weidlich, Petrik Clarberg, Jan Novák, Benedikt Bitterli, Alex Evans, Tomáš Davidovič, Simon Kallweit, and Aaron Lefohn.  Real-time neural appearance models, 2023.
  * Aila and Karras [2010] Timo Aila and Tero Karras.  Architecture considerations for tracing incoherent rays.  In _Proceedings of the Conference on High Performance Graphics_ , HPG ’10, page 113–122, Goslar, DEU, 2010. Eurographics Association.
  * Pharr et al. [2016] Matt Pharr, Wenzel Jakob, and Greg Humphreys.  _Physically based rendering: From theory to implementation_.  Morgan Kaufmann, 2016.
  * Wald et al. [2014] Ingo Wald, Sven Woop, Carsten Benthin, Gregory S. Johnson, and Manfred Ernst.  Embree: A kernel framework for efficient cpu ray tracing.  _ACM Trans. Graph._ , 33(4), jul 2014.  ISSN 0730-0301.  doi: 10.1145/2601097.2601199.  URL https://doi.org/10.1145/2601097.2601199.
  * Benthin et al. [2018] Carsten Benthin, Ingo Wald, Sven Woop, and Attila T. Áfra.  Compressed-leaf bounding volume hierarchies.  In _Proceedings of the Conference on High-Performance Graphics_ , HPG ’18, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450358965.  doi: 10.1145/3231578.3231581.  URL https://doi.org/10.1145/3231578.3231581.
  * Keely [2014] Sean Keely.  Reduced Precision for Hardware Ray Tracing in GPUs.  In Ingo Wald and Jonathan Ragan-Kelley, editors, _Eurographics/ ACM SIGGRAPH Symposium on High Performance Graphics_. The Eurographics Association, 2014.  ISBN 978-3-905674-60-6.  doi: 10.2312/hpg.20141091.
  * LeGendre et al. [2019] Chloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevec.  Deeplight: Learning illumination for unconstrained mobile mixed reality, 2019.  URL https://arxiv.org/abs/1904.01175.
  * Gupta et al. [2015] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan.  Deep learning with limited numerical precision, 2015.
  * Moshier [1992] Stephen L Moshier.  Cephes mathematical library, 1992.  URL http://www.netlib.org/cephes/.
  * Community [2018] Blender Online Community.  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.  URL http://www.blender.org.
  * Mantiuk et al. [2011] Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich.  Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.  _ACM Trans. Graph._ , 30(4), jul 2011.  ISSN 0730-0301.  doi: 10.1145/2010324.1964935.  URL https://doi.org/10.1145/2010324.1964935.
  * Fouladi et al. [2022] Sadjad Fouladi, Brennan Shacklett, Fait Poms, Arjun Arora, Alex Ozdemir, Deepti Raghavan, Pat Hanrahan, Kayvon Fatahalian, and Keith Winstein.  R2e2: Low-latency path tracing of terabyte-scale scenes using thousands of cloud cpus.  _ACM Trans. Graph._ , 41(4), jul 2022.  ISSN 0730-0301.  doi: 10.1145/3528223.3530171.  URL https://doi.org/10.1145/3528223.3530171.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  _CoRR_ , abs/1901.05103, 2019.  URL http://arxiv.org/abs/1901.05103.
  * Graphcore [2023] Graphcore.  IPU ray library, 2023.  URL https://github.com/graphcore-research/ipu-ray-lib.
  * Valiant [1990] Leslie G Valiant.  A bridging model for parallel computation.  _Communications of the ACM_ , 33(8):103–111, 1990\.
  * Cheatham et al. [1996] Thomas Cheatham, Amr Fahmy, Dan Stefanescu, and Leslie Valiant.  Bulk synchronous parallel computing—a paradigm for transportable software.  In _Tools and Environments for Parallel and Distributed Systems_ , pages 61–76. Springer, 1996.  URL https://dash.harvard.edu/handle/1/26506457.
  * Blackman and Vigna [2018] David Blackman and Sebastiano Vigna.  Scrambled linear pseudorandom number generators.  _arXiv preprint arXiv:1805.01407_ , 2018.
  * Vigna [2019] Sebastiano Vigna.  It is high time we let go of the mersenne twister.  _arXiv preprint arXiv:1910.06437_ , 2019.
  * Graphcore [2021] Graphcore.  Graphcore documents, 2021.  URL https://docs.graphcore.ai/.

##  Appendix A Hardware Description

Graphcore processors have not been adopted for rendering before so we present
relevant details of the hardware and programming model. Whilst nominally
described as an AI accelerator, IPUs are relatively general purpose parallel
processors. Each IPU consists of a large number of homogeneous cores (called
tiles) that synchronise in hardware using an embodiment of the bulk
synchronous parallel (BSP) computation paradigm Valiant [1990], Cheatham et
al. [1996]. In this paradigm all cores compute, then synchronise at a barrier,
then exchange data, and repeat. This computation model is well suited for
expressing massively parallel computation without the programmer needing to
micro-manage synchronisation and race conditions: essentially it is a hardware
barrier synchronisation scheme. IPUs can be arranged into multi-IPU devices
where the inter-IPU synchronisation is also achieved via a BSP sequence. Like
GPUs, IPUs are accelerators, not standalone processors, and therefore require
a host computer to run an operating system and orchestrate computation.

###  A.1 IPU Processor Architecture

The GC200 processor contains 1472 tiles (cores) and each tile has a six barrel
scheduled hardware threads called workers, each receiving one execution slot
in round-robin fashion. This fixed schedule hides the SRAM latency, simplifies
the compiler’s view of each thread, and eliminates the need for out of order
execution or branch prediction. The barrel contains an extra slot reserved for
a supervisor thread which is responsible for initialisation of worker threads
and inter-core communication. The workers can execute completely independent
programs (each with their own control flow) and hence this processor is
genuinely capable of multiple-instruction-multiple-data (MIMD) computations: a
single GC200 can be running 8832 independent programs at once. Each tile also
has the following features that are expanded upon below:

  * •

Private SRAM: 624KiB per tile, 897MiB aggregate per chip.

  * •

A vector floating point unit (VFPU) capable of half and single precision
calculations with (optional) stochastic rounding.

  * •

An accumulating matrix product (AMP) unit (systolic array).

  * •

A hardware random number generator.

  * •

All to all data exchange with other tiles.

While the local SRAM could be viewed as a cache the BSP architecture and graph
programming model (Section A.2) ensure its contents is completely
deterministic. There is no memory hierarchy and tiles cannot issue an external
memory fetches directly themselves. Instead, DRAM I/O requires a separate step
in the BSP schedule. In effect, all tiles must sync before data can be
streamed on/off chip. There is, however, an option to have a subset of tiles
perform off-chip I/O while all the other tiles compute facilitating
asynchronous host I/O or memory fetch pipelines.

The VFPU is 2-wide for float and 4-wide half precision FLOPs. The AMP unit
achieves the device’s peak floating point throughput and accelerates the large
matrix multiplies and convolutions in deep learning.

The hardware random number generator (HW-RNG) is based on the xoshiro family
Blackman and Vigna [2018] and can directly produce uniformly distributed
single precision floating point values with superior statistical properties to
the popular Mersenne-Twister Vigna [2019].

Inter-tile communication is achieved by a proprietary graph compiler (Poplar)
scheduling all of the program’s communication in advance. This allows the
inter-tile communication to be protocol free: no bandwidth is lost to
addressing or packet headers. Inter-chip communication is packet based but
still prearranged for efficiency and to maintain the BSP programming paradigm
throughout the communication hierarchy.

####  A.1.1 IPU System Architecture

We will not go into full details of all possible IPU system configurations but
we note that GC200s are usually available in groups of 4 in an IPU Pod 4: a 1U
chassis which also contains the external DRAM (up to 450GB). These chips also
have high-speed (320GB/sec) interconnects directly between themselves allowing
for direct data exchange amongst their 3.5GiB of aggregate SRAM. Effectively,
currently available IPU systems have traded external memory bandwidth for
larger external memory size, targeting applications where larger internal
SRAM, inter-core, and inter-chip communication bandwidth is more useful. It is
important to take advantage of this trade-off when designing or selecting
algorithms suitable for the current generation of IPU systems.

###  A.2 IPU Programming Model

The core SDK for IPU programming consists of a C++ API for describing and
compiling computational graphs (Poplar), a codelet (kernel) LLVM compiler
(PopC), and a utility library that builds on these (Poplibs).

####  A.2.1 The Compute Graph

Poplar is used to construct a compute graph consisting of vertices, describing
computations, connected by edges, describing data flow. Data can be n-way
tensors of various floating point and integer data types. Arbitrary structures
and arrays of plain old data can be transferred to the device in byte tensors
which can then be reinterpret cast for use by the compute kernels. A
computation graph is built by adding compute vertices to the program and
connecting fields in those vertices to tensors to specify the data
dependencies.

####  A.2.2 Compute Kernels (Codelets)

A compute kernel in Poplar is called a codelet. Codelets define the
calculations performed by a worker at each compute vertex. Vertex code is
written in C++ and compiled by PopC. These vertices can also be written using
assembly for maximum performance, or optimised using inline assembly or
intrinsics to access IPU hardware specific functionality.

####  A.2.3 Programs and Execution

Once the compute graph is constructed Poplar also needs a separate description
of how to execute it in the form of the graph program. The simplest form of
program is just to execute a group of vertices (compute set) but execution can
also be arranged into sequences and loops allowing for complex and data
dependent control flow within the fixed graph. This control flow is executed
on device with no interaction from the host machine. The host tells the device
which program to run then monitors when to sync with the device in the BSP
schedule for I/O purposes. The final component needed for a complete program
are data streams between the host and device or between the device and DRAM.
Full details can be found in the Poplar SDK documentation Graphcore [2021].

##  Appendix B HDRI Dataset and Credits

The HDRIs used to train the HDR-NIFs in this work were all distributed under
the CC0 license:

  * •

urban_alley_01, borghese_gardens, and green_sanctuary by Andreas Mischok

  * •

country_club and studio_small_09 by Sergej Majboroda

  * •

shanghai_bund and syferfontein_1d_clear by Greg Zaal