# ReShader: View-Dependent Highlights for Single Image View-Synthesis

Avinash Paliwal  1234-5678-9012-3456 Texas A&M UniversityCollege StationTXUSA
<EMAIL_ADDRESS>,  Brandon G. Nguyen  Texas A&M UniversityCollege
StationTXUSA<EMAIL_ADDRESS>,  Andrii Tsarov  Leia Inc.Rono-HillsMenlo ParkCAUSA
<EMAIL_ADDRESS>and  Nima Khademi Kalantari  Texas A&M
UniversityCollege StationTXUSA<EMAIL_ADDRESS>
###### Abstract.

In recent years, novel view synthesis from a single image has seen significant
progress thanks to the rapid advancements in 3D scene representation and image
inpainting techniques. While the current approaches are able to synthesize
geometrically consistent novel views, they often do not handle the view-
dependent effects properly. Specifically, the highlights in their synthesized
images usually appear to be glued to the surfaces, making the novel views
unrealistic. To address this major problem, we make a key observation that the
process of synthesizing novel views requires changing the shading of the
pixels based on the novel camera, and moving them to appropriate locations.
Therefore, we propose to split the view synthesis process into two independent
tasks of pixel reshading and relocation. During the reshading process, we take
the single image as the input and adjust its shading based on the novel
camera. This reshaded image is then used as the input to an existing view
synthesis method to relocate the pixels and produce the final novel view
image. We propose to use a neural network to perform reshading and generate a
large set of synthetic input-reshaded pairs to train our network. We
demonstrate that our approach produces plausible novel view images with
realistic moving highlights on a variety of real world scenes.

View synthesis, neural network, reshading, relocation

††journal: TOG††copyright: rightsretained††journal: TOG††journalyear:
2023††journalvolume: 1††journalnumber: 1††article: 1††publicationmonth:
1††doi: 10.1145/3618393††ccs: Computing methodologies Image-based rendering

Figure 1. To properly handle the view-dependent effects, we propose to break
down the view synthesis process into two tasks of pixel reshading and
relocation. During reshading, we use a neural network to generate a new
version of the input image (shown on the left) with the shading computed based
on the novel view. As shown on the middle, our reshading network correctly
leaves the diffuse areas intact (the dog’s head), but moves the highlights on
the specular areas (wooden floor). The relocation process takes this reshaded
image and generates the novel view image. The red crosses mark the same
location on the wooden floor to make it easier to observe the effect of
reshading and relocation.

This is the teaser figure for the article.

##  1\. Introduction

Creating novel views of a scene from a single image is a compelling way to
breathe life into still photographs. When displayed on virtual reality (e.g.,
HTC vive and Meta Quest) or light field (e.g., Lume Pad (Leia, 2023)) devices,
these “3D photographs” provide a highly immersive experience for users,
allowing them to vividly relive moments captured in still photographs as if
they have been transported back in time and place.

Figure 2. We compare our results against 3D Moments by Wang et al. (2022). 3D
Moments reconstructs the novel image by moving the input pixels according to
their depth values. As such, the highlights are treated as textures and appear
to be glued to the wooden table. Our approach, however, is able to properly
move the highlights over the table. The red crosses mark the same location on
the table. Note that the cross is inside the highlight in the input and 3D
Moment’s results, but it appears to be outside the highlight in our results.

The rapid advancements in 3D scene representation and image inpainting
techniques have led to remarkable progress in single image view synthesis in
recent years. Despite this, the existing techniques focus on producing
geometrically consistent novel views and mostly ignore the view-dependent
effects. For example, a number of techniques (Shih et al., 2020; Jampani et
al., 2021), handle this application in a _modular_ manner. These approaches
estimate the depth from the input and use it to decompose the scene into
multiple layers. These depth layers are then warped to the novel view and
composed together to form the final image. Unfortunately, these methods treat
the highlights, which are quite common in real scenes, as textures and warp
them to the novel views along with other areas. Therefore, as shown in Fig. 2,
the highlights in their synthesized views appear to be glued to the surfaces,
making their results unrealistic.

On the other hand, several approaches (Srinivasan et al., 2017; Li and
Kalantari, 2020; Yu et al., 2021) handle this problem by learning the process
in an _end-to-end_ manner. These techniques learn the entire view synthesis
pipeline either directly (Srinivasan et al., 2017), or through various scene
representations, such as neural radiance field (NeRF) (Yu et al., 2021) and
multiplane images (MPI) (Li and Kalantari, 2020; Tucker and Snavely, 2020).
Although they could potentially handle the view-dependent effects, these
techniques often struggle to properly reconstruct the moving highlights.

Our main observation is that both the shading and projected pixel location of
a 3D surface point change between the input and novel view images. Modular
approaches overlook the view-dependent shading, focusing solely on pixel
relocation. The end-to-end approaches, on the other hand, aim to learn to move
the pixels and change their shading within a unified system. However, the
majority of effort is dedicated to learning pixel relocation, as the
contribution of the shading mismatch to their training loss is often minimal.

Guided by this observation, we make a key contribution to break down the novel
view synthesis process into two tasks: pixel reshading and relocation (see
Fig. 1). During the reshading process, we only adjust the shading of the input
image according to the novel camera. We then perform pixel relocation on the
reshaded image, using the modular method by Wang et al. (2022), to obtain the
final novel view image. We propose to learn the reshading process using a
neural network that takes a single image as well as the relative novel camera
position as the input and produces the reshaded image. Since there are no
publicly available datasets of input-reshaded image pairs, we render a large
number of synthetic image pairs for training. We train our reshading network
on this newly introduced dataset using a perceptual loss to ensure producing
plausible, but detailed reshaded images. We demonstrate that our method
produces high-quality novel view images with plausible moving highlights on a
wide range of real scenes.

##  2\. Related Work

The problem of view synthesis has been extensively studied and many powerful
multi- and single-image methods have been developed (Mildenhall et al., 2020;
Wizadwongsa et al., 2021; Shih et al., 2020; Tucker and Snavely, 2020). A
complete literature review is beyond the scope of this paper. Here, we mainly
focus on approaches that use a single image as the input. We also discuss
image relighting methods as they are relevant to the focus of our paper.

###  2.1. Single Image View Synthesis

We discuss these approaches by categorizing them into two classes of modular
and end-to-end. The modular methods (Niklaus et al., 2019; Kopf et al., 2019,
2020; Shih et al., 2020; Jampani et al., 2021; Wang et al., 2022) break down
the process into multiple components and address each component separately.
Specifically, these techniques divide the view synthesis pipeline into depth
estimation, image warping, and image inpainting. The individual methods differ
in how they handle each stage of the pipeline. For example, Niklaus et al.
(2019) train a depth estimation network and use it to directly reproject the
input image to the novel view. On the other hand, Shih et al. (2020) obtain
the depth using an existing method (Ranftl et al., 2022) and reconstructs
layered depth image (LDI) representation (Shade et al., 1998) to warp the
input image to the novel view. These techniques, however, primarily focus on
pixel relocation and overlook the pixel reshading process. As a result, they
produce results with incorrect view-dependent effects, where the highlights
appear to be glued to the surfaces (see Fig. 2).

A category of modular methods focus on handling the view-dependent effects by
first decomposing the image(s) into multiple layers (e.g., diffuse and
reflective), warping each layer separately, and blending them to generate the
final image. However, most of these techniques are either specifically
designed for rendering (Lochmann et al., 2014; Zimmer et al., 2015) where
ground truth scene information (e.g., geometry and material) is available, or
require multiple images (Blake, 1985; Roth and Black, 2006; Sinha et al.,
2012).

In contrast to the modular approaches, a number of techniques (Srinivasan et
al., 2017; Wiles et al., 2020; Li and Kalantari, 2020; Tucker and Snavely,
2020; Han et al., 2022; Yu et al., 2021) attempt to learn the entire view
synthesis process in an end-to-end manner. Zhou et al. (2016) propose to
estimate optical flows at novel views and use the estimated flow to backward
warp the input image. The flow estimation network is trained by minimizing the
loss between the synthesized and ground truth novel view images. Srinivasan et
al. (2017) propose to estimate a light field from a single image using a
convolutional neural network (CNN). Several approaches use a network to
estimate intermediate representations, such as point cloud (Wiles et al.,
2020), multiplane images (MPI) (Li and Kalantari, 2020; Tucker and Snavely,
2020; Han et al., 2022), and neural radiance field (NeRF) (Yu et al., 2021).
Since these approaches perform end-to-end training, they could potentially
learn to handle the view-dependent effects. However, highlights are usually
concentrated in small regions, and thus the shading mismatch does not
significantly contribute to the loss function. As such, these methods often
are not able to produce results with proper moving highlights.

Recently, several approaches (Poole et al., 2022; Gu et al., 2023; Shue et
al., 2022; Watson et al., 2022; Fridman et al., 2023; Chan et al., 2023) have
proposed to address this problem using diffusion models (Ho et al., 2020).
Some of these techniques (Shue et al., 2022; Watson et al., 2022) produce
novel view images of only single objects or simple scenes. Others (Fridman et
al., 2023; Chan et al., 2023) handle complex scenes and produce impressive
walkthroughs from a single image. However, when synthesizing views that are
relatively close to the input, the quality of their synthesized images are not
on par with the existing modular or MPI-based techniques.

###  2.2. Image Relighting

Image relighting is the process of reconstructing images of a scene under
different illumination. This problem is highly related to inverse rendering
where the aim is to estimate the image formation factors (e.g., shape,
reflectance, lighting) of a scene. Several methods propose to handle this
application either by directly estimating the relit images (Xu et al., 2018),
estimating the individual factors (Xu et al., 2019), or by utilizing NeRF (Bi
et al., 2020b, a; Boss et al., 2021; Srinivasan et al., 2021; Zhang et al.,
2021). However, these approaches focus on simple scenes or single objects, and
require multiple images as the input. For more complex scenes, Philip et al.
(2019) propose a relighting approach for outdoor scenes, while Philip et al.
(2021) and Wu et al. (2022) focus on indoor scenes. However, both of these
techniques use several images of the scene as the input.

Several techniques (Sengupta et al., 2019; Li et al., 2020; Wang et al., 2021;
Li et al., 2022) propose to estimate all the image formation factors including
shape, reflectance, and lighting, from a single image. Sengupta et al. (2019)
propose an inverse rendering network to estimate albedo, normal, and a single
environment lighting. Li et al. (2020) extend this work to estimate per-pixel
lighting, as well as roughness and depth. Wang et al. (2021) further propose
to estimate 3D lighting of the scene through volumetric spherical Gaussian.
Moreover, Li et al. (2022) present a holistic scene reconstruction system that
estimates the reflectance, shape, and parameteric 3D lighting. These
techniques demonstrate impressive results for object insertion, material
editing, and dramatic lighting change (Li et al., 2022) (e.g., covering a
window). While they could potentially be used to perform pixel reshading,
these methods do not meet the quality requirement for our application.

##  3\. Algorithm

Given a single RGB image $I$, captured with a camera at location ${\bf c}$,
our primary goal is to synthesize an image $I^{\prime}$ from a novel view
${\bf c}^{\prime}$. Similar to most existing methods (Jampani et al., 2021;
Han et al., 2022), we assume the depth can be obtained with a reasonable
accuracy using single image depth estimation techniques (Ranftl et al., 2022).

We begin by discussing the rendering equation (Kajiya, 1986), a reasonably
expressive rendering model, to describe the relationship between the input and
novel view images. Formally, the rendering equation describes the total
outgoing radiance $L_{o}({\bf x},{\bf\omega}_{o})$ at a 3D point ${\bf x}$
along the viewing direction $\omega_{o}$ as follows:

(1) $L_{o}({\bf x},{\bf\omega}_{o})=L_{e}({\bf
x},{\bf\omega}_{o})+\int_{\Omega}f_{r}({\bf
x},{\bf\omega}_{o},{\bf\omega}_{i})\ L_{i}({\bf x},{\bf\omega}_{i})\
\cos(\theta_{i})\ d{\bf\omega}_{i},$

where $L_{e}$ and $L_{i}$ are the emitted and incoming radiances,
respectively, ${\bf\omega}_{i}$ is the incoming direction, and $f_{r}$ is the
bidirectional reflectance distribution function (BRDF). Moreover, $\theta_{i}$
is the angle between ${\bf\omega}_{i}$ and the surface normal, and the
integral is taken over the entire hemisphere $\Omega$ over the surface point.

Figure 3. We visualize the image formation process for the input (${\bf c}$)
and novel (${\bf c}^{\prime}$) cameras. A surface point ${\bf x}$ appears at
two different locations (${\bf p}_{{\bf x}}$ and ${\bf p}^{\prime}_{{\bf x}}$)
in the input and novel images. Moreover, the shading of point ${\bf x}$ in the
two images is determined by $L_{o}({\bf x},{\bf\omega}_{o}^{{\bf
x}\shortrightarrow{\bf c}})$ and $L_{o}({\bf x},{\bf\omega}_{o}^{{\bf
x}\shortrightarrow{\bf c}^{\prime}})$, and thus is different. Note that the
incoming radiance $L_{i}$, surface normal (and consequently $\theta_{i}$), and
the BRDF (shown with curly black line), are the same for both the input and
novel view images. Figure 4. We show an input and a novel view image. The same
point on the table appears at different locations and with different shadings
in the input and novel images. Therefore, the view synthesis process can be
divided into two tasks of pixel reshading and relocation.

As shown in Fig. 3, the appearance of a surface point ${\bf x}$ in the input
and novel images is determined by the outgoing radiance $L_{o}({\bf
x},{\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}})$ and $L_{o}({\bf
x},{\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}^{\prime}})$, respectively.
Here, ${\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}}$ is the direction from
the surface point to input camera location ${\bf c}$, while
${\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}^{\prime}}$ represents the
direction to the novel camera at position ${\bf c}^{\prime}$.

Based on this analysis, we observe that the appearance of point ${\bf x}$ in
the input and novel images differs in two major ways: 1) The point ${\bf x}$
appears with different shadings in the input and novel images as its
appearance is determined by $L_{o}({\bf x},{\bf\omega}_{o}^{{\bf
x}\shortrightarrow{\bf c}})$ and $L_{o}({\bf x},{\bf\omega}_{o}^{{\bf
x}\shortrightarrow{\bf c}^{\prime}})$, respectively. 2) The location of this
point in the two images is different; ${\bf p}_{{\bf x}}$ and ${\bf
p}^{\prime}_{{\bf x}}$ in the input and novel images, respectively. This is
determined by the intersection of the rays along directions
${\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}}$ and ${\bf\omega}_{o}^{{\bf
x}\shortrightarrow{\bf c}^{\prime}}$ with the image planes of the input and
novel cameras, respectively.

Therefore, we can describe the view synthesis process through two tasks of
pixel reshading and relocation, as shown in Fig. 4. Existing modular
approaches (Shih et al., 2020; Jampani et al., 2021), synthesize novel view
images by warping the input image to the novel view using the input depth. As
such, they mainly focus on the pixel relocation task and ignore the pixel
reshading process, which is responsible for the view-dependent effects. The
end-to-end systems (Li and Kalantari, 2020; Han et al., 2022), on the other
hand, attempt to learn both pixel reshading and relocation processes by
minimizing the loss between the estimated and ground truth novel view images.
However, these systems often ignore the pixel reshading task as the
contribution of the shading differences to the appearance loss is small; view-
dependent highlight are often concentrated in small regions in each scene. As
such, these techniques are not able to properly handle the view-dependent
effects.

To address this problem, we propose to treat pixel reshading and relocation as
two independent tasks. Specifically, we first adjust the shading of the input
image according to the novel view camera. We then use the reshaded image as
the input to the approach by Wang et al. (2022) to relocate the pixels and
produce the final image. Below we discuss our approach in detail.

###  3.1. Pixel Reshading

Our goal is to take the input image $I$ and produce a reshaded image $I_{s}$
that has the same shading as the novel view image. This necessitates changing
the shading of input pixel ${\bf p}_{{\bf x}}$ from $L_{o}({\bf
x},{\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}})$, to the shading of the
corresponding pixel in the novel image ${\bf p}^{\prime}_{{\bf x}}$, i.e.,
$L_{o}({\bf x},{\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}^{\prime}})$.
Note that at this stage, we are not interested in pixel relocation and
reshading occurs in the input camera frame.

According to the rendering equation (Eq. 1), performing the reshading process
requires estimating various components: the lighting $L_{e}$ (emitters),
material properties $f_{r}$, incoming radiance from all directions going
through the hemisphere $L_{i}$, and the normals (to compute $\theta_{i}$).
Once these quantities are estimated, it is possible to recompute the shading
of pixel ${\bf p}_{{\bf x}}$ in the input image, by evaluating the integral in
Eq. 1 using the outgoing direction of the corresponding pixel in the novel
view image ${\bf\omega}_{o}^{{\bf x}\shortrightarrow{\bf c}^{\prime}}$. Note
that the outgoing direction can be easily inferred from the input depth and
the camera positions (provided relatively to avoid the need for estimating the
input camera pose).

Unfortunately, estimating all of the aforementioned quantities from a single
image is an extremely challenging problem. While there are existing techniques
(Sengupta et al., 2019; Li et al., 2020; Wang et al., 2021; Li et al., 2022)
that estimate these various factors to a great extent, the quality of their
re-rendered images falls short of the requirements for our view synthesis
application.

Therefore, we instead propose to directly learn the reshaded image from the
input image using a neural network. Although simple, as shown in Sec. 4 and in
the supplementary video, our method is able to handle this challenging problem
reasonably well and produce results with plausible moving highlights. In the
following sections, we describe our dataset, inputs, architecture, and
training process.

Figure 5. We visualize our modification to the path tracer to render the
reshaded images. We trace a primary ray to find the first intersection from
the input camera. We then find the ray from the novel camera to this point
(novel primary ray). This ray is then used for shading computation at the
intersection point and generation of the secondary ray. Figure 6. For each
training example in our dataset, we store the input and ground truth reshaded
images, as well as the depth and validity mask. The red arrows point to the
highlights in the input image that are moved in the reshaded image. Note that
the objects in the reshaded image are in the same location as the input image,
since reshading happens in the input camera frame. Small areas in the reshaded
image (indicated by the green arrow) contain incorrect shading. We mask these
out using the validity mask in our training loss.

###  3.2. Dataset

To train our reshading network, we need a dataset of input-reshaded image
pairs, which is currently not available. Unfortunately, obtaining such a
dataset from real scenes is extremely challenging. Capturing the reshaded
image necessitates taking a picture of the scene from the input camera view,
but with the light rays going towards a different camera. One potential
solution is to take a large number of images of a scene and use neural
radiance field (NeRF) (Mildenhall et al., 2020) to reconstruct the radiance
field of the scene. This radiance field can then be used to produce the
reshaded images. However, generating a large scale dataset using this approach
is difficult. Additionally, even the state-of-the-art approaches (Kopanas et
al., 2022; Kerbl et al., 2023; Verbin et al., 2022) struggle to produce high-
quality view-dependent effects on arbitrary surfaces.

Therefore, we propose to generate our input-reshaded image pairs
synthetically. Specifically, we use the Tungsten renderer (Bitterli, 2014) and
render our input images using a large number of samples per pixel. We then
slightly modify the path tracer to obtain the corresponding reshaded images,
as shown in Fig. 5. To do this, we trace primary rays from the input camera
(input primary ray) to find the first intersection points. We then calculate
the rays connecting the novel camera to these intersection point (novel
primary ray). These novel primary rays are then used for shading and
generating all the additional secondary rays. An example input-reshaded image
pair from our dataset is shown in Fig. 6 (top row).

Note that some regions from the input image are occluded in the novel camera.
We could easily detect and mask these areas by performing a visibility check
with the novel primary ray. However, we choose not to do so to provide more
content for our network to learn from. Most of the occluded areas will be
shaded correctly as if they are not obscured from the camera. However, small
regions (see the green arrow in Fig. 6), typically along the boundaries of
objects, will be incorrectly shaded. These are the cases where the angle
between the surface normal and novel primary ray is greater than 90 degrees.
We detect these regions and create a validity mask, as shown in Fig. 6, which
is used to mask out such areas when computing our training loss. Note that
since we are using Monte Carlo rendering, each pixel is rendered by tracing a
large number of rays. We mark a pixel as invalid if any of such rays does not
satisfy our constraint. This is why the line in the validity mask appears to
be thicker than the problematic region in the reshaded image.

Figure 7. Scenes used to create the synthetic dataset.

We use the above approach to generate our synthetic dataset using 9 scenes,
shown in Fig. 7, provided by Bitterli (2016). For each scene, we render 200
input-reshaded pairs by randomly placing the input and novel cameras inside
the scene. We randomly choose the novel cameras inside a sphere, centered on
the input camera, with radii ranging from 0.1 to 0.3. Note that since all the
scenes have similar global scale, the chosen radius range corresponds to a
reasonable and uniform camera movement in all the training scenes. For every
image pair, we randomly change the texture and material properties of the
objects in the scene. By default, most scenes only use the environment map as
the light source. To increase the robustness of our approach, we add multiple
random colored orbs into the scene at random locations. We render $1280\times
720$ high dynamic range (HDR) images with 8K samples per pixel and for each
example, we store the input and reshaded images, as well as the depth,
validity mask, and the metadata of the cameras. Our training data for one
example is shown in Fig. 6.

Figure 8. We show the architecture of our reshading network. Each
convolutional layers (shown in orange) is followed by a LeakyReLU activation,
except the last layer that uses tanh activation. We use average $2\times 2$
pooling for downsampling, while we use bilinear upsampling to increase the
resolution. We use an MLP to convert the 3 channel novel camera position
vector to a 125-channel feature vector. We then concatenate the original
camera position vector with this feature vector. The result is then replicated
and attached to the bottleneck feature map. The dashed lines represent skip
connections. Note that our network estimate the residual image which is added
to the input to obtain the reshaded image.

###  3.3. Inputs

For our network to be able to properly reshade an input image, we need to
provide the depth information along with the novel camera position to our
network. The novel camera position is a 3-channel vector containing position
of the novel camera relative to the input camera. Similar to most current
single image view synthesis methods, we estimate the depth map using an
existing single image depth estimation method (Ranftl et al.’s approach (2021;
2022) in our implementation). Instead of passing the depth to our network,
however, we first convert it to disparity. We then scale it by a factor of 1/4
and clamp it to one. This ensures that the disparity is in the range [0, 1]
and it covers the depth from 0.25 to infinity. Moreover, we apply frequency
encoding (Mildenhall et al., 2020) with 5 frequencies (11 channels; original
plus 5 sines and 5 cosines) to the input disparity to allow the network to
effectively use the disparity, particularly for far away regions. Frequency
encoding essentially increases the resolution of the disparity, while
remaining in the range [0, 1]; similar disparity values will have
significantly different representation in the frequency domain.

To summarize, we use the input RGB image, frequency encoded disparity map, and
the relative novel camera position as the input to our network to produce the
reshaded image. The effect of using the disparity map and frequency encoding
are shown in Figs. 11 and 12, respectively.

###  3.4. Architecture

We utilize a UNet (Ronneberger et al., 2015) style encoder-decoder style
architecture consisting of 5 downsampling/upsampling layers. The encoder takes
the input image and frequency encoded disparity (3+11 channels) and produces a
bottleneck feature map of size $H/32\times W/32\times 512$, where $H$ and $W$
are the height and width of the input image, respectively. The three channel
novel camera position vector is converted to a 125-channel feature vector
using a multilayer perceptron (MLP) with a series of fully connected layers.
This feature vector is then concatenated with the original 3-channel camera
position vector to produce our novel camera features. This is then replicated
and concatenated with the bottleneck feature map from the encoder (map of size
$H/32\times W/32\times 640$). The concatenated feature map is then used as the
input to the decoder to produce a 3-channel residual image. The residual is
then added to the input to produce the reshaded image. Our architecture is
shown in Fig. 8.

###  3.5. Training

We perform a series of augmentations to improve the generalization ability of
our network. We take $384\times 384$ random crops of the HDR synthetic dataset
and convert the input and ground truth reshaded pairs to low dynamic range
images by applying random exposure (scale factor between 3 and 10) and gamma
correction ($\gamma$ between 2.2 and 5). In addition, we randomly scale the
disparity by a factor of $f$ and the camera position by a factor of $1/f$
simultaneously. This increases the range of scene scales in our training data.

Since this problem is highly ill-posed, we perform the training using a
combination of $\mathcal{L}_{1}$ and perceptual losses. Specifically, our loss
consists of the following three terms:

(2)
$\mathcal{L}=\lambda_{\mathrm{1}}\mathcal{L}_{\mathrm{1}}+\lambda_{\mathrm{vgg}}\mathcal{L}_{\mathrm{VGG}}+\lambda_{\mathrm{style}}\mathcal{L}_{\mathrm{style}},$

where the first term is the $\mathcal{L}_{1}$ loss between the estimated and
ground truth reshaded images and is defined as follows:

(3) $\mathcal{L}_{\mathrm{1}}=\|\tilde{I}_{s}-I_{s}\|_{1}.$

Moreover, the second term is a perceptual VGG-based loss and is defined as:

(4) $\mathcal{L}_{\mathrm{VGG}}=\|\phi(\tilde{I}_{s})-\phi(I_{s})\|_{2}^{2},$

where $\phi$ represents the output features from the conv4_4 layer of VGG-19
(Simonyan and Zisserman, 2014). Furthermore, the third term is a perceptual
VGG-based style loss and is defined as:

(5)
$\mathcal{L}_{\mathrm{style}}=\|G(\phi(\tilde{I}_{s}))-G(\phi(I_{s}))\|_{2}^{2},$

where $G$ computes the Gram matrix of the VGG features extracted from the
estimated and ground truth reshaded images. Finally, $\lambda_{\mathrm{1}}$,
$\lambda_{\mathrm{vgg}}$, and $\lambda_{\mathrm{style}}$ define the weight of
each term in Eq. 2 and we set them to 1e-1, 1e-2, and 1, respectively. Note
that we multiply the estimated and ground truth reshaded images by the
validity mask before computing each loss term.

Figure 9. We show comparisons against SVMPI (Tucker and Snavely, 2020) and 3D
Moments (Wang et al., 2022). Only our approach is able to move the highlights
in different views. Note that we carefully select the insets to cover roughly
the same regions in the two views to be able to demonstrate the view dependent
effects.

###  3.6. Pixel Relocation

Once our reshading network is trained, we can use it to reshade the input
image during inference. We then use the reshaded image as the input to the
approach by Wang et al. (2022) to reconstruct the final novel view image. This
approach is designed to perform view and time interpolation using near
duplicate photos. However, all the operations related to view synthesis
utilize a single image. Therefore, we isolate the view synthesis component and
use it to generate novel views from a single image.

The view synthesis component of this approach is an enhanced version of the
technique by Shih et al. (2020). Specifically, using the depth, this method
first constructs a layered depth image (LDI) representation (Shade et al.,
1998). It then inpaints the occluded regions and produces LDI features using a
network. The LDI features are then warped to the novel view and combined using
a subsequent network to produce the final image. Note that our reshaded image
is different for each view, which could potentially change the inpainting
results, and consequently affect coherency of the synthesized views. However,
we did not observe this effect in practice. As shown in the supplementary
video, our results are coherent.

We note that our approach can be combined with any view synthesis technique
that focuses on pixel relocation. We demonstrate this in Table 1, where we
examine the performance of our approach using Shih et al.’s method (2020) (3D
Photo) for pixel relocation.

##  4\. Results

We implement our approach in PyTorch and use Adam (Kingma and Ba, 2015) with
the default parameters for training. We use a learning rate of 1e-4 for 300K
iterations and 1e-5 for another 200K iterations. Our training takes 5 days on
an Nvidia 2080 Ti GPU.

We compare our approach against single image view synthesis approaches by
Tucker and Snavely (2020) (SVMPI) and Wang et al. (2022) (3D Moments). SVMPI
is trained in an end-to-end manner on a multi-view dataset and ideally should
be able to handle the view-dependent effects. On the other hand, 3D Moments,
which we use for pixel relocation, is a modular approach that is not able to
move the highlights. We use the code provided by the authors to generate the
results. We use images from several datasets, including Holopix50K (Hua et
al., 2020), Open Images V7 (Kuznetsova et al., 2020) and Shiny (Wizadwongsa et
al., 2021). Here, we show the image results, but the differences can be better
observed in the supplementary video.

Table 1. We show numerical comparisons against the other approaches on three
synthetic scenes by evaluating the error between the ground truth and novel
view images in terms of PSNR, SSIM, and LPIPS.

Scene | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
Veach Ajar | SVMPI | 22.72 | 0.877 | 0.0428
3D Photo | 30.06 | 0.962 | 0.0200
3D Photo + Ours | 30.70 | 0.962 | 0.0198
3D Moments | 29.78 | 0.962 | 0.0149
3D Moments + Ours | 30.41 | 0.962 | 0.0147
Bathroom | SVMPI | 20.27 | 0.602 | 0.1255
3D Photo | 29.96 | 0.907 | 0.0329
3D Photo + Ours | 30.84 | 0.910 | 0.0323
3D Moments | 32.03 | 0.951 | 0.0284
3D Moments + Ours | 33.12 | 0.953 | 0.0281
Modern Hall | SVMPI | 22.73 | 0.763 | 0.0759
3D Photo | 32.63 | 0.950 | 0.0230
3D Photo + Ours | 32.99 | 0.951 | 0.0229
3D Moments | 30.98 | 0.951 | 0.0197
3D Moments + Ours | 31.21 | 0.953 | 0.0196

In Fig. 9, we show comparisons against the other techniques on five scenes.
For each scene, we show the results for two different views. We have carefully
selected the insets, so they roughly cover the same region in the two views.
Therefore, each approach’s ability to adjust the shading based on the view can
be observed by comparing the two views. Overall, 3D Moments produce results
where the shading of the two views are almost identical. In some cases, SVMPI
slightly alters the position of the highlights, but when doing so, it disturbs
the texture underneath. Additionally, it produces slightly overblurred
results. Our approach, on the other hand, produces detailed images with moving
highlights. For example, in the first and fourth rows, our approach moves the
highlight to the right and left, respectively, when transitioning from view 1
to 2. Note that our method does not leak the highlights to the dark region in
the top row and the diffuse key fob in the fourth row. In the second row, our
method produces results with slightly darker shading in the second view, while
keeping the underlying texture intact. Finally, in the third and last rows,
our approach is able to properly move the highlights (to the left from view 1
to 2) on the red structure and the burger bun, respectively.

Furthermore, we numerically compare our view synthesis results against the
other techniques on three synthetic scenes in Table 1. To demonstrate that our
approach can be used with any pixel relocation method, we show results with
both 3D moments (Wang et al., 2022) and Shih et al.’s approach (2020) (3D
Photo). As seen, our approach improves the performance of both modular
relocation methods. Note that SSIM and LPIPS are highly sensitive to the
textures, but are not sensitive to the smooth highlights. As such, these
metrics do not fully reflect our quality improvement. Moreover, we evaluate
our reshading network in isolation (see Table 2), by measuring the error
between our synthesized and ground truth reshaded images. By appropriately
moving the highlights, our approach produces results that are significantly
closer to the ground truth than the input images (without reshading). This is
shown visually in Fig. 10 for the Modern Hall example. Our approach properly
moves the highlights (top row), and thus is able to synthesize a novel view
image that better matches the ground truth than 3D Moments (bottom row).

Table 2. We numerically evaluate the effect of reshading in isolation. Our
reshading network produces results that are closer to the ground truth than
the input.

Scene | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
Veach Ajar | Input | 35.45 | 0.993 | 0.0012
Ours | 40.10 | 0.994 | 0.0008
Bathroom | Input | 36.50 | 0.991 | 0.0024
Ours | 41.20 | 0.992 | 0.0020
Modern Hall | Input | 39.99 | 0.989 | 0.0015
Ours | 42.71 | 0.989 | 0.0012

Figure 10. We show our reshading (top) and view synthesis (bottom) results on
the Modern Hall scene. Our approach is able to properly move the highlights
during the reshading process (top) and produce novel view images that better
match the ground truth than existing techniques (bottom).

Next, we discuss the effect of several design choices in our approach
numerically (Table 3) and visually (Figs. 11, 12, and 13). In Fig. 11, we
demonstrate that without the disparity as the input, our reshading network is
not able to detect the depth discontinuities and smears the shading of the
tomato on the bowl. Moreover, as shown in Fig. 12, without frequency encoding,
our network has difficulty handling the objects that are far away and
incorrectly changes their shading. Finally, in Fig. 13 we show the result of
directly concatenating the camera pose with the bottleneck features (w/o MLP).
As seen, without the MLP, our network cannot effectively utilize the camera
information and incorrectly changes the shading of the background areas.

Table 3. We show numerical comparisons against variations of our approach
without disparity, frequency encoding, and MLP. The results are averaged over
the three synthetic scenes. As shown, all these components are necessary to
achieve the best results.

Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
w/o disparity | 30.98 | 0.950 | 0.0213
w/o FE | 31.11 | 0.950 | 0.0210
w/o MLP | 31.17 | 0.951 | 0.0211
Ours | 31.58 | 0.956 | 0.0208

Figure 11. We evaluate the effect of using disparity as the input to our
shading network. Figure 12. We compare our results against a version of our
approach where we do not apply frequency encoding to the input disparity.
Figure 13. We compare our results against a version of our approach where the
camera position is directly concatenated to the bottleneck features of the
UNet.

##  5\. Limitations

Although we have demonstrated that our simple network can produce reasonable
results, this is an extremely challenging problem and, as shown in Fig. 14,
our approach has several limitations. For example, we are currently not able
to handle highly specular surfaces, such as mirrors. As shown in Fig. 14
(mirror on the right wall), our technique is not able to correctly move the
content inside the mirror between the two reshaded images. Additionally, in
cases where the light sources are very close to diffuse surfaces, they create
strong saturated regions (see the area underneath the mirror). In these cases,
our reshading network interpret these as highlights and moves them between
different views.

Figure 14. We show the input image as well as two reshaded images
corresponding to different views. As seen, our method is not able to properly
move the content of the mirror on the right wall in the two reshaded images.
Additionally, while our method correctly moves the highlights on the ground,
it detects the strong saturated regions under the mirror as highlights and
move them in the reshaded images.

##  6\. Conclusion

We have presented a method to handle view dependent effects in single image
novel view synthesis. Specifically, we propose to split the task of view
synthesis into pixel reshading and relocation processes and treat them
independently. We use a network to adjust the shading of the input image
according to the novel camera. We then use the reshaded image as the input to
an existing view synthesis method to perform the pixel relocation task. We
demonstrate that our method produces plausible results with view-dependent
highlights that are better than the existing methods.

###### Acknowledgements.

The authors would like to thank the reviewers for their comments and
suggestions. This work was funded by Leia Inc. (contract #415290). Portions of
this research were conducted with the advanced computing resources provided by
Texas A&M High Performance Research Computing.

## References

  * (1)
  * Bi et al. (2020a) Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020a.  Neural reflectance fields for appearance acquisition.  _arXiv preprint arXiv:2008.03824_ (2020).
  * Bi et al. (2020b) Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020b.  Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 294–311.
  * Bitterli (2014) Benedikt Bitterli. 2014\.  Tungsten Renderer.  https://github.com/tunabrain/tungsten.
  * Bitterli (2016) Benedikt Bitterli. 2016\.  Rendering resources.  https://benedikt-bitterli.me/resources/.
  * Blake (1985) A. Blake. 1985.  Specular Stereo. In _Proceedings of the 9th International Joint Conference on Artificial Intelligence - Volume 2_ (Los Angeles, California) _(IJCAI’85)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 973–976.
  * Boss et al. (2021) Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021\.  Nerd: Neural reflectance decomposition from image collections. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12684–12694.
  * Chan et al. (2023) Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. 2023\.  GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models. In _arXiv_.
  * Fridman et al. (2023) Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. 2023.  Scenescape: Text-driven consistent scene generation.  _arXiv preprint arXiv:2302.01133_ (2023).
  * Gu et al. (2023) Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. 2023.  NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion. In _International Conference on Machine Learning_.
  * Han et al. (2022) Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. 2022\.  Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images. In _ACM SIGGRAPH 2022 Conference Proceedings_ (Vancouver, BC, Canada) _(SIGGRAPH ’22)_. Association for Computing Machinery, New York, NY, USA, Article 14, 8 pages.  https://doi.org/10.1145/3528233.3530755
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Hua et al. (2020) Yiwen Hua, Puneet Kohli, Pritish Uplavikar, Anand Ravi, Saravana Gunaseelan, Jason Orozco, and Edward Li. 2020.  Holopix50k: A Large-Scale In-the-wild Stereo Image Dataset. In _CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, 2020._
  * Jampani et al. (2021) Varun Jampani, Huiwen Chang, Kyle Sargent, Abhishek Kar, Richard Tucker, Michael Krainin, Dominik Kaeser, William T Freeman, David Salesin, Brian Curless, and Ce Liu. 2021.  SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting. In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Kajiya (1986) James T Kajiya. 1986\.  The rendering equation. In _Proceedings of the 13th annual conference on Computer graphics and interactive techniques_. 143–150.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (July 2023).  https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR)_.
  * Kopanas et al. (2022) Georgios Kopanas, Thomas Leimkühler, Gilles Rainer, Clément Jambon, and George Drettakis. 2022.  Neural Point Catacaustics for Novel-View Synthesis of Reflections.  _ACM Transactions on Graphics_ 41, 6 (2022), Article–201.
  * Kopf et al. (2019) Johannes Kopf, Suhib Alsisan, Francis Ge, Yangming Chong, Kevin Matzen, Ocean Quigley, Josh Patterson, Jossie Tirado, Shu Wu, and Michael F. Cohen. 2019\.  Practical 3D photography.  _CVPR Workshop on Computer Vision for AR/VR_ (2019).
  * Kopf et al. (2020) Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, Peizhao Zhang, Zijian He, Peter Vajda, Ayush Saraf, and Michael Cohen. 2020.  One Shot 3D Photography.  _ACM Trans. Graph._ 39, 4, Article 76 (aug 2020), 13 pages.  https://doi.org/10.1145/3386569.3392420
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020\.  The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ (2020).
  * Leia (2023) Leia. 2023.  Leia Inc.  https://www.leiainc.com/.
  * Li and Kalantari (2020) Qinbo Li and Nima Kalantari. 2020.  Synthesizing Light Field From a Single Image with Variable MPI and Two Network Fusion.  _ACM Transactions on Graphics_ 39, 6 (2020).  https://doi.org/10.1145/3414685.3417785
  * Li et al. (2020) Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2020\.  Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Li et al. (2022) Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Miloš Hašan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. 2022.  Physically-Based Editing of Indoor Scene Lighting from a Single Image. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI_. Springer, 555–572.
  * Lochmann et al. (2014) Gerrit Lochmann, Bernhard Reinert, Tobias Ritschel, Stefan Müller, and Hans-Peter Seidel. 2014\.  Real-time Reflective and Refractive Novel-view Synthesis, Jan Bender, Arjan Kuijper, Tatiana von Landesberger, Holger Theisel, and Philipp Urban (Eds.). Eurographics Association, Darmstadt, Germany, 9–16.  https://doi.org/10.2312/vmv.20141270
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _ECCV_.
  * Niklaus et al. (2019) Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 2019\.  3D Ken Burns Effect from a Single Image.  _ACM Trans. Graph._ 38, 6, Article 184 (nov 2019), 15 pages.  https://doi.org/10.1145/3355089.3356528
  * Philip et al. (2019) Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A. Efros, and George Drettakis. 2019.  Multi-View Relighting Using a Geometry-Aware Network.  _ACM Trans. Graph._ 38, 4, Article 78 (jul 2019), 14 pages.  https://doi.org/10.1145/3306346.3323013
  * Philip et al. (2021) Julien Philip, Sébastien Morgenthaler, Michaël Gharbi, and George Drettakis. 2021.  Free-viewpoint Indoor Neural Relighting from Multi-view Stereo.  _ACM Transactions on Graphics_ (2021).  http://www-sop.inria.fr/reves/Basilic/2021/PMGD21
  * Poole et al. (2022) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022.  DreamFusion: Text-to-3D using 2D Diffusion.  _arXiv_ (2022).
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021\.  Vision Transformers for Dense Prediction.  _ICCV_ (2021).
  * Ranftl et al. (2022) René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2022\.  Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44, 3 (2022).
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 234–241.
  * Roth and Black (2006) S. Roth and M.J. Black. 2006.  Specular Flow and the Recovery of Surface Structure. In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , Vol. 2. 1869–1876.  https://doi.org/10.1109/CVPR.2006.290
  * Sengupta et al. (2019) Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. 2019\.  Neural Inverse Rendering of an Indoor Scene From a Single Image. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Shade et al. (1998) Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998.  Layered Depth Images. In _Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’98)_. Association for Computing Machinery, New York, NY, USA, 231–242.  https://doi.org/10.1145/280814.280882
  * Shih et al. (2020) Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 2020\.  3D Photography using Context-aware Layered Depth Inpainting. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Shue et al. (2022) J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 2022\.  3D Neural Field Generation using Triplane Diffusion.  _arXiv preprint arXiv:2211.16677_ (2022).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Sinha et al. (2012) Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012\.  Image-Based Rendering for Scenes with Reflections.  _ACM Trans. Graph._ 31, 4, Article 100 (jul 2012), 10 pages.
  * Srinivasan et al. (2021) Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. 2021.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7495–7504.
  * Srinivasan et al. (2017) Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. 2017\.  Learning to synthesize a 4D RGBD light field from a single image. In _Proceedings of the IEEE International Conference on Computer Vision_. 2243–2251.
  * Tucker and Snavely (2020) Richard Tucker and Noah Snavely. 2020.  Single-view View Synthesis with Multiplane Images. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Verbin et al. (2022) Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. 2022.  Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields.  _CVPR_ (2022).
  * Wang et al. (2022) Qianqian Wang, Zhengqi Li, Brian Curless, David Salesin, Noah Snavely, and Janne Kontkanen. 2022\.  3D Moments from Near-Duplicate Photos. In _CVPR_.
  * Wang et al. (2021) Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. 2021\.  Learning indoor inverse rendering with 3d spatially-varying lighting. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12538–12547.
  * Watson et al. (2022) Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. 2022.  Novel view synthesis with diffusion models.  _arXiv preprint arXiv:2210.04628_ (2022).
  * Wiles et al. (2020) Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020.  SynSin: End-to-end View Synthesis from a Single Image. In _CVPR_.
  * Wizadwongsa et al. (2021) Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021\.  NeX: Real-time View Synthesis with Neural Basis Expansion. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wu et al. (2022) Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu. 2022.  Scalable neural indoor scene rendering.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–16.
  * Xu et al. (2019) Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. 2019\.  Deep view synthesis from sparse photometric images.  _ACM Transactions on Graphics (ToG)_ 38, 4 (2019), 1–13.
  * Xu et al. (2018) Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018.  Deep image-based relighting from optimal sparse samples.  _ACM Transactions on Graphics (ToG)_ 37, 4 (2018), 1–13.
  * Yu et al. (2021) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021\.  pixelNeRF: Neural Radiance Fields from One or Few Images. In _CVPR_.
  * Zhang et al. (2021) Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. 2021.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–18.
  * Zhou et al. (2016) Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. 2016.  View Synthesis by Appearance Flow. In _Computer Vision – ECCV 2016_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 286–301.
  * Zimmer et al. (2015) Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015.  Path-space Motion Estimation and Decomposition for Robust Animation Filtering.  _Computer Graphics Forum_ 34, 4 (2015), 131–142.


#  Customisable X-ray fluorescence photodetector with submicron sensitivity
using a ring array of silicon p-i-n diodes

Phil S. Yoon111Email<EMAIL_ADDRESS>P.O. Box 5000, Brookhaven National
Laboratory, Upton, NY 11973-5000, USA

###### Abstract

The research and development of silicon-based $X$-ray fluorescence detectors
achieved its submicron sensitivity. Its initial use is intended for _in-situ_
beam monitoring at advanced light-source facilities. The effectively
functioning prototype fully leveraged technologies and techniques from a wide
array of scientific disciplines: $X$-ray fluorescence technique, photon
scattering and spectroscopy, astronomical photometry, semiconductor physics,
materials science, microelectronics, analytical and numerical modelling, and
high-performance computing. In the design stage, the systematic two-track
approach was taken with the aim of attaining its submicron sensitivity:
Firstly, the novel parametric method, devised for system-wide full
optimisation, led to a considerable increase in detector’s total solid angle
(0.9 steradian), or integrated field-of-view ($\sim$ 3000 deg2), thus in turn
yielding a substantial enhancement of its photon-detection efficiency.
Secondly, the minimisation of all types of limiting noise sources identified
resulted in a boost to detector’s signal-to-noise ratio, thereby achieving its
targeted range of sensitivity. The subsequent synchrotron-radiation experiment
with this $X$-ray detector demonstrated its capability to respond to 8-keV
photon beams with 600-nanometre sensitivity. This Article reports on the
innovative and effective design methods formulated for systematising the
process of custom-building ultrasensitive photodetectors, as well as future
directions.

††journal: Scientific Reports 8, 15926 (2018)

##  1 Introduction

Contemporary synchrotron-based light-source facilities worldwide have been
placing greater and greater user demand for _in-situ_ instruments having
ultrahigh spatial sensitivity. The purpose of utilising such an ultrahigh-
precision instrument is to fully realise the benefits of higher brightness and
minuscule dimensions of $X$-ray beams radiating from a light source. To this
end, numerous groupsalkire:quad_bpm ; decker:pac2007 ; decker:biw2010 ;
owen:pindiodes ; southworth:quad_pin_diode_xbpm ; kenney:xbpm ;
plankett:pixelsensors ; carg:indus2  have developed $X$-ray detectors of this
type to fulfil their own needs over the past decades. In the meantime, the
advent of new synchrotron-radiation (SR) facilities of this decadensls2:jsr ;
nsls2:cdr  is sparking a strong need to further increase photodetector’s
sensitivity to a scale of a fraction of one micron. Driven by such growing
needs for ultrahigh-precision beam-monitoring devices, intensive R$\&$D
efforts were dedicated to developing hard/tender $X$-ray detectors capable of
monitoring nanometre-size photon beams in situ. Apart from the pinpoint
spatial sensitivity, the $X$-ray fluorescence (XRF)serpell:xrf ; kinebuchi:xfa
technique was utilised, by design, for the semiconductor-based detector. Its
initial applications are intended for downstream photon-intensive experiments.
Using this photodetector of submicron sensitivity, $X$-ray beams focused onto
beam-defining optical slits can be kept on the upstream side of minute
biological samples under study (Fig. 1).

At the inception of this R&D project, a handful of design considerations were
taken into account and thus led to building effectively functioning
prototypes. The principal design goals pursued are listed in order of priority
as follows: (1) submicron-scale sensitivity, (2) preservation of $X$-ray beam
properties, such as photon flux and relative energy spread ($\Delta E/E$), on
the downstream side of the detector, (3) ultralow-noise operation, (4) high-
vacuum compatibility, and (5) compact and cost-effective design. As
identified, these five considerations are foundational for the design of
position-sensitive detectors that are highly efficient for detecting
photonsgruner:xray_det . The model-based full optimisations, based on which
the detector prototype was constructed, gave the design a boost to its photon-
detection efficiency and enabled a resulting enhancement of signals. In
support of the low-noise design, active and fast detector-cooling modules and
a dedicated control system for photon-counting application were integrated en
masse into the photon-detection system. Consequently, it was made possible to
create a novel avenue towards a boost to detector’s signal-to-noise (SNR)
ratio and a pathway of achieving its ultrahigh spatial sensitivity.
Additionally, utilising the parametric optimisation method can empower
detector designers to react nimbly to satisfy varying needs for individual
beamline programs at light-source facilities. Illustrated as a whole in Fig. 1
is the codified process of prototyping a custom detector attaining the
targeted sensitivity.

Figure 1: Prototyping pipeline for the detector development. Block diagram
illustrating the workflow of developing an $X$-ray fluorescence detector with
ultrahigh precision; shown in blue boxes is the four-phase design stage of
optimising the core sensor-target-frontend subsystems. The calculation results
from analytical modelling and supercomputer-aided Monte-Carlo simulations
provided quantitative and practical guidance (dashed line) on conducting time-
efficient beamline experiments.

##  2 Results

### Principle of operation

As illustrated in Fig. 2(a), the detector system design consists of four key
functional components: (1) a ring array of Si(PIN) diodes serving as a
multisegmented photosensor, (2) a thin metallized film as a source of
fluorescence radiation, (3) its dedicated ASIC-based readout electronics, and
(4) single-stage micro-thermoelectric (Peltier) coolers and a water-cooling
module, all of which are interfaced with a copper heat sink, or heat
exchanger. In essence, the sensor-target subsystems, surrounded by the readout
electronics and cooling modules, are situated at the very heart of the
detector system. Importantly, the thin fluorescing film–henceforth referred to
as fluo-film, or fluo-target, or target for short–is introduced to the design
in order to act as a source of secondary $X$-ray radiation emitted upon
impingement of primary $X$-ray radiation. As an addition, a lead (Pb) scatter
shield, or visor, is inserted immediately upstream from the sensor array, so
as to safeguard the ultrasensitive photosensors against potential stray
signals that may arise during operation. And both upstream and downstream
beryllium windows on a multi-port vacuum chamber are designed to be thin
enough in consideration for downstream photon-starving experiments.

Figure 2: Principle of detector operation. (a) Schematic presentation of the
operational principle of the $X$-ray fluorescence detector. Drawn with the
blue dashed line is the cutaway view of solid angle’s imaginary sphere having
its radius equal to the working distance $Z^{*}$. The projection angle $\Psi$
is formed between the sensor segment (solid blue) in the transverse plane and
the curvilinear sensor (solid orange) tangential to the spherical surface;
note that the thin film, diode segments, and lead shield shavings are not to
scale. (b) Solid angle, subtended at a secondary radiation source on the thin
film, is defined on the surface of the imaginary sphere.

As depicted in Fig. 2(b), the effective beam-through apertures are set to
accept, at least, a 5-$\sigma$ footprint of an incident beam. The dimension of
this central aperture accordingly limits the linear working range to a few
millimetres for a highly focused nanometre-size beam, based on the
0.5-$\mu$rad tolerance for the beam-pointing stability required by the
National Synchrotron Light Source II (NSLS-II) facilitynsls2:cdr . Focused
$X$-ray beams are then guided to propagate unimpeded in a high vacuum, passing
through the circular apertures on both the sensor ring and the scatter shield.
Moreover, the fluorescing target is arranged in a configuration orthogonal to
the propagating direction of incident $X$-ray radiation. When primary
radiation impinges upon the fluo-film, a paucity of $X$-ray
radiationserpell:xrf  emanates in the backward direction in the form of
characteristic $X$-ray emission. As a result, the emission of nascent
fluorescence radiation illuminates the backside of the sensor array. At the
same time, the vast majority of incident photons are transmitted. In this
process, the orthogonal beam-target topology becomes a critical factor for
monitoring a monochromatised beam with ultrahigh precision. On this account,
the orthogonal target configuration is required for the following three
reasons: First, the normal incidence forces an incident beam to experience
from beam’s vantage point an uniform effective thickness of each fluo-target,
regardless of its point of incidence. Thus, the normal incidence of primary
radiation on the target makes it possible to preserve the incident beam
properties during the process of transmission and propagation towards a sample
at the opposite end of the beamline from the light source. Second, the on-
target normal incidence, in turn, ensures isotropic illumination and uniform
photon detection over the entire active area of the planar sensor array. And
third, during transmission of $X$-ray beams through the thin film, scientists
can derive its beam centroid from its entire beam profile, benefiting from
uniform illumination over the diode array. On top of the orthogonal
configuration, suppression of background-event signals is another critical
issue. Taken all together, the backward-scattering mode of operation was
favoured over the forward-scattering mode in the light of minimising potential
systematic uncertainties arising during operation. The choice of the operation
mode was borne out by the fact that the former mode can suppress elastically
scattered photons (i.e., Rayleigh scatterings) coming from the target.
Consequently, the sensor array captures backward-scattered fluorescent
radiation with higher detection efficiency and spectral purity. Furthermore,
three decisive advantages of using secondary $K_{\alpha}$-fluorescent
radiation are the followingperujo:k_alpha : (1) distinctively high intensity
and variable photon flux, (2) high spectral purity, and (3) the availability
of a wide selection of fluo-film materials with reference to the energy of
incident radiation.

Figure 3: Beamline layout. An envisioned beamline layout is illustrated with
the inclusion of the $X$-ray fluorescence detector assembly installed
downstream from a string of various optical components (i.e., a double-crystal
monochromator (DCM), mirrors, beam-shaping pre-slits, and an exit-slit
collimator). One representative unit is shown to be positioned upstream from
an experimental sample.

To visualise its full range of beamline operation, Fig. 3 shows a schematic
setup for monochromatic beamline components and the $X$-ray detector. One of
the design features of note is that its compact lateral form factor ($<$ 1
ft.) makes it possible for each beamline to accommodate multiple colocated
detectors. This way a set of $X$-ray detectors operational along a beamlne
enables extracting more beam parameters, such as divergence angles, beam
emittances, and so forth.

###  2.1 Photosensor design

As an optical receiver, the photodiode ring array is coupled with a fluo-
target and its dedicated front-end readout circuitry, all of which take the
form of the trio core subsystems. It is thus of foremost importance in
designing a sensor ring array, that is most optimal for both the target and
the readout electronics, and a photodetector with low-noise signal processing.

Figure 4: Prototype sensors. (a) A finished $Si$ wafer containing four
32-segment diode rings arranged in each quadrant. (b) [Prototype-II-32] A
fully optimised layout of the photodiode ring array composed of 32 segments,
each of which having its surface area of 1.0 $mm^{2}$. (c) [Prototype-II-64] A
ring array of the 64 diode segments, each measuring 0.44 $mm^{2}$. (d)
[Prototype-I-32] An unoptimised version of the 32-segment diode array.

Achieving the targeted sensitivity level presents two main challenges to the
design: (1) how to give rise to a substantial increase in photon-detection
efficiency and (2) how to realise ultralow-noise photon-detection and
operation under a high photon flux ($\gtr$ $10^{13}$ photons/s) environment.
The two-track approach was correspondingly taken to rise to the said
challenges: The first approach was to develop an effectual method of enhancing
a higher degree of photon-detection efficiency, resulting in acquiring far
more signals from the sensor array. Unlike the prior developments
elsewherealkire:quad_bpm ; decker:pac2007 ; decker:biw2010 ; owen:pindiodes ;
southworth:quad_pin_diode_xbpm , the sensor designs have the form of a
multisegmented annular ring, covering the full range of 2$\pi$-azimuthal
angle, thus enabling highly efficient capture of isotropic fluorescent
radiation (Fig. 4(a)–(d)). As the first step towards the photometric
optimisation, calculations of solid angle $\Omega_{ring}$bradt:astronomy ,
subtended by the diode ring array at a source of fluorescence radiation, were
worked out by virtue of finding real-valued analytical functions (Eqs. (S1)
and (S2)). Referring to Fig. 2(b), the four-parameter solid angles, subtended
at a point source, are expressed by Eqs. (1a) and (1b). Here,
$\Omega_{ring,4}$ and $\widetilde{\Omega}_{ring,4}$ denote respectively a
solid angle for the planar sensor and a solid angle projected onto the
spherical surface for the curvilinear sensor.

$\displaystyle\Omega_{ring,4}=~{}2N_{s}\Biggl{\\{}\arctan\Biggl{(}\frac{\theta^{+}z^{*}}{\sqrt{(\rho^{-})^{2}+(\rho^{-}\theta^{+})^{2}+(z^{*})^{2}}}\Biggr{)}-\arctan\Biggl{(}\frac{\theta^{+}z^{*}}{\sqrt{(\rho^{+})^{2}+(\rho^{+}\theta^{+})^{2}+(z^{*})^{2}}}\Biggr{)}\Biggr{\\}}$
(1a)
$\displaystyle\widetilde{\Omega}_{ring,4}(z^{*};\theta^{+},\rho^{+},\rho^{-})=~{}\textup{Pr}(\psi)\cdot\Omega_{ring,4},$
(1b)

where the projection factor $\textup{Pr}(\psi)$ is defined as $\cos\psi$ (Fig.
2(a)). The variables $z^{*}$ and $N_{s}$ represent respectively the working
distance of the target from the sensor plane and the total number of diode
segments in the array. And $\theta^{+}$, $\rho^{+}$, and $\rho^{-}$ signify
polar angle, radial coordinate components of the sidelines of each diode
segment, respectively (Supplementary Fig. 1). In the case of a planar sensor
(Fig. 2(a)), the projection factor $\textup{Pr}(\psi)$ is put into play for
calculating solid angles with precision. First and foremost, the application
of this parametric method allows for quantitatively determining a practical
range of working distance optimal for the target in use. As observed in Fig.
5(a), the optimised sensor array attains its total peak solid angle of 0.90
steradian, or integrated field of view of $\sim$ 3000 $deg^{2}$, under
irradiation with photon beams having a 90-$\mu$m diameter at an optimal
working distance of 4.5 mm–the number of photon counts peaks out when target’s
working distance is at this calculated optimum position. Given these
parameters, the photon-detection efficiency $\eta_{\small det}$ of the
optimised core subsystems is estimated to be $\eta_{\small
det}=\Omega_{ring}/4\pi=\sum^{N_{d}}_{id=1}\Omega_{id}/4\pi=0.072$.
Ultimately, this parametric method provides a quantitative basis for
determining the optimal layout and dimensions of the sensor array with the
inclusion of multiple guard rings and the central aperture. The foregoing
analytical model made it possible to perform a series of rigorous analyses for
fine-tuning the optimisation of the core subsystem. According to the
calculation results, two optimised versions were fabricated—dubbed Prototype-
II-32 and Prototype-II-64— in reference to the unoptimised Prototype-I. As
Fig. 4(b)–(d) shows, the total surface areas of Prototype-II were halved,
whereas its peak solid angle and photon-detection efficiency doubled as a
direct result of the photometric optimisation. In addition, the full
optimisation, by including spatial constraints imposed by indirect-bandgap
semiconductors, produced a compact design, thus creating enough room for
multiple concentric-circular guard rings. Accordingly, a system of two inner
and three outer guard rings was incorporated in order to smoothly step down
the electric potential to the ground (Fig. 4(b) and (c)). As a side benefit,
the resulting miniaturisation made its way for lower-cost and higher-volume
manufacturability on a single wafer. And the second approach was the
minimisations of the system-wide noise levels achieved in various ways. As
part of the optimisation process, the idea of multisegmentation was introduced
to the design of ring-shaped diodes. It was based on the fact that the
multisegmented design offers a few benefits to recognise: In the first place,
the multisegmentation allows for the reduction in junction capacitance,
thereby suppressing series noise and enabling sensor’s faster response. In the
second place, this multisegmentation provides the benefit of creating fine-
tuning knobs for enabling capacitive matching between each diode segment and
the front-end readout circuit. Upon sensor fabrication, full electrical
characterisation was performed on individual diode segments. Follow-up
extraction of the characteristic parameters in detail rendered quantification
of detector’s performance, which was fed into the optimisation loop, as
illustrated in Fig. 1.

Figure 5: Sensor characterisation. (a) [Prototype-II-32] Profiles of the total
solid angles $\Omega_{ring}$ over working distance $Z^{\star}$. The lower
trace (red solid) corresponds to the optimised planar sensor array
$\Omega_{ring}$, and the upper trace (blue dashed) to the optimised
hemispheric array of sensors $\Omega_{hemi}$. (b) [Prototype-II-32/-64]
Measured $J_{r}$–$V_{r}$ characteristics of the photodiode segments at $\sim$
300 K and 1 atm.; the blue trace corresponds to the 32-segment ring and the
red trace to the 64-segment ring. Under the reverse-biased DC voltage across
the device up to 200 V, leakage current densities $J_{r}$ of the 32-segment
and 64-segment ring array increase at $\sim$ 300 K. The vertical green dashed
line marks the operating detector bias of -150 V. (c) [Prototype-II-32]
Measured $C_{A}$–$V_{r}$ characteristics of the Si(PIN) diode segment at
$\sim$ 300 K and 1 atm.; plotted against the reverse DC bias on a log scale is
the areal capacitance undergoing the depletion and the inversion processes.
The upper trace in red corresponds to the 64-segment array and the lower trace
in blue to the 32-segment array. (d) [Prototype-II-32] Plot of $1/C_{A}^{2}$
as a function of reverse-biased DC sweep $-V_{r}$ showing the dopant
concentration of the semiconductor junction device. In the depletion region, a
linear fit (red) is applied to the data points with systematic errors
propagated.

When irradiated with photon beams of 90-$\mu m$ diameter, the total solid
angle of the planar sensor array peaks at 4.5 mm, whereas an optimised
hemispheric sensor array is expected to bring about a further increase of 30
$\%$ in its solid angle (Fig. 5(a)). It was observed from Fig. 5(b) that
surface leakage current density $J_{r}$ increases linearly with the reverse-
biased voltage applied across the p-i-n junction with ohmic contacts. In
particular, the $C_{A}$–$V_{r}$ measurements took into account leakage
currents as a dissipation factor (Fig. 5(c)). Furthermore, dopant
concentration was obtained from the slope of the $1/C_{A}^{2}$–$V_{r}$ curve
in the depletion region (Fig. 5(d)). And it was ascertained from Fig. 5(b)
that the level of leakage current density $J_{r}$ is held below 100
$pA/mm^{2}$ under reverse-biased DC sweep down to $-150$ $V$ at room
temperature, while satisfying a set of the design criteria. On the other hand,
under the applied electric field, sensor capacitance decreases until full
depletion is reached at around $-150$ $V$ (Fig. 5(c)). In the inversion
region, the areal capacitance $C_{A}$ as low as 0.5 pF was measured at the
maximum depletion depth on 100 $kHz$, based upon which the capacitive matching
was conducted (Fig. 5(c))radeka:psd ; poc:cmos . Listed as a summary in Table
1 are salient detector parameters extracted from the $J_{r}$–$V_{r}$ and
$C_{A}$–$V_{r}$ measurements. A silicon p-i-n (Si(PIN)) diode of high-speed
response produces sufficiently low leakage currents even at room temperature
without requiring a cryogenic cooling system. Moreover, a wide dynamic range
and excellent linearity are inherent to the silicon material. Cognisant of
these properties, silicon was opted over other semiconducting materials –e.g.,
Ge and Si(Li)–for the detection of hard/tender $X$-ray radiation (2 $\sim$ 25
keV). Notably, the sensor array operates as a double-side junction–the
frontside (device side) and the backside (window side)-illuminated
structurerehak:sdd ; carini:sdd .

Table 1: Figures of merit extracted for the array of 32-segment $Si$ diodes; unless otherwise stated, each parameter was obtained from $J_{r}$–$V_{r}$ and $C_{A}$–$V_{r}$ measurements taken at $\sim$ 300 K and 1 atm with the reverse bias of 150 V and of an AC signal at 100 kHz. Note that _e_ and _h_ denote electron and hole, respectively. Parameters | Values
---|---
total sensor thickness, $T$ | 480 $\mu$m
silicon oxide thickness, $T_{ox}$ | 560 nm
depletion voltage, $V_{d}$ |  $\sim$ 150 $V$
depletion depth, $W_{d}$ | 200 $\mu$m
leakage current density, $J_{r}$ | 0.12 [260 K] / 104 [300 K] $pA/mm^{2}$
effective dopant concentration, $N_{eff}$ | 5.0 $\e{12}$ $cm^{-3}$
resistivity, $\rho$ | 1.0 $k\Omega$-$cm$
extrinsic Debye length, $L_{D}$ | 1.85 $\mu$m
junction capacitance, $C_{j}$ | 0.5 $pF$
shunt resistance, $R_{sh}$ | 180 $G\Omega$
carrier transit time, $t_{tr}$ | 10.2 [$e$]/34.1 [$h$] $ns$
rise time response, $t_{r}$ | 1.9 $ns$
RC time constant, $\tau_{\small RC}$ | 0.9 $ns$
time resolution, $\tau_{r}$ | 2.8 $ns$
bandwidth, $f_{\small BW}$ | 180 $MHz$
noise equivalent power ($\lambda$ = 229 nm) | 1.6 $\e{-11}$ $W/\sqrt{Hz}$
specific detectivity, $D^{\star}$ | 6.4 $\e{10}$ $mm\sqrt{Hz}/W$

###  2.2 Thin fluorescing film.

A high-transmission fluo-film optimal for the incident beam energy was
constructed with the identification of the following elements to consider: (1)
the range of the energy of the $X$-ray beam in operation, (2) the optimal
range of beam energy to which the detector responds, (3) K-shell absorption
edge of the fluorescing target material, and (4) the intensity expected in the
$X$-ray beam. A priori information described above sets standards for
selecting appropriate film materials. Hence, the selection of materials
becomes a necessity, based on their K-shell values having sufficient
separation below the energy of incident $X$-ray beams. The following is a list
of the selection along with respective values of $K_{\alpha}$ and wavelength
$\lambda$: _22Ti ($K_{\alpha}$ = 4.511 keV; $\lambda$ = 2.749 $\mathring{A}$),
_24Cr ($K_{\alpha}$ = 5.415 keV; $\lambda$ = 2.290 $\mathring{A}$), _25Mn
($K_{\alpha}$ = 5.899 keV; $\lambda$ = 2.102 $\mathring{A}$), _26Fe
($K_{\alpha}$ = 6.404 keV; $\lambda$ = 1.936 $\mathring{A}$), _27Co
($K_{\alpha}$ = 6.930 keV; $\lambda$ = 1.789 $\mathring{A}$). The $X$-ray
transmission efficiency and fluorescence signals contend with each other. The
optimal thickness of each of the film materials was determined from the
existing experimental data of mass-absorption coefficients $\frac{\mu}{\rho}$
from the literature hubbell:survey ; hubbell:photon_mass_atten_coeff ;
alkire:mass_absorp_coeff  and numerical simulations that follow (Supplementary
Fig. 2). In such a way, transmission rates of incident photon energy of 8-keV
are above 90 $\%$ with a selection of differing metallic media for use as a
thin film. The transmission efficiency of an incident $x$-ray beam is
important particularly for photon-starving experiments being conducted at
synchrotron light sources. The thickness of thin metal film is often too
feeble to enable a self-supporting fluo-target of adequate quality. As a
solution, a robust $X$-ray transmission window was introduced to the fluo-film
design. The substrate material of choice is the commercial silicon nitride
($Si_{3}N_{4}$) window (Norcada, Inc., Edmonton, AB, Canada: NX10500E,
Supplementary Fig. 3), which is known to be a low-stress ($<$ 250 MPa)
radiation-hard substrate for $X$-ray applicationsgw:si3n4 ; toermae . For
providing sufficient mechanical strength, a 500-nm thick single-layered film
was vacuum-sputtered through a mask onto the silicon-nitride window (5 mm
$\times$ 5 mm $\times$ 500 nm) at the NSLS-II in-house
facilityconley:multilayer . Displayed in Fig. 6 (a) and (b) are the structure
of the $X$-ray window and a thin film supported by the window substrate
mounted onto an aluminium placeholder.

Figure 6: Fluorescing target. (a) Cutaway view of the chromium (Cr) film
vacuum-sputtered on the $Si_{3}N_{4}$ substrate window. (b) Image of a
fluorescing film mounted on an aluminium placeholder is shown with a US dime
coin for comparison of dimensions.

###  2.3 Photon-transport Monte-Carlo simulations

During detector operation, background signals from coherent elastic and
incoherent inelastic scattering may be emitted anisotropically in addition to
fluorescent signals, owing to the polarisation of synchrotron radiation in the
plane of synchrotronlandau:synchrad ; schwinger:synchrad . For the
quantitative assessment of signals and backgrounds, Monte-Carlo (MC)
simulation techniques were employed to study $X$-ray absorption, scattering,
and fluorescent radiation. Relying on the use of EGS4-based RÖntgen SImulator
(RÖSI version 0.19) packagerosi ; namito:egs4 , the MC simulations were
conducted with the inclusion of the geometry of the entire detector assembly
and the vacuum chamber. Detector simulations, using realistic statistics of
photon flux ($\sim$ $10^{12}$ photons/s), were carried out on high-performance
supercomputing platformsnersc . The primary purposes of such high-fidelity MC
simulations are the following: (1) quantification analysis of detector
performance and (2) estimation of potential background events coming from
ambient scatterings, both of which are important for attaining its submicron-
scale sensitivity. This MC model collectively included signals from $X$-ray
fluorescence and Rayleigh and Compton scatterings. In particular, the angular
distribution of Rayleigh photons of a few keV, which are scattered from
medium- or high-Z materials, is confined mostly in a wide-angle cone open in
the forward directionklein_nishina ; nelms_oppenheim . For the photon energy
ranging from 10 to 50 keV, the photoelectric interaction is the dominant
signal process, having sole responsibility for the fluorescence phenomenon. On
the other hand, Rayleigh scattering is the dominant background process over
Compton scattering in the energy range of 5 $\sim$ 25 keV. With the selection
of materials (i.e., _24Cr, _25Mn, _26Fe, and _27Co), MC studies show that
estimated contributions from the two competing interactions –Rayleigh and
Compton scatters–are as insubstantial as below the level of 1.0 $\%$ and 0.1
$\%$, respectively. As defined in equation (2), the photon purity $P_{K,fluo}$
of the K-shell fluorescent radiation is the ratio of K-shell fluorescent
radiation $N_{K,fluo}$ to a sum of scattered radiation $N_{scatt}$ and
$N_{K,fluo}$:

$\centering P_{K,fluo}=N_{K,fluo}/(N_{K,fluo}+N_{scatt})\@add@centering$ (2)

Further, the MC simulations indicated that the purity $P_{K,fluo}$ for each
individual element in the selection amounts to nearly 100 $\%$ in a vacuum.

###  2.4 Readout electronics and peripheries

Figure 7:  Sensor-ASIC interconnections and front-end readout noise
performance. (a) Image of the High-Energy Resolution Multi-Element
Spectrometer (HERMES4) ASIC chip of which layout dimension is 3711 $\mu$m
$\times$ 6287 $\mu$m. (b) Details of the sensor-ASIC layout displaying on-chip
interconnections. (c) Analysed was noise originated from the HERMES4 ASIC chip
alone without photogenerated current at 300 K. The blue circles in the plot
represent rms voltages in units of mV, while the red squares do the measured
equivalent noise charge ($e^{-}$ rms) at different selections of peaking time
$T_{peak}$ (0.5, 1, 2, and 4 $\mu$s). Note that the gain is set to 1500 mV/fC.
(d) Plotted as a function of $T_{peak}$ are the rms voltages arising from the
sensor array wire-bonded to the two ASIC chips at near cryogenic temperature
($\sim$ 180 K).

Compact in-vacuum readout electronics were designed, utilising dedicated
application-specific integrated circuits (ASICs) for the high-rate photon-
counting application. The HERMES4 ASIC, based upon 350-nm CMOS technology,
offers 32 channels of low-noise charge amplification, high-order shaping with
baseline stabilisation, and peak detection for various low-noise
analogue/digital processing (Fig. 7(a))degeronimo ; cook:czt . For reducing
stray capacitance and for suppressing signals stemming from electrical
interconnections, each sensor segment in the array is wire-bonded directly to
the input of the front-end readout channel (Fig. 7(b)). As such, Al-wire wedge
bonding and the ultrasonic technique were utilised between the input of an
ASIC channel and a bonding pad implemented on each diode segment. According to
the measurements, ASIC noise optimisation enables an electronic resolution of
15 $e^{-}$ rms (equivalent noise charge, or ENC) with a choice of 4-$\mu$s
peaking time at room temperature. With multisegmented photosensor’s high
finesse, the event rate per each ASIC channel is diminished to 40 kHz in a
high-flux SR condition. As a result, the front-end readout with low-power
consumption (8 mW/channel) was realised. For high-flux applications, one major
challenge is how to design associated readout electronics capable of
processing sufficiently high photon statistics without reaching charge
saturation. It was observed during a beamline experiment that the processing
rate of this ASIC readout chip can cope with $\sim$ 100 kHz per channel.
Additionally, the integrated cooling subsystem functions to lower the
operating temperature of the Si(PIN)-diode array nestled on the readout
printed circuit board. Hence, the active and fast detector cooling is an added
functional feature that helps minimise persistent system-wide parallel noise.
In this respect, the detector system is outfitted with efficient cooling
modules in three ways: (1) A ring array of thermoelectrically-cooled diodes is
included in the power budget for minimising bulk leakage currents. (2) Both
the readout electronics and the thin film make exceptionally good thermal
contact with a water-cooled heat sink copper block. And, (3) applying adhesive
with high thermal conductivity (5.77 W/m K) for interconnecting individual
components yields a remarkable enhancement of the heat-transfer process. As a
consequence, its operating temperature can plummet to $-40~{}\celsius$ in as
fast as a few seconds after power-up. As an integral component of the detector
system, the mechanical support made of an Invar 36 (Alloy 36)patent:invar  was
designed together. This instrument support system of high thermal stability
(100 nm/$\celsius$/hr) ensures the steady maintenance of detector’s high-
precision alignment and spatial sensitivity in a temperature-controlled
environment throughout each year.

###  2.5 Radiation experimentation

Figure 8: Beamline experimentation. Real-time images of pulse-shaped output
waveforms captured on oscilloscope (Tektronix MSO 4000 series) display during
detector operation at $-40~{}\celsius$, (a) when irradiated with 5.9-keV
radiations from Fe-55 sealed sources in a high-vacuum condition at the base
pressure of $10^{-4}$ torr, and (b) when irradiated by a synchrotron radiation
in a high-vacuum condition of 3 $\times$ $10^{-6}$ torr. (c) Mesh plot showing
photon counts processed from the ASIC channel 34 at each vertex (red dot) over
the 7 $\times$ 7 mesh grid. (d) Photon counts processed from the channel 46
are normalised to the minimum count at each vertex and are then projected onto
the same 7 $\times$ 7 mesh grid.

As a proof of concept, a table-top experiment was carried out first at room
temperature with an Fe-55 sealed radioactive source, confirming the
acquisition of noise-free detector response (Fig. 8 (a)). Next, the prototype
detector, loaded with the fully optimised 32-segment sensor (Prototype-II-32)
and target, was experimented with polychromatic beams at the X7A beamline of
the National Synchrotron Light Source. Prior to the SR experiment, high-
precision alignment ($\sigma$ $<$ 60 $\mu m$) of the detector assembly was
ensured at the beamline in an effort to minimise systematic uncertainties. The
amount of detector noise, intrinsic to the radiation sensor and the front-end
readout system, was minimised consistently during both the table-top and the
SR beamline experiment. As evidenced by the experimental observations, the
noise floor was brought down to the bottom level at $-40~{}\celsius$ in high-
vacuum conditions (Fig. 8(a) and (b)). Then, a subsequent SR experiment was
dedicated to one full round of beam-based transverse aperture scan with
$X$-ray beams of 90 $\mu m$ in diameter in regular steps of 600 nanometres
(nm) over 7 $\times$ 7 mesh grid. As cogent evidences, the variations of
photon counts from each channel are mapped to the colour scales on the right-
hand column and projected over the mesh grid (Fig. 8(c) and (d)). This way the
mesh plots visualise the variations of photon counts at each of the 49
vertices with the submicron interval. Hence, these colour maps clearly
delineate that the detector design has its capability to respond with 600-nm
sensitivity and better under intense irradiation of $X$-ray photon beams at
the given energy.

##  3 Discussion

Leveraging the systematic design method, developed for fully optimising the
detector system and for eliminating noise sources, maximised its SNR and thus
boosted its spatial sensitivity. Consequently, the long-standing barrier of
detector’s submicron sensitivity has been broken down. This breakthrough was
achieved by amalgamating technologies from manifold areas of research fields.
Eventually, the custom-design method, offering great flexibility to add more
new ideas and features, became a product of the transdisciplinary approach,
thereby ushering in the realm of nanometre-range sensitivity. In a
conventional prototyping process, a significant amount of lead time and cost
is needed for completing an entire detector prototyping process. By contrast,
taking the two-prong approach reported in this Article makes it possible to
design an archetype of the photodetector reaching a desired level of
sensitivity at a fraction of the previously required time and cost. Above all,
the detector design, ensuring normal incidence of primary radiation, is
apposite particularly to highly focused monochromatised $X$-ray beams,
expecting its surpassing photometric performance. Further enhancements of its
detection efficiency and sensitivity will involve optimising the sensor-target
subsystem based on a beam dimension and developing hemispheric, or semi-
hermetic 2$\pi$-photodetectors open in the forward direction. The calculations
in Fig. 5(a) also corroborate the concave photodetector systems designed
elsewhereko:eye_camera ; someya:eyeball . Hence, one practicable solution to
suggest is building such an optimised concave photodetector in a honeycombed
structure. The benefits from utilising such ultrahigh-precision instruments
are expected to ripple out across the scientific community. Further, the
position-sensitive detector of fluorescence type, realised by the tailored
design process, has a likely impact on applications of wide and far-reaching
appeal to the scientific and industrial research alike.

## References

  * [1] R. W. Alkire, G. Rosenbaum, and G. Evans.  Design of a vacuum-compatible high-precision monochromatic beam-position monitor for use with synchrotron radiation from 5 to 25 keV.  J. Synchrotron Rad., 7:61–68, 2000.
  * [2] G. P. Decker, Den Hartog, O. Singh, and G. Rosenbaum.  Progress toward a hard X-ray insertion device beam position monitor at the advanced photon source.  Proc. of Particle Accelerator Conference (PAC) 2007, Albuquerque, New Mexico, USA., 2007.
  * [3] G. Decker, B. X. Yang, R. Lill, and H. Bui.  APS Beam Stability Studies at the 100-Nanoradian Level.  Proc. of Beam Instrumentation Workshop (BIW) 2010, Santa Fe, New Mexico, USA., 2010.
  * [4] R. L. Owen, J. M. Holton, C. Schulze-Briese, and E. F. Garman.  Determination of X-ray flux using silicon pin diodes.  J. Synchrotron Rad., 16:143–151, 2009.
  * [5] S. H. Southworth and P. L. Cowan.  X-ray beam position monitor using a quadrant PIN diode.  Nucl. Instrum. Methods Phys. Res., Sect. A, 319:51–55, 1992.
  * [6] C. J. Kenney et al.  Use of active-edge silicon detectors as X-ray beam monitors.  Nucl. Instrum. Methods Phys. Res., Sect. A, 582:171–181, 2007.
  * [7] R. Plankett et al.  X-ray metrology of an array of active edge pixel sensors for use at synchrotron light sources.  Nucl. Instrum. Methods Phys. Res., Sect. A, 879:106–111, 2018.
  * [8] A. D. Carg et al.  Design of $x$-ray diagnostic beam line for a synchrotron radiation source and measurement results.  Nucl. Instrum. Methods Phys. Res., Sect. A, 754:15–23, 2014.
  * [9] Brookhaven’s Next Big Machine, NSLS-II, makes progress.  J. Synchrotron Rad., 16:693–694, 2009.
  * [10] NSLS II Conceptual Design Report.  BNL-77977-2006-V1-V2, 2006.
  * [11] C. J. Serpell et al.  Carbon nanotubes allow capture of krypton, barium and lead for multichannel biological $X$-ray fluorescence imaging.  Nat. Commun., 7:13118, 2016.
  * [12] M. Kinebuchi et al.  Diagnostic copper imaging of Menkes disease by synchrotron radiation-generated X-ray fluorescence analysis.  Sci. Rep., 6:33247, 2016.
  * [13] S. M. Gruner, E. F. Eikenberry, and M. W. Tate.  X-ray Detectors.  International Tables for Crystallography, Crystallography of biological macromolecules, Springer Netherlands, F:143–147, 2006.
  * [14] A. Perujo et al.  Deviation of the ${K}_{\beta}$/${K}_{\alpha}$ intensity ratio from theory observed in proton-induced $x$-ray spectra in the 22 $\leq$ Z $\leq$ 32 region.  J. Phys. B: At. Mol. Phys. 20 4973, 20:4973, 1987.
  * [15] H. Bradt.  Astronomy Methods: A Physical Approach to Astronomical Observations, 1 edn., chapter 3, pages 45–47.  Cambridge University Press, Cambridge, UK, 2004.
  * [16] V. Radeka.  Semiconductor Position-Sensitive Detectors.  Nucl. Instrum. Methods Phys. Res., 226:209–218, 1984.
  * [17] P. O’Connor and G. De Geronimo.  Prospects for charge sensitive amplifiers in scaled CMOS.  Nucl. Instrum. Methods Phys. Res., Sect. A., 480:713–725, 2002\.
  * [18] P. Rehak et al.  Arrays of silicon drift detectors for an extraterrestrial X-ray spectrometer.  Nucl. Instrum. Methods Phys. Res., Sect. A., 624:260–264, 2010\.
  * [19] A. G. Carini et al.  Performance of a Thin-Window Silicon Drift Detector X-ray Fluorescence Spectrometer.  IEEE Trans. Nucl. Sci., 56:2843–2849, 2009.
  * [20] J. H. Hubbell.  Survey of Photon-Attenuation-Coefficient Measurements 10 eV to 100 GeV.  Atomic Data, 3:241–297, 1971.
  * [21] J. H. Hubbell.  Photon Mass Attenuation and Energy-absorption Coefficients from 1 keV to 200 MeV.  Int. J. Appl. Radiat. Isot., 33:1269–1290, 1982.
  * [22] R. W. Alkire.  Approximating the near-edge mass absorption coefficients for Ni using an ultra-thin bimetal foil.  J. Appl. Cryst., 50:1–13, 2017.
  * [23] J. Steinlechner et al.  Optical absorption of silicon nitride membranes at 1064 nm and at 1550 nm.  Phys. Rev. D, 96:022007, 2017.
  * [24] P. T. Törmä et al.  Ultra-Thin Silicon Nitride X-ray Windows.  IEEE Trans. Nucl. Sci., 60:1311–1314, 2013.
  * [25] R. Conley et al.  The NSLS-II Multilayer Laue Lens Depositon System.  Proc. of SPIE, 7448:74480U–1, 2009.
  * [26] L. Landau and E. Lifschitz.  The Classical Theory of Fields, 3 edn., chapter 74, pages 197–202.  Pergamon Press Ltd., Oxford, UK, 1971.
  * [27] J. Schwinger.  On the Classical Radiation of Accelerated Electrons.  Phys. Rev., 75:1249, 1949.
  * [28] J. Giersch, A. Weidemann, and G. Anton.  ROSI–an object-oriented and parallel-computing Monte Carlo simulation for $x$-ray imaging.  Nucl. Instrum. Methods Phys. Res., Sect. A., 509:151–156, 2003\.
  * [29] Y. Namito, H. Hirayama, and S. Ban.  Improvements of low-energy photon transport in EGS4.  Radiation Physics and Chemistry, 53:283–294, 1998.
  * [30] W. E. Johnstone.  Computational and data grids in large-scale science and engineering.  Future Generation Computer Systems, 18:1085–1100, 2002.
  * [31] O. Klein and Y. Nishina.  Über die Streuung von Strahlung duruch freie Elektronen nach der neuen relativistischen Quantendynamik von Dirac.  Zeitschrift für Physik, 52:853–868, 1929.
  * [32] A. T. Nelms and I. Oppenheim.  Data on the Atomic Factor: Computation and Survey.  J. Res. Nat. Bur. Stds., 55:53–63, 1955.
  * [33] G. De Geronimo et al.  ASIC With Multiple Energy Discrimination for High-Rate Photon Counting Applications.  IEEE Trans. Nucl. Sci.,, 54(2):303–312, 2007.
  * [34] W. R. Cook, J. A. Burnham, and F. A. Harrison.  Low-noise custom VLSI for CdZnTe pixel detectors.  Proc. of SPIE: SPIE Conference on EUV, X-Ray, and Gamma-Ray Instrumentation for Astronomy IX,, 3445:347–354, 1998.
  * [35] W. M. Sokolowski, M. S. Lane, C. H. Hsieh, and T. P. O’Donnell.  Ultrahigh-purity dimensionally stable INVAR 36, 12 1995.
  * [36] H. C. Ko et al.  A hemispherical electronic eye camera based on compressible silicon optoelectronics.  Nature, 454:748–753, 2008.
  * [37] T. Someya.  Optics: Electronic Eyeballs.  Nature, 454:703–704, 2008.

##  4 Acknowledgements

This R&D work was conducted under the aegis of the U.S. Department of Energy,
Office of Basic Energy Sciences, by Brookhaven National Laboratory (BNL) under
contract number DE-AC02-98CH10886. The author acknowledges the provisions and
arrangements made by the detector development group and the beamline support
group of National Synchrotron Light Source-II, and the Instrumentation
Division of BNL throughout.